Wayland: type mobile text_input via clipboard paste on Portal and compositors without spare keycodes (#119)

* Wayland: type mobile text_input via clipboard paste on Portal and compositors without spare keycodes

* Wayland: address review comments for mobile text_input clipboard fallback

- Guard external clipboard commands in unit tests to eliminate desktop side effects
- Narrow paste fallback to Wayland and Portal backends only
- Document clipboard override and two-way sync behavior in README (both zh and en)
- Unify disable switch via UU_WAYLAND_TEXT=0

---------

Co-authored-by: shofukantkts <[email protected]>
This commit is contained in:
shofukantktsandshofukantkts authored and GitHub committed 2026-10-27 22:14:56 +08:00
1 parent 7a86fab340
commit 003da254a6
3 files changed
+140 -6

No files matched your search

+3 -1
View File
@@ -45,7 +45,7 @@
| Mouse / keyboard | Absolute and relative motion, clicks, wheel, keys; no spurious key repeat on high-latency links; optional uinput virtual devices on Wayland (layout independent, see below) |
| Mac shortcuts | **Recommended**: turn off 「使用 macOS 快捷键」 (use macOS shortcuts) in the Mac viewer's remote-window menu; then Cmd+C/V/A/F/X/Z work as on a Mac (Cmd+C/V become Ctrl+Shift+C/V in terminals) and Ctrl+C always interrupts. With the option on, the viewer sends Cmd as Ctrl: in a terminal Ctrl+V / Cmd+V paste, Ctrl+C interrupts, copy is Cmd+Shift+C (X11) |
| Pointer shape | Remote cursor shape is mirrored to the viewer (X11) |
| Clipboard | Two-way text sync |
| Clipboard | Two-way text sync; Wayland mobile IME text inputs use clipboard paste fallback (see below) |
| Quality / frame rate / resolution | Auto, Original 20M, Ultra 8M, HD 2M, custom; 30/60/90/144 fps; on X11 a resolution choice changes the desktop resolution itself like the official Windows host (xrandr, adding the mode when missing; restored for "original" or on disconnect), `UU_DESKTOP_RESIZE=0` scales in the encoder instead; Wayland scales in the encoder |
| Sound | System audio captured from PulseAudio / PipeWire and sent as Opus |
| File transfer | Upload, download (including folders), create folder, rename, delete |
@@ -84,6 +84,8 @@ plugins, permissions and encoders only, no account data, tokens or addresses) an
Wayland limits: captures one screen by default; Hyprland can enable monitor switching as described below. Resolution is fixed. The native Hyprland backend now pipelines two shm screencopy buffers and passes the viewer's requested fps to both capture pacing and the encoder. Actual fps remains limited by the compositor, rendering and readback (the old serial path measured about 16–22 fps at 2560x1600; the pipeline is VM-measured only, not a hardware guarantee). The portal backend does not support dynamic resolution; see below for GNOME wake and lock recovery; the clipboard syncs while the viewer's remote window has focus. On the first connection the desktop asks for "Remote Desktop / Screen Sharing"
permission; the permission is remembered once granted.
**Mobile IME typing and clipboard on Wayland**: Mobile IME text (`text_input`) on Wayland is typed through spare keycodes on wlroots compositors. On Portal (GNOME / KDE) or compositors where keycodes are exhausted, it automatically falls back to typing via "system clipboard write + `Shift+Insert` (or `UU_WAYLAND_PASTE_KEY=ctrl_v`)". **Note**: This fallback replaces the host's existing clipboard contents and may echo back to the viewer's clipboard via two-way sync; set `UU_WAYLAND_TEXT=0` to completely disable Wayland mobile text input and clipboard paste fallback.
Scaled GNOME desktops using native Portal input use EIS on the same authorized session when `ConnectToEIS` and a screen `mapping_id` are available. Each axis is mapped using the matched region and actual frame dimensions, with no fixed resolution or scale factor. This avoids scaled-coordinate rejection by `NotifyPointerMotionAbsolute`. Portal builds require libei >= 1.1. Missing EIS support retains the legacy input path and its existing scaling limitations; other initialization failures are reported and trigger session recovery. After a successful EIS connection, all input uses EIS. Capture alone and uinput do not initialize EIS; missing optional logical dimensions do not block capture. Keyboard input uses the existing Windows VK to evdev mapping. Recovery keeps the encoder frame dimensions fixed but accepts logical-size changes after scaling. On GNOME, a display-change notification enables earlier recovery when previously streaming Portal capture remains paused without a valid format for a 200 ms grace after both conditions are present. Healthy renegotiation cancels the early check; repeated notifications do not postpone the same interruption. The five-second fallback remains without a related notification. Static video and valid-format pauses do not trigger recovery. Tested while connected on Ubuntu 26.04, GNOME Wayland, one 3840x2160 display: 200% -> 166.67% -> 200%, with continuously updating video and full-screen pointer control after recovery. Switching briefly freezes and goes black before recovering; five seconds is the fallback threshold, not total recovery time. See [#35](../../pull/35) for timing measurements and validation limits. Lock recovery still needs hardware acceptance testing; multi-monitor support, live resolution changes, and rotation/reflection transforms are outside this fix's scope. Capture settings remain at protocol version 4: advertised resolutions use pixel coordinates and DPR 1; Portal logical dimensions are used only for local input mapping.
GNOME Portal CONTROL no longer waits for a consent dialog or the first video frame. It announces the selected desktop before reporting secure-desktop state and receiving the official viewer's “auto unlock host” request. The official Android viewer requires this desktop selection before entering its auto-unlock flow, so the notification must precede capture readiness; actual screen dimensions, codec information and capture permissions still follow their real readiness state. Enable auto unlock in the viewer and use the current Linux user's login password. The host verifies it with PAM, unlocks the existing GNOME session, requests display wake, and restores capture, input and clipboard through the shared Portal session. Initial consent or revoked permissions still require local approval. Reconnect if physical or logical dimensions change.
+3 -1
View File
@@ -39,7 +39,7 @@
| 鼠标 / 键盘 | 绝对/相对移动、点击、滚轮、按键;修复了网络延迟导致的按键连发;Wayland 下可用 uinput 虚拟键鼠(不依赖键盘布局,见下文) |
| Mac 快捷键 | **推荐**:在 Mac 远控窗口菜单里关闭「使用 macOS 快捷键」,之后和在 Mac 上一样:Cmd+C/V/A/F/X/Z 是复制、粘贴、全选、查找、剪切、撤销(终端里 Cmd+C/V 自动变成 Ctrl+Shift+C/V),Ctrl+C 在终端里始终是中断。保持开启时,客户端会把 Cmd 和 Ctrl 都当成 Ctrl 发送:终端里 Ctrl+V / Cmd+V 自动转成粘贴,Ctrl+C 是中断,复制用 Cmd+Shift+C(X11) |
| 鼠标指针样式 | 远端指针形状实时同步(X11) |
| 剪贴板 | 双向文本同步 |
| 剪贴板 | 双向文本同步;Wayland 手机输入法输入通过剪贴板兜底粘贴,见下文 |
| 画质 / 帧率 / 分辨率 | 自动、原画 20M、超清 8M、高清 2M、自定义;30/60/90/144 帧;X11 下切换分辨率会像官方 Windows 被控端一样真正修改桌面分辨率(xrandr,缺少该模式时自动添加;选「原始」或断开后恢复),`UU_DESKTOP_RESIZE=0` 改为仅由编码器缩放;Wayland 仍由编码器缩放 |
| 声音 | 采集系统声音(PulseAudio / PipeWire),Opus 编码传到控制端 |
| 文件传输 | 上传、下载(含文件夹)、新建文件夹、重命名、删除 |
@@ -75,6 +75,8 @@ GNOME Wayland 的 Portal/EIS 路径已在 Ubuntu 26.04、GNOME 50、单屏 4K/20
Wayland 下的限制:默认采集单屏,Hyprland 可按上文启用切屏;分辨率固定;Hyprland 原生后端默认使用双缓冲流水线 shm screencopy,控制端帧率设置会同步到采集节拍和编码器,但实际帧率仍受合成器、渲染和读回速度限制(旧串行路径在 2560x1600 曾测得约 16~22 帧;流水线只在虚拟机里测量过,不代表实机保证);portal 后端暂不支持动态分辨率;GNOME 的屏幕唤醒与锁屏恢复见下;剪贴板只在控制端的远控窗口处于前台时同步。
**Wayland 手机输入法打字与剪贴板**:Wayland 下手机端输入法提交的文字(`text_input`)优先通过虚拟键盘空闲键码上屏;在 Portal(GNOME / KDE)或键码耗尽的环境下,自动通过「写系统剪贴板 + 触发 `Shift+Insert`(或 `UU_WAYLAND_PASTE_KEY=ctrl_v`)」兜底粘贴。**注意**:此兜底过程会覆盖被控端原有的剪贴板内容,且会被双向同步回传到控制端剪贴板;设置 `UU_WAYLAND_TEXT=0` 可完全关闭 Wayland 文字输入及该粘贴兜底。
GNOME 缩放桌面在使用 Portal 原生键鼠时,若支持 `ConnectToEIS` 和屏幕 `mapping_id`,通过同一授权会话的 EIS 接口注入输入,按匹配的区域与实际帧尺寸分别映射横纵坐标,没有固定分辨率或缩放倍率,可避免 `NotifyPointerMotionAbsolute` 在缩放时拒绝坐标。Portal 构建需要 libei ≥ 1.1。缺少 EIS 支持时保留旧输入路径及其既有缩放限制;其他初始化错误会明确报告并触发会话恢复。EIS 连接成功后全部输入统一使用 EIS。只采集画面或使用 uinput 时不会初始化 EIS;缺少可选的逻辑尺寸也不阻断采集。键盘沿用项目现有的 Windows VK → evdev 映射。恢复时保持编码器的采集帧尺寸不变,允许缩放后的逻辑尺寸变化。GNOME 显示配置变化后,已推流的 Portal 采集若持续「暂停且无有效格式」,会在关联通知后的 200 毫秒宽限结束后提前进入现有恢复流程;正常重协商恢复会取消提前判定,连续通知不会推迟同次异常的判定。未收到相关通知时保留五秒兜底;静态画面或有效格式下的暂停不会触发恢复。Ubuntu 26.04、GNOME Wayland 单屏 3840×2160 已实测保持连接切换 200%→166.67%→200%,恢复后画面持续更新且鼠标可控制全屏;切换期间会短暂停顿、黑屏后恢复,五秒是兜底判定阈值,并非总恢复时间。提前恢复的测量结果和验证范围见 [#35](../../pull/35)。此路径仍需实机验证锁屏恢复;多屏、连接中改变分辨率及旋转/镜像变换不属于本次修复范围。 捕获设置协议仍为版本 4:对客户端上报的分辨率统一使用像素坐标、DPR 为 1;Portal 逻辑尺寸仅用于本地输入映射。
GNOME Portal 的控制通道不再等待授权窗口或第一帧画面:先通知客户端当前选择桌面,再上报锁屏状态、接收官方客户端的「自动解锁被控端」请求。官方 Android 客户端需要桌面目标通知才会进入自动解锁流程,因此该通知不能等待采集就绪;真实屏幕尺寸、编码信息及采集权限仍按实际就绪状态上报。请在客户端启用自动解锁,使用当前 Linux 用户的登录密码;主机通过 PAM 验证后解锁当前 GNOME 会话、请求点亮屏幕,并恢复同一 Portal 会话中的画面、键鼠和剪贴板。首次授权或权限被撤销时仍需本机确认。物理或逻辑分辨率变化后需要重连。
+134 -4
View File
@@ -372,8 +372,93 @@ impl InputInjector {
if self.backend==crate::desktop_backend::DesktopBackend::Wayland { crate::wayland::key(symbol,pressed) }
else { unsafe { x11_key_symbol(symbol as libc::c_ulong,i32::from(pressed)) } }
}
/// Set clipboard content for text injection on Wayland / Portal.
/// Tries Portal RemoteDesktop.SetSelection first, then wl-copy as fallback.
fn set_clipboard_text(&self, text: &str) -> bool {
if text.is_empty() || text.len() > crate::clipboard::MAX_TEXT {
return false;
}
// In unit tests, never invoke external clipboard binaries (wl-copy)
// or alter the system clipboard to prevent side effects on developer desktops.
#[cfg(test)]
{
let _ = text;
false
}
#[cfg(not(test))]
{
// 1. Try Portal clipboard if session is bound and active
if let Some(portal) = self.portal.lock().unwrap().as_ref() {
if portal.clipboard_set(text).is_ok() {
return true;
}
}
// 2. Try wl-copy in Wayland session
if std::env::var_os("WAYLAND_DISPLAY").is_some() {
if let Ok(mut child) = std::process::Command::new("wl-copy")
.args(["--type", "text/plain;charset=utf-8"])
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
{
if let Some(mut stdin) = child.stdin.take() {
use std::io::Write;
let _ = stdin.write_all(text.as_bytes());
}
if let Ok(status) = child.wait() {
if status.success() {
return true;
}
}
}
}
false
}
}
/// Injects text by copying it to the clipboard and simulating a paste shortcut.
/// Uses Shift+Insert by default (broadest Linux compatibility across GUI apps and terminals),
/// or Ctrl+V if configured via UU_WAYLAND_PASTE_KEY=ctrl_v.
fn paste_text_via_clipboard(&self, text: &str) -> bool {
if !self.set_clipboard_text(text) {
return false;
}
// Give the clipboard daemon / compositor a brief moment to register the selection.
std::thread::sleep(std::time::Duration::from_millis(25));
let use_ctrl_v = std::env::var("UU_WAYLAND_PASTE_KEY")
.is_ok_and(|v| v.eq_ignore_ascii_case("ctrl_v") || v.eq_ignore_ascii_case("ctrl+v"));
if use_ctrl_v {
// Control_L (0xffe3) + 'v' (0x76)
self.inject_key(0xffe3, true);
std::thread::sleep(std::time::Duration::from_millis(15));
self.inject_key(0x76, true);
std::thread::sleep(std::time::Duration::from_millis(20));
self.inject_key(0x76, false);
std::thread::sleep(std::time::Duration::from_millis(15));
self.inject_key(0xffe3, false);
} else {
// Shift_L (0xffe1) + Insert (0xff63)
self.inject_key(0xffe1, true);
std::thread::sleep(std::time::Duration::from_millis(15));
self.inject_key(0xff63, true);
std::thread::sleep(std::time::Duration::from_millis(20));
self.inject_key(0xff63, false);
std::thread::sleep(std::time::Duration::from_millis(15));
self.inject_key(0xffe1, false);
}
// Brief delay allowing the target application to consume the paste event
std::thread::sleep(std::time::Duration::from_millis(25));
true
}
/// Mobile input-method text (`text_input`). X11 types it directly; Wayland re-maps a spare
/// keycode in the virtual keyboard's keymap. Other backends log once.
/// keycode in the virtual keyboard's keymap. Portal and other Wayland environments inject
/// via clipboard paste (Shift+Insert). Set UU_WAYLAND_TEXT=0 to disable Wayland text injection.
pub fn text_input(&self, text: &str) {
let _serial=self.serial.lock().unwrap();
let Ok(c_text)=std::ffi::CString::new(text.replace('\0',"")) else { return };
@@ -381,17 +466,31 @@ impl InputInjector {
let typed=unsafe { x11_type_utf8(c_text.as_ptr()) };
log::info!("Desktop text input: chars={} result={}", text.chars().count(), typed);
} else if self.backend==crate::desktop_backend::DesktopBackend::Wayland {
// Escape hatch: keep the old behaviour without rebuilding.
// Escape hatch: disable Wayland text typing/pasting entirely without rebuilding.
if std::env::var("UU_WAYLAND_TEXT").is_ok_and(|v|v=="0") { return; }
let typed=crate::wayland::type_utf8(text);
if typed>=0 {
log::info!("Desktop text input: chars={} result={}", text.chars().count(), typed);
return;
}
// A keymap whose keycodes are all mapped cannot carry text; report it once.
// Fallback to clipboard paste if keymap has no spare keycodes or native typing failed
if self.paste_text_via_clipboard(text) {
log::info!("Desktop text input (Wayland clipboard paste fallback): chars={} result=ok", text.chars().count());
return;
}
static WARNED:std::sync::atomic::AtomicBool=std::sync::atomic::AtomicBool::new(false);
if !WARNED.swap(true,std::sync::atomic::Ordering::Relaxed) {
log::warn!("Mobile text input unavailable on the Wayland backend (no spare keycode in this keymap)");
log::warn!("Mobile text input unavailable on the Wayland backend (native and clipboard fallback failed)");
}
} else if self.backend==crate::desktop_backend::DesktopBackend::Portal {
if std::env::var("UU_WAYLAND_TEXT").is_ok_and(|v|v=="0") { return; }
if self.paste_text_via_clipboard(text) {
log::info!("Desktop text input (Portal clipboard paste): chars={} result=ok", text.chars().count());
return;
}
static WARNED:std::sync::atomic::AtomicBool=std::sync::atomic::AtomicBool::new(false);
if !WARNED.swap(true,std::sync::atomic::Ordering::Relaxed) {
log::warn!("Portal clipboard text paste failed: ensure clipboard permission or wl-copy is available");
}
} else {
static WARNED:std::sync::atomic::AtomicBool=std::sync::atomic::AtomicBool::new(false);
@@ -728,3 +827,34 @@ mod x11_monitor_pointer_tests {
crate::wayland::select(0).unwrap();
}
}
#[cfg(test)]
mod clipboard_paste_text_tests {
use super::*;
#[test]
fn rejects_empty_and_oversized_text() {
let input = InputInjector::new();
assert!(!input.set_clipboard_text(""));
let oversized = "a".repeat(crate::clipboard::MAX_TEXT + 1);
assert!(!input.set_clipboard_text(&oversized));
}
#[test]
fn text_input_handles_portal_backend_safely_without_side_effects() {
let mut input = InputInjector::new();
input.backend = crate::desktop_backend::DesktopBackend::Portal;
// In unit tests, set_clipboard_text avoids external commands so this
// will not mutate the host's actual clipboard.
input.text_input("测试输入法文字");
}
#[test]
fn text_input_respects_disable_env() {
let mut input = InputInjector::new();
input.backend = crate::desktop_backend::DesktopBackend::Portal;
std::env::set_var("UU_WAYLAND_TEXT", "0");
input.text_input("测试输入法文字");
std::env::remove_var("UU_WAYLAND_TEXT");
}
}