* feat(audio): capture and encode Opus in process
* feat(portal): encode VAAPI from leased DMA-BUF frames
* fix(input): route Portal controls and committed ASCII through uinput
* docs(portal): shorten setup notes and colocate hardware acceptance guidance
* fix(input): preserve GNOME IME guard for committed uinput text
* build: link pulse/opus after the static C archive (--as-needed)
With --features inproc-av on X11 and UU_VIDEO_ENCODER=vaapi, VAAPI runs in process: a DRI3 DMA-BUF of the root
(XShm upload without DRI3) goes through VPP to NV12 and the hardware encoder. SPS/PPS(/VPS) and slice headers are
packed by the host because radeonsi (Mesa 26) does not write them. libva/libva-drm are dlopen'ed; release builds do
not enable inproc-av. The NVENC path is unchanged.
Accepted by the contributor on AMD HawkPoint1 (Mesa 26.2.3), 1920x1200@60, with the official Mac viewer
(HEVC, presets, fps, scaling). Intel iHD and 4K are not verified. Thanks @ferstar.
H.264 High 4:4:4 / HEVC FRExt with chromaFormatIDC 3 and AQ off when 真彩 is requested; GPUs without YUV444 encode fail the capability check and fall back. full_chroma_available() no longer excludes the in-process path. UU_PROBE_CHROMA=444 for the probe.
Co-authored-by: ferstar <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Replaces the FFmpeg-based in-process encoder with a native NVENC engine (vendored nv-codec-headers n12.1.14.0, libcuda/libnvidia-encode dlopen'ed at runtime, primary context reuse). Fallback: native NVENC -> FFmpeg NVENC process -> libx264. inproc-av no longer links FFmpeg; VAAPI/CPU always use the process pipeline. Default builds are unchanged.
Co-authored-by: ferstar <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The in-process encoder opened its context at the raw capture size, so an
odd logical desktop (e.g. two stacked 150% monitors, 2560x2507) made every
H.264/HEVC/NVENC encoder fail to open. The encoder size is now rounded up to
even, and a native capture is copied into a zeroed frame of that size so the
extra row/column stays black. Scaled output paths are unchanged.
Co-authored-by: ferstar <[email protected]>
One video track carries H.264 or H.265 (RFC 7798 payloader); the host
announces its encoders with CONTROL Message 18 CodecNegotiation and picks
H.265 when the viewer negotiated it (the official Mac viewer offers H265
first; CaptureSettingRequest field 13 only reports the current codec).
hevc_nvenc / libx265 / hevc_vaapi with fallback to software HEVC, then
H.264; UU_VIDEO_CODEC=h264 forces H.264. True color 4:4:4 applies to H.265.
Verified with the official Mac viewer on NVENC.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Viewer PLI/FIR: dedupe only while an IDR is pending; drop the 1 s gap so a lost IDR cannot block recovery.
- XShm: release the local mapping on every attach failure; mark the segment for removal in all cases.
- Damage events no longer defer an overdue capture.
- After idle, the next tick is one frame after now instead of now.
The in-process encoder can now capture through XShm and XDamage in pure Rust
(x11rb) instead of libav's x11grab. Damage marks the picture dirty; a dirty
picture is captured and encoded on a fixed frame tick so the stream keeps a
steady cadence. An idle screen sends a keepalive every 40-67 ms (15-25 fps,
random) so the viewer does not treat a static desktop as stalled.
- src/damage_capture.rs: XDamage NonEmpty reports and XShm copy-out
- src/inproc_encoder.rs: run_damage loop; x11grab stays as the fallback
- UU_CAPTURE=grab forces the x11grab path
- x11rb is an optional dependency, enabled by the inproc-av feature only
Measured on a static desktop, CPU drops from about 12% to about 1% versus
x11grab. Moving content costs about the same CPU as x11grab.
The GPU path skipped the CPU colour conversion to YUV420P: frames already in
BGR0 at the encoder size go straight to h264_nvenc, other sizes are scaled to
BGR0. The libx264 path keeps YUV420P. Probe: 300 frames cut CPU from ~2.4 s to
~0.8 s with the same bitrate.
Address PR review:
- The FFmpeg 6 bindings (AVChannelLayout, ffmpeg-next 6.1) do not build on
the 22.04 runners, which ship FFmpeg 4.4. The default build no longer
depends on libav at all, so `cargo test --locked` and the release job on
22.04 no longer compile that code.
- The release tarball no longer links the build machine's libav SONAMEs.
Default builds use the FFmpeg child process for video and audio again.
- `UU_FFMPEG_ROOT` / config `ffmpeg_root` now selects the child-process path
even in an inproc-av build, so a bundled FFmpeg with working NVENC is not
silently bypassed.
Build with `--features inproc-av` (FFmpeg 6 dev packages, Ubuntu 24.04) to
get in-process capture, encoding and Opus. CI has a test-inproc job for it.
`UU_VIDEO_PIPELINE=process` still forces the child path for video.
Probes on ros2go-gpu (440 Hz tone, 5 s):
default (ffmpeg child): 44.0 packets/s, 132 kbps, max gap 43 ms
inproc-av: 49.7 packets/s, 133.5 kbps, max gap 22 ms
desktop-video-probe under inproc-av with UU_FFMPEG_ROOT=/usr uses the child
path (no `pipeline=in-process` in the log); without it, in-process.
The Opus encoder for desktop sound now runs in this process: PulseAudio
or PipeWire-pulse monitor input, resampled to s16 stereo 48 kHz, grouped
into 20 ms libopus frames, and sent as raw Opus packets. The Ogg muxing
and OpusTags skipping from the child-process path are gone.
Probe on ros2go-gpu with a 440 Hz tone on auto_null (desktop-audio-probe,
8 s): 49.7 packets/s, 134 kbps (target 128 kbps), largest gap between
packets 22 ms. No ffmpeg child process is started while it runs.
Not covered: the full viewer session start has not been checked against a
live UU client.
Also enables ffmpeg-next's software-resampling feature.
Capture (x11grab) and H.264 encoding (h264_nvenc / libx264) now run in
this process through ffmpeg-next, with the same codec options as the old
ffmpeg CLI. Each encoder packet is one access unit, fed to the existing
DependencyGate. A PLI or FIR from the viewer forces the next frame to be
an IDR. Wayland and portal capture still use the child-process path.
Set UU_VIDEO_PIPELINE=process to fall back to the old ffmpeg child.
desktop-video-probe accepts UU_PROBE_FRAMES to lengthen the sample.
Measured with desktop-video-probe on ros2go-gpu (RTX 4090 D, Xvfb
1920x1080 @60fps, 600 frames, mostly static desktop). CPU is the sum of
the probe process and its children, sampled from /proc once per second.
Probe only: no WebRTC send, no audio.
encoder pipeline CPU (%) wall (s) bitrate (kbps)
nvenc process ~138 10.17 5356
nvenc in-process ~43 10.34 5176
libx264 process ~135 10.09 5073
libx264 in-process ~72 10.08 4986
All four outputs decode without errors and start with an IDR.
Not covered: audio still runs its own ffmpeg child (src/audio.rs), PLI to
IDR has not been checked against a live viewer, and the CI jobs need the
libav*-dev and libclang-dev packages (added to ci.yml, release.yml,
AGENTS.md and the READMEs).