[Bug]: Streamed browser hover and clicks lag behind stale queued pointer events #17538

Before submitting

• I searched existing issues and did not find an equivalent queued-input fix.
• I included a deterministic reproduction, a recording and raw measurements.

Area

apps/server

Steps to reproduce

1. Open a streamed Browser tab in the web client, take control and set an 800 × 800 viewport.
2. Run the supplied fixture.py and open http://127.0.0.1:18777/?busy=55. The synthetic page does 55 ms of synchronous work per mousemove event.
3. Send 120 hover positions at 50 Hz, then stop and click. The supplied stream driver sends this sequence through the web client's active viewer WebSocket.
4. Observe the page processing obsolete positions after movement stops, while the click waits behind them.

Scripts, candidate patch, capture protocol and raw timestamps: https://gist.github.com/gillesgw/42c7ca4df0e429ba84b122d93906d6ab

A smaller reproduction loads the actual original and candidate SessionControl classes and simulates 55 ms per dispatch:

node queue-repro.mjs /path/to/patched/t3code ec80933ac8cd02fec5c97b342462ccc9567cdb1e

Expected behavior

Hover input should converge quickly to the latest position under load. Clicks, keys, drag points, navigation and ownership changes must preserve their ordering and control guarantees. Scroll distance must not be lost.

Actual behavior

Hover effects and clicks can be seconds late. Every obsolete pointer position remains queued, even after the viewer stops moving.

Before/after recording (MP4, 15 seconds) compares the actual Chromium page in T3: original FIFO on the left, candidate on the right, with identical offered inputs.

Measurement limits: the recorded driver bypasses client animation-frame batching to keep server input at 50 Hz during nested capture. Latency comes from sender and page-handler timestamps on the same host; it includes localhost transport, queue and CDP/page processing, and excludes image decode/display. These are single stress samples with recording overhead, not production latency estimates. Obsolete waiting moves are intentionally skipped by the candidate.

The separate queue-only reproduction also shows the backlog with bounded dispatch: original median wait 2,119.8 ms and drain time 4,269 ms; candidate 11.3 ms and 107.1 ms. These simulated queue measurements exclude network and graphics.

Impact

Major degradation or frequent failure

Version or commit

Recorded against main @ ec80933ac8cd02fec5c97b342462ccc9567cdb1e. The relevant runtime files are unchanged on main @ 22ccf8a3dfc9094b4ccdb689e128c67bacbbcdf5, checked before submission.

Environment

Linux x64; Chromium 154.0.8037.92, software rendering; 800 × 800 page viewport. Node 26.8.2 for recorded browser runs, Node 24.19.0 for queue reproduction and focused tests.

Diagnosis and related work

ServerBrowser.attachViewer submits each input through tab.control.human; SessionControl.enqueue chains every action onto tail. When input outpaces bounded CDP/page processing, stale positions continue delaying newer input.

• fix(web): desktop opens remote environments' browser tabs locally #17316, which closed [Bug]: Streamed server-browser previews are high-latency and CPU-heavy on remote GPU-less hosts, with no local-render path #16827, restores local rendering for user-opened desktop tabs. Web/mobile and agent-opened tabs still stream.
• fix(server): a slow typed address no longer freezes input in a server browser tab #16691 addresses slow viewer navigation.
• fix(server): isolate browser request timeouts and revoke expired work #16941 addresses stalled/expired browser work.
• feat(preview): add automatic browser control preference #17081 addresses automatic control acquisition/release.

Those fixes address different causes and do not merge queued hover bursts.

Workaround and candidate scope

A candidate server patch merges adjacent waiting continuous inputs: latest no-button hover with matching modifiers, compatible accumulated wheel deltas, and latest valid resize. It preserves running inputs and click/key/text/drag/navigation/system/ownership boundaries. It changes four server implementation/test files, with no dependency or wire-contract change.

All 67 focused tests pass, including 13 added batching/control cases; lint, formatting and server typecheck pass. Earlier nightly checks exercised the real canvas handler, clicks, text, scrolling and control handoff, and the patched nightly improved ordinary use for the reporting user. Native desktop and mobile interaction have not been manually tested.

Please triage this failure and intended scope before its candidate PR is submitted, following CONTRIBUTING.md. No prior approval is claimed.

Generated with GPT-6.1 Sol. Harness: Codex through T3 Code.