Skip to content

fix(browser): agent clicks give your keyboard back (3/3) - #28454

Merged
nwparker merged 1 commit into
mainfrom
nwparker/agent-browser-focus-3-press
Oct 14, 2026
Merged

nwparker merged 1 commit into
mainfrom
nwparker/agent-browser-focus-3-press

Conversation

@nwparker

@nwparker nwparker commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 11 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​994 $\color{#cf222e}{\Huge{\mathbf{−}}}$​163 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​831
Prod 20 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​560 $\color{#cf222e}{\Huge{\mathbf{−}}}$​104 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​456

ELI5

When an agent clicked something in Orca's browser, your keyboard silently moved into that page, so what you typed next went to the browser instead of your terminal. Now, after an agent's click, Orca puts your keyboard back where it was. The page still behaves as if it was clicked normally.

What Changed

  • Before:
    • click and mouse down called webContents.focus() on the page before pressing.
    • Clicks from the bundled browser helper also moved focus, because Chromium hands focus to any <webview> that receives a mouse press, however the press is sent.
    • Nothing ever gave focus back, so the user's next keystrokes went into the page. That includes Enter on an approval prompt.
  • After:
    1. Before an agent presses on a page embedded in an Orca window, main turns on CDP focus emulation for that page. The page keeps believing it's focused, so a menu or picker the agent opens doesn't close when your focus returns.
    2. Main then asks the pane that hosts the page to remember the element you had focused, and waits for its answer (up to 150 ms).
    3. When the press ends, the pane puts focus back on that element, and checks once more 50 ms later in case Chromium's focus move arrived late. If you were already typing in that page, nothing moves.
  • The input from PR 2 that still needs CDP is wrapped in the same hand-back: keys and text sent while focus is inside a frame on the page, text with control characters, unknown keys, and root-session input-method composition. It focuses the page only for that one command.
  • Overlapping agent input (two pages at once, back-to-back presses) all returns to the element you had before the first one. A guard whose end message never arrives releases itself after 30 s.
  • The input is refused when the user's focus can't be protected. If the pane doesn't answer within 150 ms, or no pane hosts the page, the command fails with browser_focus_guard_unavailable instead of being sent.
  • Agent Tab keys borrow focus the same way. Tab from the page's last field otherwise moves focus out of the page and onto Orca's next element (measured).
  • Agent keys are marked as the agent's. While an agent's input is being sent, Orca's shortcut and grab handlers for the page ignore keys, and setIgnoreMenuShortcuts keeps unhandled keys away from Orca's app menu. An agent's ⌘/Ctrl+T, ⌘/Ctrl+W or Ctrl+Tab stays in the page. In testing, synthetic input never reached those handlers anyway; the mark guarantees it.
  • Pages outside an Orca window are unchanged: they still focus natively, which their caret and input methods need.
  • The end-to-end spec gains a click test:
    • the agent clicks a button, presses the mouse, clicks a field, and fills multi-line text;
    • the terminal keeps focus, and the clicked field stays focused inside the page;
    • your next keystroke reaches the terminal.

Why

Measured in a hidden Electron window: every kind of press moves focus into the <webview> (CDP mouse, touch, tap gestures, sendInputEvent), and neither inert nor tabindex on the element stops it. So focus has to be given back. Only the renderer that hosts the page can do that, which is why main asks the pane.

Putting focus back the instant it moves would make the page see a blur, and pages close menus on blur. Focus emulation plus restoring when the press ends keeps the page's menus open. The cost is the press itself, a few milliseconds during which your keystrokes can reach the page.

Offscreen rendering (#24047) would remove even that window, but rebuilds every native popup and is about 8,000 lines.

Linked Issue

Fixes #20595. Also addresses #19297, #8139 and #20533. #16674's other paths (window activation on Windows) are out of scope.

Visual Proof

N/A. Focus moves are not visible in a screenshot. The end-to-end click test is the before/after: on the PR 2 code the terminal loses focus after the agent's click (terminalFocused: false); on this branch it keeps it.

Testing

  • Unit:

    • main guard: ready handshake, timeout, replies from the wrong renderer ignored, release exactly once;
    • proxy: presses wait for the guard and releases stay queued behind them; IME composition borrows focus; an insert abandoned mid-way still gives focus back; offscreen pages are unguarded;
    • bridge: click and mouse down no longer focus embedded pages;
    • renderer guard: restore on end, late focus move, user already in the page, user moved elsewhere meanwhile, two pages at once, back-to-back presses, abandoned guard.
  • End-to-end (browser-agent-input-focus-isolation.spec.ts, run hidden):

    • covers keys and text, clicks and presses, last-field Tab, and agent ⌘/Ctrl+T, ⌘/Ctrl+W and Ctrl+Tab;
    • macOS 3/3;
    • Linux (Arch, Hyprland, Wayland) 3/3;
    • Windows 3/3;
    • the click test fails on the PR 2 code.
  • Visible window: browser-agent-input-focus-isolation-native.spec.ts (@headful) runs the same scenarios in a window that has OS focus, then types a real OS keystroke (xdotool). The keystroke must reach the terminal, not the page.

    • Results on Linux, Xvfb with openbox: 3/3 on this branch, and the click scenario fails on the PR 2 code.
    • webContents.getFocusedWebContents() reports the page whenever a <webview> exists, so it can't show where typing goes. That's why the spec uses a real keystroke.
    • CI's Xvfb lane now installs xdotool.
  • pnpm tc, suites and the changed-code quality gate pass.

  • Platforms: hidden-window runs on macOS, Linux (Wayland) and Windows; visible-window runs on Linux (Xvfb, locally and in CI).

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Known gaps (listed in the doc):
    • during an agent press your focus is briefly in the page, so an in-progress input-method composition is lost;
    • focus emulation stays on while the debugger is attached, so a page you really leave sees no blur;
    • a cross-site frame's own menu can still close.
  • Security: one new IPC pair (browser:agentInputFocusGuard / ...Ready). It carries a page id, a random token and a boolean. Replies are matched to the renderer that was asked.
  • Cross-platform: no platform branches.
  • SSH / remote: a page hosted on a paired client is guarded on that client's own window. The web client has no <webview>, so it doesn't subscribe.
  • Performance: one IPC round trip per agent press or Tab, about a millisecond. A pane that doesn't answer fails the command after 150 ms.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from bbc5e98 to 98149a0 Compare October 14, 2026 04:36
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-2-keys branch from a895224 to 103fd93 Compare October 14, 2026 04:49
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 98149a0 to 4a749dd Compare October 14, 2026 04:49
@nwparker
nwparker marked this pull request as ready for review October 14, 2026 04:49
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 4a749dd to 39fa406 Compare October 14, 2026 05:48
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-2-keys branch from 103fd93 to fba4e8b Compare October 14, 2026 06:04
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 39fa406 to 22b9280 Compare October 14, 2026 06:07
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-2-keys branch from fba4e8b to 9c73fee Compare October 14, 2026 06:30
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 22b9280 to 9b26935 Compare October 14, 2026 06:30
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-2-keys branch from 9c73fee to ba51be1 Compare October 14, 2026 06:50
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 9b26935 to 8583146 Compare October 14, 2026 06:50
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-2-keys branch from ba51be1 to 5631757 Compare October 14, 2026 07:22
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 8583146 to 29517fc Compare October 14, 2026 07:22
Any press on a <webview> guest moves Chromium focus into it, so an agent click
took the user's terminal focus (#20595). Main now asks the hosting pane to guard
focus for each agent press (Orca's click/mouse down and the CDP proxy) and the
pane puts it back inside the webview's focus event, and again when the press
ends. Offscreen pages keep their native focus.
@nwparker
nwparker force-pushed the nwparker/agent-browser-focus-3-press branch from 29517fc to aa565ae Compare October 14, 2026 08:13
@nwparker
nwparker changed the base branch from nwparker/agent-browser-focus-2-keys to main October 14, 2026 08:13
@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 35007bcc-a63a-430e-a75e-5f07a916a3c5
📥 Commits

Reviewing files that changed from the base of the PR and between 8d88c44 and aa565ae.

📒 Files selected for processing: 31
  • .github/workflows/e2e.yml
  • src/main/browser/agent-browser-bridge-interaction-commands.ts
  • src/main/browser/agent-browser-bridge-lifecycle.ts
  • src/main/browser/agent-browser-bridge-mouse-commands.ts
  • src/main/browser/agent-browser-bridge-mouse-input.test.ts
  • src/main/browser/agent-browser-bridge-pointer-commands.ts
  • src/main/browser/agent-browser-bridge-pointer-input.test.ts
  • src/main/browser/agent-browser-bridge-test-harness.ts
  • src/main/browser/agent-input-focus-guard.test.ts
  • src/main/browser/agent-input-focus-guard.ts
  • src/main/browser/agent-input-injection.test.ts
  • src/main/browser/agent-input-injection.ts
  • src/main/browser/browser-guest-grab-shortcuts.ts
  • src/main/browser/browser-guest-shortcut-forwarding.ts
  • src/main/browser/browser-manager-visibility.ts
  • src/main/browser/cdp-dom-focus-replay.ts
  • src/main/browser/cdp-page-input-commands.ts
  • src/main/browser/cdp-ws-proxy-input-focus.test.ts
  • src/main/browser/cdp-ws-proxy-test-harness.ts
  • src/main/browser/cdp-ws-proxy.ts
  • src/main/browser/guest-direct-key-input.test.ts
  • src/main/browser/guest-direct-key-input.ts
  • src/preload/api/browser-api.ts
  • src/preload/api/browser-bridge-page-interaction-and-sessions.ts
  • src/renderer/src/components/browser-pane/host-guest/agent-input-focus-guard.test.ts
  • src/renderer/src/components/browser-pane/host-guest/agent-input-focus-guard.ts
  • src/renderer/src/hooks/ipc-events/browser-state-ipc-bridge.ts
  • src/renderer/src/hooks/useIpcEvents-lifecycle.test.ts
  • tests/e2e/browser-agent-input-focus-isolation-native.spec.ts
  • tests/e2e/browser-agent-input-focus-isolation.spec.ts
  • tests/e2e/helpers/browser-agent-focus-scenarios.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

brennanb2025 pushed a commit that referenced this pull request Oct 15, 2026
Any press on a <webview> guest moves Chromium focus into it, so an agent click
took the user's terminal focus (#20595). Main now asks the hosting pane to guard
focus for each agent press (Orca's click/mouse down and the CDP proxy) and the
pane puts it back inside the webview's focus event, and again when the press
ends. Offscreen pages keep their native focus.

(cherry picked from commit 9c0420f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Agent browser commands steal the user's keyboard focus or type into their terminal

1 participant