Skip to content

fix(desktop): preview screenshots work while the window is not painted - #152

Merged
lukemaj merged 2 commits into
mainfrom
fix/preview-capture
Oct 2, 2026
Merged

lukemaj merged 2 commits into
mainfrom
fix/preview-capture

Conversation

@lukemaj

@lukemaj lukemaj commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

What: Agent browser screenshots (preview_snapshot, preview screenshots) work again while the Chromeria window is minimized, hidden or covered.
Why: After first reveal the main window renderer is background-throttled. While the window is not painted it never composites a guest that automation moved into the viewport, so capturePage on the guest stalls (1 s timeouts) or fails with UnknownVizError.
So what: The maintainer reviews and merges. After install, the live check below confirms it on the installed build.

Closes #151

Root cause and evidence

  • Live trace (~/.t3/userdata/logs/desktop.trace.ndjson*, read-only): every PreviewManager.capturePageWithRetry from 2026-10-02 10:06:49 to 10:57:07 CEST failed on WebContents 2, 3, 4 and 5. That includes guests created fresh at 10:07:40 and 10:22:23, and the app and GPU processes were the same throughout (up since 02:14), so the failure was global, not one wedged guest. desktop.lifecycle.activate fired at 12:36:57; a background-thread snapshot at 12:45:44 then succeeded in 4 ms.

  • Standalone Electron 44.4.2 repro (/tmp/pc-repro/app/main.js): a <webview> guest moved into the viewport from -100000px (what hostedBrowserWebviewStyle.ts does for background automation), then capturePage on the guest.

    Window Throttled (before) Main window unthrottled (after) Only guest unthrottled
    visible ok ok ok
    minimized TIMEOUT 1500 ms ok, real guest pixels TIMEOUT
    hidden TIMEOUT 1500 ms ok, real guest pixels TIMEOUT
    occluded TIMEOUT 1500 ms ok, real guest pixels TIMEOUT

Change

  • capturePageWithRetry holds a page capture lease for the whole retried capture. The first lease unthrottles the main window unless a recording or picture-in-picture session already has it unthrottled. When the last lease ends and no session remains, the window is throttled again. A failed toggle logs a warning and never fails the capture.
  • The lease count changes only under frameCaptureSessionsRef, so it is serialized with session start and stop. stopFrameCapture, the failed-start rollback and setMainWindow respect in-flight captures.
  • Upstream files touched: apps/desktop/src/preview/Manager.ts, Manager.test.ts. They are listed in scripts/fork-upstream-edits.txt and docs/fork-features.md (preview-capture-unthrottle). The change is self-contained and can be offered upstream as is.

Proof

  • pnpm exec vp test run apps/desktop/src/preview/Manager.test.ts: 93 passed. The new test "keeps the main window unthrottled while a guest page capture runs" fails on the unfixed Manager.ts (expected [ 'capture' ] to deeply equal [ 'throttle:false', 'capture', …(1) ]), and also fails if the stopFrameCapture guard is dropped.
  • pnpm exec vp lint on both Manager files: exit 0. The one warning (no-extraneous-class at Manager.test.ts:211) is pre-existing.
  • Desktop typecheck (tsc --noEmit in apps/desktop): exit 0. vp fmt --check: clean. bash scripts/fork-check.sh: OK.
  • Not verified: the installed app. Remaining live-install item for the maintainer: with the new build running, minimize Chromeria (or cover it fully), then have an agent in any thread run preview_open with url: "https://example.com" and then preview_snapshot. Before: Desktop preview operation failed: automationSnapshot.capturePage ... TimeoutError. After: a screenshot is returned.

Review

Independent review: APPROVE, no blocking findings, on exact head d0cdd010bce1d5536a9e22d4dce1e610ac93d1a6. Reviewer: Prism reviewer thread sub.sub.46b03420-8643-4703-8756-8602f14b39fb.dispatcher-0f3d8c3717b2.reviewer-9dfffb3b6221 (codex / gpt-6-luna). Findings; review/independent = success.

Reversible assumptions

  • Toggling throttling per capture is cheap enough: it wakes a hidden renderer only for the duration of a snapshot.
  • The web client's move of the guest into the viewport commits without requestAnimationFrame. The capture's existing 3 x 1 s retries cover the first paint after unthrottling.

Out of scope

preview_open with url: "about:blank" fails with the generic "Preview automation open failed on client ..." error. That is a separate input-handling issue.

No versioning by design: this fork has no version policy, and builds are identified by commit SHA.

Elon record

  • Requirements and who asked: agent preview screenshots must work while the user is not looking at Chromeria (user via coordinator, 2026-10-02); a hidden window should stay cheap once captures end (existing DesktopWindow first-reveal design).
  • Deleted: the "background-thread offscreen host" and "wedged guest" hypotheses (fresh guests failed too, and background-thread snapshots work with the window visible); no retry tuning, web-side change or new setting.
  • Bottleneck: the throttled, unpainted embedder window, not the guest or the retry policy.
  • Checked myself: live trace timeline and process start times; live preview_open/preview_snapshot from a background thread; Electron 44.4.2 repro across all window states; reran the focused test on the commit.

Written by Claude Opus 5.5 (planner) and a Claude Opus 5.5 writer thread, in Claude Code inside Chromeria.

🤖 Generated with Claude Code

lukemaj and others added 2 commits October 2, 2026 13:00
After first reveal the main window renderer is background-throttled. While
Chromeria is minimized, hidden or covered, it never composites a guest that
automation moved into the viewport, so capturePage on the guest stalls or
fails with UnknownVizError. Hold the main window unthrottled for each guest
capture, as recording and picture-in-picture already do, and restore it once
no capture or frame-capture session needs it.

Closes #151

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Oct 2, 2026
@lukemaj

lukemaj commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Independent review: APPROVE

Reviewed exact head d0cdd010bce1d5536a9e22d4dce1e610ac93d1a6 against fe1302fff80654d62051080fa9028f009e3251f8. No blocking findings.

  • Capture lease release is scoped with acquireUseRelease, including capture errors, retry exhaustion, interruption during use, and stale guest detection after capture. Lease accounting and frame-session transitions share frameCaptureSessionsRef; start rollback, stop, and setMainWindow preserve the unthrottled state while another owner remains. The capture call sites do not hold that lock, so they do not recursively wait on it.
  • The standalone Electron repro matched the stated cause: visible captures succeeded; minimized, hidden, and occluded captures timed out with main-window throttling, and succeeded with main-window throttling disabled, returning guest pixels.
  • pnpm exec vp test run apps/desktop/src/preview/Manager.test.ts: 1 file passed, 93 tests passed. The added behavioral test checks ordering, retry failure cleanup, and overlap with recording, and would fail without the fix and without the stop guard.
  • Diff scope is limited to Manager.ts, its test, and the corresponding entries in scripts/fork-upstream-edits.txt and docs/fork-features.md.

Proof was limited to the focused test, source review, and the provided standalone repro. I did not run the installed-app recipe or broader lint/typecheck checks.

@lukemaj
lukemaj merged commit aed1a66 into main Oct 2, 2026
20 checks passed
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.5 KiB 13.5 KiB −12 B (−0.1%) 15.1 KiB ✅
Codex Thread snapshot wire 7.1 KiB 7.1 KiB +5 B (+0.1%) 7.3 KiB ✅
Codex Live turn WebSocket wire 6.5 KiB 6.4 KiB −17 B (−0.3%) 7.8 KiB ✅
Codex Live turn WebSocket decoded 56.3 KiB 56.2 KiB −44 B (−0.1%) 66.4 KiB ✅
Codex Live turn messages 10 9 −1 (−10.0%) 21 ✅
Claude Total thread wire 13.5 KiB 13.5 KiB +1 B (+0.0%) 15.1 KiB ✅
Claude Thread snapshot wire 7.1 KiB 7.1 KiB −4 B (−0.1%) 7.3 KiB ✅
Claude Live turn WebSocket wire 6.4 KiB 6.4 KiB +5 B (+0.1%) 7.8 KiB ✅
Claude Live turn WebSocket decoded 57.0 KiB 57.0 KiB 0 B (0.0%) 66.4 KiB ✅
Claude Live turn messages 9 9 0 (0.0%) 21 ✅

Baseline: 48ecc69 · PR result: d0cdd01 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 114.0 KiB
  • Claude decoded thread snapshot: 114.7 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(desktop): preview screenshots fail while the Chromeria window is not painted

1 participant