Skip to content

feat(native-chat): desktop reads chat visuals as ordinary files on the chat's host - #28310

Closed
brennanb2025 wants to merge 2 commits into
brennanb2025/cvs-s9-scratch-directoryfrom
brennanb2025/cvs-s10-desktop-visual-reads
Closed

brennanb2025 wants to merge 2 commits into
brennanb2025/cvs-s9-scratch-directoryfrom
brennanb2025/cvs-s10-desktop-visual-reads

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​735 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​735
Prod 11 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​633 $\color{#cf222e}{\Huge{\mathbf{−}}}$​65 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​568

ELI5

When an agent draws a chart or page in a chat reply, it writes an HTML file into the chat's own scratch folder on the computer the chat runs on, and Orca shows that file inside the reply. Until now the Orca window fetched each picture through a special "visuals only" request, and it fetched it once: if the agent rewrote the file later, an earlier reply kept showing the old picture until you reopened the chat. This PR makes the window read the picture as an ordinary file at its real path on the chat's own computer, and read it again at the moments the file may have changed.

What Changed

The problem.

  • An inline visual (::orca-visual{file="x.html"} in a reply) was read through agentSession.readVisual, a visuals-only request that names the chat and the file name. It is a separate way of reading files, with its own revision protocol, beside the ordinary file reads every other file uses.
  • It read a visual when the reply first scrolled into view, and not again. An agent that updates the same file (the usual way to revise a chart) left earlier replies showing the old version until the chat was reopened.

User-facing change.

  • The same inline visuals, read the same way for chats on this computer, on a paired machine and on an SSH host.
  • An earlier reply shows a rewritten file after any of these:
    • the visual scrolls back into view, or its chat tab is shown again;
    • the agent's turn ends;
    • the chat's host comes back after losing contact;
    • you save that same file in Orca's editor.
  • A page that comes back unchanged keeps its frame, so whatever you did inside it (scroll, selected tab, expanded section) survives. If the host can't be reached during a refresh, the page already shown stays; losing contact says nothing about the file.
  • Older hosts are unchanged: they keep the visuals-only reader and its current behavior.

Mechanism.

  • Where the file is: the chat's host publishes the chat's scratch folder as scratchDirectory on the chat status it already streams (S9). The visual's path is that folder joined with the reply's validated bare file name, in the host's own path style (\ for a Windows host, / otherwise). The join is the shared nativeChatVisualHostPath, which the phone (S12) uses too.
  • Which host: always the chat's recorded host (the host stored on its tab), never the selected workspace's host. Two hosts can share a workspace id.
  • How it is read, always with the restricted chat-visual access kind (S4), which only reads a visual file directly inside a chat folder under that host's scratch root and allows no other operation:
    • a chat on this computer: the desktop file read over IPC;
    • a chat on a paired machine or an SSH host (both go through Orca's managed server): the host's ordinary files.read by path. Nothing is read from this computer's disk for a remote chat.
  • When the path route is used:
    • a chat on this computer: whenever its status carries scratchDirectory (this build's own file read always serves the chat-visual kind, so it never waits on a capability check);
    • a remote chat: when its status carries scratchDirectory and the host advertises files.host-path.v1 (S5);
    • otherwise (older host, WSL chat, a host that publishes no folder) the visual keeps the existing visuals-only reader, unchanged.
  • While the remote capability check is actually running, a visual reads nothing (so it is not fetched twice) but still shows a page either reader already cached. The wait ends when the check answers or after 15 seconds (the remote call timeout). A failed or unanswered check means "not known to read by path", and the visual uses the visuals-only reader, which every host serves. When contact with the host is regained, the check runs again and can move the visual to path reads. The page content is the same, so nothing remounts.
  • That choice lives in a separate React context from the chat's owner, so learning it never remounts a visual that is already shown. The shared capability hook gains a variant that also says whether its check is still running; existing callers are unchanged.
  • Identity and cache (native-chat-visual-path-read-client.ts): a visual is its host plus its absolute path. A small bounded cache (24 pages, 6 MB of text) paints a remounted visual at once; every load still reads the file. Visuals of the same file asking at once share one read. Each refresh trigger takes a "read mark", and a refresh never joins a read that began before its mark, so it always sees the file as it is after the trigger. A slower, older read can't overwrite a newer cached page. Failures are never cached; a refusal (missing, not a file, too large, not text, outside a chat folder) drops the cached page, and losing contact keeps it.
  • Refresh triggers (native-chat-visual-refresh.ts, native-chat-visual-owner.tsx, NativeChatInlineVisual.tsx). No new watchers.
    • Reveal: the visual keeps its existing in-view observer and reads again when it comes back into view after being out of it. A hidden chat tab is display: none, so showing the tab counts too.
    • Turn end: the chat view's existing "turn finished" edge, next to the one file links already use.
    • Host return: the existing "contact with this host regained" signal.
    • Editor save: the existing editor "file saved" event. Its file id is resolved to the open file's owning host and path, and only a save of this visual's file on this host triggers a read.
  • One document hook serves both readers, so switching reader keeps what is shown; it compares HTML before replacing the frame, and the frame itself is rebuilt only for different HTML.

Deletes. On hosts that read by path, inline visuals no longer use the visuals-only request or its revision protocol. The legacy reader stays for older hosts and for the existing "Open in new tab" visual tab, which S11 replaces.

Stack position. S10 of 14 in the chat-visuals scratch-folder stack. Base and parent: S9 (#28263, host publishes each chat's scratchDirectory), which contains S0 to S5. Depends on this: S11 (Open a visual like any file) and S14 (inline Share).

Why

  • Ordinary file reads over the visuals-only request. A visual is a file; reading it like one means the same host rules, the same SSH and paired transport, and one less special path to keep working. It is also what lets later PRs open and share the visual as a file.
  • The chat's host plus the host-published folder, not anything derived on this computer. Only the host knows its state folder and path style, and only the chat's recorded host is guaranteed to be the one the agent wrote to.
  • Refresh at known moments, not a file watcher. The scratch folder is not a watched root, and a watcher per visual on every remote host would cost far more than the few re-reads these triggers cause. A rewrite made outside these moments shows at the next one.
  • Keep the old reader for old hosts. Mixed versions are normal; an older host has no path reads, and its visuals must keep showing.

Differences from the common pattern:

  • intended: rewrites show at refresh moments (reveal, turn end, host return, editor save), not live. Safe because every one of those moments is when a user would next look, and it avoids watching files outside workspaces.
  • temporary: the legacy visuals-only reader stays for hosts without path reads and for the existing visual tab. Removed by follow-ups F4/F5 once supported hosts all read by path, and the visual tab by S11.

Linked Issue

N/A, maintainer stack PR (chat-visuals scratch-folder plan, PR 12 of the signed plan).

Visual Proof

N/A in this PR: visuals look the same; what changes is when a rewritten file shows. Live end-to-end QA (local, paired and SSH chats) runs once at the end of the stack.

Testing

  • I manually tested these changes locally (stack-end QA covers the live flows)
  • Automated tests added/updated, or explained why not below

New tests:

  • native-chat-visual-path-read-client.test.ts (14): local chat reads over IPC with the chat-visual kind and never over RPC; paired and SSH (Windows host path) chats read with files.read by path and never from this computer; the same file name in two chats and the same path on two hosts stay apart; concurrent reads share one read; a fresh read never joins an earlier one but shares a later one, and an older read finishing last does not replace the newer cached page; error classification (missing, refused kinds, transport); binary and cut-off reads refused; contact loss keeps the cached page, a verdict drops it.
  • NativeChatInlineVisual.host-file.test.tsx (12): local and SSH (Windows) reads by path; older host and no-folder chats keep agentSession.readVisual; nothing read while the route is pending, but a page cached by either reader still shows; turn end re-reads, unchanged content keeps the same frame element, changed content shows; host return re-reads and a failing host keeps the page through every retry; scrolled away and back re-reads; an editor save of this file re-reads, a save of another file or the same path on another host does not; a refused file shows the existing "Visualization unavailable" line.
  • native-chat-visual-owner.test.tsx (11, real capability hook): routing by published folder plus capability; waits while the check runs and switches to the visuals-only reader exactly when the 15 s wait is up; a failed check, a host without path reads and a host with no folder use the visuals-only reader; a failed check still shows the visual through that reader (no blank box); a local chat reads by path at once with its capabilities unknown; floating chats; host return refreshes and re-checks; the owner object stays the same when the route changes.
  • src/shared/native-chat-visual-host-path.test.ts: the shared join (identical copy shared with S12).
  • Mutation checks: removing the fresh-read rule, the reveal trigger, or keep-page-on-contact-loss each fails a test.
  • All native-chat renderer tests: 370 files, 3930 tests pass; renderer runtime tests (246 files) pass. Typecheck: web and node clean.

AI Disclosure

Review

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Security: reads use the restricted chat-visual kind; the host and desktop refuse any other path or operation with it. No write path is added.
  • Cross-platform: the path is joined in the host's path style; tested with a Windows host path from a non-Windows client.
  • SSH and paired: both go through the managed server's files.read by path; nothing is read from this computer for a remote chat.
  • Mobile: unchanged here (S12 does the phone side).
  • Mixed versions: the route needs both the published folder and the files.host-path.v1 capability; otherwise the existing reader runs.
  • Performance: no watchers; at most one read per visual per trigger, shared between visuals of the same file.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

…e chat's host

Inline visuals on hosts that publish the chat's scratch folder and read files by
path are read as the file at that folder plus the reply's bare name, with the
chat-visual access kind: desktop IPC for a local chat, the host's files.read for
paired and SSH chats. They read again on reveal, turn end, host return and an
editor save of the same file; unchanged pages keep their frame and a lost host
keeps the shown page. Older hosts keep the per-session reader.
A failed or silent probe left a capable host's visuals as blank placeholders.
Visuals now wait only while the files.host-path.v1 probe is running, for at most
15 s, then read through the older endpoint, which every host serves; a later
answer on regained contact moves them to path reads. A local chat never waits,
and a pending visual paints a page either reader already cached. The hook calls
the shared retry rule.
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Superseded by the rebuilt six-PR chat scratch stack (#28385 merged, #28389, #28397, #28437, #28394, #28479). This piece is replaced by #28437 (readVisual stays the inline reader, with a fresh re-read after a matching save). Closing.

@brennanb2025
brennanb2025 deleted the brennanb2025/cvs-s10-desktop-visual-reads branch October 14, 2026 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant