Repository navigation
feat(native-chat): desktop reads chat visuals as ordinary files on the chat's host - #28310
Closed
brennanb2025 wants to merge 2 commits into
Closed
brennanb2025 wants to merge 2 commits into
brennanb2025 wants to merge 2 commits into
Conversation
…e chat's host Inline visuals on hosts that publish the chat's scratch folder and read files by path are read as the file at that folder plus the reply's bare name, with the chat-visual access kind: desktop IPC for a local chat, the host's files.read for paired and SSH chats. They read again on reveal, turn end, host return and an editor save of the same file; unchanged pages keep their frame and a lost host keeps the shown page. Older hosts keep the per-session reader.
6 of 9 tasks
A failed or silent probe left a capable host's visuals as blank placeholders. Visuals now wait only while the files.host-path.v1 probe is running, for at most 15 s, then read through the older endpoint, which every host serves; a later answer on regained contact moves them to path reads. A local chat never waits, and a pending visual paints a page either reader already cached. The hook calls the shared retry rule.
8 of 9 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI5
When an agent draws a chart or page in a chat reply, it writes an HTML file into the chat's own scratch folder on the computer the chat runs on, and Orca shows that file inside the reply. Until now the Orca window fetched each picture through a special "visuals only" request, and it fetched it once: if the agent rewrote the file later, an earlier reply kept showing the old picture until you reopened the chat. This PR makes the window read the picture as an ordinary file at its real path on the chat's own computer, and read it again at the moments the file may have changed.
What Changed
The problem.
::orca-visual{file="x.html"}in a reply) was read throughagentSession.readVisual, a visuals-only request that names the chat and the file name. It is a separate way of reading files, with its own revision protocol, beside the ordinary file reads every other file uses.User-facing change.
Mechanism.
scratchDirectoryon the chat status it already streams (S9). The visual's path is that folder joined with the reply's validated bare file name, in the host's own path style (\for a Windows host,/otherwise). The join is the sharednativeChatVisualHostPath, which the phone (S12) uses too.chat-visualaccess kind (S4), which only reads a visual file directly inside a chat folder under that host's scratch root and allows no other operation:files.readby path. Nothing is read from this computer's disk for a remote chat.scratchDirectory(this build's own file read always serves thechat-visualkind, so it never waits on a capability check);scratchDirectoryand the host advertisesfiles.host-path.v1(S5);native-chat-visual-path-read-client.ts): a visual is its host plus its absolute path. A small bounded cache (24 pages, 6 MB of text) paints a remounted visual at once; every load still reads the file. Visuals of the same file asking at once share one read. Each refresh trigger takes a "read mark", and a refresh never joins a read that began before its mark, so it always sees the file as it is after the trigger. A slower, older read can't overwrite a newer cached page. Failures are never cached; a refusal (missing, not a file, too large, not text, outside a chat folder) drops the cached page, and losing contact keeps it.native-chat-visual-refresh.ts,native-chat-visual-owner.tsx,NativeChatInlineVisual.tsx). No new watchers.display: none, so showing the tab counts too.Deletes. On hosts that read by path, inline visuals no longer use the visuals-only request or its revision protocol. The legacy reader stays for older hosts and for the existing "Open in new tab" visual tab, which S11 replaces.
Stack position. S10 of 14 in the chat-visuals scratch-folder stack. Base and parent: S9 (#28263, host publishes each chat's
scratchDirectory), which contains S0 to S5. Depends on this: S11 (Open a visual like any file) and S14 (inline Share).Why
Differences from the common pattern:
Linked Issue
N/A, maintainer stack PR (chat-visuals scratch-folder plan, PR 12 of the signed plan).
Visual Proof
N/A in this PR: visuals look the same; what changes is when a rewritten file shows. Live end-to-end QA (local, paired and SSH chats) runs once at the end of the stack.
Testing
New tests:
native-chat-visual-path-read-client.test.ts(14): local chat reads over IPC with thechat-visualkind and never over RPC; paired and SSH (Windows host path) chats read withfiles.readby path and never from this computer; the same file name in two chats and the same path on two hosts stay apart; concurrent reads share one read; a fresh read never joins an earlier one but shares a later one, and an older read finishing last does not replace the newer cached page; error classification (missing, refused kinds, transport); binary and cut-off reads refused; contact loss keeps the cached page, a verdict drops it.NativeChatInlineVisual.host-file.test.tsx(12): local and SSH (Windows) reads by path; older host and no-folder chats keepagentSession.readVisual; nothing read while the route is pending, but a page cached by either reader still shows; turn end re-reads, unchanged content keeps the same frame element, changed content shows; host return re-reads and a failing host keeps the page through every retry; scrolled away and back re-reads; an editor save of this file re-reads, a save of another file or the same path on another host does not; a refused file shows the existing "Visualization unavailable" line.native-chat-visual-owner.test.tsx(11, real capability hook): routing by published folder plus capability; waits while the check runs and switches to the visuals-only reader exactly when the 15 s wait is up; a failed check, a host without path reads and a host with no folder use the visuals-only reader; a failed check still shows the visual through that reader (no blank box); a local chat reads by path at once with its capabilities unknown; floating chats; host return refreshes and re-checks; the owner object stays the same when the route changes.src/shared/native-chat-visual-host-path.test.ts: the shared join (identical copy shared with S12).AI Disclosure
Review
Agent skill upstream boundary
Notes
chat-visualkind; the host and desktop refuse any other path or operation with it. No write path is added.files.readby path; nothing is read from this computer for a remote chat.files.host-path.v1capability; otherwise the existing reader runs.Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)