Repository navigation
feat(mobile): phone reads chat visuals as ordinary files on the chat's host - #28309
Closed
brennanb2025 wants to merge 2 commits into
Closed
brennanb2025 wants to merge 2 commits into
brennanb2025 wants to merge 2 commits into
Conversation
…s host On a host that advertises files.host-path.v1 and publishes the chat's scratchDirectory, the phone reads a visual with files.read by host path and the read-only chat-visual access. Older hosts, and chats the host gives no folder, keep agentSession.readVisual.
The session route clears its capability answer on every disconnect, which flipped shown visuals back to readVisual and re-downloaded them. Keep the last probe answer per client in memory until the next one replaces it.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI5
When a chat agent draws a chart or diagram, it writes an HTML file into the chat's own scratch folder on the computer that runs the chat, and its reply says "show
chart.htmlhere". The phone then shows that page inside the chat and full screen.Until now the phone could only get that page through a special chat-only call: "give me file
chart.htmlof chat X". With this PR, on hosts that support it, the phone reads the page the ordinary way. It asks the host to read the file at its full path, with a read-only permission that only works for chat visuals. The page you see is the same. Older hosts keep the old way.What Changed
The problem. The phone fetched every chat visual through
agentSession.readVisual, a call that exists only for visuals. The rest of this stack moves visuals onto ordinary host files:chat-visualpermission.files.host-path.v1.scratchDirectory) on the status stream.The phone was the last client still tied to the visual-only call on hosts that no longer need it. The plan's later clean-up (retiring
readVisualonce no supported client calls it) depends on this.User-facing change. None visible: the same inline visual and the same full-screen sheet, including in floating chats. A visual the host refuses still shows "Visualization unavailable", and tapping it still tries again. No new phone editor, Share button or outside-file viewer.
Mechanism.
useMobileNativeChatVisualSource(new) builds the chat's visual source:files.host-path.v1and the status stream, and its status summary for this chat carriesscratchDirectory. The phone reads by path in that folder.agentSession.readVisual. That covers an older host, a chat the host gives no folder (Pi, WSL), and the moment before the status stream has said anything.nativeChatVisualHostPath(scratchDirectory, file)(new,src/shared/native-chat-visual-host-path.ts) joins the folder and the bare file name in the host's own path syntax: a backslash for a Windows host, a slash otherwise, whatever the phone runs.files.readwith{ path, access: { kind: 'chat-visual' } }. The host's protected visual reader serves it: only a visual directly inside a chat folder under the host's scratch root, read-only, size-capped.mobile-native-chat-visual-file-read.ts, new). These map to the same three outcomes the old reader produced, so the screen is unchanged:StructuredAgentSessionHostSupportgainshostPathFiles, read from the same capability probe as the other structured-chat flags. The probe's answer is also remembered per connection (rememberStructuredAgentSessionHostSupport), so the visual's reader doesn't change while a reconnect waits for the next answer.useMobileStructuredSessionStatusValue(new) is the one subscription to the per-host status stream. The existing "Stopping…" read now uses it too, with no behaviour change.Routing (direct and relayed).
files.readis already an execution-host method on the phone allowlist. A phone connected directly reaches the host's own dispatcher. A phone relayed through its desktop to a server has the frame forwarded with params intact and the routing tag stripped (rewriteRelayedPhoneRequest).assertHostPathServedHere). No allowlist or relay change was needed; a new test pins both facts.Stack position. S12 of 14 in the chat-visuals scratch-folder stack (plan PR 14).
brennanb2025/cvs-s9-scratch-directory), which contains S0 to S5.src/shared/native-chat-visual-host-path.tsand its test are byte-identical to the copies in S10 (desktop visual reads), so the two merge cleanly.Why
files.readby path with thechat-visualpermission (S4). The phone already holds the folder from the status stream (S9). Reusing both adds no new wire surface.Differences from the common pattern:
readVisual, that first-open read must be held until the status answer instead of falling back.ENOENT, access denied,file_too_large,binary_file, not a regular file), as its file preview already does. The host sends these asruntime_errormessages, not typed codes.Mixed versions.
files.host-path.v1or publishscratchDirectory, so a new phone keeps usingreadVisualexactly as before.Linked Issue
Part of the chat-visuals scratch-folder stack (no separate issue).
Visual Proof
N/A: the phone shows the same visual through a different read; there is no visible change. Live phone QA is at the end of the stack.
Testing
mobile/src/session/mobile-native-chat-visual-path-read.test.ts(new, 8):files.readwith the joined path andchat-visualaccess, and caches the page;invalid_argument,forbidden, a truncated reply) takes the page down and drops the cache;mobile/src/session/use-mobile-native-chat-visual-source.test.tsx(new, 7):mobile/src/session/use-mobile-native-chat-visual-reconnect.test.tsx(new, 2), using the real source, reader and visual hook:readVisualincluded;mobile/src/session/use-mobile-native-chat-visual.test.tsx(+1): the same content keeps the same state object (no frame reload) and new content replaces it. The existing refused-to-"unavailable" and retry tests still pass.mobile/src/session/mobile-native-chat-visual-read.test.ts: the old reader's tests are unchanged except for the ready result no longer carrying a revision.mobile/src/session/mobile-structured-agent-session-host-support.test.ts(+2):files.host-path.v1maps tohostPathFiles; each client keeps its last answer until the next one replaces it.src/shared/native-chat-visual-host-path.test.ts(new): POSIX, Windows drive and UNC joins; invalid names and an empty folder build nothing.src/main/native-chat/native-chat-visual-host-path-reader-agreement.test.ts(new): the shared join's output passes the host reader's own path check (nativeChatVisualPathParts) for a POSIX and a Windows host.src/main/runtime/runtime-rpc/runtime-rpc-mobile-chat-visual-path-routing.test.ts(new):files.readis an execution-host phone method, and a relayed path read keeps its params and loses its routing tag.Local runs:
src/session,src/filesandsrc/transport: 423 files pass. One pre-existing failure,transport/mobile-e2ee-v2-client-session.test.ts"preserves full large text frames…", fails identically on the base branch without this change.tsc: node, web and mobile pass; the mobile tests typecheck ratchet is OK.check:code-quality:changedand CI's anti-slop oxlint step on the changed files are clean.Tested on macOS only (unit level). The Windows host path is covered by the join and reader-agreement tests.
I manually tested these changes locally
Automated tests added/updated, or explained why not below
Notes
chat-visualis read-only and limited by the host to visual files directly inside a chat folder under its scratch root, andfiles.readis already a phone method under the same permission.