Skip to content

feat(mobile): phone reads chat visuals as ordinary files on the chat's host - #28309

Closed
brennanb2025 wants to merge 2 commits into
brennanb2025/cvs-s9-scratch-directoryfrom
brennanb2025/cvs-s12-phone-visual-reads
Closed

brennanb2025 wants to merge 2 commits into
brennanb2025/cvs-s9-scratch-directoryfrom
brennanb2025/cvs-s12-phone-visual-reads

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 12 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​605 $\color{#cf222e}{\Huge{\mathbf{−}}}$​12 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​593
Prod 11 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​251 $\color{#cf222e}{\Huge{\mathbf{−}}}$​36 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​215

ELI5

When a chat agent draws a chart or diagram, it writes an HTML file into the chat's own scratch folder on the computer that runs the chat, and its reply says "show chart.html here". The phone then shows that page inside the chat and full screen.

Until now the phone could only get that page through a special chat-only call: "give me file chart.html of chat X". With this PR, on hosts that support it, the phone reads the page the ordinary way. It asks the host to read the file at its full path, with a read-only permission that only works for chat visuals. The page you see is the same. Older hosts keep the old way.

What Changed

The problem. The phone fetched every chat visual through agentSession.readVisual, a call that exists only for visuals. The rest of this stack moves visuals onto ordinary host files:

  • S3/S4 let a host read a file by its absolute path, including the read-only chat-visual permission.
  • S5 advertises that as files.host-path.v1.
  • S9 publishes each chat's scratch folder (scratchDirectory) on the status stream.

The phone was the last client still tied to the visual-only call on hosts that no longer need it. The plan's later clean-up (retiring readVisual once no supported client calls it) depends on this.

User-facing change. None visible: the same inline visual and the same full-screen sheet, including in floating chats. A visual the host refuses still shows "Visualization unavailable", and tapping it still tries again. No new phone editor, Share button or outside-file viewer.

Mechanism.

  • Deciding how to read. useMobileNativeChatVisualSource (new) builds the chat's visual source:
    • Path read: the host advertises files.host-path.v1 and the status stream, and its status summary for this chat carries scratchDirectory. The phone reads by path in that folder.
    • Old reader: everything else keeps agentSession.readVisual. That covers an older host, a chat the host gives no folder (Pi, WSL), and the moment before the status stream has said anything.
    • The folder is always the host's word. The phone never derives it and never names the folder.
  • Reading by path.
    • nativeChatVisualHostPath(scratchDirectory, file) (new, src/shared/native-chat-visual-host-path.ts) joins the folder and the bare file name in the host's own path syntax: a backslash for a Windows host, a slash otherwise, whatever the phone runs.
    • A name that is not a valid bare visual file name builds nothing, so nothing is sent.
    • The phone calls files.read with { path, access: { kind: 'chat-visual' } }. The host's protected visual reader serves it: only a visual directly inside a chat folder under the host's scratch root, read-only, size-capped.
  • What the answer means (mobile-native-chat-visual-file-read.ts, new). These map to the same three outcomes the old reader produced, so the screen is unchanged:
    • refused (the page comes down, showing "Visualization unavailable"): missing, outside the folder, not a regular file, too large, not text, a refused request, or a host that doesn't serve the method;
    • no verdict (the page stays, with the existing bounded automatic re-asks): a host fault, a timeout or an unreadable answer;
    • ready: the page.
  • Cache. The phone's small visual cache is shared by both readers, keyed by host, chat and file:
    • a path read stores no revision;
    • the old reader still revalidates by revision, and sends none for bytes a path read cached;
    • the visual compares content before replacing the page, so an identical re-read never reloads the frame.
  • Host capability. StructuredAgentSessionHostSupport gains hostPathFiles, read from the same capability probe as the other structured-chat flags. The probe's answer is also remembered per connection (rememberStructuredAgentSessionHostSupport), so the visual's reader doesn't change while a reconnect waits for the next answer.
  • Status stream reads. useMobileStructuredSessionStatusValue (new) is the one subscription to the per-host status stream. The existing "Stopping…" read now uses it too, with no behaviour change.

Routing (direct and relayed).

  • files.read is already an execution-host method on the phone allowlist. A phone connected directly reaches the host's own dispatcher. A phone relayed through its desktop to a server has the frame forwarded with params intact and the routing tag stripped (rewriteRelayedPhoneRequest).
  • The host then serves the path form only as its own request (S3's assertHostPathServedHere). No allowlist or relay change was needed; a new test pins both facts.

Stack position. S12 of 14 in the chat-visuals scratch-folder stack (plan PR 14).

  • Base: S9 (feat(native-chat): publish each chat's scratch folder #28263, brennanb2025/cvs-s9-scratch-directory), which contains S0 to S5.
  • Nothing later in the stack depends on it.
  • src/shared/native-chat-visual-host-path.ts and its test are byte-identical to the copies in S10 (desktop visual reads), so the two merge cleanly.

Why

  • The phone uses the ordinary file read instead of a new phone call. The host already serves files.read by path with the chat-visual permission (S4). The phone already holds the folder from the status stream (S9). Reusing both adds no new wire surface.
  • The path is built from the host's folder, not this phone's idea of it. Hosts run on macOS, Linux and Windows. The host's reader refuses any spelling it would not build itself, and a test checks the shared join against that reader for POSIX and Windows hosts.
  • Fall back per chat, not per host. A capable host can still have chats with no folder (Pi has no visuals; WSL chats have no published folder yet). Those keep the old reader rather than showing nothing.
  • Refusal versus no verdict is kept. The old reader took a page down only on the host's real verdict and kept it through a dropped connection. The path read keeps that split, so a flaky network never blanks a visual that is already showing.

Differences from the common pattern:

  • intended: once a host has answered the capability probe, the phone keeps that answer per connection object (in memory, never stored), and the next probe answer replaces it. The session route clears its capability flags on every disconnect and until the probe answers again; without this memory, every visual on screen would flip back to the old reader and be downloaded again on each reconnect. A host that answers "not capable" after a reconnect switches the phone to the old reader at once.
  • temporary: on a host not yet known to publish a folder for the chat (before the host's first status answer for it), the phone still reads through the old reader. This happens at first open only, never after a reconnect, and reads the same file. When F4 retires readVisual, that first-open read must be held until the status answer instead of falling back.
  • intended: the phone matches the host's refusal text (ENOENT, access denied, file_too_large, binary_file, not a regular file), as its file preview already does. The host sends these as runtime_error messages, not typed codes.

Mixed versions.

  • An older host doesn't advertise files.host-path.v1 or publish scratchDirectory, so a new phone keeps using readVisual exactly as before.
  • An older phone never reads the new field or sends the path form, so nothing changes for it.

Linked Issue

Part of the chat-visuals scratch-folder stack (no separate issue).

Visual Proof

N/A: the phone shows the same visual through a different read; there is no visible change. Live phone QA is at the end of the stack.

Testing

  • mobile/src/session/mobile-native-chat-visual-path-read.test.ts (new, 8):

    • a path read sends files.read with the joined path and chat-visual access, and caches the page;
    • a Windows host gets a backslash path;
    • every host refusal (missing, access denied, not a regular file, too large, binary, invalid_argument, forbidden, a truncated reply) takes the page down and drops the cache;
    • a host fault, timeout, runtime-unavailable or unreadable reply is no verdict and keeps the page;
    • a transport failure is no verdict;
    • an invalid name sends nothing;
    • bytes from the old reader paint first and are then replaced by the path read;
    • the old reader gets no revision for bytes a path read cached.
  • mobile/src/session/use-mobile-native-chat-visual-source.test.tsx (new, 7):

    • a capable host's published folder is used;
    • a floating chat uses its scratch folder, not its workspace id or pinned launch folder;
    • a summary without a folder (Pi, WSL, older host), a host without the capability, a host without the status stream, and an unanswered probe all keep the old reader, and the last three open no stream;
  • mobile/src/session/use-mobile-native-chat-visual-reconnect.test.tsx (new, 2), using the real source, reader and visual hook:

    • a visual shown by path stays on screen through a disconnect and reconnect, with no request sent at all, readVisual included;
    • a probe that answers "not capable" after the reconnect switches to the old reader.
    • Removing the remembered answer fails the first test.
    • unrelated status changes keep the same source object.
  • mobile/src/session/use-mobile-native-chat-visual.test.tsx (+1): the same content keeps the same state object (no frame reload) and new content replaces it. The existing refused-to-"unavailable" and retry tests still pass.

  • mobile/src/session/mobile-native-chat-visual-read.test.ts: the old reader's tests are unchanged except for the ready result no longer carrying a revision.

  • mobile/src/session/mobile-structured-agent-session-host-support.test.ts (+2): files.host-path.v1 maps to hostPathFiles; each client keeps its last answer until the next one replaces it.

  • src/shared/native-chat-visual-host-path.test.ts (new): POSIX, Windows drive and UNC joins; invalid names and an empty folder build nothing.

  • src/main/native-chat/native-chat-visual-host-path-reader-agreement.test.ts (new): the shared join's output passes the host reader's own path check (nativeChatVisualPathParts) for a POSIX and a Windows host.

  • src/main/runtime/runtime-rpc/runtime-rpc-mobile-chat-visual-path-routing.test.ts (new): files.read is an execution-host phone method, and a relayed path read keeps its params and loses its routing tag.

  • Local runs:

    • mobile src/session, src/files and src/transport: 423 files pass. One pre-existing failure, transport/mobile-e2ee-v2-client-session.test.ts "preserves full large text frames…", fails identically on the base branch without this change.
    • Host-side tests above plus S3/S4's path-read tests: 6 files, 42 tests pass.
  • tsc: node, web and mobile pass; the mobile tests typecheck ratchet is OK. check:code-quality:changed and CI's anti-slop oxlint step on the changed files are clean.

  • Tested on macOS only (unit level). The Windows host path is covered by the join and reader-agreement tests.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

Notes

  • Security: the phone gains no new reach. chat-visual is read-only and limited by the host to visual files directly inside a chat folder under its scratch root, and files.read is already a phone method under the same permission.
  • Cross-platform: the join follows the host's path syntax, not the phone's.
  • SSH and remote: paired and SSH hosts are both served by Orca's server on that host, and the path read runs there. Nothing is read from another computer's disk.
  • Mobile: no allowlist or relay change.
  • Backwards compatibility: capability-gated, with a per-chat fallback to the old reader.

…s host

On a host that advertises files.host-path.v1 and publishes the chat's
scratchDirectory, the phone reads a visual with files.read by host path
and the read-only chat-visual access. Older hosts, and chats the host
gives no folder, keep agentSession.readVisual.
The session route clears its capability answer on every disconnect, which
flipped shown visuals back to readVisual and re-downloaded them. Keep the
last probe answer per client in memory until the next one replaces it.
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Superseded by the rebuilt six-PR chat scratch stack (#28385 merged, #28389, #28397, #28437, #28394, #28479). This piece is replaced by no PR: the phone stays display-only and keeps reading visuals through readVisual. Closing.

@brennanb2025
brennanb2025 deleted the brennanb2025/cvs-s12-phone-visual-reads branch October 14, 2026 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant