Remote agent sessions #154

Start an agent session on a remote server.

Design doc https://github.com/joelmoss/workroom/blob/master/docs/designs/remote-workrooms.md

Inspiration

• https://x.com/dannypostma/status/2091881232997233111?s=20

Phase 1 — complete

Landed in #181, #182, #183, #192 (2026-09-15). wr-agent serve | attach owns every new pty. See the Phase 1 comment below.

Phase 2 — merged to master (2026-09-23)

The whole stack merged as ce284372…827033d2: #201 → #202 → #204 → #210 → #212 → #213 → #214 → #215 → #216 → #217 → #218

• feat(macapp): add host-aware repository routing #201 host-aware repository routing
• feat(macapp): manage host service connection lifecycles #202 host connection lifecycle management
• feat(macapp): route local VCS reads through the agent #204 agent-backed local VCS reads
• feat(macapp): route local VCS writes through the agent #210 agent-backed VCS writes (Agent-backed VCS writes #205)
• feat(wr-agent): agent-backed file service — listing, reads and change notification #212 agent-backed file service: listing, raw reads, change notification (Agent-backed file service: listing, raw reads and change notification #211)
• feat(macapp): agent-aware GitHub status — identity-based gh probes (#207) #213 agent-aware GitHub status, identity-based gh probes (Agent-aware GitHub status #207)
• Interchangeability services: port forwarding, terminal durability, wakefulness #208 interchangeability services:
  • OQ19 measured: the busy/idle signal works, on the hold-out and on boxd (#208, measurement half) #214 OQ19 measurements: busy/idle signal on the hold-out and on boxd
  • feat(wr-agent): the wakefulness service, ported from the OQ19 measurements #215 wr-agent wakefulness service
  • feat(macapp): show the agent's busy/idle verdict and prompt at the awake ceiling #216 app shows busy/idle verdict, prompts at the awake ceiling
  • feat(wr-agent): Service::Forward, loopback port forwarding over the multiplex (#208, agent half) #217 Service::Forward, loopback port forwarding (agent half)
  • feat(macapp): forward a loopback port over the agent multiplex (#208, app half) #218 port forwarding (app half)

Review follow-ups: #220 (closed 2026-09-24), fixed in #221 (high), #222 (medium), #223 (low) and #225 (OQ19 harness). #226 fixed three fork-window test flakes (#224).

Phase 2 — what is left on #208

• Port forwarding for a remote process. The agent half works over the remote stream: over_ssh_every_service_answers_through_the_relay forwards to a process on the container. Still owed:
  • The app's Ports panel reaches only the local agent (PortForwardingModel uses the .local host), so a remote workroom's ports can't be forwarded from the app yet. Now tracked by Remote pane parity: pane close, cmd-click, forwarded ports and idle state #254 (Phase 4), which closes Interchangeability services: port forwarding, terminal durability, wakefulness #208's remaining criterion. PR Remote pane parity: cmd-click on the host, Ports from the host, busy/idle in the sidebar (#254) #326 (merged 2026-10-04) forwards from the remote host and closed Interchangeability services: port forwarding, terminal durability, wakefulness #208.
  • The provider hostname that "come back to a running preview URL" needs.
• Terminal-state durability after a stop-and-reboot — done in Stop-and-reboot screen restoration #232 (merged in feat(agent): show a pane its last screen after its host reboots (#232) #242, 2026-09-25).

#206 closed as a duplicate of #211 (2026-09-22).

Phase 3 — checklist complete (2026-09-26)

The stream drivers, per the design doc's Phase 3 and Next Steps item 4. All of it is exercised end to end over ssh against the container fixture, but nothing in the shipped app creates a remote pane or connects a remote host yet: registerRemoteSession and AgentBootstrap.ensure are only called from tests until Phase 4 wires remote workrooms in.

• Linux wr-agent as a published build artifact #227 Linux wr-agent as a published build artifact (merged in Bundle static Linux wr-agents in Release and Nightly builds #233, 2026-09-24). Its first Nightly DMG is still to be checked end to end (now part of Phase 4 gate: the success criteria on two real providers #260).
• Persistent remote agent: supervised serve, stdio relay and the container fixture #228 persistent remote agent: supervised serve, stdio relay, container fixture (merged in feat(wr-agent): relay to a supervised remote agent, proven over ssh #235, 2026-09-24)
• App-side remote transport: HostDriver, ssh-stdio connection and the container driver #229 app-side remote transport: HostDriver, ssh-stdio connection, container driver, remote writes and terminal panes (merged in feat(macapp): remote hosts through a HostDriver: services, writes and terminal panes #237, 2026-09-24)
• Agent hand-off: execve upgrade that keeps every shell, local Mac first #230 agent hand-off: execve upgrade that keeps every shell, local Mac first (merged in feat(wr-agent): hand a running agent to a new binary, keeping every session #238, 2026-09-25; on for Nightly and Dev. Remote hand-off has been on in every build since feat(macapp): remote workrooms in every build #375, 2026-10-08)
• Remote bootstrap: push the Linux agent on first connect, hand off on version mismatch #231 remote bootstrap: push the agent on first connect, hand off on version mismatch (merged in feat(macapp): push the Linux agent to a remote host and hand off to it #240, 2026-09-25)
• Stop-and-reboot screen restoration #232 stop-and-reboot screen restoration (merged in feat(agent): show a pane its last screen after its host reboots (#232) #242, 2026-09-25). serve --screens <dir> keeps each session's screen on disk, and a restored pane is shown it frozen, marked as ended. Only the container fixture's supervisor passes --screens so far; a real host's supervisor has to pass it too (Portable derivation on the container driver: the base machine and deriveFromBase #252).
• Remote panes: push xterm-ghostty terminfo and the shell integration with the agent #239 remote panes: xterm-ghostty terminfo and the shell integration go with the agent (merged in feat(macapp): push Ghostty's terminfo and shell integration to remote hosts #244, 2026-09-26), and the pane's footer follows the shell through the host's agent, since libghostty drops a remote OSC 7 (merged in feat(macapp): a remote pane's footer follows its shell through the host's agent #245, 2026-09-26). The probe re-checks the pushed set on every connect and pushes it again if a file is missing or changed.
• Remote panes: keep reconnecting a restored pane until a rebooted host is back #241 remote panes: keep reconnecting a restored pane until a rebooted host is back (merged in fix(macapp): keep reconnecting a remote pane until its host is back #243, 2026-09-26). It backs off to every 30 s and stops only for a host that refuses for good (changed host key, refused key).

Phase 3 follow-ups:

• wr-agent: a session list just under 1 MiB panics its connection's thread #246 wr-agent: a session list just under 1 MiB panics its connection's thread — fixed in fix(wr-agent): refuse a session list whose frame would overflow its envelope #247 (merged 2026-09-30). The frame cap and the envelope cap disagreed by the 5-byte frame header; the list is now capped at the smaller of the frame cap and what an envelope leaves room for.
• Known limit, from feat(macapp): a remote pane's footer follows its shell through the host's agent #245's review: the agent builds a session list under its session-store lock on the connection's read thread, so a List that lands during a client's reattach repaint holds up that connection's other requests until the repaint ends. A per-session cwd request would remove it (ponytail: note at AgentVCSConnection.workingDirectory(of:)).

OQ21 (where the HostDriver lives) is answered: a Swift protocol in the Mac app.

Phase 4 — filed 2026-09-27

Provisioning, credentials, lifecycle and the UI, per the design doc's Phase 4 and Next Steps item 5. OQ20 is decided by ADR 0049 (2026-10-12), which supersedes the credential broker PR #248 first chose: Workroom runs no broker. A provider's host fetches and pushes with the provider's own GitHub credentials, so pushes are in the user's name and a remote workroom needs no Codaset account. A boxd machine uses the user's boxd git integration; an exe.dev VM uses an exe.dev GitHub integration the user adds per repository. A local container workroom keeps the Mac's gh relay (ADR 0039). The broker, the Codaset sign-in and the Mac's device key were removed in 8d7b0502 and ce812d4a. ADR: docs/adr/0049-provider-hosts-bring-their-own-git-credentials.md. Research: docs/designs/oq20-remote-git-credentials.md (PR #248). The two real providers are boxd and exe.dev.

• Remote host descriptors in config, and host-gated warnings #249 remote host descriptors in config, and host-gated warnings (merged in feat: remote host descriptors in config, and host-gated warnings #275, 2026-10-01; docs in docs(remote-workrooms): record #249 as merged, with its review fixes and gaps #276, docs(contributing): note the host key in the config file section #277)
• Credential broker service: GitHub App, sign-in, and installation-token minting #250 credential broker service: GitHub App, sign-in, installation-token minting. Built into Codaset at codaset.dev, under joelmoss/codaset#40. Workroom users become Codaset users. Built in joelmoss/codaset#43 (merged 2026-10-01); closed 2026-10-02. The design doc's OQ20 records it as built. Superseded by ADR 0049 (2026-10-12): Workroom no longer uses the broker.
• Credential broker clients: Mac sign-in, agent enrolment, and the git credential helper #251 broker clients: Mac Secure Enclave sign-in, agent enrolment, git credential helper (after Credential broker service: GitHub App, sign-in, and installation-token minting #250). Merged in Credential broker clients: Mac sign-in, agent enrolment, git credential helper #263 (2026-09-30). fix(macapp): Debug builds talk only to a local Codaset; requests name the build #278 (2026-10-01) keeps Debug builds on a local Codaset, gives a Debug remote agent a reverse forward to it, and names the build on every request (Codaset side joelmoss/codaset#45). Closed 2026-10-02; the design doc's OQ20 has the "As built (Credential broker clients: Mac sign-in, agent enrolment, and the git credential helper #251)" entry. Removed by ADR 0049 (2026-10-12, 8d7b0502, ce812d4a); only the Mac's gh relay for local container workrooms remains.
• Portable derivation on the container driver: the base machine and deriveFromBase #252 portable derivation on the container driver: the base machine and deriveFromBase (after Remote host descriptors in config, and host-gated warnings #249, Credential broker clients: Mac sign-in, agent enrolment, and the git credential helper #251). Merged in feat: portable derivation on the container driver (#252) #280 (2026-10-01, ac20de4d), with all six acceptance criteria met on the container fixture.
  • The container driver now creates bases, derives instances (each mints its own ssh host keys and machine-id at first boot) and destroys them.
  • The base clones with a broker clone token that never reaches disk, argv or the remote URL.
  • Each workroom enrols with its own key and checks out its own branch. A failure at any step rolls back, and names what is still live if the rollback fails too. Operations on one base run one at a time.
  • Also new: a hidden workroom host set|clear CLI command records the base. delete-project now refuses a project that has a base.
  • The fixture gained a fake GitHub and broker, so a derived workroom is shown fetching and pushing with the Mac disconnected.
  • Gaps left for Remote workrooms in the app: create, open and delete (Nightly only) #253, boxd driver with portable derivation #256 and Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257:
    • nothing re-adopts a base after the app relaunches — done in Remote workrooms in the app: create, open, delete, restore (#253) #289: RemoteHosts.adopt takes back each project's recorded bases and workrooms at launch;
    • broker.json sits on a tmpfs /run on real hosts — moot: ADR 0049 removed the broker;
    • a base's clone must finish within 10 minutes;
    • a runtime command whose CLI is killed can leave a container or image behind that appears after the cleanup sweep (deferred to Remote workrooms in the app: create, open and delete (Nightly only) #253's reconciler).
• Remote workrooms in the app: create, open and delete (Nightly only) #253 remote workrooms in the app: create, open, delete, Nightly only (after Portable derivation on the container driver: the base machine and deriveFromBase #252). Merged in Remote workrooms in the app: create, open, delete, restore (#253) #289 (2026-10-02), on the container driver (Docker on this Mac), behind the hidden remoteWorkroomsPreview setting in Nightly and Dev builds. A stable build shows no remote UI. feat(macapp): remote workrooms in every build #375 (2026-10-08) removed that gate: every build now has remote workrooms.
  • Relaunch: HostDescriptor carries the driver, provisioner, host ID, grant and container record, so ContainerHostDriver adopts recorded hosts. A once-per-launch sweep removes labelled leftovers that no record names. Each build owns its own hosts.
  • Create: New Remote Workroom builds the project's base once, then derives each workroom from it on branch workroom/<name>. The full descriptor is written last. make remote-host-image builds the workroom-host image.
  • Open: a reachable remote workroom is registered on its host. Changes, History, Files, diffs, file viewers and PR/CI status read through the agent there.
  • Delete: cancels the grant, removes the box, then the CLI drops the entry. Delete Project takes down the remote workrooms, then the base. A teardown that fails part-way leaves the entry failed, so deleting again finishes the job.
  • Restore: restored panes reattach to this Mac's sessions only for a local target that is still there. An unreachable remote workroom's saved session is held and restored once it is reachable.
  • Follow-ups from its review: Closing a remote pane leaves its session running on the host #283 (closing a remote pane ends its session, fixed in fix: closing a remote pane ends its session on the host (#283) #306), Tombstone a project's workrooms while Delete Project tears down its remote hosts #287 (tombstones during Delete Project, fixed in fix(macapp): tombstone a project while Delete Project tears it down #294), Build and smoke-test the workroom-host image (FIXTURE=0) in CI #288 (build workroom-host in CI, fixed in test(ssh-fixture): build and smoke-test the workroom-host image in CI #298), Allow deleting a remote workroom while signed out of Codaset #285 (delete while signed out of Codaset, fixed in feat: choose where to create a workroom: this Mac, Docker or Apple container (#309) #320). Also closed: Make the remote host sweep safer and make it run when nothing is recorded #284 (sweep safety, fixed in fix(macapp): make the launch container sweep safe and complete (#284) #374, 2026-10-08) and Debug builds: restore broker reverse forwards for adopted remote workrooms after a relaunch #286 (Debug broker forwards after a relaunch, closed as won't-fix on 2026-10-09: it affects Debug builds only).
• Remote pane parity: pane close, cmd-click, forwarded ports and idle state #254 remote pane parity: cmd-click, forwarded ports, idle state (after Remote workrooms in the app: create, open and delete (Nightly only) #253). Pane close was done first: closing a remote pane ends its session on the host (Closing a remote pane leaves its session running on the host #283, fixed in fix: closing a remote pane ends its session on the host (#283) #306). The rest merged in PR Remote pane parity: cmd-click on the host, Ports from the host, busy/idle in the sidebar (#254) #326 (2026-10-04), which closed Remote pane parity: pane close, cmd-click, forwarded ports and idle state #254 and Interchangeability services: port forwarding, terminal durability, wakefulness #208.
  • Ports: the Ports section forwards from the selected workroom's host, with one PortForwardingModel per host.
  • Idle state: each remote workroom's sidebar row shows its host's busy/idle badge. Its "Keep awake" connects the host first, and quitting drains every host's keep.
  • Cmd-click (C7): a click resolves on the host, against the shell's directory there, and opens the file in the in-app viewer.
    • The click gate decides from the word's shape alone (a /, a name.ext or a :line), and only the click asks the host. The owner chose this after a gate backed by a cache of the host's answers regressed on three review passes.
    • Accepted costs: a path-shaped word with no file opens nothing, and a dotless bare name like Gemfile isn't clickable remotely.
  • Follow-ups from its review:
    • CredentialRelayTests.testIdleStrangersDoNotBlockAKnownWorkroom fails under full-suite load #322: a CredentialRelay test fails under full-suite load — fixed in fix: the credential relay keeps a full accept queue and gives back reset peers' permits (#322) #330 (merged 2026-10-04): the relay keeps a full accept queue and gives back a reset peer's reading permit. Not caused by Remote pane parity: pane close, cmd-click, forwarded ports and idle state #254.
    • Remote port forwards are reachable by every account on the Mac #323: remote port forwards are reachable by every account on the Mac. Closed as won't-fix (2026-10-07): never seen in real use.
    • A failed remote "Keep awake" shows nothing #324: a failed remote "Keep awake" shows nothing. Closed as done (2026-10-07): a failed keep shows a card, via Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257.
    • Remote cmd-click fails for relative paths when the workroom root goes through a symlink #325: relative cmd-clicks fail where the workroom root goes through a symlink. Closed as won't-fix (2026-10-09): never seen in real use, and absolute paths still work.
    • Remote cmd-click on link/../file opens the file beside a symlink, not beside its target #327: link/../file with a symlinked link opened the file beside the link — fixed in PR fix: remote cmd-click resolves .. on the host, through the symlink (#327) #340 (merged 2026-10-04): the agent's File service gains a resolve request that resolves .. on the host, through the link, and the click falls back to the Mac's lexical path only for an agent that predates it.
      • Follow-ups from fix: remote cmd-click resolves .. on the host, through the symlink (#327) #340's review:
        • wr-agent resolve through a link onto a dead mount holds a File read slot until the agent restarts #334: a resolve through a link onto a dead mount holds a read slot (and its permit) until the agent restarts. Fixed by fix(wr-agent): a resolve answers at a deadline, so a hung mount can't hold a read slot (#334) #346 (merged 2026-10-04). Follow-ups:
          • wr-agent read through a link onto a hung mount holds a read slot and permit forever #343: give file reads a deadline too.
          • A remote cmd-click that times out on a hung mount shows the user nothing #344: show feedback when a cmd-click opens nothing.
          • A remote cmd-click whose :line comes before its .. loses its healthy variant when the resolve times out #345: a :line before .. loses a candidate variant.
        • Remote cmd-click on ../x.rb opens nothing when the shell's directory was deleted #335: a deleted shell directory makes ../x.rb open nothing.
        • Remote cmd-click still opens the lexical decoy when a .. comes before the workroom root as written #336: the lexical decoy still opens when the written path climbs above the workroom root.
        • Check the shape of the path a host's resolve returns before it reaches app state #337: shape-check the path a host's resolve returns before it reaches app state.
        • Remote cmd-click: resolve a .. prefix once, not once per candidate #338: resolve a .. prefix once per click, not once per candidate.
        • wr-agent resolve races a root symlink that changes between its two lookups #339: resolve races a root symlink that changes between its two lookups.
• Cross-machine reattach: session enumeration and workroom UI state on the host (OQ8) #255 cross-machine reattach: session enumeration and UI state on the host, OQ8 (after Remote workrooms in the app: create, open and delete (Nightly only) #253). Merged in feat: a remote workroom's layout and sessions live on its host (#255) #349 (2026-10-05, 01062fb2).
• boxd driver with portable derivation #256 boxd driver with portable derivation (after Portable derivation on the container driver: the base machine and deriveFromBase #252). Merged in feat(macapp): boxd driver with portable derivation (#256) #281 (2026-10-02). BoxdHostDriver drives the boxd CLI over the same ssh transport as the container driver.
  • deriveFromBase snapshots the base, creates a machine from the snapshot, mints a fresh identity, reboots, and removes the snapshot. Any failure rolls back the machine and the snapshot.
  • boxd.sh installs a workroom-identity unit and a supervised workroom-agent (wr-agent serve --screens). Enrolment and screens live on the home disk, so they survive a stop/start.
  • An org guard checks boxd auth before every create, derive, destroy and rollback.
  • All 5 live boxd integration tests pass (WR_BOXD_TESTS=1).
  • Known gaps: machines are not --isolated (owner's decision, because Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257 needs the in-VM CLI); the org check is not a pin; a timed-out CLI call can still finish on boxd's side; hibernation after 4 h idle is left to Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257.
• Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257 far-side lifecycle shim, boxd first (after boxd driver with portable derivation #256). Merged in feat: keep a busy remote box awake with an agent heartbeat (#257) #353 (2026-10-06).
• boxd live-fork derivation, as an accelerator behind deriveFromBase #258 boxd live-fork derivation as an accelerator (after boxd driver with portable derivation #256). Closed as not planned (2026-10-09). It had already been deferred and taken off Phase 4 gate: the success criteria on two real providers #260 (2026-10-06).
• exe.dev driver: the second real provider #259 exe.dev driver, the second real provider (after Portable derivation on the container driver: the base machine and deriveFromBase #252). Merged in feat(macapp): exe.dev as a remote provider (#259) #383 (2026-10-09).
• Wire boxd into the app: create, open and delete remote workrooms on boxd #356 wire boxd into the app: create, open and delete remote workrooms on boxd (after boxd driver with portable derivation #256, Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257). Merged in feat: wire boxd into the app (#356) #372 (2026-10-07); follow-ups in boxd in the app: follow-ups from #356 (PR #372) #371.
• Phase 4 gate: the success criteria on two real providers #260 Phase 4 gate: the success criteria on boxd and exe.dev (after Remote pane parity: pane close, cmd-click, forwarded ports and idle state #254, Cross-machine reattach: session enumeration and workroom UI state on the host (OQ8) #255, Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257, exe.dev driver: the second real provider #259). The remote UI already left Nightly in feat(macapp): remote workrooms in every build #375 (2026-10-08), so Phase 4 gate: the success criteria on two real providers #260 now verifies it after the fact.

Body updated by Claude Code at the owner's request, 2026-10-06.

#356 added to the checklist by Claude Code at the owner's request, 2026-10-07 (#348 closed as not needed).

Updated by Claude Code at the owner's request, 2026-10-08: PR #375 removed the remote-workrooms gate.

Updated by Claude Code at the owner's request, 2026-10-09: #259 shipped as #383, and #284, #286, #323, #324 and #325 are closed.

Updated by Claude Code at the owner's request, 2026-10-12: #258 closed as not planned, and ADR 0049 replaced the Codaset credential broker.