feat(macapp): remote hosts through a HostDriver: services, writes and terminal panes

docs/designs/remote-workrooms.md

@@ -11,8 +11,9 @@ Mode: Builder
 **Phase 2 is merged to master** (2026-09-23, #201 through #218; review follow-ups #220–#226), and
 **Phase 3 has started.** Its milestones are sub-issues of #154: the Linux agent artifact (#227,
 merged; its first Nightly DMG is still to be checked), the supervised far-side agent with its stdio
-relay and container fixture (#228, in progress), the
-app-side transport with `HostDriver` and the container driver (#229), `execve` hand-off (#230),
+relay and container fixture (#228, merged), the
+app-side transport with `HostDriver` and the container driver (#229, in progress: services first,
+terminal panes second), `execve` hand-off (#230),
 push-on-first-connect bootstrap (#231) and stop-and-reboot screen restoration (#232). The rest of
 this section is the 2026-09-17 status, kept for the Phase 2 detail it records and corrected where
 it had gone stale.
@@ -993,6 +994,69 @@ these are the subsystems that actually gate "a remote workroom is a real workroo
     not die with it: their jobs run on, orphaned, while the replacement agent reports no
     sessions. systemd's default `KillMode=control-group` does this. The fixture's loop does not,
     which is fine for a test box and wrong for a host with real work on it.
+  **App side, services built (#229):** `HostDriver` (`Core/HostDriver.swift`) has the four methods
+  and the declared traits, and `ContainerHostDriver` implements `openStream` as
+  `ssh -F <its own config> <host> wr-agent relay --socket <path>`. The stream is one end of a
+  socketpair whose other end is ssh's stdin and stdout, and `AgentVCSConnection.connect(host:stream:)`
+  runs the same negotiation over it as over a local socket, so VCS, File, Status and Forward need
+  no change. ssh exiting is the connection ending, through the same generation and
+  fail-pending-calls rules as a dead local agent. Terminal panes through the driver are the second
+  half of #229.
+  **Remote writes (#229)** keep every decision in the Swift writer (`CLIVCSWriter`) and move only
+  two things to the host:
+  - *The disk the failure classifier reads.* It decides a parked rebase, a leftover lock or the last
+    fetch from the repository's git directory, and on a remote host that directory is there. The
+    agent's `stat` request reports the facts (existence, directory, modification time, a worktree's
+    `.git` text) for the paths the classifier names, read after the command they explain.
+  - *The jj working-copy barrier.* Locally the app's gate holds it for a whole operation, and exec
+    never takes it. A Mac cannot flock a file on another host, so an exec request for a remote jj
+    repository names its shared root (`barrier_root`) and the agent takes the barrier around that
+    command. **Known ceiling:** that is per command, where the local gate is per operation, so a
+    remote snapshot can land between two commands of one write. The upgrade is an
+    operation-scoped barrier the client acquires and releases, held by the connection and dropped
+    when it goes. Two kinds of command skip it:
+    - A read that cannot snapshot (`--ignore-working-copy`), which has nothing to protect.
+    - A network command (`jj git fetch`/`push`), because the barrier is handed to the child, and
+      git's detached helpers would inherit it and hold it after jj exits. jj's op log reconciles a
+      snapshot that runs alongside one.
+  Both came with exec service version 3, and the app refuses a remote write to an older agent.
+  **Known residual, wider remotely:** a jj commit tells "committed, then failed" from "failed" by
+  whether the op head moved, and a snapshot also moves it. A snapshot in the gap between the
+  before-read and the commit can make a failed commit read as one that landed. Locally the after-read
+  already sits outside the gate; the per-command barrier widens the gap. The fix is to compare
+  `@-`, which a snapshot never rewrites, for jj's `.commit` mode.
+  **Terminal panes (#229).** A pane is a process libghostty spawns on its own pty, which a socketpair
+  made in the app cannot become, so a pane does not use `openStream`. A driver that can host panes
+  implements `HostTerminalDriver.attachCommand`. For ssh that runs
+  `ssh -t <host> 'env TERM=xterm-256color WORKROOM_SESSION_{ID,SOCKET,CWD}=… wr-agent attach
+  --no-spawn [--no-create]'`, the far side's attach client in the pty ssh allocates there.
+  - *`--no-spawn`:* with no agent listening, a shell rather than a spawned agent. Only the
+    supervisor starts one there.
+  - *`--no-create` (a restored pane):* the agent refuses a session it does not hold (`CREATE=0` in
+    the attach request, which older agents ignore). The pane becomes a shell that says the session
+    ended. Locally the app asks first (`confirmBeforeAttach`); remotely one request answers and
+    attaches, so the answer cannot go stale in between.
+  - *A dropped link reconnects.* ssh's own failure (exit 255, as after sleep or a network change)
+    reattaches the pane as a restored one, backing off to 30s while the host stays unreachable,
+    and stopping after five quick failures in a row so a permanent one (a mismatched host key)
+    leaves its message on screen. A session that itself exited 255 is reported as 254 by the
+    remote attach, so it never reads as a dropped link. The pane's ssh has `EscapeChar none`: what
+    the user types is theirs, and `~.` would otherwise disconnect it.
+  - *Closing a remote pane does not end its session yet.* Nothing on the service connection can
+    ask the host's agent to, so the session is left running and reported as not killed. That
+    belongs with the host's lifecycle (Phase 4).
+  - *`CREATE=0` is only as good as the agent that reads it.* An agent older than the flag ignores
+    it and creates the session. The version hand-off (#230) and the bootstrap (#231) keep the
+    host's agent current; until they land, a host running an older agent can still hand a
+    restored pane a fresh shell.
+  - *Degraded until the agent bootstrap (#231):* `TERM` is `xterm-256color`, since a host rarely
+    has `xterm-ghostty` terminfo, and there is no shell integration there (its resources are in the
+    Mac's app bundle), so the title and footer do not follow the shell.
+  - *Not persisted:* a session is marked remote in memory (`registerRemoteSession`). Phase 4's
+    remote workrooms re-register their panes on relaunch.
+  - *An SDK-exec driver has no command to hand libghostty.* It needs a local bridge process, which
+    is a Phase 4 question.
+
 - **Local needs no supervisor at all. DECIDED — and an earlier claim in this document was wrong.**
   A previous revision asserted that `SessionDaemon.swift:97-107`'s self-exit
   (`sessions.isEmpty && connections.isEmpty`) had to go, because the agent would also own VCS reads
@@ -1023,8 +1087,30 @@ these are the subsystems that actually gate "a remote workroom is a real workroo
   product uses everywhere, the container fixture exercises the real path rather than a fallback
   nobody runs. What remains uncovered by CI is boxd's *live-process* derivation specifically — a
   narrower gap than the whole fork path, and one a nightly boxd job can cover if it earns it.
-- **State a host-key policy** (see Provider Decision). Fresh-key-per-fork plus `BatchMode=yes` cannot prompt, so
-  the driver has to deliver the expected host key out of band or pin the base's.
+- **Host-key policy: pinned and delivered out of band. DECIDED (#229).** The driver writes its own
+  `ssh_config` and passes it with `-F`, so nothing in `~/.ssh/config` or `/etc/ssh/ssh_config`
+  applies. The host's key is the only known key (`GlobalKnownHostsFile /dev/null`), with
+  `StrictHostKeyChecking yes` and `BatchMode yes`: a mismatched key fails at once with "Host key
+  verification failed." and there is no prompt to hang on. A fresh-key-per-fork driver delivers each
+  instance's key the same way, from its provider's API.
+- **Auth resolution for the ssh transport: host-resolved credentials, no forwarding. DECIDED
+  (#229).** Nothing from the Mac's environment crosses. A remote exec request carries only policy
+  pins (`GIT_OPTIONAL_LOCKS`, `GIT_TERMINAL_PROMPT`, `LC_ALL`), and the agent builds the rest from
+  its own environment (`host_environment` in `vcs.rs`), less variables that redirect git to
+  another repository. That makes the supervisor's environment what every remote git gets, so a
+  supervisor should start the agent from a clean one. ssh is kept from prompting without setting
+  `GIT_SSH_COMMAND`, which would outrank a repository's own `core.sshCommand`: no askpass helper,
+  and a supervised agent has no terminal. ssh itself runs with an empty environment,
+  `ForwardAgent no` and `ClearAllForwardings yes`. git on the host authenticates with what the host
+  has, which in production is a per-workroom deploy key minted on the box (Decisions Made; Phase 4).
+  Agent forwarding stays rejected (premise 6). The container driver also logs in with a key file
+  and `IdentityAgent none`; whether a real driver may use the Mac's ssh agent to LOG IN, which is
+  not forwarding it, is that driver's decision.
+- **Peer authentication to shell grade, on every driver. DECIDED (#229).** The exec service runs
+  `git` with arbitrary arguments, which is arbitrary code execution, so a stream that reaches an
+  agent is a shell whatever it is called. ssh authenticates to that grade. Every later driver (SDK
+  exec, WebSocket) must show that it does too before it counts as a driver; `HostDriver`'s doc
+  carries the requirement.
 - **Agent bootstrap over the transport.** Workroom.app ships the Linux agent binary and pushes it on
   first connect rather than baking provider images. One source of truth, no image maintenance, and
   it works for SDK-transport providers with no image concept. (This is the one piece of Coder's and

macapp/WorkroomApp/Core/AgentCommandRunner.swift

@@ -10,6 +10,10 @@ import Foundation
 /// distinction is load-bearing — see `neverRan` and `outcomeUnknown`.
 struct AgentCommandRunner: StatusCommandRunning, Sendable {
   let connection: AgentVCSConnection
+  /// The shared root of a jj repository on a REMOTE host, whose working-copy barrier the agent
+  /// takes around each command (`barrier_root` in `vcs.rs`). Nil locally, where the caller's gate
+  /// holds it for the whole operation and the agent must never take it again.
+  var barrierRoot: String?
 
   func run(_ executable: String, _ args: [String], in directory: String, timeout: TimeInterval)
     async -> CommandResult
@@ -55,11 +59,9 @@ struct AgentCommandRunner: StatusCommandRunning, Sendable {
       }
       payload = encoded
     }
-    let request = AgentExecRequest(
-      executable: executable, args: args, dir: directory,
-      timeoutMs: Int((timeout * 1000).rounded(.up)),
-      stdin: payload,
-      env: StatusCommandRunner.childEnvironment(network: network))
+    let request = Self.request(
+      executable, args, in: directory, timeout: timeout, stdin: payload, network: network,
+      host: connection.host, barrierRoot: barrierRoot)
     let reply: Data
     do {
       // Slack above the command's own timeout: the round trip and the agent's own bookkeeping
@@ -70,7 +72,12 @@ struct AgentCommandRunner: StatusCommandRunning, Sendable {
       // once it holds the flock — see the comment there. Shielding HERE instead would also cover
       // `remoteState`'s ungated reads, and a superseded `RemoteStateModel` refresh would then squat
       // one of this connection's 32 shared slots until the agent answered.
-      reply = try await connection.request(request, timeout: timeout + 15)
+      //
+      // Plus the barrier's own wait where the agent takes one (a remote jj command): it can wait
+      // up to 30s for the barrier before the command starts (`SnapshotLock::acquire`), and a
+      // deadline that did not cover that would give up on a command still running.
+      let barrierWait: TimeInterval = request.barrierRoot == nil ? 0 : 30
+      reply = try await connection.request(request, timeout: timeout + 15 + barrierWait)
     } catch let error as VCSError {
       // Raised before anything left this process — today only the 1 MiB single-envelope request
       // ceiling. Nothing ran, and the workroom is fine.
@@ -163,6 +170,38 @@ struct AgentCommandRunner: StatusCommandRunning, Sendable {
   }
 }
 
+extension AgentCommandRunner {
+  /// The exec request for `host`. **Nothing from the Mac's environment goes to a remote host.**
+  ///
+  /// Locally the request carries the app's whole child environment, which the agent adopts in place
+  /// of its own stale one (see `StatusCommandRunner.childEnvironment`). A remote host is a different
+  /// machine: the Mac's `HOME` and `PATH` name nothing there, and its `SSH_AUTH_SOCK`, git identity
+  /// and credential variables are the user's Mac credentials, which must not cross (design doc,
+  /// premise 6: agent forwarding was rejected, and git on a remote host authenticates with what the
+  /// host has). So a remote request sends only `remoteEnvironment`'s policy pins and asks the agent
+  /// to build the rest from the host's own environment (`host_environment` in `vcs.rs`), which also
+  /// pins ssh to fail rather than prompt, as `networkEnvironment` does here.
+  static func request(
+    _ executable: String, _ args: [String], in directory: String, timeout: TimeInterval,
+    stdin: String?, network: Bool, host: HostID, barrierRoot: String? = nil
+  ) -> AgentExecRequest {
+    let remote = host != .local
+    return AgentExecRequest(
+      executable: executable, args: args, dir: directory,
+      timeoutMs: Int((timeout * 1000).rounded(.up)),
+      stdin: stdin,
+      env: remote
+        ? StatusCommandRunner.remoteEnvironment
+        : StatusCommandRunner.childEnvironment(network: network),
+      hostEnvironment: remote ? true : nil,
+      // Not for a network command (`jj git fetch`/`push`): the barrier is handed to the child, and
+      // git's detached helpers (`git maintenance --auto`, a credential cache daemon) would inherit
+      // it and hold it long after jj exits. jj's own op log reconciles a snapshot that runs
+      // alongside one of those.
+      barrierRoot: remote && !network ? barrierRoot : nil)
+  }
+}
+
 struct AgentExecRequest: Encodable, Sendable {
   var version = 1
   var kind = "exec"
@@ -172,6 +211,11 @@ struct AgentExecRequest: Encodable, Sendable {
   let timeoutMs: Int
   var stdin: String?
   var env: [String: String]
+  /// Set only for a remote host, and omitted otherwise: an agent that predates the field rejects
+  /// any request carrying it (`deny_unknown_fields`), and a local agent may be one.
+  var hostEnvironment: Bool?
+  /// Remote only, for the same reason: see `AgentCommandRunner.barrierRoot`.
+  var barrierRoot: String?
 }
 
 struct AgentExecResult: Decodable, Sendable {

macapp/WorkroomApp/Core/AgentPortForwarding.swift

@@ -278,7 +278,17 @@ final class PortForward: @unchecked Sendable {
     // blocking `read` is the whole design.
     _ = fcntl(client, F_SETFL, fcntl(client, F_GETFL) & ~O_NONBLOCK)
     var enabled: Int32 = 1
-    setsockopt(client, SOL_SOCKET, SO_NOSIGPIPE, &enabled, socklen_t(MemoryLayout<Int32>.size))
+    // Checked, because it can only be set here, after `accept`: a client that has already reset
+    // leaves the socket shut down, `setsockopt` fails with EINVAL, and the pump's first write would
+    // then raise SIGPIPE and kill the app. Such a client is gone, so there is nothing to forward.
+    // The slot is released as it goes out of scope.
+    guard
+      setsockopt(client, SOL_SOCKET, SO_NOSIGPIPE, &enabled, socklen_t(MemoryLayout<Int32>.size))
+        == 0
+    else {
+      Darwin.close(client)
+      return
+    }
     // What a forward carries is small writes in both directions (chunked HTTP, HMR frames, a wire
     // protocol); Nagle plus delayed ACK on a loopback hop is latency for nothing.
     setsockopt(client, IPPROTO_TCP, TCP_NODELAY, &enabled, socklen_t(MemoryLayout<Int32>.size))