fix(wr-agent): a resolve answers at a deadline, so a hung mount can't hold a read slot (#334)

TODOS.md

@@ -54,6 +54,25 @@ target or at a second listener answering the port.
 
 **Priority:** P0 (chosen at ship time on #327)
 
+### `a_relay_with_no_agent_fails_fast_and_says_so` flaked under load (vcs) — #334 ship follow-up
+
+**What:** The test (`vcs/crates/wr-agent/tests/remote_transport.rs:1236`) failed once with
+`assertion left == right failed: ".../wr-relay-none-<pid>/stale.sock"`, `left: Some(0)`,
+`right: Some(92)`: the relay, pointed at a socket file whose listener had been dropped, connected
+(exit 0) instead of reporting no agent (exit 92). Seen on `fix/334-resolve-deadline` (2026-10-04)
+while the full `make app-test` ran at the same time, starting agents of its own. Run alone it passed
+5 times out of 5, and the whole Rust suite passed straight after. The branch doesn't touch the relay,
+the transport or the test.
+
+**Why:** Either the test is flaky under load or the relay can mistake a stale socket for a live
+agent, and then the "no agent listening" error a supervisor relies on would be missed.
+
+**How to start:** Run the test in a loop while `make app-test` runs, and log what the relay
+connected to (the peer's pid). Check whether `Scratch::new`'s per-pid directory can collide with a
+path another process binds.
+
+**Priority:** P0 (chosen at ship time on #334)
+
 ## P1 — before GA
 
 ## P2 — perf, correctness, and the next VCS phase

docs/designs/remote-workrooms.md

@@ -1775,10 +1775,29 @@ these are the subsystems that actually gate "a remote workroom is a real workroo
     `..` come back as their target. A `..` after a link out of the workroom can still tell
     whether the directory it names exists, which the VCS service's `stat` already does for any
     path; what follows the `..` is never looked up.
-    `resolve` shares `read`'s cap of four in flight, since a link can lead into a slow mount. A
-    newer click replaces the one before it, and an answer whose pane has closed opens nothing. An
-    error reading one path moves on to the next candidate; only a transport failure stops the
-    probe.
+    `resolve` shares `read`'s cap of four in flight, since a link can lead into a slow mount, and
+    answers within 10 seconds whether or not its walk has returned (#334): a walk hung on a dead
+    mount is left behind, holding no read slot and no permit, so reads and idle exit carry on.
+    Once four such walks are counted, `resolve` answers busy without starting another (up to seven
+    can be parked, since walks already waiting can still be left behind). The walks' threads are
+    not bounded in time, and a kill doesn't always end one: a FUSE request the daemon has already
+    read waits uninterruptibly. A hand-off's `execve` would wait on such a thread and freeze every
+    session, so the hand-off is refused while any walk is left behind, as it was when the walk held
+    its permit. On a remote host that means each connect pushes the update again and is refused,
+    and the host stays on its older agent with only a log line until the mount answers or the host
+    reboots. Restarting the agent doesn't clear it either: a process with a thread in such a FUSE
+    wait can't finish exiting, so it keeps its lock and listener, and a new agent finds the lock
+    taken; only the mount answering, aborting its FUSE connection, or a reboot does. The agent logs
+    each walk it leaves behind, and each one that returns. A click stops at the first resolve that
+    times out: its other candidates resolve the same prefix, or would read through the same link
+    with no deadline, so nothing more in that click reaches the host. One click on a dead mount
+    costs 10 seconds and one walk; a mount that is only slow to wake can take a second click, and
+    a word whose `:line` comes before its `..` (`good.rb:12/link/../t.rb`) loses its healthy
+    `good.rb` variant (#345). `read` itself has no deadline (#343), so a read opened through a link
+    onto a hung mount still holds its slot. A newer click replaces the one before it, and an
+    answer whose pane has closed opens nothing. An error reading one path moves on to the next
+    candidate; a transport failure, a resolve that timed out, or the agent refusing a resolve
+    because too many earlier walks are stuck stops the probe.
 
 ## Phase 0 Results
 
@@ -2962,6 +2981,9 @@ disagreement passes every test on either side alone while presenting as an empty
       (they are answered `LockContention`), then waits up to 2 s for running ones to finish, and
       refuses if they do not. Stopping first is what lets the wait end under steady traffic. Every
       app connection closes at the exec, and the next request reconnects.
+    - *No file lookup is stuck on a mount* (#334). A `resolve` left behind at its deadline holds no
+      request slot, so the wait above doesn't see it, but the exec would wait for its thread, which
+      a FUSE request can hold uninterruptibly. The hand-off is refused while any is left behind.
     - *The pre-check is the restore without the descriptors.* It runs
       `<binary> handoff-check <table>` on the real table, painting every screen. A binary that
       predates hand-off has no such command. A build without `terminal-state` refuses a table with
@@ -3002,7 +3024,9 @@ disagreement passes every test on either side alone while presenting as an empty
       - The binary is read three times by path (hash, check, exec), so a file swapped in between
         is not caught. Only a same-user process can do that, per the trust model above.
       - A refused hand-off (a repository command still running, or a session being ended or
-        repainted) is not retried until the next launch.
+        repainted) is not retried until the next launch. (A remote host's is retried on every
+        connect; a file lookup stuck on a mount only happens there, see Path handling under
+        As built (#254, remote pane parity).)
       - An agent that could not read its own binary at startup hands off on every request, even to
         the same binary.
     - *Tests* (`wr-agent/tests/hand_off.rs`):

macapp/WorkroomApp/Core/AgentFileProvider.swift

@@ -9,6 +9,16 @@ struct AgentFileProvider: FileProviding {
   /// predates `resolve` (#327) from one that answered.
   static let unknownMethod = "unsupported file method"
 
+  /// How the agent's answer begins when a `resolve` hit its deadline (#334): a walk through a link
+  /// onto a hung mount. Every candidate of the same click resolves the same prefix, so the click
+  /// stops there. `RESOLVE_TIMED_OUT` in `file.rs`, whose tests pin the prefix.
+  static let resolveTimedOut = "resolving timed out"
+
+  /// The agent's `Busy` answer to a `resolve` while too many earlier walks are still stuck (#334).
+  /// A click ends on it as on a timeout: its other candidates could read through the same hung
+  /// link, and a read has no deadline (#343). Pinned by the cap test in `file.rs`.
+  static let resolveWalksBusy = "too many earlier resolves are still walking"
+
   let context: FileContext
   let connection: AgentVCSConnection
 

macapp/WorkroomApp/Core/TerminalSessions.swift

@@ -2270,6 +2270,15 @@ final class TerminalSessions: ObservableObject {
             // An agent from before `resolve`: probe the lexical path, as every agent did before
             // #327. Any other failure (busy, unreadable, a loop) skips the candidate below, since
             // the lexical path is the wrong file whenever the host's answer would differ.
+          } catch FileServiceError.failed(let message)
+            where message.hasPrefix(AgentFileProvider.resolveTimedOut)
+            || message == AgentFileProvider.resolveWalksBusy
+          {
+            // The walk hung on a mount, or earlier ones still are (#334). The other candidates
+            // would resolve the same prefix or read through the same link, and a read has no
+            // deadline (#343), so nothing more in this click goes to the host.
+            NSLog("Workroom: resolving %@ on the host: %@", onHost, message)
+            return nil
           }
         }
         let found = TerminalLinkOpener.PathCandidate(

macapp/WorkroomAppTests/TerminalLinkOpenerTests.swift

@@ -654,6 +654,31 @@ final class TerminalLinkOpenerTests: XCTestCase {
       let failed = await sessions.remoteFile("link/../file.rb", cwd: nil, target: target)
       XCTAssertNil(failed, "the decoy opened after the host failed with \(failure)")
     }
+
+    // Value: protects=a resolve that hit its deadline (a link onto a hung mount, #334) ends the
+    // click, so its other candidates don't each wait out another deadline and leave another walk
+    // behind; fails_when=`remoteFile` treats the timeout like any other failure and goes on to the
+    // next candidate; why_new=every other failure here skips one candidate only; seam=none
+    sessions.remoteFiles = provider(
+      resolved: ["link/../file.rb": "nested/file.rb"],
+      failing: ["link/../file.rb:4": FileServiceError.failed("resolving timed out after 10s")])
+    let timedOut = await sessions.remoteFile("link/../file.rb:4", cwd: nil, target: target)
+    XCTAssertNil(timedOut, "a click went on to the next candidate after its resolve timed out")
+
+    // So does the agent's refusal while too many earlier walks are stuck: the next candidate of
+    // `link/file.rb:12/../t.rb` is plain `link/file.rb`, a read through the same hung link with no
+    // deadline (#343). It exists here, so reading it would return it.
+    sessions.remoteFiles = { location in
+      HostFiles(
+        context: FileContext(location: location, sharedLocation: nil),
+        files: ["link/file.rb": Data("x\n".utf8)],
+        failing: [
+          "link/file.rb:12/../t.rb": FileServiceError.failed(
+            AgentFileProvider.resolveWalksBusy)
+        ])
+    }
+    let busy = await sessions.remoteFile("link/file.rb:12/../t.rb", cwd: nil, target: target)
+    XCTAssertNil(busy, "a click read through the hung link after the agent refused its resolve")
   }
 
   /// Only what the host's file service can read: nothing outside the workroom's root, no `~` (this