[Bug]: SSH reconnect kills a busy remote server, cancelling all running agents (2s reuse probe)

[bloom-street]

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/desktop (SSH environment launch script), with apps/server startup recovery

Steps to reproduce

1. On laptop A, run T3 Code desktop with an SSH environment pointing at machine B. The launch script starts a managed server on B (~/.t3/ssh-launch/<key>/, managed).
2. Start several long agent runs on B, with background shells and subagents, and leave B under heavy CPU load (load average about 2-3x core count from test suites).
3. Close the lid on laptop A, on battery. macOS DarkWakes it about every 15 minutes, and the desktop app reconnects the SSH environment on each wake.
4. Each reconnect reruns REMOTE_LAUNCH_SCRIPT on B.

Expected behavior

A live, healthy server that is only slow to answer is reused. Running agents and their background work survive a client reconnect.

Actual behavior

On each reconnect the reuse probe (REMOTE_REUSE_READY_TIMEOUT_MS = 2e3, SSH_READY_PROBE_TIMEOUT_MS = 1e3) does not get an answer in time on the loaded host. The script hits elif ! wait_ready ...; then kill "$REMOTE_PID" and starts a new server. The new server's V2 orchestration recovery then stops every provider session (stoppedSessions: 8) and emits run.background-work-cancelled for every running background shell and subagent.

In one night this happened 29 times, about every 15 minutes from 00:34 to 08:29. Each server restart was about 10-15 s after a laptop DarkWake, for example:

Laptop DarkWake	Server log Relay client stopped then Listening on
00:33:44	00:33:57 → 00:34:06
00:46:23	00:46:33 → 00:46:40
01:02:08	01:02:20 → 01:02:24
01:21:42	01:21:59 → 01:22:03
01:37:27	01:37:37 → 01:37:42

No agent work survived the night. The server never crashed; every shutdown was a clean SIGTERM.

Related cases in the same script:

• RUNNER_CHANGED=1, when a client on a different nightly reconnects after an auto-update, also kills a busy managed server and cancels all of its work without warning.
• If server-runtime.json points at another live server (for example the desktop app's own server on B), the next reconnect kills the managed server even though it is healthy and running agents.

Impact

Major degradation or frequent failure

Version or commit

0.0.46-nightly.20261004.2657 (overnight), then 0.0.46-nightly.20261005.2702. macOS on both ends.

Suggested fixes

• Do not kill a managed server whose pid is alive just because a 2 s readiness probe failed. Retry with a longer budget (for example 30-60 s) or check liveness via the pid and server-runtime file before deciding.
• On RUNNER_CHANGED with active runs, defer the restart or ask the user, rather than SIGTERMing immediately.
• Consider making a client reconnect never restart the server at all. Restarts only from explicit user action or update.

[juliusmarminge]

Note

Grok responding on behalf of Julius.

Triage

Thanks @bloom-street for the overnight log analysis, the wake pattern and the clean SIGTERMs made this much easier to trace. Reading current main, a desktop SSH reconnect can stop a managed remote server, and the new process's startup recovery is what ends the running agents. It doesn't look like a crash or a provider bug.

What I found

• Stale local forward after sleep. ensureTunnelEntry probes the existing local forward for 2 seconds. After sleep that forward is dead, so the probe fails even if the remote server would have answered immediately. The tunnel finalizer then calls stopRemoteServer(), and the stop script SIGTERMs the recorded pid unless the host is marked external. The next launch starts a replacement. Host load isn't required for this path.
• The elif ! wait_ready branch is a second kill. In packages/ssh/src/tunnel.ts, a managed pid that's still alive is SIGTERM'd when RUNNER_CHANGED=1, or when wait_ready misses REMOTE_REUSE_READY_TIMEOUT_MS (2s, each GET / capped at 1s). That runs when the finalizer didn't already remove the pid file. A loaded host can miss that budget. The logs don't show ssh.environment.tunnel.existing.stale vs. the launch script, so it's not clear which of the two fired each time; both produce the same SIGTERM and replacement.
• Recovery on the new server. V2 orchestration recovery completed is startup reconciliation: stoppedSessions: 8 is every provider session not already stopped or errored, and run.background-work-cancelled clears background shells and subagents. continueThreadsAfterServerUpdate (off by default) would start a new continuation turn, but wouldn't preserve the killed provider process.
• server-runtime.json case. The managed process is started with --base-dir ~/.t3, so the file it reads on the next launch is usually its own record. If that probe succeeds and state says managed, the script kills that pid and flips the host to external. fix(ssh): stop killing the managed server on reconnect #13341 and fix(ssh): reconnecting no longer restarts the remote server #13521 skip the kill only when the runtime pid is the managed pid; they don't change the timeout kill or the tunnel finalizer.
• RUNNER_CHANGED case. This is the update path. Whether to defer an update while runs are active is a separate question for maintainers.
• Related PRs and issues. fix(desktop): give the SSH readiness probe room for high-RTT links #7815 lengthens the desktop tunnel probe for high-RTT links and leaves the remote 2s reuse budget alone. fix(ssh): keep remote work running when the desktop closes #10951 would have stopped the finalizer from killing the remote on teardown but was closed unmerged. [Bug]: Restarting the desktop client breaks running remote Codex threads over SSH #10934 (quit/restart, same finalizer), SSH reconnect can replace an external T3 service with a competing managed runtime #5749 (a competing server beside an external service), and [Bug]: SSH remote environment never connects when link RTT exceeds ~1s, eg on in-flight wifi (readiness probe deadline is 1000ms) #7733 (desktop never connects because the local probe is 1s) are different failures.
• Docs. The user docs say removing the connection stops a server T3 Code launched; a lid close isn't removing the connection, but the finalizer treats any stale tunnel as a managed-server stop.

Likely fix area

packages/ssh/src/tunnel.ts and the tunnel finalizer. Some options:

• On a stale tunnel, wake, or client teardown, close only the local forward and keep stopRemoteServer() for explicit Disconnect / Remove.
• In the launch script, avoid SIGTERMing a live managed pid just because a 2s loopback probe missed, e.g. retry with a budget closer to the 60s cold-start wait, or treat a live pid plus a bound port as enough to reuse.
• Adopt the same-pid guard from fix(ssh): stop killing the managed server on reconnect #13341 / fix(ssh): reconnecting no longer restarts the remote server #13521 so a healthy server isn't killed when server-runtime.json names itself.

Workaround: for a host that needs to outlive the laptop, npx t3 service install on the remote and then a normal paired connection avoids the managed-server lifecycle.

A maintainer will decide on the fix direction.