Repository navigation
fix(ssh): retry a stalled SSH reconnect when the machine wakes or the network returns - #29237
Conversation
📝 Walkthrough
Priority: ➖ Normal Severity of issue fixed: Medium Merge Risk: 🟡 Moderate · up to
|
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
4e100ebf-ccc0-4e14-9ed7-d29bf3515dfb
📥 Commits
Reviewing files that changed from the base of the PR and between 1b1d09c and 4a295270257db957868201abfdbef35a5d554e56.
📒 Files selected for processing: 7
src/main/ipc/runtime-environment-recovery-handler.test.tssrc/main/ipc/runtime-environment-recovery-handler.tssrc/main/ipc/ssh-host-sleep-reconnect.test.tssrc/main/ipc/ssh-host-sleep-reconnect.tssrc/main/ssh/ssh-connection-manager.tssrc/main/ssh/ssh-connection-wake-retry.test.tssrc/main/ssh/ssh-connection.ts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.
… network returns After sleep, an SSH host could sit in reconnection-failed forever: the retry ladder spent its nine attempts while the Mac was asleep or in dark wakes, and nothing restarted it. Resume only re-probed managed tunnels, and the window's online signal never reached SSH at all. - SshConnection.retryAfterNetworkReturn restarts a spent ladder, a pending backoff, or a background handshake older than 5s with a fresh budget. - Resume, unlock-screen and the window's resume/online signal (runtimeEnvironments:retryConnectionsNow) all call it via the manager and re-probe managed tunnels, so a transport that still looks connected after a network change is caught by the probe instead of keepalive. Co-Authored-By: Claude <noreply@anthropic.com>
…ement's transport A wake retry can now replace an in-flight handshake. Closing a system-SSH probe only signals the process, so its late close used to reset the transport state the replacement had just set up, leaving a connected host whose exec failed with 'Not connected'. Co-Authored-By: Claude <noreply@anthropic.com>
…proxy process Co-Authored-By: Claude <noreply@anthropic.com>
7cf4ce2 to
a0f5930
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
🟠 Major · Guard the configuration assignment against superseded attempts. · ssh-connection.ts:851
src/main/ssh/ssh-connection.ts:851
🔒 Security & Privacy | 🟠 Major | ⚡ Quick winGuard the configuration assignment against superseded attempts.
When this retry replaces an attempt waiting in
resolveWithSshG, the old attempt can resume after the replacement assignshostKeyResolvedConfig. The old attempt overwrites that field before its generation check. If the replacement is still checking security-key transport,doSsh2Connectcan then use the old attempt’sstrictHostKeyCheckingor known-hosts configuration. Check the generation before assigninghostKeyResolvedConfiginattemptConnect, or keep the resolved configuration local to each attempt.
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
15db1282-88e0-4a16-99ef-a8fc131f17ce
📥 Commits
Reviewing files that changed from the base of the PR and between 4a295270257db957868201abfdbef35a5d554e56 and a0f5930.
📒 Files selected for processing: 2
src/main/ssh/ssh-connection-wake-retry.test.tssrc/main/ssh/ssh-connection.ts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.
ELI5
If your computer sleeps for a while, Orca can use up all of its automatic SSH reconnect attempts while the computer is still asleep. Once those attempts were used up, nothing tried again. You had to click Reconnect or restart Orca, even though a plain
sshto the same host worked fine. Now, when the computer wakes, the screen unlocks, or the network comes back, Orca retries right away, with a fresh set of attempts.What Changed
Before
reconnection-failed) stayed down until you reconnected it by hand. In [Bug]: SSH host stays in reconnection-failed after macOS sleep; no retry on wake #26888 it was still down 40 minutes after wake. On macOS, the attempts are used up during the short "dark wakes" while the lid is closed. Those dark wakes give no wake signal, so the attempts are gone before the real wake.onlineevent), that signal reached the remote-runtime connections but not SSH.After
resume), screen unlock (unlock-screen), and the window's existing resume/online request (runtimeEnvironments:retryConnectionsNow) all run one function,retrySshConnectionsAfterNetworkReturn. That function restarts every SSH connection that is stalled in one of three ways: it gave up, it is waiting out a backoff delay, or a background attempt has been running for more than 5 s. The retry starts with a full set of attempts. The same function then re-checks the managed-server tunnels, as wake already did. A connection that still looks connected but whose route died is found by that check within about 10 s, rather than by keepalive.Mechanism.
SshConnection.retryAfterNetworkReturn()sits on top of the existingreconnect()path: it closes the transports, resets the backoff ladder, and makes one immediate attempt. If that attempt fails, the normal ladder takes over from the first step.SshConnectionManager.retryStalledReconnects()calls it for each connection. Healthy connections are left alone, and so are a first connect, a closed connection, and a connection waiting on a password or passphrase prompt (so an open dialog is never closed). Auth failures and permanent errors are also left alone.Why
reconnect()and the managed-tunnel resume check.Linked Issue
Fixes #26888
Fixes #21127
Not fixed here:
Visual Proof
N/A. Nothing new is drawn. The difference is that a host showing "reconnection-failed" or "reconnecting" after sleep comes back by itself after wake or when the network returns.
Testing
New
src/main/ssh/ssh-connection-wake-retry.test.ts(fake timers, realSshConnectionwith the ssh2 mock). Covers: a ladder used up while asleep stays down for 10 minutes, then reconnects on the wake retry. A wake retry that fails gets a fresh ladder. A 30 s backoff is skipped. A hanging handshake older than 5 s is replaced, and a younger one is kept. Healthy, first-connect and closed connections are untouched. The manager restarts each stalled connection.src/main/ipc/ssh-host-sleep-reconnect.test.ts:resumeandunlock-screenrestart stalled reconnects, the network-back path also re-checks tunnels, and the listeners go away once unregistered.New
src/main/ipc/runtime-environment-recovery-handler.test.ts: the window's resume/online request now also reaches SSH.With the fix reverted, 10 of the 13 tests fail. The other three cover behavior that is unchanged.
pnpm test src/main/ssh/ssh-connection src/main/ssh/orcad-managed-tunnel src/main/ipc/ssh src/main/ipc/runtime-environment: 82 files, 675 tests pass.pnpm tcpasses, and so doespnpm run check:code-quality:changed.I manually tested these changes locally
Automated tests added/updated, or explained why not below
Review
resumefires on macOS, Windows and Linux.unlock-screenfires only on macOS and Windows, and on Linux the listener just never fires. The browseronlineevent works everywhere.runtimeEnvironments:retryConnectionsNowchannel.Agent skill upstream boundary
Notes
Overlap: none with #29210 (renderer pane recovery) or #29220 (move refusal). The work on serving-probe latency changes
orcad-managed-serving.tsand the shared-control files, which this PR does not touch.Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)