Skip to content

fix(remote-terminal): a remote pane comes back on its own once its host answers again - #29444

Merged
nwparker merged 4 commits into
mainfrom
nwparker/remote-outage-recovery
Oct 16, 2026
Merged

nwparker merged 4 commits into
mainfrom
nwparker/remote-outage-recovery

Conversation

@nwparker

@nwparker nwparker commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 8 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​836 $\color{#cf222e}{\Huge{\mathbf{−}}}$​4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​832
Prod 9 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​196 $\color{#cf222e}{\Huge{\mathbf{−}}}$​11 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​185

ELI5

When the network between Orca and a paired remote Orca computer drops for a few minutes (a Tailscale re-key, a VPN reconnect, a long tunnel) and then comes back, the open terminal used to stay frozen for good, even though the remote shell was still running. You had to click Reconnect or restart Orca. Now the terminal comes back by itself as soon as Orca can talk to the remote computer again.

What Changed

  • Before: each remote terminal pane retries on its own for about 3 minutes. After that it stops and shows "disconnected", and only three things could wake it: the OS saying the network is back (online), the machine waking from sleep, or a Reconnect click. A Tailscale or VPN route that comes back does not change the OS network state, so none of those fire. Orca's control connection to the host reconnected (it retries forever), but the pane never tried again. Reproduced on current main: a paired pane after a 200 s silent outage stays at disconnected with the shell alive and nothing typed reaching it.
  • After: when an environment's control connection becomes ready again, every pane of that environment that is still recovering retries at once, and a pane that gave up replays its attach exactly as a Reconnect click does. Same PTY, same shell, typing works.
  • Mechanism: the renderer already received the control connection's state over onSharedControlDiagnostics but nobody listened. useRemoteRuntimeRecoveryTriggers now listens and, on a not-ready to ready transition, calls reviveRemoteRuntimePtyRecoveriesForEnvironment. Each remote pane transport registers a reviver for its environment only while it is recovering or latched (detach and destroy return it to idle, which unregisters it, so a discarded transport is never retained).
  • Also included, rebuilt from fix(remote-runtime): recover remote terminals promptly after sleep or network change #23359 (credit @Meapri): on wake or network return, every remote-runtime socket pings once and is replaced if it stays silent for 8 s (re-probed instead if the timer itself fired late because the process stalled); the shared-control retry ladder restarts from its short delays; and a resubscribe that fails or opens after the pane's window latched is parked so wake/online can revive it.

Why

The control connection answering is positive evidence the host is reachable, which is exactly the moment a pane retry can succeed. Alternatives considered:

No wire change: only client-side reactions to state the client already has. Loss of contact still never marks anything exited; a pane goes from disconnected back to live only by reattaching to the same PTY.

Linked Issue

Fixes #9092

Refs #25783: re-run on the managed-server SSH path at pane level, 30 s silent and reset outages recover on current main without this change (and a 200 s / 300 s silent outage too). The new ssh-orcad-terminal-network-outage.spec.ts keeps that contract in the managed SSH Docker lane.

Visual Proof

N/A: no new UI. The existing reconnect banner now clears on its own once the host is reachable instead of staying until Reconnect.

Testing

  • tests/e2e/paired-remote-terminal-network-outage.spec.ts (paired runtime through a cuttable TCP proxy): 30 s silent and 30 s reset outages pass on main and with the fix; the 200 s silent outage fails on main (pane stays disconnected, input never reaches the PTY) and passes with the fix.

  • tests/e2e/ssh-orcad-terminal-network-outage.spec.ts (managed SSH Docker host, profile reached through the same proxy): 30 s silent and reset pass, same shell PID afterwards.

  • Unit: remote-runtime-pty-transport-stream-reconnect.test.ts (latched pane revived by its environment, not another one; no registration while healthy or after detach), use-remote-runtime-recovery-triggers.test.ts (ready transition revives once per transition). Both new tests fail with the fix reverted. Plus the fix(remote-runtime): recover remote terminals promptly after sleep or network change #23359 tests and a stalled-timer re-probe test.

  • pnpm tc, pnpm run check:code-quality:changed pass. macOS host, Linux Docker SSH host.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

AI Disclosure

Review

Codex (gpt-6.1-sol, high): (1) the reviver registry retained transports discarded by detach-only cleanup: fixed, it now registers only while recovering. (2) The 8 s wake probe could kill a healthy socket if the process stalled past the deadline: fixed, a late timer re-probes.

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Cross-platform: timers and Electron IPC only. SSH: managed SSH panes use the same transport and control connection, so they get the same revive. Mobile: shared liveness code keeps its unref guard for DOM timer types. Backward compatible with old hosts (WebSocket ping only).

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

nwparker and others added 4 commits October 15, 2026 21:36
…p late resubscribes revivable

Rebuilt from #23359 on current main. On OS resume or network return every
remote-runtime socket pings once and is replaced if it stays silent for 8 s,
the shared-control retry ladder restarts from its short delays, and a
resubscribe that fails or opens after the pane's recovery window latched is
parked so resume/online can revive it.

Co-authored-by: Meapri <104259736+Meapri@users.noreply.github.com>
…ers again

A pane whose ~3 min auto-recovery window ran out only came back on an OS
online/resume event or a Reconnect click. A Tailscale or VPN route that
returns fires neither, so the pane stayed dead over a working link even
though the host shell was alive. The environment's shared control socket
reaching ready again is now the revive signal: the pane advances its backoff
or replays its attach, exactly as Reconnect does.

Adds e2e repros for paired runtimes and managed SSH hosts: 30 s silent and
reset outages, and a silent outage longer than the recovery window.
@coderabbitai

coderabbitai Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 15 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9f5c3708-d1db-44dd-8c9d-8fb6fc9d041a
📥 Commits

Reviewing files that changed from the base of the PR and between 90849ed and 6496b70.

📒 Files selected for processing: 17
  • src/main/ipc/runtime-environment-request-connections.ts
  • src/main/ipc/runtime-environment-retry-connections-probe.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-deadline-reattach.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-recovery-state.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport-stream-reconnect.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-resubscribe-after-deadline.test.ts
  • src/renderer/src/runtime/use-remote-runtime-recovery-triggers.test.ts
  • src/renderer/src/runtime/use-remote-runtime-recovery-triggers.ts
  • src/shared/remote-runtime-passthrough-socket.ts
  • src/shared/remote-runtime-shared-control-open.ts
  • src/shared/remote-runtime-shared-control-reconnect.ts
  • src/shared/remote-runtime-socket-liveness.ts
  • src/shared/remote-runtime-socket-resume-probe.test.ts
  • src/shared/remote-runtime-subscription-transport.ts
  • tests/e2e/paired-remote-terminal-network-outage.spec.ts
  • tests/e2e/ssh-orcad-terminal-network-outage.spec.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote runtime does not recover after macOS lock or sleep over Tailscale

1 participant