Skip to content

fix(remote-runtime): recover remote terminals promptly after sleep or network change - #23359

Closed
Meapri wants to merge 7 commits into
stablyai:mainfrom
Meapri:fix/remote-runtime-recover-after-sleep
Closed

Meapri wants to merge 7 commits into
stablyai:mainfrom
Meapri:fix/remote-runtime-recover-after-sleep

Conversation

@Meapri

@Meapri Meapri commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

ELI5

When your laptop sleeps, the network connections to a remote Orca computer often die without either side noticing. After you wake the laptop, Orca could take a long time to realise, or give up on a terminal for good, so you had to click Reconnect or restart Orca. Now, as soon as the laptop wakes or the network comes back, Orca quickly checks every connection, replaces the dead ones right away, and brings back terminals that had stopped retrying on their own.

What Changed

  • Before: A Mac paired with a remote Orca runtime (for example over Tailscale) sleeps or locks. On wake:
    • The connections usually still looked open, so the wake signal found nothing to retry. Orca only noticed they were dead through its regular 10-second health check, about 30–40 seconds later.
    • By then the one wake signal was used up. The control connection's next attempt could still be waiting on the long delay it had built up during the outage (up to 5 minutes for an idle connection).
    • If a terminal's reconnect attempt was still in flight when its ~3-minute automatic retry window ended, its late result was dropped. The terminal stayed disconnected until the user clicked Reconnect, even after the host was reachable again.
  • After: On wake or when the network comes back, dead connections are replaced within about 8 seconds. The next retry starts from the short delays (250 ms). A terminal that ran out its automatic window mid-attempt is picked up again by the next wake or network-back event.
  • Mechanism: three small changes in the existing recovery code.
    1. A quick check on wake. Each remote-runtime connection already has a health monitor (the shared control connection and each terminal or browser stream). The monitors are now kept in one list inside the app's main process. When the wake / network-back handler runs, every monitor sends one ping. A connection that is still being set up is skipped, because it has nothing to answer with yet and could otherwise be cut off mid-handshake. A monitor that hears nothing back within 8 seconds reports its connection dead. That hands the connection to the existing close-and-reconnect path. Repeated wake events while a check is running are merged into one.
    2. Start retries fresh after wake. A wake or network-back event sends the control connection's retry delays back to the start instead of leaving them at the long delay built up during the outage. This happens whether a reconnect was waiting on its timer or was already under way.
    3. Keep a late-finishing terminal revivable. A terminal's reconnect attempt can finish after its automatic window has already ended. If it fails, or connects too late to be used (Orca discards a connection from a window that has ended), the retry is now held for the next wake / network-back event or Reconnect click. Before, it was dropped. This reuses the hold mechanism that fix(remote-runtime): make every advertised recovery attempt reachable, and stop two recovery latches #17822 added for the first-connect path. This PR applies it to the two reconnect paths fix(remote-runtime): make every advertised recovery attempt reachable, and stop two recovery latches #17822 did not cover.

Why

The wake and network-back signals were already wired up. They just did nothing when the connections still looked open, and nothing when a terminal had already given up. This fix makes those signals effective and keeps everything else as it is. Alternatives considered:

Context: #17822 (merged) fixed the retry schedule and added the "hold the retry after the window ends" mechanism for first connects. This PR is its follow-up for wake detection and for the reconnect paths it did not reach.

Linked Issue

Fixes #9092

Visual Proof

No visual change: this changes when and how Orca reconnects, not anything it draws. The visible difference is how long the existing reconnect banner stays up after wake. Before, it took about 30–40 seconds or needed a manual Reconnect. After, it takes a few seconds once the network is back.

Testing

  • New src/shared/remote-runtime-socket-resume-probe.test.ts: a silent connection is declared dead within the wake-check deadline, one that answers is kept, stopped monitors are skipped, repeated wake events merge into one check, and a connection still being set up is not cut off. A wake also restarts the retry delays from the beginning, including when a reconnect is already under way. 6 of its 7 tests fail with the fix reverted. The seventh checks that a wake with no pending reconnect does nothing, which was already true.

  • New src/renderer/src/components/terminal-pane/remote-runtime-resubscribe-after-deadline.test.ts: a terminal whose reconnect is still in flight when the automatic window ends is revived by the wake trigger, both when that late attempt fails and when it connects. Both fail with the fix reverted (expected +0 to be 1, nothing held for the wake event).

  • New case in remote-runtime-pty-deadline-reattach.test.ts: the same situation on the path where the host has published a new terminal and its stream is still opening when the window ends. It fails when only that one-line change is reverted, which the test above does not catch.

  • New src/main/ipc/runtime-environment-retry-connections-probe.test.ts: the existing wake / network-back request from the window now runs the connection check. It fails with the fix reverted.

  • Existing remote-runtime terminal, shared-control, connection-health, runtime-environment and direct-SSH wake tests: 285 files, 2210 tests pass.

  • pnpm tc, pnpm run check:code-quality:changed upstream/main, pnpm run audit:anti-slop, check:max-lines-ratchet, oxlint and oxfmt on the changed files all pass. Full pnpm lint and pnpm build are left to CI.

  • Unit tests were run on macOS only. I did not reproduce a real sleep over Tailscale by hand. The timing in the tests is driven with fake timers.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

AI Disclosure

Claude Opus 5.5 (via Claude Code) wrote the fix and the tests, directed by a human.

Review

AI code review summary:

  • Cross-platform: No platform checks. The wake event comes from Electron's system-resume notification and the browser online event, both of which exist on macOS, Linux and Windows. The new timer uses the same checked unref call as other main-process timers, so it never keeps the app alive on its own. It has no path, shell or shortcut impact.
  • SSH / remote / local: Only connections to paired remote Orca runtimes are checked. Local terminals are untouched. Direct-SSH runtimes already send the same wake request (covered by direct-ssh-runtime-wake-isolation.test.tsx). There is no wire change: the check is a standard WebSocket ping that every supported server, old or new, already answers. Losing contact still only resets the connection. Nothing here reports remote work as exited, which is in line with docs/reference/ssh-execution-boundary.md.
  • Agents / integrations: Agent sessions in remote terminals come back through the existing reconnect-and-replay path. Agent status reporting is unchanged. Browser streams to the remote host are checked the same way as terminal streams.
  • Performance: One ping per open remote connection per wake / network-back event, merged when events repeat, and an 8-second timer per connection. Nothing is added to hot paths or to the regular health-check interval.
  • UI: N/A. The existing reconnect banner and its wording are unchanged.
  • Security: No new IPC channel. It reuses the existing runtimeEnvironments:retryConnectionsNow request. There is no new input parsing, logging or permission.

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • A link that takes more than 8 seconds to answer right after wake gets reconnected once. That costs one reconnect and a screen replay, not lost work, and the terminal keeps running on the host.
  • The wake check lives in shared code, but the mobile app does not use this module, so mobile is unaffected.
  • Commits: the two fixes, a commit with the two extra tests described above, and a small follow-up that replaces a type cast with a checked 'unref' in timer test (no behaviour change). Three more commits address review findings: the wake check skips connections still being set up, the retry delays reset on wake even while a reconnect is under way, and a reconnect that connects after the window ended is held like one that fails.

Author

X: @mea_pri

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

🤖 Generated with Claude Code

Meapri and others added 4 commits September 27, 2026 14:52
…ter the deadline (stablyai#9092)

A resubscribe attempt still in flight when the auto-recovery deadline
latched (the last attempt's 15 s RPC budget ends at the deadline by
design) rejected into a catch that only acted on a live epoch, so the
recoverable failure was dropped and nothing was parked. recoverAfter-
SubscribeFailure returned early the same way. System resume and network
online then had nothing to fire, and the pane stayed disconnected until
the user clicked Reconnect, e.g. after a laptop slept through a
Tailscale outage.

Both paths now park the resubscribe with parkRetryAfterDeadline, the
same mechanism the connect path already uses, so
retryAllRemoteRuntimePtyRecoveriesNow() opens a fresh recovery window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tablyai#9092)

On wake, sockets usually still read OPEN, so the resume/online trigger
found no reconnect timer to advance and did nothing. The dead path was
only noticed by the regular liveness cadence ~30-40 s later, by which
point the one-shot resume signal was spent and shared-control backoff
could sit at the 5-minute idle cap earned during the outage.

- Liveness monitors register in a process-wide set; a resume probe pings
  each socket (shared control and per-stream terminal/browser sockets)
  and declares it dead unless anything arrives within 8 s, which hands
  it to the existing close/reconnect path. Repeated triggers coalesce.
- runtimeEnvironments:retryConnectionsNow now also runs that probe.
- SharedControlReconnectScheduler.retryNow resets the attempt count, so
  a failure right after wake retries from 250 ms.

Loss of contact still only resets the connection; nothing here reports
remote work as exited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… IPC probe (stablyai#9092)

Two paths from the fix had no failing test without it: a rebound
subscribe whose stream open is still pending when a second recovery
deadline latches (recoverAfterSubscribeFailure), and
runtimeEnvironments:retryConnectionsNow reaching the socket probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… assertion

Uses the checked `'unref' in timer` narrowing already used elsewhere, so
the changed-code casting gate stays clean. Behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 27, 2026 06:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: d4bfb725-e7f1-493c-a5ba-c794d6760546

📥 Commits

Reviewing files that changed from the base of the PR and between c7153d6 and 99e0caf.

📒 Files selected for processing (7)
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-resubscribe-after-deadline.test.ts
  • src/shared/remote-runtime-shared-control-open.ts
  • src/shared/remote-runtime-shared-control-reconnect.ts
  • src/shared/remote-runtime-socket-liveness.ts
  • src/shared/remote-runtime-socket-resume-probe.test.ts
  • src/shared/remote-runtime-subscription-transport.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/shared/remote-runtime-socket-liveness.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

The change adds immediate probes for active remote runtime sockets and invokes them after shared-control retries. It resets the reconnect attempt counter when advancing a scheduled retry. It also parks PTY resubscription retries for recoverable failures that occur after recovery enters the disconnected phase. New regression tests cover socket probes, reconnect counters, and PTY retries after recovery deadlines.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 99e0c

The recovery changes are ready to merge after normal checks; no outstanding failure was established.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 99e0c

The recovery changes act on existing remote connections and terminal attachments rather than adding a new remote command or connection-selection route. No material security regression was established, but broader security coverage remains incomplete.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • observed — The main-process recovery operation can affect every cached shared-control connection and every active socket monitor registered in that process. The monitor registry is process-local; the evidence does not establish an application-wide or cross-process probe.

Trust Boundaries and Controls

  • inferred — The changed resume path does not supply attacker-selected environment identities or terminal commands to main-process recovery. Existing connection ownership and PTY handle-and-epoch checks remain the controls on which it relies.

Resilience and Maintainability Implications

  • observed — Overlapping socket probes are skipped, stopping a monitor cancels its probe deadline, and an unsuccessful ping does not start one. Shared-control retry consumes its pending callback before opening, so repeated triggers cannot dispatch that same scheduled callback twice.
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning Issue [#9092] requires shared-control subscriptions to retry until explicit closure with a maximum delay of 15 seconds. The current scheduler still defines ACTIVE_RECONNECT_DELAYS_MS with a 30,000 m… Use a shared-control subscription retry sequence capped at 15,000 ms. Preserve retries until explicit closure. Add a test that repeated subscription failures never schedule a delay above 15 seconds.
Docstring Coverage ⚠️ Warning Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 10 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The production changes and tests address socket liveness after wake or network return, shared-control reconnect timing, and terminal resubscription recovery. These changes support issue [#9092]. No un…
Title check ✅ Passed The title clearly and concisely describes the main change: prompt recovery of remote terminals after sleep or network changes.
Description check ✅ Passed The description is detailed and covers the required sections, including user impact, mechanism, rationale, linked issue, visual impact, testing, AI disclosure, compatibility considerations, and checkl…
Full details: Linked Issues check

Explanation

Issue [#9092] requires shared-control subscriptions to retry until explicit closure with a maximum delay of 15 seconds. The current scheduler still defines ACTIVE_RECONNECT_DELAYS_MS with a 30,000 ms maximum. This PR adds wake probes, resets retry state, and parks late terminal retries, which address other recovery paths. It does not change the shared-control delay cap. Tests cover the new probe and terminal paths but do not establish the required 15-second cap.

  • Fix all pre-merge checks with AI

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: d59f3757-25d2-4c1c-915b-9b7ace491c2f

📥 Commits

Reviewing files that changed from the base of the PR and between 7ac9c8d and c7153d6.

📒 Files selected for processing (8)
  • src/main/ipc/runtime-environment-request-connections.ts
  • src/main/ipc/runtime-environment-retry-connections-probe.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-deadline-reattach.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-resubscribe-after-deadline.test.ts
  • src/shared/remote-runtime-shared-control-reconnect.ts
  • src/shared/remote-runtime-socket-liveness.ts
  • src/shared/remote-runtime-socket-resume-probe.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — two follow-ups on the recovery paths, both inline.

Reviewed changes

  • Resume probe on wake / network-back — startRemoteRuntimeSocketLiveness now exposes probeNow and registers each monitor in a process-wide activeMonitors set; probeAllRemoteRuntimeSocketsNow() pings every live remote-runtime socket with an 8 s deadline, and retryRemoteRuntimeSharedControlConnectionsNow() invokes it alongside the existing retryNow() fan-out.
  • Backoff reset on resume — SharedControlReconnectScheduler.retryNow() zeroes attempt before firing a pending reconnect so the ladder restarts from 250 ms instead of the idle cap.
  • Park late resubscribe failures — a recoverable resubscribe failure landing after the auto-recovery deadline latched now goes through recovery.parkRetryAfterDeadline(...) in both recoverAfterSubscribeFailure and the resubscribeAfterTransportClose(...).catch, so online / system-resume / Reconnect can revive it.
  • Tests — IPC wiring test for the probe, unit tests for probeNow/coalescing and the backoff reset, plus two renderer cases covering the parked late failure on the attach and rebound-subscribe paths.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

Meapri and others added 3 commits September 27, 2026 16:20
…l connecting

A liveness monitor is installed while its socket may still be CONNECTING,
where ping() sends nothing. The resume probe still armed its 8 s deadline,
so a slow handshake with no inbound frame could be killed mid-connect.
ping() now reports whether it sent, and the probe arms only after a send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en is in flight

retryNow() only reset the attempt count when a reconnect timer was
pending. During an in-flight open the timer is already null, so a wake
left the pre-wake attempt and the next failure waited at the long cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne (stablyai#9092)

The deadline latch invalidates the in-flight subscription, so a stream
that opened just after it was discarded and returned normally; nothing
was parked and resume/online had nothing to revive. Park it the same way
as the late-failure path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Parked the late success of an in-flight resubscribe — subscribeToHandle's stale-subscription discard now calls parkResubscribeAfterDeadline when the handle is unchanged, the phase latched disconnected, and the recovery epoch is still owned, so a stream that resolves after the auto-recovery window closed is held for the next resume / online / Reconnect trigger instead of being dropped. This closes the last asymmetry from the prior review.
  • Restarted the shared-control backoff on wake while an open is in flight — SharedControlReconnectScheduler.retryNow() now zeroes attempt before the pending-timer guard, so a wake that lands mid-handshake still restarts the ladder at 250 ms; it continues to return false and open nothing when no reconnect is pending.
  • Made the resume probe skip a socket that is still connecting — the liveness ping callback now returns a boolean (both the shared-control and subscription closures return false when the socket is not OPEN), and probeNow returns without arming the 8 s deadline when nothing was sent, so a mid-handshake socket is not killed by a probe that never left.
  • Tests — a renderer case that holds the rebound stream open across the whole window and asserts it is parked on success, a unit case that a connecting socket arms no deadline while an open one does, and a scheduler case that a wake during an in-flight open still resets the ladder.

Independent checks of the two fixed paths found no duplicate/conflicting recovery or spurious reconnect: parkRetry is single-slot, a successful superseding subscribe clears any parked retry via markHealthy, and the probe's connecting-skip does not strand a dead socket (half-open reads OPEN and is still probed; CONNECTING is bounded by the 12 s handshake timeout and the unchanged 25 s cadence).

Pullfrog  | View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

@Meapri

Meapri commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Hi @Jinwoo-H, gentle ping on this PR — clean mergeable state with all checks passing. Whenever you have time, could you please take a look? Thank you!

nwparker added a commit that referenced this pull request Oct 16, 2026
…st answers again (#29444)

* fix(remote-runtime): probe sockets and reset backoff on wake, and keep late resubscribes revivable

Rebuilt from #23359 on current main. On OS resume or network return every
remote-runtime socket pings once and is replaced if it stays silent for 8 s,
the shared-control retry ladder restarts from its short delays, and a
resubscribe that fails or opens after the pane's recovery window latched is
parked so resume/online can revive it.

Co-authored-by: Meapri <104259736+Meapri@users.noreply.github.com>

* fix(remote-terminal): revive a latched remote pane when its host answers again

A pane whose ~3 min auto-recovery window ran out only came back on an OS
online/resume event or a Reconnect click. A Tailscale or VPN route that
returns fires neither, so the pane stayed dead over a working link even
though the host shell was alive. The environment's shared control socket
reaching ready again is now the revive signal: the pane advances its backoff
or replays its attach, exactly as Reconnect does.

Adds e2e repros for paired runtimes and managed SSH hosts: 30 s silent and
reset outages, and a silent outage longer than the recovery window.

* fix(remote-terminal): hold the control-ready reviver only while a pane is recovering

* fix(remote-runtime): re-probe a wake check whose deadline fired late, and trim the SSH outage spec

---------

Co-authored-by: Meapri <104259736+Meapri@users.noreply.github.com>
@nwparker

Copy link
Copy Markdown
Contributor

Thanks, @Meapri! This was rebuilt on current main and merged in #29444 (you're credited as co-author). Closing as superseded.

@nwparker nwparker closed this Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote runtime does not recover after macOS lock or sleep over Tailscale

3 participants