Skip to content

fix(ssh): suppress resume after relay replacement - #16839

Closed
brennanb2025 wants to merge 2 commits into
mainfrom
brennanb2025/sta-5698-relay-absence-fallback-r1
Closed

brennanb2025 wants to merge 2 commits into
mainfrom
brennanb2025/sta-5698-relay-absence-fallback-r1

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 10 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​743 $\color{#cf222e}{\Huge{\mathbf{−}}}$​8 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​735
Prod 30 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​408 $\color{#cf222e}{\Huge{\mathbf{−}}}$​132 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​276

Summary

  • treat a reachable SSH relay's "that terminal isn't here" as typed evidence with an authority check, instead of flattening it into a generic session-expired error
  • each relay stamps a unique process identity at startup; the client persists it with the SSH PTY lease and accepts positive absence as authoritative (exited) only when the answering relay is the exact instance that owned the binding — any other answer (replacement relay, older relay with no identity, malformed or failed status) degrades to unverifiable, never exited
  • on unverifiable, retire the unusable binding but start a plain shell with no agent launch authority, launch metadata, claim, or telemetry, and surface the existing resume-unavailable banner

What this means for users (ELI5)

Orca keeps a small helper process — the relay — running on your SSH host. It owns the terminal processes for your remote workspace, and Orca reconnects to it whenever you come back.

Before this fix. You reopen an SSH workspace whose relay was replaced while you were away — an Orca update restarted it, the host rebooted, or host-side cleanup swapped it out. The replacement relay truthfully answers "I don't have that terminal", but it isn't the relay that was holding your terminal, so it cannot actually know what happened to the process. Orca took that answer as proof your agent had exited: it marked the pane as a clean exit and automatically re-ran the original agent launch/resume command in a fresh terminal. If the old process was in fact still alive on the host, you now had a second agent silently resuming into the same session — duplicate work, contested session state, and agent-launch claims and telemetry for a launch you never asked for.

After this fix. Orca checks who is answering before believing "it's gone". If the same relay that owned your terminal says the terminal is gone, that is authoritative and nothing changes: normal exit handling, resume works as before. If a replacement relay answers, Orca refuses to guess: the pane opens as a plain shell — no automatic agent relaunch, no session claim, no launch telemetry — with the existing "resume unavailable" banner, so you decide what to run next. Nothing is silently resumed on an answer that could not prove anything.

When you'd hit it: reconnecting to an SSH workspace after its relay restarted — typically after an Orca upgrade, a host reboot, or host-side cleanup — while a persisted terminal binding for that workspace still exists.

Root cause

The stable-pane fallback replayed the original launchAgent / resume command after a reachable SSH relay answered that the persisted PTY was absent. It did not distinguish the relay instance that owned the binding — and could authoritatively prove process exit — from a younger replacement relay that could never have observed the bound process.

Remote wire compatibility

The relay mints its process identity at startup and reports it in relay.status, and — only when the client asks via a new optional requestRelayProcessId request field — in spawn/attach results; the client persists it on the SSH PTY lease as a new optional field. These are additive optional fields on existing RPCs: no new stream opcode, no removed field. An older host simply omits the identity, so the client classifies its positive absence as unverifiable and takes the conservative plain-shell path instead of the old replay — a deliberate behavior change, since such a host cannot prove ownership. An older client never requests the identity and keeps its prior behavior.

Verification

  • failing-first proof: before the production fix, all 4 initial relay-age tests failed because absenceVerdict was absent
  • pnpm test across 7 focused/adjacent files: 38 passed
  • final focused relay suite: 3 files, 12 passed
  • pnpm tc and pnpm tc:node
  • direct code-quality, type-aware, and React Doctor oxlint scans on all changed files
  • max-lines ratchet, oxfmt, and git diff --check
  • same-model adversarial review: CLEAN after fixing IPC and runtime-owned suppression paths
  • isolated Electron/CDP identity verified for this worktree; production resume-unavailable banner rendered visibly in a real terminal pane

Scope note

This focused PR is based directly on main. It overlaps the relay-age fallback area in draft PR #16763, but it does not stack on, retarget, close, or modify that PR.

Closes STA-5698.

@coderabbitai

coderabbitai Bot commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The change tracks SSH relay process identities through relay status, PTY spawn and attach responses, PTY results, and persisted leases. Stable-pane adoption compares persisted and current relay identities to produce exited or unverifiable liveness verdicts. Unverifiable absence suppresses agent launch inputs, session claims, launch bookkeeping, telemetry, and launch configuration. Spawn responses report suppressed startup or unavailable resume status. Tests cover identity validation, relay replacement fallback, liveness callbacks, lease persistence, and startup suppression.

Merge Risk: 🟡 Moderate · up to 96b41

The change can suppress normal agent startup for older SSH leases and can carry a stale relay identity across retries, leading to incorrect recovery behavior; the relay ownership check also needs explicit boundary validation. Merge should wait for the two actionable fixes, with security-contract hardening tracked by the owner.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 5.56% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 36 functions across 39 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description provides a detailed summary, user impact, root cause, linked issue, compatibility notes, and verification results. However, it omits the required Visual Proof section and most template… Add the required template sections and complete the checklist. Include Visual Proof with N/A and a reason if no visual change applies, document manual testing and tested platforms, add the Agent skill upstream boundary response, and con…
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the primary change: preventing automatic SSH session resume after relay replacement.
Full details: Description check

Explanation

The description provides a detailed summary, user impact, root cause, linked issue, compatibility notes, and verification results. However, it omits the required Visual Proof section and most template checklist sections, including explicit platform coverage and completion status.

Resolution

Add the required template sections and complete the checklist. Include Visual Proof with N/A and a reason if no visual change applies, document manual testing and tested platforms, add the Agent skill upstream boundary response, and confirm lint, typecheck, test, and build status.

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2a5a1da5-6822-4b99-8d5c-d9c859c1293e

📥 Commits

Reviewing files that changed from the base of the PR and between c0a884e and 96b413b.

📒 Files selected for processing (25)
  • src/main/ipc/pty-pane-reservation-settlement.test.ts
  • src/main/ipc/pty-relay-age-fallback.test.ts
  • src/main/ipc/pty/ipc/spawn-commit-persist.ts
  • src/main/ipc/pty/pane/ssh-relay-absence-verdict.ts
  • src/main/ipc/pty/pane/stable-owner.ts
  • src/main/ipc/pty/pane/stable-pane-relay-age-fallback.test.ts
  • src/main/ipc/pty/pane/stable-pane-types.ts
  • src/main/ipc/pty/runtime/spawn-commit.ts
  • src/main/persistence/leasing-ssh-ptys/ssh-normalization.ts
  • src/main/persistence/leasing-ssh-ptys/ssh-relay-process-id-normalization.test.ts
  • src/main/providers/pty-spawn-result.ts
  • src/main/providers/ssh-agent-session-create-operation.ts
  • src/main/providers/ssh-pty-provider-agent-session-create-operation.test.ts
  • src/main/providers/ssh-pty-provider-reattach-incarnation.test.ts
  • src/main/providers/ssh-pty-provider-spawn.test.ts
  • src/main/providers/ssh-pty-session-reattach.ts
  • src/main/providers/ssh-pty-spawn-request.ts
  • src/main/providers/ssh-relay-process-id.ts
  • src/relay/pty-handler-spawn-admission.test.ts
  • src/relay/pty-handler-test-harness.ts
  • src/relay/pty-handler.ts
  • src/relay/relay-daemon.ts
  • src/relay/relay-runtime-services.ts
  • src/relay/subprocess.test.ts
  • src/shared/ssh-types.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 1 remains after this review.

Comment thread src/main/ipc/pty/pane/stable-owner.ts Resolved
Comment thread src/relay/pty-handler.ts Resolved
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Exact-head review verification — 96b413bde43c287821d9c31fc2c5c486274d5d12 (96b413bde4)

Verdict: CLEAN. I re-reviewed the full 40-file PR diff at this exact head, then ran an independent same-model adversarial reviewer through two passes. No in-scope P0/P1/P2 findings remain.

  • Execution boundary: only an exact persisted/current relayProcessId match certifies exited. Missing RPC support, missing or malformed identity, mismatch, or status failure remains unverifiable; no age boundary is used.
  • Lifecycle: the IPC-owned and runtime-owned spawn paths suppress command delivery, launch token/config, agent-session claims/create-operation identity, launch bookkeeping, and telemetry after unverifiable absence, while preserving the fresh-shell degraded state and user notice.
  • Compatibility: the request/result/lease fields are optional. Older relays that do not return identity degrade conservatively to unverifiable; no required RPC, stream opcode, mobile contract, or persisted-field rename was introduced.
  • Performance/resources: the only new probe is one absence-only relay.status RPC bounded to 5 seconds. No polling, broad inventory scan, subprocess fanout, listener/timer leak, renderer churn, or unbounded collection was added.
  • Review threads: both current non-outdated threads were independently adjudicated as non-findings and resolved. Restoring an age fallback would violate the exact-identity boundary; cached create-operation replay stays within one immutable relay handler/process lifetime and production retries preserve the request shape.

Verification performed at this head:

  • focused SSH/PTY/relay suite: 10 files, 134 tests passed
  • adjacent liveness/resume suite: 4 files, 42 tests passed
  • rendered-banner contracts: 4 files, 22 tests passed
  • pnpm tc:node: passed
  • direct type-aware and React-doctor oxlint on every changed TS file: passed with warnings denied
  • git diff --check: passed

The changed-code-quality wrapper itself could not parse pnpm's Node-26 engine warning as JSON; the two underlying oxlint scans passed directly. Live relay-replacement fault injection and native Windows/Linux execution remain environmental gaps, not claimed passes.

@brennanb2025

Copy link
Copy Markdown
Contributor Author

QA evidence — 96b413bde43c287821d9c31fc2c5c486274d5d12 (96b413bde4)

Rendered banner arm: PASS. I launched an isolated dev instance with its own user-data directory, CDP port 9334, and renderer port 5176. CDP reported devRepoRoot as this exact worktree, devBranch as 96b413bde4, and devLabel as pr-16839-96b413-evidence before capture.

Through the real Orca surface and production main/preload/renderer path, I started a real local Codex PTY with deliberately unverifiable resume provenance. Main dropped the resume request, the terminal remained live (inspectProcess: foreground codex, child process present), and the full rendered pane visibly showed the exact degraded-state notice:

--- previous session unavailable, started fresh ---

Exact-head rendered resume-unavailable banner

Live SSH relay-replacement arm: UNVERIFIABLE in this isolated environment. The disposable loopback SSH probe was:

ssh -o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null 127.0.0.1 true

It failed with exit 255: Permission denied (publickey,password,keyboard-interactive). I did not substitute another host or touch an existing Orca/SSH session. The exact relay-identity/mismatch/missing/malformed/status-failure behavior is instead covered by the passing focused suites above; native Windows/Linux and real network-drop execution are not claimed.

@brennanb2025

Copy link
Copy Markdown
Contributor Author

Decision record: what this PR actually does, and what should outlive it

Written after reading the full diff at 96b413bde43c287821d9c31fc2c5c486274d5d12 against origin/main at 2c86d2a3bd677ab432f3eca02a5ae738866e2d91 (96 behind / 2 ahead, 6 conflicts). No code was pushed to this branch, and it was deliberately not rebased — the point was to decide it, not to refresh it.

1. The mechanism is a per-relay nonce, not an age comparison

This is the one place where the "it's a duplicate of #16989" summary is wrong, and it deserves to be on the record.

src/main/ipc/pty/pane/ssh-relay-absence-verdict.ts:

return status?.relayProcessId === args.bindingRelayProcessId
  ? { status: 'exited' }
  : { status: 'unverifiable', reason: UNPROVEN_RELAY_OWNER_REASON }

relay-daemon.ts mints randomUUID() once per daemon start, publishes it on relay.status, and returns it on spawn/attach only when the client sets the new optional requestRelayProcessId param. The client persists it on the SshRemotePtyLease (src/shared/ssh-types.ts) and compares for equality.

That is an identity check. #16989 (answeringRelayPredatesBinding) compares relay.status.uptimeMs against the lease's age with 30s of slack — a proxy for the same question. So the two PRs implement the same predicate ("did the relay that just answered actually own this binding?") with different evidence, and the identity version is the sound one. The test files here are still named *-relay-age-*, which is what makes it read as an age fix from the outside; it isn't.

But that does not rescue the approach, because the objection in #16843 was never that the age evidence was imprecise. It was that the predicate itself convicts the safe case:

an age comparison convicts any younger relay, including after a host reboot where no orphan can exist, so every SSH pane on that host silently loses its agent resume

After a host reboot the relay mints a new UUID. Mismatch → unverifiable → plain shell, no resume. grep -ci boot over this diff is 0; there is no boot-id, no host-identity persistence, nothing that distinguishes "relay restarted" from "machine rebooted." The identity mechanism reproduces the rejected harm exactly. #16843's close comment also listed "daemon nonce" by name among the guards already considered and rejected as a class.

2. Its failure policy is stricter than #16989's, not looser

This is the part that decides it. The two guards differ in what they do when the evidence is missing:

#16989 (age) #16839 (nonce)
Relay demonstrably replaced suppress suppress
Host rebooted suppress suppress
relay.status times out keep resume (2s budget) suppress (5s budget)
relay.status malformed / rejected keep resume suppress
Host runs a relay without the field keep resume suppress
Lease predates this change (every lease on disk today) keep resume suppress

resolveSshRelayAbsenceVerdict short-circuits to unverifiable on !provider.requestHostRpc || !bindingRelayProcessId, and its catch returns unverifiable too. Every unknown is fail-closed. #16989 returns false on every unknown and stays on today's behavior.

Two consequences worth naming:

  • A transient hiccup costs you the resume. The probe runs on the reconnect path — precisely when the SSH link is least reliable. A 5s relay.status timeout is converted into "your agent did not come back," and the user has to relaunch by hand.
  • It is an admitted, permanent regression against un-upgraded hosts. The PR body says so plainly: "An older host simply omits the identity, so the client classifies its positive absence as unverifiable … a deliberate behavior change." Per docs/reference/remote-wire-compatibility.md, mixed client/host versions are the normal state, so this is not a transition cost — it is the steady state for every host that hasn't been redeployed. The lease does get stamped on the next successful spawn/reattach, so the on-disk migration is one-shot per binding, but the old-host case never resolves.

If #16989 is being closed for convicting too much, #16839 convicts a strict superset.

Verdict vocabulary: this PR handles it correctly — { status: 'unverifiable', reason } from src/shared/pty-liveness-verdict.ts, no synonyms, and loss of contact is never promoted to death. That is not the problem here.

3. About a quarter of the diff already shipped on main

The first bullet of this PR's summary — "treat a reachable relay's that terminal isn't here as typed evidence instead of flattening it into a generic session-expired error" — landed on main in #16763 (25fa6a9920, merged 17 minutes after this PR opened). Main already has:

  • SshPtyAbsentFromRelayError + isSshPtyAbsentFromRelayError in src/main/providers/ssh-pty-errors.ts
  • the identical reattachSshPtySession catch-block rewrite in src/main/providers/ssh-pty-session-reattach.ts
  • isSshPtyAbsentFromRelayError folded into isPtyAlreadyGoneError in src/main/ipc/pty/provider/liveness.ts
  • a file already named src/main/providers/ssh-pty-relay-absence-verdict.test.ts

clearProviderPtyState(id, { preserveAgentSessionOwners }) predates both (#15172).

Four of the six merge conflicts are exactly those files (liveness.ts, ssh-pty-errors.ts, ssh-pty-session-reattach.ts, and an add/add on ssh-pty-relay-absence-verdict.test.ts). The remaining two are src/shared/ssh-types.ts (the one-line lease field) and src/main/ipc/pty/runtime/spawn-commit.ts.

So "+1151 / −140 across 40 files" overstates the unique surface. Netting out the #16763 overlap, the genuinely new content is: the nonce transport and persistence (~11 files), the verdict + suppression plumbing (~15 files), and 10 test files (+743). Nothing in it is off-topic for STA-5698 — there is no incidental fix in here worth keeping for its own sake, and the stable-pane-types.ts / stable-pane-spawn-fallback.ts split is a mechanical extraction that only exists because stable-owner.ts grew.

4. What is genuinely worth salvaging

#16843's close said the renderer half "is not lost work; it is the correct outcome if a guard is ever justified." That is right, and this PR's version of it is materially better than #16989's — that is the real asset here, and it is independent of which guard (if any) ever ships.

#16989 strips intent at the provider spawn-options layer only (stripStartupIntent) and gates two call sites on agentResumeUnavailable. #16839 does it at the runtime layer (orca-runtime.ts, earlyAgentStartupSuppressed), which additionally suppresses:

  • launchToken is never minted, and ORCA_AGENT_LAUNCH_TOKEN is deleted from the child env
  • buildClaudeAgentTeamsLaunchPlan is skipped entirely (no shim dir built for a launch that won't happen)
  • effectiveLaunchConfig → undefined, so no sleeping launch config is persisted on the pane
  • resumeProviderSession, telemetry, and agentSessionClaim are all dropped
  • the published launchConfig / launchToken / launchAgent on the pty record are cleared

Two of those gaps in #16989 are substantive, not cosmetic. Under #16989 the pane still gets a published launch config, so it is still an agent pane holding resume intent; and #16989 leaves runtime?.onPtyExit(owner.ptyId, 0, …) and the bare clearProviderPtyState(owner.ptyId) untouched — meaning it publishes exit code 0, a clean-exit death certificate, for the very PTY it just classified as unverifiable, and releases the agent-session claim for a process it cannot prove is dead. #16839 gets both right:

if (absenceVerdict.status === 'unverifiable') {
  runtime?.markPtyLivenessUnverifiable(owner.ptyId, absenceVerdict.reason)
  runtime?.onPtyExit(owner.ptyId, -1, owner.incarnationId)
} else {
  runtime?.onPtyExit(owner.ptyId, 0, owner.incarnationId)
}
clearProviderPtyState(owner.ptyId, {
  preserveAgentSessionOwners: absenceVerdict.status === 'unverifiable'
})

One thing to carry the other way: withoutAgentStartupIntent here misses startupIngress, which #16989's stripStartupIntent does strip. Whichever survives should be the union.

The second salvage is the scaffolding, not the nonce. #16843 said that if prevention proves insufficient, "the mechanism to build is a pid + start-time probe, which Orca already implements for its local daemon." The plumbing this PR builds — an optional per-relay stamp on relay.status, an opt-in requestRelayProcessId on spawn/attach, a normalized optional field on SshRemotePtyLease, and a PtyLivenessVerdict threaded from the absence fallback into the runtime spawn path — is mechanism-agnostic. Swapping randomUUID() for pid + startTime is a change to one field, not to the transport. If that probe is ever built, start from this wiring rather than from scratch. What must not be carried over is the fail-closed policy on unknowns.

Nothing here helps #16894 / #16895; the host-side prevention work shares no surface with this PR.

5. Recommendation: salvage, then close as superseded

Superseded by #16901 (mint PTY ids carrying the relay incarnation, so a stale id can never resolve to a different live process) plus the host-side prevention pair #16894 / #16895 — prevent the orphan and remove the ambiguity, rather than detect the collision on the client. This PR does not clear the bar #16843 was closed against, and its failure policy is stricter than #16989's, which is itself recommended for closure.

One thing that argues the other way, on the record

The "three PRs remove both ingredients" framing is not quite complete, and it should be visible to whoever revisits this:

So the residual risk is real but narrow: a relay on a Windows SSH host killed by something other than an uncaught exception. On POSIX, the live repro showed the pty master closing and SIGHUP reaping the shell and agent in 0.3s — no orphan to collide with.

That residual is an argument for revisiting a guard later, on the pid + start-time mechanism #16843 named, using the scaffolding above. It is not an argument for merging a fail-closed nonce guard that would cost every SSH user their agent resume after any host reboot, any un-upgraded relay, and any 5-second network hiccup on reconnect — to cover a case that only arises on Windows hosts once #16895 is made to actually work.

@OrcaWin

OrcaWin commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

SSH still works: Orca connects to the host over SSH and routes workspace operations through the managed remote Orca server (orcad). These merges remove the legacy SSH relay, not SSH support.

Closing as obsolete after the merged SSH relay retirement in #27214 and #27215. This PR concerns the SSH relay PTY attach, recovery, replay, or lease lifecycle. #27214 removed the desktop relay connection/fallback path; #27215 deleted the SSH relay server, deployment assets, and relay-only tests. SSH hosts now use the managed Orca server (orcad), and unsupported hosts fail explicitly, so this relay-specific work is no longer planned. This closes the proposed relay implementation; it does not claim that any independent local, WSL, or managed-server behavior was fixed.

@OrcaWin OrcaWin closed this Oct 11, 2026
@OrcaWin

OrcaWin commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Thanks @brennanb2025 — the relay itself is gone, but the idea behind this (don't auto-resume an agent when the host's "terminal not found" was never observed as a real exit) still applies to managed hosts and paired runtimes. I've captured it on #13716 with credit to this PR; this one stays closed since its mechanism was relay-specific.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants