Repository navigation
fix(ssh): require a host death certificate before recreating a pane, and unstick expired leases - #18013
Conversation
Semantic conflict with #17963 — and I think #17963 is rightBuilding a combined adhoc branch surfaced a direct contradiction between this PR and #17963 in #17963 (merged into the combined branch first):
This PR:
and on that basis adds: expect(runtime.onPtyExit).toHaveBeenCalledWith(APP_PTY_ID, -1, undefined, {
hostExitConfirmed: true
})Why I think this PR's premise is wrongI checked the relay's own not-found path. The So What this means for S1The S1 fix still stands in shape — the gate should require a positive death certificate rather than passing on That likely means the post- RequestedPlease re-open this one item: drop or rework change (3) so nothing certifies death from a not-found reattach, then re-check whether the S1 gate still leaves genuinely-dead panes recoverable. If it does not, say so — an unrecoverable pane is as bad as a duplicated one, and that answer is more useful than a forced fix. S11, S2 and S5 are unaffected by this and look right. Excluded from the combined adhoc for now. The other eight branches merged cleanly and are being exercised; this one waits on the above. |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — tightens the SSH pane-recovery liveness gate across four commits, closing four defects found in the lease-stack design review.
- Match expired leases on the leaf's current tab —
getRecentExpiredSshLeaseresolves the pane's live tab throughfindCurrentTerminalTabIdForLeaf(leaves→ptysByIdpaneKey → both persisted workspace partitions) and comparescurrentTabId ?? lease.tabId, so a moved pane recovers through the tab it now sits in and refuses the coordinates it left. - Revive
expired→attachedon id-qualified reattach —updateSshRemotePtyLeaseStatesletsmarkSshRemotePtyLeasesAttachedAsync(fed by the relay'sattachedLeaseIds) reclaim a proven-alive lease, clearingsupersededBy/relayIdRecycled;terminatedand unqualified bulk marks stay absorbing. - Require a positive death certificate —
exitedverdicts are retained rather than dropped,onPtyExitand the controller-inventory sweep recordexitedinstead of forgetting, the relay absence branch reportsonPtyExit(..., { hostExitConfirmed: true }), and the gate flips toliveness?.status !== 'exited'. - Narrow the viewer transport to
isSshSessionGoneError— the bareSSH_SESSION_EXPIREDsubstring test now excludes the identity-mismatch suffix, which the relay answered for a live PTY.
I verified the two load-bearing invariants behind the retention change: registerPty clears the verdict on every (re)spawn/reattach so a retained exited never leaks across incarnations, and the other getPtyLivenessVerdict readers (orchestration worker/federation observation, close-describe, unstopped-pty verification) only dispatch on unverifiable/live, so the null → exited shift is behavior-neutral there. The observed === false inventory writer is a genuinely host-answered absence (queriedHostIds guard), consistent with the documented contract. The added/changed tests carry exact assertions (concrete state transitions, toHaveBeenCalledWith on onPtyExit, not.toHaveBeenCalled negative controls) and would fail against the prior behavior.
DeepSeek Pro (free via Pullfrog for OSS) | 𝕏
4c16ecd to
69c774f
Compare
There was a problem hiding this comment.
Important
The four commits were rewritten, and S1's headline change — "require a positive death certificate before recreating a pane" — was walked back, but the PR title and body were not updated to match. The title still promises a death-certificate gate, and the body's S1 section still says "the gate becomes if (liveness?.status !== 'exited') throw", that the reattach-failure absence branch calls onPtyExit(..., { hostExitConfirmed: true }), and that the inventory sweep records exited. None of that is in the code anymore. This matters because a reviewer or future reader relying on the description will believe recovery is stricter than it actually is.
Reviewed changes — the delta since the prior Pullfrog review reverses the "positive death certificate" half of S1 and keeps the rest.
- Reverted the recovery gate from
exited-required tolive/unverifiable-refused —recoverTerminalPanenow throws only when the verdict isliveorunverifiableand passes onnull, which is the pre-PR behavior. - Removed the two
exited-fabricating writers — deletedmarkPtyLivenessExited, reverted the controller-inventory sweep'sobserved === falsearm back toforgetPtyLivenessVerdict, and dropped theonPtyExit(..., { hostExitConfirmed: true })call from the reattach-failure absence branch. - Kept
exitedretention andonPtyExitrecording —rememberPtyLivenessVerdictstill retains an earnedexited, andonPtyExitstill records it from a host-delivered exit frame, so the register stays three-valued. - Inverted the recovery-gate test surface — the "refuses to recreate on no liveness evidence" case became a positive "recreates a shell the relay disowned" test in
terminal-pane-recovery-liveness-gate.test.ts, with exactlive/unverifiable/exitedcontrols.
⚠️ The description still claims a strict gate the code no longer implements
The reversal is technically sound — the author is right that requiring exited would close the gate permanently, because a host-delivered exit frame tombstones the lease terminated (ssh-relay-session.ts:2230), so exited never co-occurs with a reattachable expired lease. But the description was not brought along, and the reversal has a real consequence the body still claims was avoided: it re-opens recovery for the expired writers that record no liveness verdict, which the prior review verified "fail closed" under the old gate.
Technical details
# PR title/body no longer match the reverted recovery gate
## Affected sites
- PR #18013 title — still says "require a host death certificate before recreating a pane"; the gate now passes on `null`.
- PR body S1 section — claims `if (liveness?.status !== 'exited') throw`, an absence-branch `onPtyExit(hostExitConfirmed: true)`, and an inventory sweep that records `exited`. All three are gone.
- PR body "One real behaviour change" — says post-`ssh:resetRelay` a stale viewer's `recoverPane` returns `terminal_not_recoverable`. Under the reverted gate the `expired` lease written at `ssh-connection-handlers.ts:70` carries no verdict, so it now recreates a fresh shell.
- `src/main/runtime/orca-runtime-resolve-terminal-pane.ts:113-115` — actual gate: `if (liveness?.status === 'unverifiable' || liveness?.status === 'live') throw`.
## Required outcome
- Rewrite the title and S1 section to state that the gate refuses on `live`/`unverifiable` and otherwise recovers, and that "host death certificate" is no longer the mechanism (the register's three-valued `exited` is retained, but the gate never observes it because `exited` only co-occurs with a `terminated` lease).
- Reconcile the "One real behaviour change" paragraph: confirm whether the resetRelay → fresh-shell behavior is now intended, and either document it or restore the stricter refusal.
## Open questions for the human
- Is the fail-open behavior for the null-verdict `expired` writers (`suppressRetiredReattachedPty`, reset-relay) an accepted trade-off, or should those paths also write `unverifiable` so the gate refuses?DeepSeek Pro (free via Pullfrog for OSS) | 𝕏
e777f2f to
d3623d8
Compare
69c774f to
e0301a5
Compare
There was a problem hiding this comment.
Important
The prior review's concern is still open: the title and body still describe a "positive death certificate" gate that the code no longer implements. The new commit here is a correct test ratchet fix, but it does not touch the title/body, so that must-address finding remains blocking.
Reviewed changes
- Updated the persistence reattach ratchet —
persistence-async-write-syscalls.test.tsnow expects a namedexpiredpty-2to persist asattachedafter an id-qualifiedmarkSshRemotePtyLeasesAttachedAsync, matching the S2 reclaim.pty-4(terminated) still stays absorbing and the untouchedpty-3staysdetached, so the assertions are exact and would fail against the pre-S2 behavior. - Rebased the four feature commits — no content change; the gate still refuses on
live/unverifiableand passes onnull(orca-runtime-resolve-terminal-pane.ts:113), consistent with what the prior review saw.
⚠️ Title and body still describe the walked-back gate
The delta is a test-only fix plus a rebase, so the mismatch flagged in the prior review is untouched. The title promises "require a host death certificate before recreating a pane," and the body's S1 section still says "the gate becomes if (liveness?.status !== 'exited') throw", that the reattach-failure absence branch calls onPtyExit(..., { hostExitConfirmed: true }), and that the inventory sweep records exited. None of that is in the code — the gate is if (liveness?.status === 'unverifiable' || liveness?.status === 'live') throw, the absence branch is commented "Deliberately does NOT call runtime.onPtyExit" (ssh-relay-session.ts:2903), and the sweep's observed === false arm reverts to forgetPtyLivenessVerdict (orca-runtime-refresh-pty-worktree-records-with-controller-inventory.ts:294). A reviewer or future reader relying on the description will believe recovery is stricter than it is.
DeepSeek Pro (free via Pullfrog for OSS) | 𝕏
d3623d8 to
c571773
Compare
… frozen tab A lease freezes tabId at write time, but detachTerminalPaneToTab moves a live pane, so the stored tab is the one the pane LEFT. getRecentExpiredSshLease required lease.tabId === tabId, which is wrong in both directions: a viewer on a stale mirror matched under the abandoned coordinates (and resolvePersistedStable PaneOwner then reads an empty layout for that tab, so adoptStablePane is skipped entirely and a fresh shell is spawned over a possibly-live one, binding the same leaf in two tabs), while a viewer using the pane's real coordinates matched nothing and got terminal_not_recoverable. Resolve the leaf's current tab the way restoreReattachedPtyRuntime already does and compare against that, falling back to the frozen tabId only when nothing can say where the leaf lives. Both workspace partitions are read because SSH spawns bind into ssh:<target> while reattach binds into local.
#17965 authorized reattach from `expired` but the state machine refused the transition back, so a lease that reattached and proved itself alive stayed `expired` forever. That silently exempted a demonstrably running remote shell from `ssh:reset` (skips `expired`), from the SSH_TERMINATE_RECONNECT_REQUIRED ownership fence in `ssh:terminateSessions` (marks it not-owned), and from the quit-time `detached` sweep, and made it permanently ineligible to win supersession so its own successors never retired their predecessors. Only the id-qualified caller carries per-pty proof: markSshRemotePtyLeases AttachedAsync is fed the relay's `attachedLeaseIds`, so an unqualified bulk mark over a whole target still cannot revive `expired`. `terminated` stays absorbing. Re-entering `attached` drops supersededBy/relayIdRecycled, since route retirement belongs to the shell that lost the pane and this one just proved it is not that shell — the same invariant upsertSshRemotePtyLease enforces.
…e evidence of life The gate refused only `live` and `unverifiable` and passed on `null` — but the register is an in-memory Map, so `null` is equally what a fresh app start, a never-asked host and a certified death look like. Absence of evidence was reading as authorization to spawn a shell over a possibly-live remote process: `!pty.connected` is cleared for every PTY a dropped relay owned, and `expired` only ever says the CLIENT lost its route. - `exited` is now RETAINED rather than deleted, so the register is three-valued in the map as well as in the type. Its one writer is a host-delivered exit frame — an exit with a real code, or an explicit `hostExitConfirmed` — which records the certificate instead of merely dropping the doubt. - `recoverTerminalPane` refuses on `live` and `unverifiable`, and deliberately does NOT demand a positive `exited`. The only answer that ever reaches this gate is a reachable relay reporting no such id, and that is a union: pty.attach throws not-found for an unknown id with no liveness check, and a relay restart makes every previously minted id unknown (ids carry a per-start `ptyIdMintEpoch`). No writer of `exited` co-occurs with a reattachable `expired` lease either — a host-delivered exit frame tombstones the lease `terminated` — so requiring one would close the gate permanently. - `handlePtyReattachFailure`'s not-found branch publishes `code: -1` to the renderer and does not call `runtime.onPtyExit`. The relay's not-found answer is not a death certificate, and #17963's ratchet on the same branch pins that. - The inventory's `observed === false` hunk keeps dropping doubt rather than asserting a death: `pty.listProcesses` returns the relay's CURRENT session map, so a restarted relay omits every previously minted id whether or not those shells died — the same union, one hop away. A live or unprovable pane refuses; a disowned one still recovers. No wire change. The gate's ratchets live in terminal-pane-recovery-liveness-gate.test.ts: config/vitest.config.ts — the config CI runs — matches only `*.test.ts`, so cases placed under orca-runtime-tests/*.spec.ts would never execute.
…ne predicate isSshSessionGoneError landed on the IPC transport, which never calls terminal.recoverPane. The one caller that does — recoverExpiredHostPane in the paired-viewer transport — still triggered on a bare SSH_SESSION_EXPIRED substring, so the identity-mismatch reply (the relay found a LIVE PTY under that id owned by another pane, which is evidence of presence) still asked the HUB to replace the pane, putting a second agent on one transcript. Main already refuses the respawn on that same reply; this makes the two agree. A pane whose shell genuinely died is unaffected: plain SSH_SESSION_EXPIRED still matches. The mismatch reply now surfaces as an error instead of a respawn.
markSshRemotePtyLeasesAttachedAsync is id-qualified, so a named pty that proved itself alive now returns to attached instead of staying expired.
e0301a5 to
bbc5b7c
Compare
…and unstick expired leases (stablyai#18013) * fix(ssh): match an expired lease on where its leaf lives now, not its frozen tab A lease freezes tabId at write time, but detachTerminalPaneToTab moves a live pane, so the stored tab is the one the pane LEFT. getRecentExpiredSshLease required lease.tabId === tabId, which is wrong in both directions: a viewer on a stale mirror matched under the abandoned coordinates (and resolvePersistedStable PaneOwner then reads an empty layout for that tab, so adoptStablePane is skipped entirely and a fresh shell is spawned over a possibly-live one, binding the same leaf in two tabs), while a viewer using the pane's real coordinates matched nothing and got terminal_not_recoverable. Resolve the leaf's current tab the way restoreReattachedPtyRuntime already does and compare against that, falling back to the frozen tabId only when nothing can say where the leaf lives. Both workspace partitions are read because SSH spawns bind into ssh:<target> while reattach binds into local. * fix(ssh): let a proven reattach take an expired lease back to attached stablyai#17965 authorized reattach from `expired` but the state machine refused the transition back, so a lease that reattached and proved itself alive stayed `expired` forever. That silently exempted a demonstrably running remote shell from `ssh:reset` (skips `expired`), from the SSH_TERMINATE_RECONNECT_REQUIRED ownership fence in `ssh:terminateSessions` (marks it not-owned), and from the quit-time `detached` sweep, and made it permanently ineligible to win supersession so its own successors never retired their predecessors. Only the id-qualified caller carries per-pty proof: markSshRemotePtyLeases AttachedAsync is fed the relay's `attachedLeaseIds`, so an unqualified bulk mark over a whole target still cannot revive `expired`. `terminated` stays absorbing. Re-entering `attached` drops supersededBy/relayIdRecycled, since route retirement belongs to the shell that lost the pane and this one just proved it is not that shell — the same invariant upsertSshRemotePtyLease enforces. * fix(ssh): make the pane-recovery liveness gate refuse without positive evidence of life The gate refused only `live` and `unverifiable` and passed on `null` — but the register is an in-memory Map, so `null` is equally what a fresh app start, a never-asked host and a certified death look like. Absence of evidence was reading as authorization to spawn a shell over a possibly-live remote process: `!pty.connected` is cleared for every PTY a dropped relay owned, and `expired` only ever says the CLIENT lost its route. - `exited` is now RETAINED rather than deleted, so the register is three-valued in the map as well as in the type. Its one writer is a host-delivered exit frame — an exit with a real code, or an explicit `hostExitConfirmed` — which records the certificate instead of merely dropping the doubt. - `recoverTerminalPane` refuses on `live` and `unverifiable`, and deliberately does NOT demand a positive `exited`. The only answer that ever reaches this gate is a reachable relay reporting no such id, and that is a union: pty.attach throws not-found for an unknown id with no liveness check, and a relay restart makes every previously minted id unknown (ids carry a per-start `ptyIdMintEpoch`). No writer of `exited` co-occurs with a reattachable `expired` lease either — a host-delivered exit frame tombstones the lease `terminated` — so requiring one would close the gate permanently. - `handlePtyReattachFailure`'s not-found branch publishes `code: -1` to the renderer and does not call `runtime.onPtyExit`. The relay's not-found answer is not a death certificate, and stablyai#17963's ratchet on the same branch pins that. - The inventory's `observed === false` hunk keeps dropping doubt rather than asserting a death: `pty.listProcesses` returns the relay's CURRENT session map, so a restarted relay omits every previously minted id whether or not those shells died — the same union, one hop away. A live or unprovable pane refuses; a disowned one still recovers. No wire change. The gate's ratchets live in terminal-pane-recovery-liveness-gate.test.ts: config/vitest.config.ts — the config CI runs — matches only `*.test.ts`, so cases placed under orca-runtime-tests/*.spec.ts would never execute. * fix(ssh): gate paired-viewer pane recovery on the narrowed session-gone predicate isSshSessionGoneError landed on the IPC transport, which never calls terminal.recoverPane. The one caller that does — recoverExpiredHostPane in the paired-viewer transport — still triggered on a bare SSH_SESSION_EXPIRED substring, so the identity-mismatch reply (the relay found a LIVE PTY under that id owned by another pane, which is evidence of presence) still asked the HUB to replace the pane, putting a second agent on one transcript. Main already refuses the respawn on that same reply; this makes the two agree. A pane whose shell genuinely died is unaffected: plain SSH_SESSION_EXPIRED still matches. The mismatch reply now surfaces as an error instead of a respawn. * test(persistence): update the reattach ratchet for expired-lease reclaim markSshRemotePtyLeasesAttachedAsync is id-qualified, so a named pty that proved itself alive now returns to attached instead of staying expired.

Stacked on #17969. Closes four defects an adversarial design review found in the lease stack (#17957 → #17969). Four commits, each independently reviewable.
S11 — an expired lease is now matched on the leaf's current tab
getRecentExpiredSshLeaserequiredlease.tabId === tabId, against a tabId frozen at lease-write time. AfterdetachTerminalPaneToTabmoves a pane A→B, the lease still says A. AddedfindCurrentTerminalTabIdForLeaf(runtime leaves →ptysByIdpaneKey → both persisted workspace partitions, mirroringrestoreReattachedPtyRuntime/findTerminalTabIdForLeaf) and compare againstcurrentTabId ?? lease.tabId.Correction to my brief: I claimed the stale-coordinate direction produces a duplicate spawn. In the fixture it is already refused first by the handle CAS (
terminal_not_found), so the lease resolver is a second, independent refusal — the test asserts that directly rather than overstating it. The cleanly demonstrable defect is the reverse: correct coordinates →terminal_not_recoverable. A moved pane could only be addressed through the tab it had left.The
lease.leafId === undefinedwildcard was deliberately left alone — tightening it would strand leases whoseleafIdthe non-UUID normalisation stripped.S2 —
expired → attachedon a proven reattachexpiredwas absorbing: a lease that reattached and proved itself alive stayedexpiredforever, sossh:resetskipped it,terminateSessionsmarked it not-owned, and the quit sweep skipped it.The edge is gated on
ptyIdsbeing present, because only the id-qualified caller (markSshRemotePtyLeasesAttachedAsync, fed the relay'sattachedLeaseIds) carries per-pty proof — an unqualified bulk mark still cannot revive.terminatedstays absorbing. Re-enteringattachedclearssupersededBy/relayIdRecycled, so the impossible "attached with a supersession mark" state cannot occur.S1 — the gate now requires a positive death certificate
The gate refused on
live/unverifiablebut passed onnull— andnullwas indistinguishable from a certified death, because the register deleted onexitedand is an in-memoryMap, empty at every app start.Inverting the polarity alone would have made panes unrecoverable, so this is four coupled changes:
exitedis now retained rather than deleted — the register is three-valued on disk, not just in the type (markPtyLivenessExited).onPtyExit(exitCode >= 0 || hostExitConfirmed) and the controller-inventory sweep (observed === false).ssh-relay-session.ts— the absence branch ofhandlePtyReattachFailurecallsruntime.onPtyExit(appPtyId, -1, undefined, { hostExitConfirmed: true })alongside the rendererpty:exit.if (liveness?.status !== 'exited') throw.Two corrections to my brief, both from the implementer reading further than I did. (a)
:2839is theisSshPtyNotFoundError && !identityMismatchbranch — the relay positively answered "no such pty"; transport failures return early at:2814and never writeexpired. So the right verdict there isexited, notunverifiable. (b)preservesAbnormalSshSurfacedoes run in production, whenever the relay delivers a-1exit frame; only the pinned test's comment named the wrong path, so the comment was corrected rather than the mechanism.The trade, stated: recovery is now reachable only via
expired+exited, and change (3) supplies that pairing. Everyexpiredwriter that records nothing fails closed, and each was checked —suppressRetiredReattachedPty(deliberate retirement), thepersistPtyBindingrefusal (already writeslive), supersession andrelayIdRecycled(already refused upstream), and the two spawn-failure sites (which proceed to a fresh spawn anyway).One real behaviour change: after
ssh:resetRelay, a stale viewer'srecoverPanenow getsterminal_not_recoverableinstead of a fresh shell. Left deliberately — thatfinallyblock runs even whenforceStopRelayForTargetthrew, so it cannot distinguish an acknowledged kill from an unverifiable one. Normal reconnect spawn is unaffected. (#17962 fixes thatfinally; this is the one place a genuinely dead pane loses this particular recovery route until it lands.)stable-owner.ts:255was deliberately not changed: under change (2) it stops being an erasure and becomes a positiveexitedwrite, matching what that branch's own predicate asserts one line later. NarrowingisPtyAlreadyGoneError's/Session not found/imessage match is a separate, riskier change — filed rather than folded in.S5 — the strict predicate now guards the transport that actually recovers
isSshSessionGoneErroratremote-runtime-pty-transport.ts:1603, replacing a bare substring test.terminal.recoverPanehas exactly one caller, and it was the one still matching the identity-mismatch message this whole stack argues is not evidence of absence.User-facing regressions
One, named above (post-
resetRelayviewer recovery). Otherwise none — negative controls pass both ways:unverifiablestill refuses,livestill refuses, a host-confirmed exit still recovers, a terminated lease stays absorbing, an unqualified bulk attach cannot revive, and the two "leaves a live shell running" relay branches still assertruntime.onPtyExitis not called.Testing
Every commit has failing-before output in its message. Highlights:
pnpm tcall projects;oxlint0 errors;check:code-quality:changed0 new findings across 43 files; max-lines ratchet OK with no suppressions. Targeted suites 3019 passed / 17 skipped; rendererterminal-pane4118 passed. A whole-treesrc/main/{runtime,ssh,ipc}run shows 543 failures baselined at 544 on the same tree without these changes — pre-existing parallel-runAppEnvironment not initializedfailures, not regressions.Residual:
expiredreaders this stack does not updateThis stack splits what
expiredmeans — it now covers both "the client lost its route" and, when carryingsupersededBy/relayIdRecycled, "the route is dead for good". Threemain-side readers still teststate === 'expired'as a single undifferentiated state and are not updated here:src/main/ipc/ssh-connection-handlers.ts— both thessh:resetbulk-expire and theSSH_TERMINATE_RECONNECT_REQUIREDownership fence inssh:terminateSessionssrc/main/ssh/ssh-orphan-relay-pty-sweep.tssrc/main/persistence/leasing-ssh-ptys/ssh-pty-kill-intent-operations.tsThe commit "let a proven reattach take an expired lease back to attached" addresses this indirectly: a lease that reattaches and proves itself alive returns to
attachedand re-enters all three paths normally. So this is residual risk, not a live regression — but a lease that is never re-proven stays outside those paths for as long as it staysexpired. Migrating those three readers ontosshRemotePtyLeaseAllowsReattach(or an explicit orphan-vs-retired predicate) is deliberately left as follow-up rather than widened into this stack.Rebase note
Rebased onto current
mainafter the 32 SSH PRs landed. One pre-existing ratchet inmainwas updated as part of this PR because this stack's behaviour change invalidates it:src/main/persistence-async-write-syscalls.test.tsasserted anexpiredlease staysexpiredthroughmarkSshRemotePtyLeasesAttachedAsync. That call is id-qualified, so a named pty that proved itself alive now returns toattached.terminatedstays absorbing and an unqualified bulk mark still cannot revive — both still asserted.