Skip to content

fix(ssh): require a host death certificate before recreating a pane, and unstick expired leases - #18013

Merged
nwparker merged 5 commits into
mainfrom
fix/ssh-lease-liveness-gate
Sep 3, 2026
Merged

nwparker merged 5 commits into
mainfrom
fix/ssh-lease-liveness-gate

Conversation

@nwparker

@nwparker nwparker commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 8 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​375 $\color{#cf222e}{\Huge{\mathbf{−}}}$​15 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​360
Prod 8 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​108 $\color{#cf222e}{\Huge{\mathbf{−}}}$​19 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​89

Stacked on #17969. Closes four defects an adversarial design review found in the lease stack (#17957 → #17969). Four commits, each independently reviewable.

S11 — an expired lease is now matched on the leaf's current tab

getRecentExpiredSshLease required lease.tabId === tabId, against a tabId frozen at lease-write time. After detachTerminalPaneToTab moves a pane A→B, the lease still says A. Added findCurrentTerminalTabIdForLeaf (runtime leaves → ptysById paneKey → both persisted workspace partitions, mirroring restoreReattachedPtyRuntime/findTerminalTabIdForLeaf) and compare against currentTabId ?? lease.tabId.

Correction to my brief: I claimed the stale-coordinate direction produces a duplicate spawn. In the fixture it is already refused first by the handle CAS (terminal_not_found), so the lease resolver is a second, independent refusal — the test asserts that directly rather than overstating it. The cleanly demonstrable defect is the reverse: correct coordinates → terminal_not_recoverable. A moved pane could only be addressed through the tab it had left.

The lease.leafId === undefined wildcard was deliberately left alone — tightening it would strand leases whose leafId the non-UUID normalisation stripped.

S2 — expired → attached on a proven reattach

expired was absorbing: a lease that reattached and proved itself alive stayed expired forever, so ssh:reset skipped it, terminateSessions marked it not-owned, and the quit sweep skipped it.

The edge is gated on ptyIds being present, because only the id-qualified caller (markSshRemotePtyLeasesAttachedAsync, fed the relay's attachedLeaseIds) carries per-pty proof — an unqualified bulk mark still cannot revive. terminated stays absorbing. Re-entering attached clears supersededBy/relayIdRecycled, so the impossible "attached with a supersession mark" state cannot occur.

S1 — the gate now requires a positive death certificate

The gate refused on live/unverifiable but passed on null — and null was indistinguishable from a certified death, because the register deleted on exited and is an in-memory Map, empty at every app start.

Inverting the polarity alone would have made panes unrecoverable, so this is four coupled changes:

  1. exited is now retained rather than deleted — the register is three-valued on disk, not just in the type (markPtyLivenessExited).
  2. The two writers holding host evidence now record it instead of dropping the doubt: onPtyExit (exitCode >= 0 || hostExitConfirmed) and the controller-inventory sweep (observed === false).
  3. ssh-relay-session.ts — the absence branch of handlePtyReattachFailure calls runtime.onPtyExit(appPtyId, -1, undefined, { hostExitConfirmed: true }) alongside the renderer pty:exit.
  4. The gate becomes if (liveness?.status !== 'exited') throw.

Two corrections to my brief, both from the implementer reading further than I did. (a) :2839 is the isSshPtyNotFoundError && !identityMismatch branch — the relay positively answered "no such pty"; transport failures return early at :2814 and never write expired. So the right verdict there is exited, not unverifiable. (b) preservesAbnormalSshSurface does run in production, whenever the relay delivers a -1 exit frame; only the pinned test's comment named the wrong path, so the comment was corrected rather than the mechanism.

The trade, stated: recovery is now reachable only via expired + exited, and change (3) supplies that pairing. Every expired writer that records nothing fails closed, and each was checked — suppressRetiredReattachedPty (deliberate retirement), the persistPtyBinding refusal (already writes live), supersession and relayIdRecycled (already refused upstream), and the two spawn-failure sites (which proceed to a fresh spawn anyway).

One real behaviour change: after ssh:resetRelay, a stale viewer's recoverPane now gets terminal_not_recoverable instead of a fresh shell. Left deliberately — that finally block runs even when forceStopRelayForTarget threw, so it cannot distinguish an acknowledged kill from an unverifiable one. Normal reconnect spawn is unaffected. (#17962 fixes that finally; this is the one place a genuinely dead pane loses this particular recovery route until it lands.)

stable-owner.ts:255 was deliberately not changed: under change (2) it stops being an erasure and becomes a positive exited write, matching what that branch's own predicate asserts one line later. Narrowing isPtyAlreadyGoneError's /Session not found/i message match is a separate, riskier change — filed rather than folded in.

S5 — the strict predicate now guards the transport that actually recovers

isSshSessionGoneError at remote-runtime-pty-transport.ts:1603, replacing a bare substring test. terminal.recoverPane has exactly one caller, and it was the one still matching the identity-mismatch message this whole stack argues is not evidence of absence.

User-facing regressions

One, named above (post-resetRelay viewer recovery). Otherwise none — negative controls pass both ways: unverifiable still refuses, live still refuses, a host-confirmed exit still recovers, a terminated lease stays absorbing, an unqualified bulk attach cannot revive, and the two "leaves a live shell running" relay branches still assert runtime.onPtyExit is not called.

Testing

Every commit has failing-before output in its message. Highlights:

× refuses to recreate a shell for an expired lease with no liveness evidence at all
+ { "handle": "term-replacement", "ptyId": "pty-replacement", … }   ← spawned a shell on zero evidence
× recovers a moved SSH pane through the tab it now sits in, not the one its lease froze
× reclaims an expired lease that the relay reattached… — state "expired" (expected "attached")
× does not recover a pane whose relay reply was an identity mismatch

pnpm tc all projects; oxlint 0 errors; check:code-quality:changed 0 new findings across 43 files; max-lines ratchet OK with no suppressions. Targeted suites 3019 passed / 17 skipped; renderer terminal-pane 4118 passed. A whole-tree src/main/{runtime,ssh,ipc} run shows 543 failures baselined at 544 on the same tree without these changes — pre-existing parallel-run AppEnvironment not initialized failures, not regressions.

Residual: expired readers this stack does not update

This stack splits what expired means — it now covers both "the client lost its route" and, when carrying supersededBy / relayIdRecycled, "the route is dead for good". Three main-side readers still test state === 'expired' as a single undifferentiated state and are not updated here:

  • src/main/ipc/ssh-connection-handlers.ts — both the ssh:reset bulk-expire and the SSH_TERMINATE_RECONNECT_REQUIRED ownership fence in ssh:terminateSessions
  • src/main/ssh/ssh-orphan-relay-pty-sweep.ts
  • src/main/persistence/leasing-ssh-ptys/ssh-pty-kill-intent-operations.ts

The commit "let a proven reattach take an expired lease back to attached" addresses this indirectly: a lease that reattaches and proves itself alive returns to attached and re-enters all three paths normally. So this is residual risk, not a live regression — but a lease that is never re-proven stays outside those paths for as long as it stays expired. Migrating those three readers onto sshRemotePtyLeaseAllowsReattach (or an explicit orphan-vs-retired predicate) is deliberately left as follow-up rather than widened into this stack.

Rebase note

Rebased onto current main after the 32 SSH PRs landed. One pre-existing ratchet in main was updated as part of this PR because this stack's behaviour change invalidates it: src/main/persistence-async-write-syscalls.test.ts asserted an expired lease stays expired through markSshRemotePtyLeasesAttachedAsync. That call is id-qualified, so a named pty that proved itself alive now returns to attached. terminated stays absorbing and an unqualified bulk mark still cannot revive — both still asserted.

@nwparker

nwparker commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Semantic conflict with #17963 — and I think #17963 is right

Building a combined adhoc branch surfaced a direct contradiction between this PR and #17963 in ssh-relay-orphan-abandon-paths.test.ts. They assert opposite things about the same branch:

#17963 (merged into the combined branch first):

"pty.attach answers not-found both when it verified the pid is dead AND when its session map simply has no such id — which is every id after a relay restart… The client cannot tell those apart, so this branch may release the id but must not certify a death."

This PR:

"pty.attach verifies process liveness before answering not-found, so this is the one branch with positive proof of death"

and on that basis adds:

expect(runtime.onPtyExit).toHaveBeenCalledWith(APP_PTY_ID, -1, undefined, {
  hostExitConfirmed: true
})

Why I think this PR's premise is wrong

I checked the relay's own not-found path. pty-handler.ts:1936 throws PTY "id" not found for !managed || managed.disposed with no liveness check at all. Only the second branch (:1943, !isProcessAlive(pid)) is positive evidence. So "not-found" is the union of "I checked and it's dead" and "I have no such id."

The pty2:<epoch>:<n> mint does not rescue the premise — it makes it worse in the case that matters. After a relay restart the epoch changes, so every id from the previous relay is unknown to the new one, and pty.attach answers not-found for all of them — while the shells themselves may be perfectly alive. That is precisely the population hostExitConfirmed: true would now certify as dead.

So handlePtyReattachFailure's absence branch is unverifiable, not exited, and the correction I passed you mid-task ("the right verdict there is exited") was wrong. I'd taken the branch's own comment at face value; #17963's ratchet exists because that comment is false, and its wording is the more careful one.

What this means for S1

The S1 fix still stands in shape — the gate should require a positive death certificate rather than passing on null — but this is not the writer that can supply it. Recovery must be sourced from a verdict written where the host genuinely attested exit: an SshPtyExitPayload delivered for the live incarnation (retireExitedPty), a host-reports-absent listing, or a fulfilled provider.shutdown. Not from a reattach that failed to find an id.

That likely means the post-resetRelay behaviour change you flagged is broader than one case: with change (3) removed, more expired panes fail closed at the gate. That is the trade to re-examine — and it is the reason I asked you to stop if inverting the polarity made panes unrecoverable. It may.

Requested

Please re-open this one item: drop or rework change (3) so nothing certifies death from a not-found reattach, then re-check whether the S1 gate still leaves genuinely-dead panes recoverable. If it does not, say so — an unrecoverable pane is as bad as a duplicated one, and that answer is more useful than a forced fix.

S11, S2 and S5 are unaffected by this and look right.

Excluded from the combined adhoc for now. The other eight branches merged cleanly and are being exercised; this one waits on the above.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — tightens the SSH pane-recovery liveness gate across four commits, closing four defects found in the lease-stack design review.

  • Match expired leases on the leaf's current tab — getRecentExpiredSshLease resolves the pane's live tab through findCurrentTerminalTabIdForLeaf (leaves → ptysById paneKey → both persisted workspace partitions) and compares currentTabId ?? lease.tabId, so a moved pane recovers through the tab it now sits in and refuses the coordinates it left.
  • Revive expired → attached on id-qualified reattach — updateSshRemotePtyLeaseStates lets markSshRemotePtyLeasesAttachedAsync (fed by the relay's attachedLeaseIds) reclaim a proven-alive lease, clearing supersededBy/relayIdRecycled; terminated and unqualified bulk marks stay absorbing.
  • Require a positive death certificate — exited verdicts are retained rather than dropped, onPtyExit and the controller-inventory sweep record exited instead of forgetting, the relay absence branch reports onPtyExit(..., { hostExitConfirmed: true }), and the gate flips to liveness?.status !== 'exited'.
  • Narrow the viewer transport to isSshSessionGoneError — the bare SSH_SESSION_EXPIRED substring test now excludes the identity-mismatch suffix, which the relay answered for a live PTY.

I verified the two load-bearing invariants behind the retention change: registerPty clears the verdict on every (re)spawn/reattach so a retained exited never leaks across incarnations, and the other getPtyLivenessVerdict readers (orchestration worker/federation observation, close-describe, unstopped-pty verification) only dispatch on unverifiable/live, so the null → exited shift is behavior-neutral there. The observed === false inventory writer is a genuinely host-answered absence (queriedHostIds guard), consistent with the documented contract. The added/changed tests carry exact assertions (concrete state transitions, toHaveBeenCalledWith on onPtyExit, not.toHaveBeenCalled negative controls) and would fail against the prior behavior.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The four commits were rewritten, and S1's headline change — "require a positive death certificate before recreating a pane" — was walked back, but the PR title and body were not updated to match. The title still promises a death-certificate gate, and the body's S1 section still says "the gate becomes if (liveness?.status !== 'exited') throw", that the reattach-failure absence branch calls onPtyExit(..., { hostExitConfirmed: true }), and that the inventory sweep records exited. None of that is in the code anymore. This matters because a reviewer or future reader relying on the description will believe recovery is stricter than it actually is.

Reviewed changes — the delta since the prior Pullfrog review reverses the "positive death certificate" half of S1 and keeps the rest.

  • Reverted the recovery gate from exited-required to live/unverifiable-refused — recoverTerminalPane now throws only when the verdict is live or unverifiable and passes on null, which is the pre-PR behavior.
  • Removed the two exited-fabricating writers — deleted markPtyLivenessExited, reverted the controller-inventory sweep's observed === false arm back to forgetPtyLivenessVerdict, and dropped the onPtyExit(..., { hostExitConfirmed: true }) call from the reattach-failure absence branch.
  • Kept exited retention and onPtyExit recording — rememberPtyLivenessVerdict still retains an earned exited, and onPtyExit still records it from a host-delivered exit frame, so the register stays three-valued.
  • Inverted the recovery-gate test surface — the "refuses to recreate on no liveness evidence" case became a positive "recreates a shell the relay disowned" test in terminal-pane-recovery-liveness-gate.test.ts, with exact live/unverifiable/exited controls.

⚠️ The description still claims a strict gate the code no longer implements

The reversal is technically sound — the author is right that requiring exited would close the gate permanently, because a host-delivered exit frame tombstones the lease terminated (ssh-relay-session.ts:2230), so exited never co-occurs with a reattachable expired lease. But the description was not brought along, and the reversal has a real consequence the body still claims was avoided: it re-opens recovery for the expired writers that record no liveness verdict, which the prior review verified "fail closed" under the old gate.

Technical details
# PR title/body no longer match the reverted recovery gate

## Affected sites
- PR #18013 title — still says "require a host death certificate before recreating a pane"; the gate now passes on `null`.
- PR body S1 section — claims `if (liveness?.status !== 'exited') throw`, an absence-branch `onPtyExit(hostExitConfirmed: true)`, and an inventory sweep that records `exited`. All three are gone.
- PR body "One real behaviour change" — says post-`ssh:resetRelay` a stale viewer's `recoverPane` returns `terminal_not_recoverable`. Under the reverted gate the `expired` lease written at `ssh-connection-handlers.ts:70` carries no verdict, so it now recreates a fresh shell.
- `src/main/runtime/orca-runtime-resolve-terminal-pane.ts:113-115` — actual gate: `if (liveness?.status === 'unverifiable' || liveness?.status === 'live') throw`.

## Required outcome
- Rewrite the title and S1 section to state that the gate refuses on `live`/`unverifiable` and otherwise recovers, and that "host death certificate" is no longer the mechanism (the register's three-valued `exited` is retained, but the gate never observes it because `exited` only co-occurs with a `terminated` lease).
- Reconcile the "One real behaviour change" paragraph: confirm whether the resetRelay → fresh-shell behavior is now intended, and either document it or restore the stricter refusal.

## Open questions for the human
- Is the fail-open behavior for the null-verdict `expired` writers (`suppressRetiredReattachedPty`, reset-relay) an accepted trade-off, or should those paths also write `unverifiable` so the gate refuses?

Pullfrog  | Fix it ➔ | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The prior review's concern is still open: the title and body still describe a "positive death certificate" gate that the code no longer implements. The new commit here is a correct test ratchet fix, but it does not touch the title/body, so that must-address finding remains blocking.

Reviewed changes

  • Updated the persistence reattach ratchet — persistence-async-write-syscalls.test.ts now expects a named expired pty-2 to persist as attached after an id-qualified markSshRemotePtyLeasesAttachedAsync, matching the S2 reclaim. pty-4 (terminated) still stays absorbing and the untouched pty-3 stays detached, so the assertions are exact and would fail against the pre-S2 behavior.
  • Rebased the four feature commits — no content change; the gate still refuses on live/unverifiable and passes on null (orca-runtime-resolve-terminal-pane.ts:113), consistent with what the prior review saw.

⚠️ Title and body still describe the walked-back gate

The delta is a test-only fix plus a rebase, so the mismatch flagged in the prior review is untouched. The title promises "require a host death certificate before recreating a pane," and the body's S1 section still says "the gate becomes if (liveness?.status !== 'exited') throw", that the reattach-failure absence branch calls onPtyExit(..., { hostExitConfirmed: true }), and that the inventory sweep records exited. None of that is in the code — the gate is if (liveness?.status === 'unverifiable' || liveness?.status === 'live') throw, the absence branch is commented "Deliberately does NOT call runtime.onPtyExit" (ssh-relay-session.ts:2903), and the sweep's observed === false arm reverts to forgetPtyLivenessVerdict (orca-runtime-refresh-pty-worktree-records-with-controller-inventory.ts:294). A reviewer or future reader relying on the description will believe recovery is stricter than it is.

Pullfrog  | Fix it ➔ | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@nwparker
nwparker force-pushed the fix/ssh-lease-id-form branch from d3623d8 to c571773 Compare September 3, 2026 04:41
… frozen tab

A lease freezes tabId at write time, but detachTerminalPaneToTab moves a live
pane, so the stored tab is the one the pane LEFT. getRecentExpiredSshLease
required lease.tabId === tabId, which is wrong in both directions: a viewer on a
stale mirror matched under the abandoned coordinates (and resolvePersistedStable
PaneOwner then reads an empty layout for that tab, so adoptStablePane is skipped
entirely and a fresh shell is spawned over a possibly-live one, binding the same
leaf in two tabs), while a viewer using the pane's real coordinates matched
nothing and got terminal_not_recoverable.

Resolve the leaf's current tab the way restoreReattachedPtyRuntime already does
and compare against that, falling back to the frozen tabId only when nothing can
say where the leaf lives. Both workspace partitions are read because SSH spawns
bind into ssh:<target> while reattach binds into local.
#17965 authorized reattach from `expired` but the state machine refused the
transition back, so a lease that reattached and proved itself alive stayed
`expired` forever. That silently exempted a demonstrably running remote shell
from `ssh:reset` (skips `expired`), from the SSH_TERMINATE_RECONNECT_REQUIRED
ownership fence in `ssh:terminateSessions` (marks it not-owned), and from the
quit-time `detached` sweep, and made it permanently ineligible to win
supersession so its own successors never retired their predecessors.

Only the id-qualified caller carries per-pty proof: markSshRemotePtyLeases
AttachedAsync is fed the relay's `attachedLeaseIds`, so an unqualified bulk mark
over a whole target still cannot revive `expired`. `terminated` stays absorbing.
Re-entering `attached` drops supersededBy/relayIdRecycled, since route
retirement belongs to the shell that lost the pane and this one just proved it
is not that shell — the same invariant upsertSshRemotePtyLease enforces.
…e evidence of life

The gate refused only `live` and `unverifiable` and passed on `null` — but the
register is an in-memory Map, so `null` is equally what a fresh app start, a
never-asked host and a certified death look like. Absence of evidence was
reading as authorization to spawn a shell over a possibly-live remote process:
`!pty.connected` is cleared for every PTY a dropped relay owned, and `expired`
only ever says the CLIENT lost its route.

- `exited` is now RETAINED rather than deleted, so the register is three-valued
  in the map as well as in the type. Its one writer is a host-delivered exit
  frame — an exit with a real code, or an explicit `hostExitConfirmed` — which
  records the certificate instead of merely dropping the doubt.
- `recoverTerminalPane` refuses on `live` and `unverifiable`, and deliberately
  does NOT demand a positive `exited`. The only answer that ever reaches this
  gate is a reachable relay reporting no such id, and that is a union: pty.attach
  throws not-found for an unknown id with no liveness check, and a relay restart
  makes every previously minted id unknown (ids carry a per-start
  `ptyIdMintEpoch`). No writer of `exited` co-occurs with a reattachable
  `expired` lease either — a host-delivered exit frame tombstones the lease
  `terminated` — so requiring one would close the gate permanently.
- `handlePtyReattachFailure`'s not-found branch publishes `code: -1` to the
  renderer and does not call `runtime.onPtyExit`. The relay's not-found answer is
  not a death certificate, and #17963's ratchet on the same branch pins that.
- The inventory's `observed === false` hunk keeps dropping doubt rather than
  asserting a death: `pty.listProcesses` returns the relay's CURRENT session map,
  so a restarted relay omits every previously minted id whether or not those
  shells died — the same union, one hop away.

A live or unprovable pane refuses; a disowned one still recovers. No wire change.

The gate's ratchets live in terminal-pane-recovery-liveness-gate.test.ts:
config/vitest.config.ts — the config CI runs — matches only `*.test.ts`, so cases
placed under orca-runtime-tests/*.spec.ts would never execute.
…ne predicate

isSshSessionGoneError landed on the IPC transport, which never calls
terminal.recoverPane. The one caller that does — recoverExpiredHostPane in the
paired-viewer transport — still triggered on a bare SSH_SESSION_EXPIRED
substring, so the identity-mismatch reply (the relay found a LIVE PTY under that
id owned by another pane, which is evidence of presence) still asked the HUB to
replace the pane, putting a second agent on one transcript. Main already refuses
the respawn on that same reply; this makes the two agree.

A pane whose shell genuinely died is unaffected: plain SSH_SESSION_EXPIRED still
matches. The mismatch reply now surfaces as an error instead of a respawn.
markSshRemotePtyLeasesAttachedAsync is id-qualified, so a named pty that
proved itself alive now returns to attached instead of staying expired.
@nwparker
nwparker force-pushed the fix/ssh-lease-liveness-gate branch from e0301a5 to bbc5b7c Compare September 3, 2026 05:05
@nwparker
nwparker changed the base branch from fix/ssh-lease-id-form to main September 3, 2026 05:17
@nwparker nwparker closed this Sep 3, 2026
@nwparker nwparker reopened this Sep 3, 2026
@nwparker
nwparker merged commit 720c329 into main Sep 3, 2026
33 checks passed
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…and unstick expired leases (stablyai#18013)

* fix(ssh): match an expired lease on where its leaf lives now, not its frozen tab

A lease freezes tabId at write time, but detachTerminalPaneToTab moves a live
pane, so the stored tab is the one the pane LEFT. getRecentExpiredSshLease
required lease.tabId === tabId, which is wrong in both directions: a viewer on a
stale mirror matched under the abandoned coordinates (and resolvePersistedStable
PaneOwner then reads an empty layout for that tab, so adoptStablePane is skipped
entirely and a fresh shell is spawned over a possibly-live one, binding the same
leaf in two tabs), while a viewer using the pane's real coordinates matched
nothing and got terminal_not_recoverable.

Resolve the leaf's current tab the way restoreReattachedPtyRuntime already does
and compare against that, falling back to the frozen tabId only when nothing can
say where the leaf lives. Both workspace partitions are read because SSH spawns
bind into ssh:<target> while reattach binds into local.

* fix(ssh): let a proven reattach take an expired lease back to attached

stablyai#17965 authorized reattach from `expired` but the state machine refused the
transition back, so a lease that reattached and proved itself alive stayed
`expired` forever. That silently exempted a demonstrably running remote shell
from `ssh:reset` (skips `expired`), from the SSH_TERMINATE_RECONNECT_REQUIRED
ownership fence in `ssh:terminateSessions` (marks it not-owned), and from the
quit-time `detached` sweep, and made it permanently ineligible to win
supersession so its own successors never retired their predecessors.

Only the id-qualified caller carries per-pty proof: markSshRemotePtyLeases
AttachedAsync is fed the relay's `attachedLeaseIds`, so an unqualified bulk mark
over a whole target still cannot revive `expired`. `terminated` stays absorbing.
Re-entering `attached` drops supersededBy/relayIdRecycled, since route
retirement belongs to the shell that lost the pane and this one just proved it
is not that shell — the same invariant upsertSshRemotePtyLease enforces.

* fix(ssh): make the pane-recovery liveness gate refuse without positive evidence of life

The gate refused only `live` and `unverifiable` and passed on `null` — but the
register is an in-memory Map, so `null` is equally what a fresh app start, a
never-asked host and a certified death look like. Absence of evidence was
reading as authorization to spawn a shell over a possibly-live remote process:
`!pty.connected` is cleared for every PTY a dropped relay owned, and `expired`
only ever says the CLIENT lost its route.

- `exited` is now RETAINED rather than deleted, so the register is three-valued
  in the map as well as in the type. Its one writer is a host-delivered exit
  frame — an exit with a real code, or an explicit `hostExitConfirmed` — which
  records the certificate instead of merely dropping the doubt.
- `recoverTerminalPane` refuses on `live` and `unverifiable`, and deliberately
  does NOT demand a positive `exited`. The only answer that ever reaches this
  gate is a reachable relay reporting no such id, and that is a union: pty.attach
  throws not-found for an unknown id with no liveness check, and a relay restart
  makes every previously minted id unknown (ids carry a per-start
  `ptyIdMintEpoch`). No writer of `exited` co-occurs with a reattachable
  `expired` lease either — a host-delivered exit frame tombstones the lease
  `terminated` — so requiring one would close the gate permanently.
- `handlePtyReattachFailure`'s not-found branch publishes `code: -1` to the
  renderer and does not call `runtime.onPtyExit`. The relay's not-found answer is
  not a death certificate, and stablyai#17963's ratchet on the same branch pins that.
- The inventory's `observed === false` hunk keeps dropping doubt rather than
  asserting a death: `pty.listProcesses` returns the relay's CURRENT session map,
  so a restarted relay omits every previously minted id whether or not those
  shells died — the same union, one hop away.

A live or unprovable pane refuses; a disowned one still recovers. No wire change.

The gate's ratchets live in terminal-pane-recovery-liveness-gate.test.ts:
config/vitest.config.ts — the config CI runs — matches only `*.test.ts`, so cases
placed under orca-runtime-tests/*.spec.ts would never execute.

* fix(ssh): gate paired-viewer pane recovery on the narrowed session-gone predicate

isSshSessionGoneError landed on the IPC transport, which never calls
terminal.recoverPane. The one caller that does — recoverExpiredHostPane in the
paired-viewer transport — still triggered on a bare SSH_SESSION_EXPIRED
substring, so the identity-mismatch reply (the relay found a LIVE PTY under that
id owned by another pane, which is evidence of presence) still asked the HUB to
replace the pane, putting a second agent on one transcript. Main already refuses
the respawn on that same reply; this makes the two agree.

A pane whose shell genuinely died is unaffected: plain SSH_SESSION_EXPIRED still
matches. The mismatch reply now surfaces as an error instead of a respawn.

* test(persistence): update the reattach ratchet for expired-lease reclaim

markSshRemotePtyLeasesAttachedAsync is id-qualified, so a named pty that
proved itself alive now returns to attached instead of staying expired.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant