Skip to content

fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn - #16763

Merged
nwparker merged 2 commits into
mainfrom
nwparker/ssh-07-absent-from-relay
Aug 28, 2026
Merged

nwparker merged 2 commits into
mainfrom
nwparker/ssh-07-absent-from-relay

Conversation

@nwparker

@nwparker nwparker commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 2 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​243 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​243
Prod 3 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​44 $\color{#cf222e}{\Huge{\mathbf{−}}}$​6 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​38

7 of 7 in the SSH stack, on top of #16752 (#16746 -> #16747 -> #16748 -> #16750 -> #16751 -> #16752 -> this). The diff below is only this change.

ELI5

If the relay program on the remote machine dies and restarts, it forgets its shells and numbers new ones from 1 again.

So Orca asks for "pty-1", and the fresh relay honestly answers "I don't have a pty-1."

Orca already knows how to handle that — throw away the stale link and start a new shell. But the message got flattened into a generic "session expired" error on the way, so the code that checks for "gone" couldn't recognise it, and the recovery never ran. You got a terminal that refused to open.

The fix: keep the specific answer instead of flattening it, so the recovery that already exists can fire.

Why this is safe: the app only treats a shell as gone when the relay is reachable and says so — and in one case only after the relay has checked the operating system and confirmed the process is dead. A dropped connection or a timeout never counts, because "I couldn't reach it" is not "it's gone." That distinction was tested by running the real check against every error the connection layer can produce.

It can never kill anything — worst case is a shell left running unattached, which was already possible before this change through a different route.


The bug

A relay renumbers its PTYs from pty-1 on every start. After the relay daemon dies and restarts, a pane still bound to pty-1 takes the attach path against a PTY that no longer exists, and pty:spawn rejects outright:

[ssh-pty] spawn() called with sessionId=ssh:…@@pty-1, attempting pty.attach
[ssh-pty] pty.attach FAILED for ssh:…@@pty-1: Error: PTY "pty-1" not found
Error occurred in handler for 'pty:spawn': Error: SSH_SESSION_EXPIRED: pty-1

The recovery machinery already existed and was already correct. attachStablePaneOwner (src/main/ipc/pty/pane/stable-owner.ts:242) has exactly the right fallback — retire the pane binding, runtime.onPtyExit, then let spawnForStablePane spawn fresh — and it is gated on isPtyAlreadyGoneError.

The defect is one layer up. reattachSshPtySession (src/main/providers/ssh-pty-session-reattach.ts:227-231) flattened the relay's PTY "pty-1" not found into a bare Error("SSH_SESSION_EXPIRED: pty-1"), discarding the evidence. isPtyAlreadyGoneError classifies on that message via /PTY ".+" not found/, so it could never match, and the error propagated to the renderer instead of falling back. Observed call path:

reattachSshPtySession → reattachSshPtySessionForSpawn → SshPtyProvider.spawn
 → attachStablePaneOwner → adoptStablePane → preparePtyIpcSpawnPreflight → runPtyIpcSpawn

This also explains why an earlier attempt in kill.ts changed nothing: wrong path entirely.

The fix

SshPtyAbsentFromRelayError carries the verdict the message cannot — a reachable relay answered for this exact id and reported it absent. Per docs/reference/ssh-execution-boundary.md that is exited, not unverifiable. isPtyAlreadyGoneError also accepts the class.

Why this is narrow, not just plausible

Negative space. pty.attach is src/relay/pty-handler.ts:1763, and only three throws are inside it:

site meaning classified absent?
:1776 no record, or already disposed yes
:1789 relay called isProcessAlive(pid), found it dead, marked the physical exit, notified its exit listener, disposed the managed PTY and removed it from its table — then threw yes
:1801 identity mismatch: the id names a live PTY owned by another pane no

:1789 is the strongest form of the argument: the relay is not reporting ignorance, it has verified death and acted on it. "Not found" reads weaker than it is. The other four not found throws (:958, :1942, :2042, :2051) are in different methods and unreachable from this path.

Loss of contact cannot be mistaken for absence. Every string ssh-channel-multiplexer.ts can synthesise, run against the real /PTY ".+" not found/i predicate:

synthesised at string matches?
:53 SSH connection lost, reconnecting... no
:53 Multiplexer disposed no
:72 Request "<m>" timed out after <n>ms no
:486 Method not found: <m> no
:229, :261 Request "<m>" was cancelled no
:607 Connection timed out (no ack received) no

The sole place an Error is built from a relay-supplied payload is handleResponse (:520), so only the two genuine relay strings can ever reach the predicate. restoreRequired (the PTY is live, only its source stream is not) keeps its plain Error and is never classified absent.

Blast radius is one call site. isPtyAlreadyGoneError's other consumers — kill.ts:66, kill.ts:266, inspect.ts:91 — all handle pty.shutdown / pty.inspectProcess errors. The new class is thrown only from reattachSshPtySession, so none of them can ever see it. stable-owner.ts:242 is the only behaviour that changes.

Both thrown error.message values are byte-identical to the previous ones. The old code built SSH_SESSION_EXPIRED: <id>${mismatchMarker} where the marker was ' SSH_PTY_IDENTITY_MISMATCH' or ''; the two branches produce exactly those strings.

The string that crosses IPC does change, and that is the more important half. Electron serializes the stack, whose first line is ${name}: ${message}, so the renderer now sees SshPtyAbsentFromRelayError: SSH_SESSION_EXPIRED: pty-1 where it previously saw Error: SSH_SESSION_EXPIRED: pty-1. That is visible in this PR's own e2e observable below. Every consumer was therefore checked against the changed string, and none anchors on Error: or on a line start — all six match on substring or on a regex with no start anchor. That is why it is safe, not an incidental detail:

  • src/main/ipc/pty/ipc/spawn-execute.ts:145-146
  • src/main/ipc/pty/runtime/spawn-execute.ts:225-226 (the paired-runtime path — easy to miss, there are two)
  • TerminalErrorToast.tsx:32, the fussiest consumer: SSH_SESSION_EXPIRED:[ \t]*\S*(?:[ \t]+SSH_PTY_IDENTITY_MISMATCH)?
  • ipc-pty-connect.ts:124, remote-runtime-pty-transport.ts:1526, pty-connection/ssh-session-connect.ts:15

This is parity with shipped local behaviour, not a new policy. The local daemon already reaches this exact fallback: SessionNotFoundError carries the message Session not found: <id> (daemon-errors.ts:59), which isPtyAlreadyGoneError already matches via /Session not found/i (liveness.ts:65). Local panes have been retiring their binding and respawning on positive absence for as long as that predicate has existed. This PR gives SSH the same treatment for the same class of evidence.

Can this abandon or duplicate a LIVE remote session?

It can never kill one — the fallback issues no shutdown or kill, only client-side bookkeeping plus a spawn. The failure mode is orphaning, never termination.

The one real risk, stated plainly: the relay is authoritative only over its own id space. If a daemon is abnormally replaced while the old daemon's PTYs survive, the new daemon truthfully answers "pty-1 not found" while a live shell exists under the old one, and we retire the binding and spawn fresh — orphaning it. This is pre-existing: the renderer already did exactly this on SSH_SESSION_EXPIRED (deferred-session-reattach-connect.ts:144-150); this PR moves the same decision into main for the stable-pane path. Bounded to one extra shell per pane per relay restart.

I previously called this unavoidable. It is not, and the PR should not claim it is. Abnormal replacement is one line away: src/relay/relay-daemon.ts:23-27 handles uncaughtException with relayLogLine + socketOwnership.cleanup() + process.exit(1) and no PTY disposal, so any uncaught throw in the relay orphans every PTY it owned. And the client is not actually blind here — relay.status already returns pid and uptimeMs (relay-daemon.ts:130-131), so a daemon demonstrably younger than the binding is detectable. Two non-blocking follow-ups are filed: dispose PTYs in that handler, and decline to re-run launchAgent / resumeProviderSession when the answering daemon is younger than the binding.

The fresh spawn uses the original request's options, so a request carrying command / launchAgent / resumeProviderSession will run it — pre-existing cold-restore behaviour, not introduced here.

Concurrency: attachStablePaneOwner re-resolves the owner before and after retiring and aborts with terminal_pane_owner_changed if anything moved; retirePersistedStablePaneOwner is fenced on the exact ptyId + incarnation. It cannot clobber a pane that re-bound underneath it, and it cannot loop.

Evidence

The e2e oracle cannot discriminate this change in either direction — ssh-lost-kill-tab-resurrection.spec.ts passes at the stack tip with SshPtyAbsentFromRelayError absent from the build. Don't go looking for oracle evidence that cannot exist. The real evidence is below.

Two grep counts, identical across both runs on this branch, carry the e2e evidence. The pass/fail is incidental.

count both runs what it proves
grep -c "Error occurred in handler for 'pty:spawn'" 0 the observable this change controls is gone
grep -c "resurrected the closed tab" 0 any red is a setup timeout, never a correctness failure

Primary e2e observable (deterministic, where the test outcome is not): the presence or absence of

Error occurred in handler for 'pty:spawn': SshPtyAbsentFromRelayError: SSH_SESSION_EXPIRED: pty-1

Disable the new clause and that line returns; enable it and it is gone, with the run otherwise byte-identical. A/B'd on one tree, one build pipeline.

Unit A/B: with the clause disabled, stable-pane-relay-absence-respawn.test.ts fails on exactly one case ("spawns fresh once the relay has positively answered for that id") and the others pass. The clause is load-bearing.

Persistence coverage (added after review). The first cut of that suite passed no store/worktreeId, so attachStablePaneOwner's guard (stable-owner.ts:258-261) short-circuited and the retirement never ran — which matters, because retireTerminalSurfaceFromPersistence deletes the parent tab and its layout when the retired leaf is the last one. retirePersistedStablePaneOwner and attachStablePaneOwner had zero coverage repo-wide. Three tests now drive the fallback against a real in-memory session: a two-leaf tab keeps the tab and the sibling's binding; a one-leaf tab drops from persistence (pinning current behaviour, since the fallback returns owner: null so tab survival then rests on the renderer); and a non-absence failure leaves persistence byte-for-byte untouched. A/B'd by removing worktreeId — the two persistence tests fail, the rest pass, confirming they exercise the path rather than the guard.

Two full oracle runs on this branch, provenance SshPtyAbsentFromRelayError=6 in out/main/index.js and tombstones=13 in the renderer each time. Both runs: test 2 passed all three cycles; test 1 failed at the cycle-2 createRemoteTerminalTab setup timeout. grep -c "resurrected the closed tab" = 0 in both — any red here is a setup timeout, never a correctness failure.

Test 1's red is a separate, pre-existing transport defect that this PR does not touch and did not introduce (own ticket). Its discriminator, in one line:

At every reconnect, [ssh-relay] Socket probe result: reads "DEAD" in test 1 and "ALIVE" in test 2. Whenever the client must launch a new relay daemon, the SSH transport drops afterwards and does not recover within 60s. (The initial-connect probe is DEAD in both tests and carries no signal.)

It is intermittent, not deterministic: red 5 of 5 in isolated single-spec runs, green in the full 14-spec lane.

Gates: pnpm tc clean · 314 files / 3849 tests passed, 0 failed (providers, ipc/pty, persistence, ssh, tombstones, toast humanizer) · check:code-quality:changed 0 findings across 55 files · oxlint + oxfmt clean · no max-lines disables.

@nwparker
nwparker force-pushed the nwparker/ssh-06-close-tombstone branch 2 times, most recently from 03d73fb to 5ac2552 Compare August 27, 2026 09:53
@nwparker

Copy link
Copy Markdown
Contributor Author

Reviewers: the file count below is temporarily wrong. This PR is 6 files, not 58.

nwparker/ssh-06-close-tombstone was force-pushed to 5ac2552fc67 after this PR opened, and this branch's parent (917f1920eb4) is no longer an ancestor of it, so GitHub's three-dot diff is currently showing the entire stack as this PR's change. The branch content is unchanged and correct.

The actual change is 6 files:

src/main/providers/ssh-pty-errors.ts
src/main/providers/ssh-pty-session-reattach.ts
src/main/ipc/pty/provider/liveness.ts
src/main/providers/ssh-pty-relay-absence-verdict.test.ts
src/main/ipc/pty/pane/stable-pane-relay-absence-respawn.test.ts

(three modified, two new test files) — commits 63062114595 and 99496a94892.

Holding the rebase until the base settles; the diff will snap back to 6 files then. Verified in advance that the rebase is clean: all three modified files, plus src/relay/pty-handler.ts and src/main/ssh/ssh-channel-multiplexer.ts, are byte-identical between the old and new base — so the safety claims about the not-found predicate need no re-verification, though I'll re-run the probe after rebasing anyway.

Review P2-1 (persistence coverage for the absence fallback) is addressed in 99496a94892.

reattachSshPtySession flattened the relay's `PTY "<id>" not found` into a bare
`SSH_SESSION_EXPIRED`, discarding the evidence. isPtyAlreadyGoneError classifies
on that message via /PTY ".+" not found/, so it could never match — and
attachStablePaneOwner's existing already-gone fallback (retire the pane binding,
onPtyExit, spawn fresh) is gated on exactly that predicate. The recovery
machinery was already correct; only the classification was broken, which is why
`pty:spawn` rejected outright after a relay restart renumbered its PTYs from
pty-1.

SshPtyAbsentFromRelayError carries the verdict the message cannot: a reachable
relay answered for this exact id and reported it absent. Both thrown messages
stay byte-identical to the previous ones, so all six message-based consumers are
untouched; only callers that can act on the stronger verdict test the class.

Deliberately not raised for a transport failure, timeout, disposed multiplexer,
identity mismatch, or restoreRequired — none of those observe the process, and
treating them as absence would orphan live remote work.
The first cut passed no store/worktreeId, so attachStablePaneOwner's guard
short-circuited and retireTerminalSurfaceFromPersistence — which deletes the
parent tab and its layout when the retired leaf is the last one — never ran.
retirePersistedStablePaneOwner and attachStablePaneOwner had no coverage at all.
@nwparker
nwparker force-pushed the nwparker/ssh-07-absent-from-relay branch from 99496a9 to bd6ff90 Compare August 28, 2026 02:48
@nwparker
nwparker changed the base branch from nwparker/ssh-06-close-tombstone to main August 28, 2026 02:48
@nwparker
nwparker marked this pull request as ready for review August 28, 2026 02:48
@nwparker
nwparker merged commit 25fa6a9 into main Aug 28, 2026
3 of 21 checks passed
@coderabbitai

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9d86175e-c205-4ff7-a700-4741c1513301

📥 Commits

Reviewing files that changed from the base of the PR and between 2b39165 and bd6ff90.

📒 Files selected for processing (5)
  • src/main/ipc/pty/pane/stable-pane-relay-absence-respawn.test.ts
  • src/main/ipc/pty/provider/liveness.ts
  • src/main/providers/ssh-pty-errors.ts
  • src/main/providers/ssh-pty-relay-absence-verdict.test.ts
  • src/main/providers/ssh-pty-session-reattach.ts

📝 Walkthrough

Walkthrough

The change adds SshPtyAbsentFromRelayError and its type guard. Reattach logic now distinguishes confirmed relay absence from identity mismatch errors. PTY liveness detection recognizes the new error as an already-gone condition. Tests cover provider verdicts, stable-pane respawning, PTY binding cleanup, sibling preservation, final-tab removal, and unchanged persistence for non-absence failures.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

paidaxingyo666 pushed a commit to paidaxingyo666/Manta that referenced this pull request Aug 30, 2026
…awn (stablyai#16763)

* fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn

reattachSshPtySession flattened the relay's `PTY "<id>" not found` into a bare
`SSH_SESSION_EXPIRED`, discarding the evidence. isPtyAlreadyGoneError classifies
on that message via /PTY ".+" not found/, so it could never match — and
attachStablePaneOwner's existing already-gone fallback (retire the pane binding,
onPtyExit, spawn fresh) is gated on exactly that predicate. The recovery
machinery was already correct; only the classification was broken, which is why
`pty:spawn` rejected outright after a relay restart renumbered its PTYs from
pty-1.

SshPtyAbsentFromRelayError carries the verdict the message cannot: a reachable
relay answered for this exact id and reported it absent. Both thrown messages
stay byte-identical to the previous ones, so all six message-based consumers are
untouched; only callers that can act on the stronger verdict test the class.

Deliberately not raised for a transport failure, timeout, disposed multiplexer,
identity mismatch, or restoreRequired — none of those observe the process, and
treating them as absence would orphan live remote work.

* test(ssh): cover the absence fallback against real persistence

The first cut passed no store/worktreeId, so attachStablePaneOwner's guard
short-circuited and retireTerminalSurfaceFromPersistence — which deletes the
parent tab and its layout when the retired leaf is the last one — never ran.
retirePersistedStablePaneOwner and attachStablePaneOwner had no coverage at all.

(cherry picked from commit 25fa6a9)
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…awn (stablyai#16763)

* fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn

reattachSshPtySession flattened the relay's `PTY "<id>" not found` into a bare
`SSH_SESSION_EXPIRED`, discarding the evidence. isPtyAlreadyGoneError classifies
on that message via /PTY ".+" not found/, so it could never match — and
attachStablePaneOwner's existing already-gone fallback (retire the pane binding,
onPtyExit, spawn fresh) is gated on exactly that predicate. The recovery
machinery was already correct; only the classification was broken, which is why
`pty:spawn` rejected outright after a relay restart renumbered its PTYs from
pty-1.

SshPtyAbsentFromRelayError carries the verdict the message cannot: a reachable
relay answered for this exact id and reported it absent. Both thrown messages
stay byte-identical to the previous ones, so all six message-based consumers are
untouched; only callers that can act on the stronger verdict test the class.

Deliberately not raised for a transport failure, timeout, disposed multiplexer,
identity mismatch, or restoreRequired — none of those observe the process, and
treating them as absence would orphan live remote work.

* test(ssh): cover the absence fallback against real persistence

The first cut passed no store/worktreeId, so attachStablePaneOwner's guard
short-circuited and retireTerminalSurfaceFromPersistence — which deletes the
parent tab and its layout when the retired leaf is the last one — never ran.
retirePersistedStablePaneOwner and attachStablePaneOwner had no coverage at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant