Skip to content

fix(ssh): stop treating a live PTY's retired source delivery as proof of death - #18016

Closed
nwparker wants to merge 1 commit into
mainfrom
salvage/ssh-restore-required-token
Closed

nwparker wants to merge 1 commit into
mainfrom
salvage/ssh-restore-required-token

Conversation

@nwparker

@nwparker nwparker commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 6 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​447 $\color{#cf222e}{\Huge{\mathbf{−}}}$​4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​443
Prod 16 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​296 $\color{#cf222e}{\Huge{\mathbf{−}}}$​131 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​165

Salvaged from #17786 (see the review there). This is the one genuine root-cause win in that PR, extracted clean. Sequence this before #17957 and #18013 — it changes the token they gate on.

ELI5

A successful RPC that proved the process alive was authorising a cold restart over the same worktree.

Chain, verified end to end on main:

relay-pty-source-activation.ts:80 returns restoreRequired for a live PTY (only its output delivery was retired) → reattachSshPtySessionForSpawn threw SSH_SESSION_EXPIRED → both spawn-execute.ts sites set isExpiredSshSession, call clearProviderPtyState, deletePtyOwnership(), markSshRemotePtyLease(…, 'expired') → ipc-pty-connect.ts:186 returns {sessionExpired: true} → reattach-result-handler.ts:135 calls startFreshColdRestoreAgentResume(…, { forceBlankRestoredViewport: true }).

That is the exact inversion docs/reference/ssh-execution-boundary.md exists to prevent, and it is reached on positive evidence of life, not loss of contact. No other PR in the sweep touches ssh-pty-session-reattach.ts.

The fix

A new src/shared/ssh-pty-failure-tokens.ts with three deliberately disjoint tokens — disjoint because every consumer matches with includes(). Plus the throw-site change, the bounded 3-attempt retry, and sameClientTransport + transportGeneration on RequestContext.

Because the tokens are disjoint, 'SSH_SOURCE_RESTORE_REQUIRED'.includes('SSH_SESSION_EXPIRED') is false, so both spawn-execute.ts sites and remote-runtime-pty-transport.ts:1603 already do the right thing — no respawn. That is the point of the split, and it is why the "unreachable consumers" concern raised in review needed no code change there.

The one real gap was the toast. It deliberately does not go in UNREATTACHABLE_SESSION_SOURCES — that copy says "Open a new terminal", which here would abandon a running agent. It gets its own pattern, its own copy, and an entry in isExplainedTerminalError to suppress the issue link.

The clear-matching regression #17786 had is gone by construction: describeReattachFailure was not taken at all. Humanizing at render in TerminalErrorToast — the existing canonical mechanism — means the store holds the raw string on both the append and clear paths, so they match. use-terminal-pane-title-state.ts is untouched and there is no compensation to drop.

isProvenSshSessionGoneError was deliberately left out. Once the tokens are disjoint it is behaviourally identical to main's isSshSessionExpiredError. Its only added value is the identity-mismatch gate (that is #18013's change) and a PTY not found broadening that widens respawn authorisation.

The second root cause — and #17786's fix for it does not work

Delivery ownership was keyed on client id alone, so a re-bound primary channel (setWrite() → resetClient() bumps generation, keeps the id) got 'existing' back for a delivery bound to the dead sink. The pane repaints and then never receives frames. dispatcher-capacity-signals.ts:96 already keys the publication ledger on ${id}:${generation}; the delivery record was the outlier.

#17786 guards one of the three release/cancel sites in activate(). That was confirmed insufficient by experiment, not argument:

configuration result
guard on all three sites (this PR) passes
guard removed entirely (main) AssertionError: expected true to be false
guard on site A only (#17786's fix) same failure

The path actually taken is site B (unadmitted/subscriber). I also rewrote the transport-generation test so isStale() stays false as production reports it — #17786's version hard-coded it true, which is what hid site B.

User-facing regressions

None expected. The change narrows what authorises a respawn; it never widens it.

Testing

New/changed tests against main sources: Test Files 5 failed (5) / Tests 10 failed | 41 passed (51). After: Test Files 6 passed (6) / Tests 62 passed (62). Full suites: src/relay/ + src/main/providers/ 2364 passed; terminal-pane/ 4113 passed. pnpm tc clean, check:code-quality:changed 0 findings, catalog synced (1 key).

One honest limitation: the ported renderer retry test still passes on main, because a renderer-only test cannot distinguish this — the change is in what the provider emits. A positive control was added (same scenario with SSH_SESSION_EXPIRED, asserting clearTabPtyId is called) so "did not respawn" is a real result rather than a build where nothing respawns. The main→branch delta is proven by the provider tests.

Sequencing

Build this first. #17957 changes ipc-pty-connect.ts:186 to isSshSessionGoneError(message); #18013 deletes the same local SSH_SESSION_EXPIRED_ERROR const in remote-runtime-pty-transport.ts. This PR deletes both consts too, importing from the shared module — a textual conflict at exactly those import lines. Both PRs should then define isSshSessionGoneError over the shared tokens and exclude SSH_SOURCE_RESTORE_REQUIRED, rather than re-deriving the vocabulary.

A relay answer of sourceRecovery.status === 'restoreRequired' is a SUCCESSFUL
RPC that proves the PTY is alive - only its output delivery was retired. The
spawn reattach path raised SSH_SESSION_EXPIRED for it, which is the one token
that authorises replacing the pane's PTY: both spawn-execute sites then call
markSshRemotePtyLease(..., 'expired') and deletePtyOwnership(), and the
renderer cold-starts a fresh agent over the same worktree. A live remote agent
was orphaned by a request that had just observed it running.

Split the vocabulary so the two verdicts cannot be confused
(docs/reference/ssh-execution-boundary.md), retry the attach first since a
retired delivery is repairable, and give the token its own user-facing copy so
it never inherits the "open a new terminal" advice.

Second root cause, same symptom: the relay's delivery record was keyed on
client id alone, so a re-bound primary channel matched a delivery bound to the
dead sink and got 'existing' back - the pane repaints and never receives
frames. Key it on the transport generation too, as the dispatcher's own
publication ledger already does, and scope every release/retire path in
activate() to a record the caller still owns.
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: bc669029-e820-4d7d-b927-cd7c88d53dd3

📥 Commits

Reviewing files that changed from the base of the PR and between 7314ada and 7c6740f.

📒 Files selected for processing (22)
  • src/main/providers/ssh-pty-errors.ts
  • src/main/providers/ssh-pty-provider-reattach-incarnation.test.ts
  • src/main/providers/ssh-pty-provider.ts
  • src/main/providers/ssh-pty-relay-absence-verdict.test.ts
  • src/main/providers/ssh-pty-session-reattach-for-spawn.ts
  • src/main/providers/ssh-pty-session-reattach-restore-retry.test.ts
  • src/main/providers/ssh-pty-session-reattach.ts
  • src/relay/dispatcher-contract.ts
  • src/relay/dispatcher-rpc-routing.ts
  • src/relay/relay-pty-source-delivery-record.ts
  • src/relay/relay-pty-source-publication.ts
  • src/relay/relay-pty-source-restore-required.ts
  • src/relay/relay-pty-source-send-scheduler.ts
  • src/relay/relay-pty-source-transport-generation.test.ts
  • src/renderer/src/components/terminal-pane/TerminalErrorToast.test.ts
  • src/renderer/src/components/terminal-pane/TerminalErrorToast.tsx
  • src/renderer/src/components/terminal-pane/ipc-pty-connect.ts
  • src/renderer/src/components/terminal-pane/pty-connection-direct-ssh-reattach-retry.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection/pty-connect-limits.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/renderer/src/i18n/locales/en.json
  • src/shared/ssh-pty-failure-tokens.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The change centralizes SSH PTY failure tokens and adds a dedicated restore-required token. Reattachment retries restore-required responses up to three times and rolls back provisional leases before retrying. Relay delivery records now track transport generations to reject stale transport ownership. Restore-required handling moves into shared relay helpers. Renderer error handling distinguishes expired sessions from live sessions that require output restoration. Tests cover retry limits, lease rollback, transport changes, and terminal messaging.

Merge Risk: 🔵 Low · up to 7c674

The PR prevents live SSH sessions from being mistaken for expired sessions and fences delivery ownership across reconnects. A bounded recovery risk remains because some rejection paths do not confirm remote delivery cancellation before retiring local state; the change is mergeable with explicit owner awareness and follow-up.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description gives a detailed ELI5, change summary, rationale, implementation details, sequencing guidance, and testing results. However, it does not provide the required Linked Issue entry using a… Add a valid linked issue using the required Fixes # format. Add BEFORE and AFTER visual proof, or write N/A with a brief reason if no visual or interaction change applies. Complete the Agent skill upstream boundary and checklist sections, a…
Docstring Coverage ⚠️ Warning Docstring coverage is 26.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 21 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the primary change: preventing a live PTY with retired output delivery from being treated as an expired SSH session.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description gives a detailed ELI5, change summary, rationale, implementation details, sequencing guidance, and testing results. However, it does not provide the required Linked Issue entry using a Fixes reference, the required Visual Proof section or an explicit N/A statement, the Agent skill upstream boundary checkbox, or the checklist confirmations.

Resolution

Add a valid linked issue using the required Fixes # format. Add BEFORE and AFTER visual proof, or write N/A with a brief reason if no visual or interaction change applies. Complete the Agent skill upstream boundary and checklist sections, and confirm the required validation commands or state that CI will cover them.

Full details: Docstring Coverage

Explanation

Docstring coverage is 26.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 21 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ One minor suggestion — the fix itself is sound, well-reasoned, and well-tested.

Reviewed changes — one commit reworking SSH PTY reattach failure signalling and relay delivery ownership around the ssh-execution-boundary live/unverifiable/exited vocabulary.

  • Disjoint failure tokens — a new src/shared/ssh-pty-failure-tokens.ts defines three includes()-disjoint SSH_*_ERROR tokens; ssh-pty-errors.ts re-exports them and the renderer consumers import them instead of re-declaring the string.
  • Correct reattach throw — reattachSshPtySessionForSpawn (moved to ssh-pty-session-reattach-for-spawn.ts) throws SSH_SOURCE_RESTORE_REQUIRED rather than SSH_SESSION_EXPIRED when sourceRecovery reports restoreRequired, so the SSH_SESSION_EXPIRED-matching consumers leave a live PTY alone.
  • Bounded restore retry — reattachSshPtySession retries pty.attach up to three times on restoreRequired, advancing only when the relay retires the retired delivery.
  • Transport-generation ownership — RequestContext gains transportGeneration and delivery records store it; sameClientTransport keys ownership on ${clientId}:${generation} so a re-bound primary (setWrite() keeping its id) no longer gets a false existing.
  • Renderer toast — TerminalErrorToast renders the new token as a distinct "reconnecting / still running" line, excluded from the "open a new terminal" copy and added to isExplainedTerminalError.

Verified end to end: the disjoint tokens genuinely split the respawn-authorising includes(SSH_SESSION_EXPIRED) from the live-PTY token, the transport-generation keying correctly matches the dispatcher-capacity-signals ledger, and the renderer retry test has a real positive control so "did not respawn" is an actual assertion rather than a build where nothing respawns.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

// restoreRequired proves the PTY is live and only its delivery was retired, so retrying the
// attach is the repair. Cancel the provisional delivery first, or the retry races the lease
// it is meant to replace; if cancellation cannot be confirmed, stop rather than stack leases.
const staleDeliveryCanceled = attachResult.sourceActivationLease

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This provisional-delivery rollback is unreachable in production: a restoreRequired response carries no sourceActivation (the relay returns only { status: 'restoreRequired', reason }), so requestSshPtyAttach never installs a lease and sourceActivationLease is always undefined here. The retry's safety rests entirely on the relay deleting the retired record in registerCanceledPtySourceRetirement's onResponseSettled; the comment two lines up claims the client cancels a provisional delivery it never holds.

Suggested change
const staleDeliveryCanceled = attachResult.sourceActivationLease
// restoreRequired proves the PTY is live and only its delivery was retired, so retrying the
// attach is the repair. The relay already canceled and queued the retired delivery for
// deletion on response-settle; the lease below is only present when a sourceActivation was
// returned (never on restoreRequired), so it is a defensive guard, not the cancellation itself.
const staleDeliveryCanceled = attachResult.sourceActivationLease
? await attachResult.sourceActivationLease.rollback()
: true

@nwparker

nwparker commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Duplicate implementation — #17951's stack already ships this fix

Found while integrating branches into the combined adhoc: this PR and #18029 both fix the same defect, independently, with different token names and different retry counts. They conflict on six files.

SSH_PTY_SOURCE_RESTORE_REQUIRED_ERROR exists in src/main/providers/ssh-pty-errors.ts on fix/duplicate-agent-respawn-gate (#17951) and every branch stacked above it — but not on main. So the lease stack has carried a restore-required token split since #17951, and I had the #17786 salvage build a second one without checking.

The two differ:

#18016 (this PR) #18029 / #17951 stack
token SSH_SOURCE_RESTORE_REQUIRED (new src/shared/ssh-pty-failure-tokens.ts) SSH_PTY_SOURCE_RESTORE_REQUIRED_ERROR (existing ssh-pty-errors.ts)
retry attempts SSH_PTY_RESTORE_REQUIRED_ATTACH_ATTEMPTS = 3 RESTORE_REQUIRED_ATTACH_ATTEMPTS = 2
classifier isProvenSshSessionGoneError (dropped as a no-op) isSshSessionGoneError in pty-connect-limits.ts

Conflicting files: ssh-pty-errors.ts, ssh-pty-session-reattach.ts, pty-connect-limits.ts, and three test files.

This is a semantic conflict, not a textual one. Merging them mechanically would produce a vocabulary that is half one scheme and half the other — exactly the failure mode this sweep has been cataloguing.

What needs deciding

One of the two should carry the fix. Points in favour of each:

This PR has the relay delivery-ownership fix applied at all three release/cancel sites in activate() — measured, with the guard-on-one-site configuration reproducing main's failure identically. It also carries the toast handling (its own pattern and copy, deliberately not UNREATTACHABLE_SESSION_SOURCES, whose "Open a new terminal" copy would abandon a running agent), and avoids the clear-matching regression by humanizing at render rather than at report.

The #17951 stack version is older, already reviewed in place, and sits under four PRs that build on it — so replacing it means rebasing that stack.

My recommendation: keep the #17951 stack's token as the vocabulary (it is upstream of more work), and port from here the three things it does not have — the three-site ownership guard, the toast pattern, and the render-time humanization. Then close this PR or reduce it to those hunks.

This also revises the sequencing note in this PR's description. It said "build this first, before #17957 and #18013." That is wrong now: #17951 is below #17957, so the vocabulary already lands ahead of both. The real constraint is that this PR and #18029 must not both land.

Flagging rather than resolving, since choosing which token survives affects four other PRs.

@nwparker

nwparker commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #18038.

This PR built a parallel vocabulary for a fix the #17951 lease stack already carries — SSH_SOURCE_RESTORE_REQUIRED vs the stack's SSH_PTY_SOURCE_RESTORE_REQUIRED_ERROR, retry 3 vs 2, isProvenSshSessionGoneError vs isSshSessionGoneError — conflicting on six files. #17951 is an ancestor of four PRs, so its vocabulary wins.

#18038 ports only what the stack genuinely lacked, in the stack's own vocabulary. Two findings from that port worth recording here:

The ownership guard was missing entirely, not partially. I reported that this PR's version guarded one of three release/cancel sites. In fact relay-pty-source-publication.ts at the stack tip is byte-identical to main for that file — zero sites were guarded. And the one-site configuration was measured to fail identically to unguarded main, because the live path is the unadmitted/subscriber branch. So this PR's guard would not have closed the bug it targeted.

Render-time humanization was already present, so that third item needed no code at all — humanizeTerminalError has exactly two references, both in TerminalErrorToast.tsx, and nothing humanizes before storing.

Two things from here were deliberately not carried over, and remain available if wanted: sameClientTransport / clientTransportGeneration (a genuinely separate root cause — the delivery record keyed on client id alone, so a re-bound primary channel gets 'existing' back for a delivery bound to a dead sink; it needs a RequestContext.transportGeneration field the stack does not have), and the context.isStale() early-out.

The underlying defect this PR was built to fix — restoreRequired raising the token that authorises a respawn, on a successful RPC that proved the PTY alive — is real and is fixed by the stack. See also #18029, which reverts a related change of mine that would have certified death from a pty.attach not-found.

@nwparker nwparker closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant