Skip to content

fix(ssh): fence reattach restore and relay backpressure - #17786

Closed
nwparker wants to merge 14 commits into
stablyai:mainfrom
nwparker:fix/ssh-reattach-orphan-pty
Closed

nwparker wants to merge 14 commits into
stablyai:mainfrom
nwparker:fix/ssh-reattach-orphan-pty

Conversation

@nwparker

@nwparker nwparker commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

ELI5

An SSH terminal can be alive even when its output stream or network path is temporarily unavailable. This change keeps that distinction intact, retries only the recoverable source-restore case, and prevents a disconnected viewer from wedging the relay indefinitely.

What changed

  • Bound SSH PTY reattach/source-restore retries and preserve live vs unverifiable vs exited outcomes.
  • Fence PTY identities and stale generations so delayed exits cannot clear a replacement pane.
  • Drop undeliverable per-client PTY backlog and resume the live PTY when the last client detaches.
  • Make split routing, cold parking, and tab cycling worktree/unified-tab aware.
  • Normalize remote workspace authority snapshots and harden native PTY/error classification.
  • Add deterministic unit, relay fault, and Docker SSH transport-drop coverage.

Why

This is the follow-up for #12448 and the remaining SSH/tab reliability findings from #17159. It also covers the long-lived SSH slowdown report by ensuring an abandoned client cannot keep a relay PTY paused forever.

Verification

  • pnpm tc
  • pnpm run check:code-quality:changed
  • pnpm exec vitest run --config config/vitest.config.ts focused SSH/relay/tab matrix: 15 files, 178 tests
  • Rebased onto origin/main at 1efd4e1a97d
  • AWIN, M4 Air, and OpenClaw exact-head validation is being rerun at this SHA

Residual risk

Physical WSL reboot/sleep-resume remains unavailable in this environment. #17748 is related but separate: its timeout fix is plausible, though its current Docker gate is non-green after an Electron startup context failure.

Checklist

  • Tests and typecheck
  • Changed-code quality and formatting
  • Rebased on latest main
  • No .llm-counsel scratch files included
  • Physical WSL reboot/sleep-resume

AI-assisted change; all behavior claims are backed by the listed tests or host runs.

@pullfrog

pullfrog Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Run failed. View the logs →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@nwparker
nwparker force-pushed the fix/ssh-reattach-orphan-pty branch from ba75847 to 7d43b39 Compare September 1, 2026 02:38
@greptile-apps

greptile-apps Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Greptile Summary

The PR strengthens SSH PTY restoration, identity fencing, relay backpressure handling, remote workspace normalization, split routing, cold parking, and tab cycling.

  • Adds bounded source-restore retries and preserves live, unverifiable, and exited outcomes.
  • Fences stale PTY generations and drops undeliverable per-client relay backlog.
  • Adds focused unit, relay-fault, and Docker SSH transport-drop coverage.
  • Updates split routing, cold parking, and unified-tab-aware navigation.

Confidence Score: 4/5

The PR is not yet safe to merge because split terminal cycling can still be swallowed when hydration leaves the active unified-tab pointer null or unmatched.

The current fallback derives both the cursor and active-target check from the terminal entity ID when no exact active unified tab is available; distinct split tabs sharing that entity therefore remain indistinguishable and cycling can return without activating the sibling.

Files Needing Attention: src/renderer/src/hooks/ipc-tab-switch.ts

Important Files Changed

Filename Overview
src/main/providers/ssh-pty-session-reattach.ts Refines SSH reattach restoration with bounded retries and evidence-preserving liveness outcomes.
src/relay/relay-pty-source-publication.ts Updates source publication and client-delivery behavior to isolate disconnected or backpressured viewers.
src/relay/pty-handler.ts Adds PTY delivery cleanup and resume behavior around client detachment.
src/renderer/src/hooks/ipc-tab-switch.ts Makes terminal cycling unified-tab aware and adds hydration fallbacks for worktree terminal order.
src/renderer/src/components/terminal-pane/terminal-pane-split-request-routing.ts Makes terminal split requests aware of exact worktree and unified-tab routing.
tests/e2e/ssh-docker-transport-drop-recovery.spec.ts Adds Docker-backed coverage for SSH transport interruption and terminal recovery.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[SSH PTY transport drops] --> B{PTY liveness evidence}
  B -->|Live| C[Bounded source restore]
  B -->|Unverifiable| D[Retain recoverable ownership]
  B -->|Exited| E[Publish terminal exit]
  C --> F[Generation-fenced relay delivery]
  F --> G{Viewer attached?}
  G -->|Yes| H[Resume ordered output]
  G -->|No| I[Drop client backlog and resume PTY]
Loading

Reviews (11): Last reviewed commit: "fix(ssh): honor cancellation during reat..." | Re-trigger Greptile

@coderabbitai

coderabbitai Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 1cbcf7ae-a0ce-4aad-8319-a9a96d14afe5

📥 Commits

Reviewing files that changed from the base of the PR and between f290ccf5037083e5904af8a08b80cc14549317f2 and 00e30b49949ad7f1994fa5e6da5547bf0364d60f.

📒 Files selected for processing (4)
  • src/relay/relay-pty-source-activation.ts
  • src/relay/relay-pty-source-publication.ts
  • src/relay/relay-pty-source-transport-generation.test.ts
  • tests/e2e/ssh-docker-transport-drop-recovery.spec.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/relay/relay-pty-source-transport-generation.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

The changes centralize SSH PTY failure tokens and refine identity-mismatch, session-expiry, and source-restore handling. Reattach flows now retry bounded source restoration and await lease rollback. Relay delivery ownership now includes transport generation, and undeliverable output is released after client detach. Renderer terminal routing now handles worktree-scoped split requests, hydration gaps, and normalized workspace status. New Docker SSH E2E tests cover transport recovery, memory bounds, and output continuity.

Merge Risk: 🔵 Low · up to 00e30

The change improves SSH recovery, relay backpressure, and terminal routing behavior, with no current merge-blocking check failure. A bounded terminal-cycling issue remains possible when stale and active terminal identifiers collide, so owner follow-up is still warranted.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 66 functions across 54 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main SSH reattach and relay backpressure fixes. It is concise and specific.
Description check ✅ Passed The description is mostly complete and explains the changes, motivation, verification, residual risk, and checklist status. It uses equivalent sections for testing and issue context, but it does not p…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is mostly complete and explains the changes, motivation, verification, residual risk, and checklist status. It uses equivalent sections for testing and issue context, but it does not provide the required Fixes # entry, visual proof or an explicit N/A, AI model details, or several template sections.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 2629599b-30ae-4dc7-bb38-a42ec66afd3c

📥 Commits

Reviewing files that changed from the base of the PR and between a5796ec and 7d43b3961a90d861058feff231c402b19d820a81.

📒 Files selected for processing (55)
  • config/reliability-gates.jsonc
  • config/scripts/pr-e2e-gate-contract.test.mjs
  • config/scripts/pr-e2e-source-routing.mjs
  • config/scripts/run-ssh-docker-e2e.mjs
  • src/main/daemon/daemon-native-pty-exception.test.ts
  • src/main/daemon/daemon-native-pty-exception.ts
  • src/main/ipc/pty/ipc/spawn-execute.ts
  • src/main/ipc/pty/runtime/spawn-execute.ts
  • src/main/providers/ssh-pty-errors.ts
  • src/main/providers/ssh-pty-provider-reattach-incarnation.test.ts
  • src/main/providers/ssh-pty-provider.ts
  • src/main/providers/ssh-pty-relay-absence-verdict.test.ts
  • src/main/providers/ssh-pty-session-reattach-for-spawn.ts
  • src/main/providers/ssh-pty-session-reattach-restore-retry.test.ts
  • src/main/providers/ssh-pty-session-reattach.ts
  • src/main/ssh/__tests__/ssh-connection-test-client.ts
  • src/relay/dispatcher-capacity-signals.ts
  • src/relay/dispatcher-contract.ts
  • src/relay/dispatcher-rpc-routing.ts
  • src/relay/pty-handler-output-streaming.test.ts
  • src/relay/pty-handler.ts
  • src/relay/relay-pty-source-delivery-record.ts
  • src/relay/relay-pty-source-publication.ts
  • src/relay/relay-pty-source-restore-required.ts
  • src/relay/relay-pty-source-send-scheduler.ts
  • src/relay/relay-pty-source-transport-generation.test.ts
  • src/renderer/src/app-shell/remote-workspace-unplaced-upload-suppression.test.tsx
  • src/renderer/src/app-shell/use-app-session-persistence.ts
  • src/renderer/src/components/terminal-pane/TerminalErrorToast.test.ts
  • src/renderer/src/components/terminal-pane/TerminalErrorToast.tsx
  • src/renderer/src/components/terminal-pane/TerminalPane.tsx
  • src/renderer/src/components/terminal-pane/ipc-pty-connect.ts
  • src/renderer/src/components/terminal-pane/pty-connection-direct-ssh-reattach-retry.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection/deferred-session-attach.ts
  • src/renderer/src/components/terminal-pane/pty-connection/deferred-session-reattach-connect.ts
  • src/renderer/src/components/terminal-pane/pty-connection/pty-connect-limits.ts
  • src/renderer/src/components/terminal-pane/pty-connection/reattach-failure-classification.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection/reattach-failure-classification.ts
  • src/renderer/src/components/terminal-pane/pty-connection/ssh-session-connect.ts
  • src/renderer/src/components/terminal-pane/pty-transport-spawn-errors.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport-expired-pane-recovery.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/renderer/src/components/terminal-pane/terminal-pane-split-request-routing.test.ts
  • src/renderer/src/components/terminal-pane/terminal-pane-split-request-routing.ts
  • src/renderer/src/components/terminal-pane/use-terminal-tab-cold-parking.test.ts
  • src/renderer/src/components/terminal-pane/use-terminal-tab-cold-parking.ts
  • src/renderer/src/hooks/ipc-tab-switch.test.ts
  • src/renderer/src/hooks/ipc-tab-switch.ts
  • src/renderer/src/hooks/remote-workspace-push-status.ts
  • src/shared/ssh-pty-failure-tokens.ts
  • tests/e2e/headless-paired-remote-terminal-stall-recovery.spec.ts
  • tests/e2e/helpers/docker-ssh-relay-faults.ts
  • tests/e2e/helpers/paired-terminal-parking-fixture.ts
  • tests/e2e/ssh-docker-transport-drop-recovery.spec.ts
  • tests/e2e/terminal-parked-cli-split.spec.ts
💤 Files with no reviewable changes (2)
  • src/renderer/src/components/terminal-pane/pty-connection/pty-connect-limits.ts
  • src/renderer/src/components/terminal-pane/pty-connection/ssh-session-connect.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 5 remain after this review.

@nwparker
nwparker force-pushed the fix/ssh-reattach-orphan-pty branch from e73cd72 to 8a998fc Compare September 1, 2026 04:31

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 3c107d01-21a1-40fb-926e-c47366beeb1f

📥 Commits

Reviewing files that changed from the base of the PR and between e73cd722b14b0a6d338fc31f7798c450c0f6b679 and 8a998fc5203b1d220abaee86a995d87d74170ad6.

📒 Files selected for processing (11)
  • src/main/providers/ssh-pty-provider-exit-race.test.ts
  • src/main/providers/ssh-pty-session-reattach-for-spawn.ts
  • src/main/providers/ssh-pty-session-reattach-restore-retry.test.ts
  • src/main/providers/ssh-pty-session-reattach.ts
  • src/relay/relay-pty-source-publication.ts
  • src/relay/relay-pty-source-transport-generation.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection/reattach-failure-classification.ts
  • src/renderer/src/hooks/ipc-tab-switch.test.ts
  • src/renderer/src/hooks/ipc-tab-switch.ts
  • tests/e2e/ssh-docker-transport-drop-recovery.spec.ts
  • tests/e2e/terminal-parked-cli-split.spec.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: f7991a92-5876-4bc4-8aba-428ef75ad383

📥 Commits

Reviewing files that changed from the base of the PR and between 8a998fc5203b1d220abaee86a995d87d74170ad6 and f290ccf5037083e5904af8a08b80cc14549317f2.

📒 Files selected for processing (2)
  • SSH-HANDOFF.md
  • src/renderer/src/i18n/locales/en.json

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.

Comment thread SSH-HANDOFF.md Resolved

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta since the prior review (e73cd72 → 8a998fc): the provisional source-delivery rollback is now awaited on every SSH reattach/spawn failure path, the relay fences stale activations, the reattach-failure copy is localized, and terminal cycling honors the unified active tab.

  • Awaited provisional delivery rollback — requestSshPtyAttach, reattachSshPtySessionWithExitFence, and reattachSshPtySessionForSpawn now await the lease rollback (preserving the original error) instead of fire-and-forget, so a failed spawn cannot outlive its own cancellation or let a retry overlap the old lease. The ledger's cached rollbackSettlement keeps repeated calls safe.
  • Fenced stale relay activations — RelayPtySourcePublication.activate() now returns false (releasing the rotation fence) when context.isStale(), so a superseded request cannot retire or rotate the delivery opened by its replacement.
  • Localized the reattach-failure message — the "reconnecting" copy routes through translate() with the existing English fallback.
  • Honored the unified active tab in terminal cycling — handleSwitchTerminalTab anchors the cycle on the active group's unified tabId and only treats single-terminal as a no-op when a terminal is already active, fixing stale legacy-id collisions during hydration.
  • Expanded focused coverage — new unit tests pin settle-before-reject timing, stale-activation fencing, and unified-tab cycle direction; e2e flood-line and parking-fixture assertions were tightened.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

A personal work-handoff document was committed to the repo root and should be removed before merging.

Reviewed changes

Reviewed the two commits pushed after the prior review (8a998fc → f290ccf): a localization-catalog entry for the new SSH restore message, and a new root-level handoff document.

  • Registered the SSH restore message — src/renderer/src/i18n/locales/en.json now carries reattachFailure.outputStreamRestoring, matching the translate() key added in reattach-failure-classification.ts. Correct and complete.
  • Added SSH-HANDOFF.md — a machine-specific working note that records a local worktree path, references untracked .llm-counsel/ and .ui-proof/ scratch directories, and lists open review-thread follow-ups. This is agent scratch, not repo documentation, and contradicts the PR checklist's "no scratch files" claim.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

Comment thread SSH-HANDOFF.md Resolved
@nwparker
nwparker force-pushed the fix/ssh-reattach-orphan-pty branch from 00e30b4 to a417ff9 Compare September 1, 2026 05:27

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta since the prior review (f290ccf → 00e30b4): the root-level handoff note was removed, a stale relay activation can no longer release a replacement transport's rotation fence, and the closed-delivery retirement check was inlined on the publish path.

  • Removed SSH-HANDOFF.md — the machine-specific scratch note flagged in the prior review is deleted from the repo root, resolving the only open finding.
  • Fenced the stale-activation rotation release — activate() now releases the rotation fence through releaseOwnedPtySourceRotationFence, which gates the release on sameClientTransport(record, context), so a superseded request from an old transport generation cannot release the fence a replacement transport armed. Legacy id-only behavior is preserved when both generations are undefined.
  • Inlined the closed-delivery retirement check — the deliveryClosedUnderRecord helper was folded into direct ptySourceDeliveryClosed(this.session, ...) calls at both the existing-delivery and publish() sites, preserving the check.
  • Hardened e2e drop/flood markers — the transport-drop spec now emits markers via printf '..._%s\n' instead of shell-interpolated echo.
  • Added a deterministic regression test — does not release a replacement fence from a stale activation asserts the stale request leaves the replacement fence armed (so publish() stays false) and that the owning transport can later release it.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the three commits pushed after the prior review (00e30b4 → a417ff9): the relay link is now declared lost when it delivers nothing inbound instead of wedging on a saturated writer, an unanswered native-deps probe is no longer misread as missing deps, and terminal cycling ignores stale non-terminal entity ids.

  • Declared a wedged relay link lost — SshChannelMultiplexer drops the writerSaturated field and no longer suppresses the dead-link check while the writer is saturated. The check still requires both no inbound data and a stale unacked frame, so a slow-but-alive peer whose relay keepalives still arrive stays connected, while a half-open link that delivers nothing inbound is torn down with connection_lost. Drain still rebases both health clocks.
  • Stopped treating an unanswered native-deps probe as proof the deps are missing — probeRequiredNativeDeps now returns an available | missing | unverifiable verdict; a throw (host never answered) maps to unverifiable, and repairInstalledNativeDeps only authorizes an npm install/rebuild on a confirmed missing. Aborts still propagate via signal?.throwIfAborted(). execCommand timeout errors also gained a machine-readable SSH_EXEC_TIMEOUT_CODE.
  • Ignored stale non-terminal ids during terminal cycling — handleSwitchTerminalTab applies the entity-id fallback only when a terminal is active, so a stale editor/browser id that collides with a terminal entity during hydration starts the cycle at the edge instead of at the colliding row.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@nwparker
nwparker force-pushed the fix/ssh-reattach-orphan-pty branch 3 times, most recently from 3442662 to 087d356 Compare September 1, 2026 06:48
@nwparker
nwparker force-pushed the fix/ssh-reattach-orphan-pty branch from 087d356 to b7d4c16 Compare September 1, 2026 06:57
A departed client wedged the shell it was watching. Output backpressure pauses the pty itself
and only resumes once pending bytes drain below the low watermark - which needs a client to
drain to. When the transport dies mid-stream there is none, so the pty stayed paused, the shell
blocked on a pty nobody was reading, and those bytes were never produced at all.

Measured on both sides before the fix. Client, instrumenting pty.attach:
  replayLen=1427 seqCount=120 first=1 last=120 sourceRecovery=no
Relay, instrumenting appendReplayBuffer in-container:
  [relay-probe] ingress saw SEQ up to 100 ... and nothing after

So the reattach replay was not returning too little - for a scrolling shell it could not contain
the outage, because the outage was never recorded.

The pending queue is per-client delivery state and unbounded, which is why pausing was the safe
move. With no client it is also undeliverable, so it is dropped rather than carried; buffered is
the reattach replay source and is capacity-bounded.

Oracle: 459 of 600 lines lost -> 0, across 10 runs. Bounded under 48MB of undeliverable output,
with reconnect-and-rebind measured at ~11.9s.

Also adds the fault-injection harness this needed (the repo had none) and wires six Docker-SSH
specs to SSH source changes - 12 of 17 previously ran only when their own spec file changed.
@nwparker
nwparker force-pushed the fix/ssh-reattach-orphan-pty branch from 5accc6f to ac961d0 Compare September 1, 2026 07:14

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the single commit pushed since the prior review (a417ff9 → 5accc6fd9f): cancellation is now honored throughout the SSH reattach restore-retry path.

  • Forwarded the spawn signal to the attach request — requestSshPtyAttach accepts signal?: AbortSignal and passes it through to mux.request, which already supports abort and emits rpc.cancel to the relay, so an in-flight pty.attach is canceled relay-side rather than only dropped client-side.
  • Fenced each retry iteration against cancellation — reattachSshPtySession calls args.options.signal?.throwIfAborted() before each attach attempt and again after the provisional-delivery rollback, so a spawn canceled mid-reattach does not burn another attach round-trip. The rollback check was refactored into a staleDeliveryCanceled boolean with unchanged fail-closed semantics.
  • Added regression coverage — honors spawn cancellation before a restore retry asserts a single attach round-trip and an AbortError rejection; the focused suite passes (7 tests).

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta since the prior review (5accc6fd9f → ac961d0c0a): a rebase plus one mechanical relocation.

  • Relocated the reattach-error-clear block into the title-state hook — the describeReattachFailure-aware onPtyErrorClearedRef body moved verbatim from TerminalPane.tsx into use-terminal-pane-title-state.ts, which use-terminal-pane-controller.ts already composes, so the ref reaches the same consumers (terminal-pane-pty-deps.ts, transport-output-callbacks.ts, session-reconcile-dispose.ts). No behavior change.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@nwparker

nwparker commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Review verdict: salvage parts, discard others — not a rebase-and-merge

Three independent reviews of this PR against main (merge base 41ef1ddd80) plus the ~70-PR sweep that has landed since it was written. Method caveat, stated up front: the reviews read files via git show pr17786:<path> and traced code paths by hand; the suites were not executed, because the review worktrees were on unrelated commits. Treat "fails on main" claims as strong but unexecuted.

Keep — these are real and fix user-visible bugs

ipc-tab-switch.ts is the strongest part of this PR. On main, handleSwitchTerminalTab ends with setActiveTab(next.id) on the legacy entity id and compares next.id === store.activeTabId. With two split copies of the same terminal entity both copies share an id, so main returns false outright and Ctrl+PageDown is dead in that split. This PR routes through the existing activateCyclableTab, which already carries the activateTab(next.tabId) disambiguation, and indexes off the unified group pointer. Reusing the existing helper rather than adding a parallel setter is the right call.

The activeGroupTerminalTabs.length >= 2 guard removal is also justified: getGroupVisibleTabOrder drops a unified terminal row whose entity has not landed in tabsByWorktree yet, and that can happen with two rows already projected — so main really could strand the third terminal. The healthy multi-group case was traced and cycling stays split-local.

Also reported as genuinely failing on main for the right reason: the multiplexer wedge, the daemon stack-frame, the relay backlog-drop, the native-deps probe, and the reattach-retry core.

Discard — the split-routing rework guards a state production cannot produce

Every new construct in terminal-pane-split-request-routing.ts — splitRequestHandlersByTab, pendingLegacyQueueFlushTabIds, routingGeneration, unscopedRouteIsUnambiguous, scheduleLegacyQueuedRequestFlush, and the detail.worktreeId === undefined listener branch — fires only for a detail whose worktreeId is undefined. Both producers always set it: terminal-ui-routing-ipc-bridge.ts:86-107 early-returns on a null resolution, and terminal-presentation-ipc-bridge.ts:171-185 reads it from an IPC payload typed worktreeId: string (required). The sole consumer always passes a string.

The mechanism itself is sound and not racy — the generation fence, the pending-flag clear before the size re-check, and the re-read from the map are all correct. It simply has no reachable trigger. Four of the five new tests here fail on main for a reason that cannot occur in production.

Two defects in the same file regardless: scheduleLegacyQueuedRequestFlush drops the request entirely if the sole handler unregisters before its microtask runs (main would have consumed it at register time), and takeQueuedTerminalPaneSplitRequests now has zero production callers — exported solely for the test file.

Either delete this machinery and narrow SplitTerminalPaneDetail.worktreeId to required, or, if the optionality is deliberate wire tolerance, say so in a comment.

Fix before any of it lands — a new error vocabulary with unreachable consumers

SSH_SOURCE_RESTORE_REQUIRED appears in 9 files, and none are the four that previously consumed SSH_SESSION_EXPIRED:

  • ipc-pty-connect.ts:186 — recognizes only SSH_SESSION_EXPIRED / belongs to SSH connection
  • remote-runtime-pty-transport.ts:1605 — the paired-hub recovery gate keys on the old token, so the new one falls through to onError(rawMessage)
  • spawn-execute.ts:144 and runtime/spawn-execute.ts:224 — never match (intended, untested)
  • TerminalErrorToast.tsx:30-34 — UNREATTACHABLE_SESSION_SOURCES gained a pattern for SSH_PTY_IDENTITY_MISMATCH but not for the token this PR introduces

Outside the two describeReattachFailure call sites, the user sees the literal string SSH_SOURCE_RESTORE_REQUIRED: ssh:conn@@pty-7 in a red toast. This is also a Rule 3 change under docs/reference/remote-wire-compatibility.md — a new host sends an unrecognized marker to an old viewer where it previously sent a recoverable one — with no mixed-version test.

Related, and the shape that most concerns me: the use-terminal-pane-title-state.ts change exists to compensate for a regression this same PR introduces. remote-runtime-pty-transport.ts:482-495 stores the raw message and markRecoveryHealthy replays it, while clearPaneTerminalError matches by string. Main stored the raw message, so it matched. Changing the report to describeReattachFailure(message) breaks that match, so the error never clears — and there is no test for it.

Two tests that pass unchanged on main

  • pty-connection-direct-ssh-reattach-retry.test.ts:285 injects the new token into onError, but main's predicate is message.includes('SSH_SESSION_EXPIRED') — false for that string — so main takes the identical no-respawn branch and every assertion holds. The regression it is named for is unreachable by the message it sends.
  • ssh-channel-multiplexer-saturation-wedge.test.ts:56 advances 205s and asserts only settled('rejected'), but REQUEST_TIMEOUT_MS is 30s and the per-request timeout is unconditional — it fires on main too, wedge or no wedge.

Recommendation

Do not rebase this whole PR onto main. Split it:

  1. Land now, on its own: the ipc-tab-switch.ts fix. It is self-contained, reuses an existing helper, and fixes a dead keyboard shortcut in split terminals.
  2. Rework: the error-vocabulary change — follow SSH_SOURCE_RESTORE_REQUIRED to all four consumers and the toast allowlist, add the mixed-version case, and drop the title-state compensation once the string mismatch is gone.
  3. Drop: the split-routing rework, unless someone can name a producer that emits an unscoped detail.
  4. Re-check against the sweep before salvaging the rest — live/unverifiable/exited preservation, identity/generation fencing, and backlog-drop are all now covered by fix(ssh): stop respawning panes on client-side-only absence evidence #17957→fix(ssh): require a host death certificate before recreating a pane, and unstick expired leases #18013, fix(ssh): give supersession and id-recycling their own lease marks #17966, fix(ssh): compare lease and runtime pty ids in one form so pane recovery runs #17969, fix(ssh): stop a request that never left the client from tombstoning its operation id #17928, fix(relay): fail an oversized response instead of closing the client #17968 and fix(ssh): declare a wedged relay link lost, and stop reading silence as a verdict #17817. Anything overlapping should be dropped in favour of the landed implementation rather than merged alongside it.

@nwparker

nwparker commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Correction and expansion — I under-weighted the core fix in my comment above

A deeper pass changes the emphasis. My previous comment led with ipc-tab-switch.ts; that is real but is scope creep. The genuinely valuable part of this PR is something no other PR in the sweep fixes.

The core bug, and nothing else addresses it

On main, src/main/providers/ssh-pty-session-reattach.ts:290 throws SSH_SESSION_EXPIRED: <id> when the relay answers sourceRecovery.status === 'restoreRequired'.

That status proves the PTY is live — only its output delivery was retired. And SSH_SESSION_EXPIRED is the one token that authorises a respawn: both spawn-execute.ts sites match it, then markSshRemotePtyLease(…, 'expired'), deletePtyOwnership(), and the renderer cold-starts a fresh agent over the same worktree.

So this is a respawn authorised by a successful RPC that proved the process alive — not by loss of contact. It is the exact inversion docs/reference/ssh-execution-boundary.md exists to prevent, and it is reachable today.

I checked: #17957, #17965, #17966, #17969 and #18013 do not touch ssh-pty-session-reattach.ts. This is unfixed anywhere else.

The fix here is small and correct — a distinct SSH_SOURCE_RESTORE_REQUIRED token plus isProvenSshSessionGoneError, which excludes both restore-required and identity-mismatch from "proven gone". The bounded 3-attempt retry is the honest completion of the protocol rather than a band-aid, and on exhaustion it fails to a token the classifier treats as unverifiable — no respawn. Two nits: no backoff between attempts, and attempt 2 only succeeds if registerCanceledPtySourceRetirement has fired on response settlement.

A second real root cause: delivery ownership keyed on client id alone

relay-pty-source-publication.ts:96,107 compares current.clientId === context.clientId. But setWrite() → resetClient() → client.generation++ revives the primary channel under the same id. So a re-bound stdio channel got 'existing' back for a delivery bound to the dead sink — the pane repaints and then never receives source frames. That is a frozen-pane root cause, and sameClientTransport is also the consistent key: dispatcher-capacity-signals.ts:96 already keys the publication ledger on ${id}:${generation}; the delivery record was the outlier.

Also worth taking regardless of this PR's fate

tests/e2e/helpers/docker-ssh-relay-faults.ts — four genuinely distinct fault shapes, each documented with why it is not the others: kill the per-connection sshd forks (transport dies, relay survives); docker pause (silence, TCP still established); network blackhole (remote keeps producing while unreachable); SIGKILL every relay.js (the only fault where exited is the correct verdict). Plus a fault-clearing helper so a failed spec cannot poison the worker. No other PR has this, and the sweep needs it.

And pty-handler.ts:504-530 — resume a PTY whose last client left. pausedOutputPtys survives client detach and self-heals only on the attach path, so an abandoned client blocks the remote shell's writes indefinitely. That is a direct contributor to the long-lived-slowdown report. Verified safe: appendReplayBuffer runs at ingest before enqueuePtyOutput, so dropping the egress queue does not truncate what a reconnect replays.

Superseded — drop these

Hunk Superseded by
ssh-relay-deploy.ts probe verdict #17979 (merged) — and the landed version is strictly better: it repairs when status !== 'ok' and carries initialProbe.missing forward when a locked re-probe is unverifiable. This PR narrows to verdict === 'missing', so an unverifiable re-probe after an answered missing initial probe silently skips the repair and leaves a broken relay unrepaired. That is a regression.
mux writerSaturated removal + its wedge test + SSH_EXEC_TIMEOUT_CODE #17817 — identical production change, identical new test filename, same companion files. This PR is a strict subset.
identity-mismatch gate in remote-runtime-pty-transport.ts #18013
ipc-pty-connect.ts guard #17957
both spawn-execute.ts hunks Pure refactor, behaviourally identical to main. Zero value.

One thing that should not be in any PR

config/reliability-gates.jsonc rewrites the prose of a historical 2026-08-29 gate-run record ("Fifteen" → "Fourteen"). The count is factually right, but editing a past evidence record inside an unrelated PR is the wrong habit regardless of correctness.

Revised recommendation

Close this PR as a unit. Open three focused ones:

  1. fix(ssh): stop treating a live PTY's retired source delivery as proof of death — the token, the reattach retry, reattach-failure-classification.ts, sameClientTransport, and the two provider tests (~400 lines). Rewrite the vacuous renderer test to actually fail on main. Sequence after fix(ssh): stop respawning panes on client-side-only absence evidence #17957/fix(ssh): require a host death certificate before recreating a pane, and unstick expired leases #18013 to avoid fighting them in ipc-pty-connect.ts / remote-runtime-pty-transport.ts.
  2. test(ssh): dockerized relay fault injection — the faults helper, the transport-drop spec, and the gate wiring. Test-only, mergeable immediately.
  3. The pty-handler last-client drain, once fix(ssh): reclaim relay PTYs the client has provably lost, on host attestation only #17831/fix(relay): retire PTYs the host proves are gone, and stop two per-poll scan storms #17832 settle.

Everything else — the split-routing rework, ipc-tab-switch.ts, the cold-parking and title-state changes, the gates-file edit — either belongs in its own non-SSH PR or should be dropped.

@nwparker

nwparker commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Closing per the review above, under the original brief's explicit option to "take it over… OR throw it all away."

The salvage is done and landed as two focused PRs:

  • fix(ssh): stop treating a live PTY's retired source delivery as proof of death #18016 — fix(ssh): stop treating a live PTY's retired source delivery as proof of death. The one genuine root-cause win here: restoreRequired proves the PTY is alive, and it was raising SSH_SESSION_EXPIRED — the one token that authorises deletePtyOwnership() + markSshRemotePtyLease(…, 'expired') and a renderer cold start over the same worktree. Nothing else in the sweep touches that file. Also carries the relay delivery-ownership fix, applied at all three release/cancel sites rather than one.
  • test(ssh): dockerized relay fault injection with verdict assertions #18017 — test(ssh): dockerized relay fault injection with verdict assertions. The fault harness, plus two new cases that assert the verdict vocabulary this PR's spec never reached: exited (SIGKILL the relay) and unverifiable (docker pause). 4 passed, 1 test.fixme.

#18017 found a defect on main that neither branch fixes — filed as #18018: after a 30s freeze the pane keeps its PTY and repaints scrollback correctly, then never accepts input again. Live-looking and deaf.

Why the rest is not being rebased

On the one file where this PR and the sweep actually met, this PR's version is worse. ssh-relay-deploy.ts is the sole merge conflict, and it is semantic: merged #17979 repairs when status !== 'ok' and carries initialProbe.missing forward when a locked re-probe is unverifiable; this PR narrows to verdict === 'missing', so an answered-missing initial probe followed by an unanswerable re-probe skips the repair entirely and leaves a broken relay unrepaired. That is a regression, and it is the clearest single signal for this decision.

Verbatim duplicates of open work: the mux writerSaturated removal is a strict subset of #17817 — identical production change, identical new test filename, same companion files. The native-deps probe hunk is superseded by merged #17979 and extended further by #18011.

Fixes that do not work as landed: the delivery-ownership guard is applied at 1 of 3 sites in activate(). Measured, not argued — with only that site guarded, the rewritten test fails identically to unguarded main; the live path is the unadmitted/subscriber branch. The test hid this by hard-coding isStale: () => true, which production reports as false.

A new error vocabulary shipped without its consumers. SSH_SOURCE_RESTORE_REQUIRED reached none of the four sites that previously consumed SSH_SESSION_EXPIRED, and no toast pattern — so the raw marker surfaced to the user. It also broke clearPaneTerminalError's string match (raw stored, humanized reported), which this PR then compensated for in use-terminal-pane-title-state.ts. #18016 removes that class of problem by construction: it humanizes at render, so the store holds the raw string on both paths.

Test issues: two tests pass unchanged on main; four split-routing tests guard a worktreeId === undefined state no producer emits (both producers type it as required); one asserts on a log string that no longer exists.

Out of scope: ipc-tab-switch.ts (a real fix for a dead Ctrl+PageDown in split terminals — worth its own non-SSH PR), the split-routing rework, cold-parking, title-state, push-status, session-persistence.

And config/reliability-gates.jsonc rewrites the prose of a historical gate-run record. The correction is factually right; editing a past evidence record inside an unrelated PR is not.

Credit where due

Three genuinely valuable ideas came out of this PR and are preserved: the restore-required vocabulary split, the transport-generation ownership key, and the Docker fault taxonomy — the last of which is the strongest test artifact in the whole sweep and had no equivalent anywhere. The ssh-connection-test-client.ts change was also checked and is a hardening: the old mock read lastHostKeyAccepted synchronously and emitted ready on a key the verifier had not yet decided.

Reopen if you disagree with any of the above — the analysis is on this PR and in #18016/#18017.

@nwparker nwparker closed this Sep 1, 2026
@AmethystLiang

Copy link
Copy Markdown
Contributor

thanks for reporting. looking into it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants