Repository navigation
Improve remote connection terminal error msg and session restore recovery - #7661
Conversation
- Release the replay guard after a fallback timeout to prevent permanent keyboard input lockouts when an unmounted terminal never parses. - Verify backing process liveness on PTY attach and reap stale entries so dead shells cleanly trigger a fresh pane spawn. - Normalize connection IDs during restore to prevent spurious mismatch errors, and treat true mismatches as expired sessions instead of crashing.
Avoid spawning SSH terminal processes against disconnected targets, which otherwise throws "No PTY provider" and leaves panes stranded. - Intercept spawning via a new connect gate that triggers the deferred connection flow when the SSH target is disconnected. - Fall back to composite worktree IDs during cold-start hydration to ensure deferred SSH session IDs are properly stashed. - Retry spawning and remounting terminal panes upon SSH reconnect if they are stranded or failed to spawn.
…remote-connection-terminal-error-msg
📝 WalkthroughWalkthroughThis PR hardens SSH terminal reconnection and PTY reattachment. On the relay side, PtyHandler now verifies the backing process is alive via a new isProcessAlive helper before reattaching, reaping stale PTYs otherwise. A new resolveSshPaneConnectGate resolver and shouldRetryPaneSpawnOnSshReconnect predicate govern deferred SSH connect flows and pane remount retries, replacing prior ad-hoc checks in pty-connection.ts and useIpcEvents.ts. pty-transport.ts now recovers from cross-connection PTY mismatches as session-expired. replay-guard.ts adds a bounded fallback release to prevent stuck replay counters. ssh-pty-id.ts normalizes execution-host style connection ids, and terminals.ts adds a repoId fallback for SSH worktrees missing from state during hydration. Extensive tests accompany each change. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant PtyHandler
participant isProcessAlive
Client->>PtyHandler: attach(id, paneKey)
PtyHandler->>isProcessAlive: check(managed.pty.pid)
isProcessAlive-->>PtyHandler: false
PtyHandler->>PtyHandler: notify exit listener + dispose + remove
PtyHandler-->>Client: throw PTY not found
sequenceDiagram
participant PtyConnection
participant resolveSshPaneConnectGate
participant SshConnect
PtyConnection->>resolveSshPaneConnectGate: input(sshStatus, sessionIds, tabPtyId)
resolveSshPaneConnectGate-->>PtyConnection: gate(pendingSessionId, enterDeferredFlow, sshConnected)
alt gate.enterDeferredFlow
PtyConnection->>SshConnect: enter deferred reconnect flow
else
PtyConnection->>PtyConnection: proceed to spawn
end
Related PRs: None identified from the provided summaries. Suggested labels: area: relay, area: terminal, area: ssh, tests Suggested reviewers: No reviewer information available from provided context. Poem 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
src/relay/pty-handler.ts (1)
900-905: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
revive()liveness check uses different (inconsistent) semantics than the newisProcessAlive.
revive()'s inlineprocess.kill(entry.pid, 0)treats any thrown error (includingEPERM) as "not alive" and skips reviving the entry, whereas the newisProcessAliveexplicitly treats non-ESRCHerrors likeEPERMas alive. A revived-but-unsignalable process would be silently dropped here while an attach-reaped one wouldn't. Consider reusingisProcessAliveinrevive()for consistent liveness semantics across both code paths.♻️ Proposed fix
- // Only re-attach if the original process is still alive - try { - process.kill(entry.pid, 0) - } catch { - continue - } + // Only re-attach if the original process is still alive + if (!isProcessAlive(entry.pid)) { + continue + }src/shared/ssh-pty-id.ts (1)
46-67: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider normalizing the embedded
parsed.connectionIdtoo, for symmetry.Both functions normalize the incoming
connectionIdparameter before comparing, but compare it againstparsed.connectionId— the connection id embedded inside an existing pty id — without normalizing that side. If any app pty id was ever created by encoding an execution-host-form id (e.g. before this fix landed, or via some other caller passing"ssh:<target>"), its embeddedconnectionIdwould be"ssh:<target>"rather than the bare target id, and a legitimate reattach from a normalized bare id would spuriously fail the equality check.The blast radius is soft today —
pty-transport.ts's newSSH_PTY_CONNECTION_MISMATCH_MARKERhandling turns that spurious mismatch into asessionExpiredrecovery rather than a crash — but it would still mean losing reattach/scrollback continuity unnecessarily for a session that is actually valid.🛡️ Proposed fix for symmetric normalization
export function toAppSshPtyId(connectionId: string, relayPtyId: string): string { const normalizedConnectionId = normalizeConnectionId(connectionId) const parsed = parseAppSshPtyId(relayPtyId) if (parsed) { - if (parsed.connectionId !== normalizedConnectionId) { + if (normalizeConnectionId(parsed.connectionId) !== normalizedConnectionId) { throw new Error(`PTY ${relayPtyId} belongs to SSH connection "${parsed.connectionId}"`) } return relayPtyId } return `${SSH_PTY_ID_PREFIX}${encodeURIComponent(normalizedConnectionId)}${SSH_PTY_ID_SEPARATOR}${relayPtyId}` } export function toRelaySshPtyId(connectionId: string, ptyId: string): string { const parsed = parseAppSshPtyId(ptyId) if (!parsed) { return ptyId } - if (parsed.connectionId !== normalizeConnectionId(connectionId)) { + if (normalizeConnectionId(parsed.connectionId) !== normalizeConnectionId(connectionId)) { throw new Error(`PTY ${ptyId} belongs to SSH connection "${parsed.connectionId}"`) } return parsed.relayPtyId }Worth confirming whether any code path could have produced (or could still produce, from another caller not in this cohort) an app pty id with a host-id-form connection embedded, to gauge if this is a live concern or purely theoretical.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 479ff287-3682-46e5-9478-1c4bcb3f5fe2
📒 Files selected for processing (19)
src/relay/pty-handler.test.tssrc/relay/pty-handler.tssrc/relay/pty-shell-utils.test.tssrc/relay/pty-shell-utils.tssrc/renderer/src/components/terminal-pane/pty-connection.test.tssrc/renderer/src/components/terminal-pane/pty-connection.tssrc/renderer/src/components/terminal-pane/pty-transport.test.tssrc/renderer/src/components/terminal-pane/pty-transport.tssrc/renderer/src/components/terminal-pane/replay-guard.test.tssrc/renderer/src/components/terminal-pane/replay-guard.tssrc/renderer/src/components/terminal-pane/ssh-pane-connect-gate.test.tssrc/renderer/src/components/terminal-pane/ssh-pane-connect-gate.tssrc/renderer/src/hooks/ssh-reconnect-pane-retry.test.tssrc/renderer/src/hooks/ssh-reconnect-pane-retry.tssrc/renderer/src/hooks/useIpcEvents.tssrc/renderer/src/store/slices/terminals-hydration.test.tssrc/renderer/src/store/slices/terminals.tssrc/shared/ssh-pty-id.test.tssrc/shared/ssh-pty-id.ts
| const fallbackTabSessionId = | ||
| !sshConnected && | ||
| !input.hasLeafSessionMap && | ||
| input.tabPtyId && | ||
| parseAppSshPtyId(input.tabPtyId)?.connectionId === input.connectionId | ||
| ? input.tabPtyId | ||
| : null |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect normalizeConnectionId's definition/usage and getConnectionId's normalization.
fd 'ssh-pty-id.ts$' src/shared
cat -n src/shared/ssh-pty-id.ts
rg -n 'normalizeConnectionId' -C3 src
rg -n 'function getConnectionId' -C10 srcRepository: stablyai/orca
Length of output: 10013
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the connectionId resolution path and the caller that feeds ssh-pane-connect-gate.
cat -n src/renderer/src/lib/connection-context.ts | sed -n '1,220p'
echo '--- folder-workspace-connection.ts ---'
cat -n src/renderer/src/lib/folder-workspace-connection.ts | sed -n '1,260p'
echo '--- pty-connection.ts ---'
cat -n src/renderer/src/components/terminal-pane/pty-connection.ts | sed -n '1,260p'Repository: stablyai/orca
Length of output: 24034
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Trace where repo/folder-workspace/project-group connectionId values are written.
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'connectionId\s*[:=]' src shared
echo '--- repo type ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'type Repo|interface Repo|connectionId\??:' src/shared src/renderer/src src/main
echo '--- folder workspace type ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'type FolderWorkspace|interface FolderWorkspace|connectionId\??:' src/shared src/renderer/src src/main
echo '--- project group type ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'type ProjectGroup|interface ProjectGroup|connectionId\??:' src/shared src/renderer/src src/mainRepository: stablyai/orca
Length of output: 50370
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- ssh-pane-connect-gate.ts ---'
cat -n src/renderer/src/components/terminal-pane/ssh-pane-connect-gate.ts | sed -n '1,120p'
echo '--- shared/project-groups.ts ---'
cat -n src/shared/project-groups.ts | sed -n '1,140p'
echo '--- shared/folder-workspaces.ts ---'
cat -n src/shared/folder-workspaces.ts | sed -n '1,150p'
echo '--- shared/execution-host.ts ---'
cat -n src/shared/execution-host.ts | sed -n '130,190p'Repository: stablyai/orca
Length of output: 16278
Compare normalized connection IDs here. parseAppSshPtyId(...) returns the decoded id, but input.connectionId comes from stored workspace/repo metadata without the same normalization, so an execution-host-style id can miss the fallback reattach and spawn a fresh shell.
46 commits from main. Conflict resolutions (branch structure wins, main semantics grafted): - pending-backlog cap (#7630): main's capPendingPtyData/droppedBacklog 2MB tail-trim EXCISED — this branch's droppedOutput sentinel mechanism (O(1) drop + query salvage + settings-scaled cap + restore marker) already bounds the unsent backlog and strictly supersedes it; carrying both would race two cap policies over one buffer. Kept main's makePtyDataPayload helper and exit-time pending flush (adapted to the sentinel). Main's duplicate never-ACK cap test removed (covered by the existing sentinel cap tests); its reload-gate tests kept with the runtime mock extended. - replay guard (#7661): main's blind REPLAY_GUARD_RELEASE_FALLBACK_MS release dropped — this branch's probe-certified stall release covers the same cold-restore reattach lockout without the auto-reply leak a timed release can cause in agent TUIs. Main's three regression tests KEPT, adapted to probe timing (explicit stallCheckMs). - hidden-pressure scenario: kept this branch's STRICT 1500ms restore budget (the background keep-tail global budget earns it; main relaxed to 4s for CI drain overhead) + grafted main's timer-drift gate const. - ssh-docker-relay-perf: kept this branch's ACK-stalled background PTY test AND grafted main's new file-streams+git-churn test (4 tests). - preload/renderer: droppedBacklog threading excised with the mechanism; main's unrelated changes (ssh importConfig reAdopt, etc.) kept. Gates: typecheck clean; ipc+daemon+terminal-pane+pane-manager suites 4,396 passed / 14 skipped. Co-authored-by: Orca <help@stably.ai>
Three respawn gates acted on evidence weaker than host-attested exit. Per docs/reference/ssh-execution-boundary.md, loss of contact, a failed reattach, an identity mismatch and absence from a client map are all `unverifiable`, never `exited`. Gate 1 (ipc-pty-connect.ts): "belongs to SSH connection" is minted by the id router from a pure client-side string compare, before any relay is asked, and still returned `sessionExpired: true` -> fresh PTY + agent resume. After an SSH target re-adoption the "other" connection is the same machine, so that puts a second `claude --resume` on the transcript the surviving PTY still owns. Now returns undefined with no error, which routes the pane to recoverUnverifiableDirectSshReattach (remount + reattach, no shell restart) and keeps #7661's no-red-toast outcome. Gate 3 (ssh-reconnect-pane-retry.ts): `!tabPtyId` read `tab.ptyId`, which is only the single-pane fallback for legacy attach. It diverges from the real records deterministically: workspace-terminal-reconnect fills ptyIdsByTabId from the leaf map but writes tab.ptyId only when a tab-level id survives, and clearTransientTerminalState nulls tab.ptyId on every hydrated row. Both leave live leaf PTYs with a null fallback field, arming a generation bump onto the fresh-spawn path. Now consults ptyIdsByTabId and the layout leaf map too; a tab with no PTY in any record still retries. Gate 2 (recoverTerminalPane): an `expired` lease plus `!pty.connected` authorized createTerminal. Every writer of `expired` records that the CLIENT lost its route, not that the shell died. Now also requires the runtime's own liveness verdict to be neither `live` nor `unverifiable`, and ssh-relay-session records markPtyLivenessLive at the persistPtyBinding refusal, which is reached only after pty.attach succeeded. See the report for why this branch is currently unreachable for SSH panes.
…17957) * fix(ssh): stop respawning panes on client-side-only absence evidence Three respawn gates acted on evidence weaker than host-attested exit. Per docs/reference/ssh-execution-boundary.md, loss of contact, a failed reattach, an identity mismatch and absence from a client map are all `unverifiable`, never `exited`. Gate 1 (ipc-pty-connect.ts): "belongs to SSH connection" is minted by the id router from a pure client-side string compare, before any relay is asked, and still returned `sessionExpired: true` -> fresh PTY + agent resume. After an SSH target re-adoption the "other" connection is the same machine, so that puts a second `claude --resume` on the transcript the surviving PTY still owns. Now returns undefined with no error, which routes the pane to recoverUnverifiableDirectSshReattach (remount + reattach, no shell restart) and keeps #7661's no-red-toast outcome. Gate 3 (ssh-reconnect-pane-retry.ts): `!tabPtyId` read `tab.ptyId`, which is only the single-pane fallback for legacy attach. It diverges from the real records deterministically: workspace-terminal-reconnect fills ptyIdsByTabId from the leaf map but writes tab.ptyId only when a tab-level id survives, and clearTransientTerminalState nulls tab.ptyId on every hydrated row. Both leave live leaf PTYs with a null fallback field, arming a generation bump onto the fresh-spawn path. Now consults ptyIdsByTabId and the layout leaf map too; a tab with no PTY in any record still retries. Gate 2 (recoverTerminalPane): an `expired` lease plus `!pty.connected` authorized createTerminal. Every writer of `expired` records that the CLIENT lost its route, not that the shell died. Now also requires the runtime's own liveness verdict to be neither `live` nor `unverifiable`, and ssh-relay-session records markPtyLivenessLive at the persistPtyBinding refusal, which is reached only after pty.attach succeeded. See the report for why this branch is currently unreachable for SSH panes. * fix(ssh): let the respawn gate see the relay's own absence answer Gate 3 refused to respawn a pane whose records still named a PTY, which is right for a transport drop and wrong for a killed relay: after the relay is SIGKILLed and comes back, the leaf map still holds `pty2:<dead-epoch>:1` while the new relay answers that it has no such id. #18017's "replaces the pane only when the host proves the session is gone" regressed on exactly that. The gap was not the predicate, it was its inputs. `handlePtyReattachFailure` already distinguishes the three reattach outcomes and only its not-found branch publishes anything — a lost link and an identity mismatch send nothing. But it published `pty:exit { code: -1 }`, and `-1` is the sentinel every reader resolves to `stop_unverified`, so the one branch holding positive host evidence of absence arrived looking exactly like loss of contact. The renderer had no host answer at all, which the gate's own comment conceded. The exit now carries `livenessVerdict: 'exited'` beside the unchanged `-1`, so the code keeps meaning "no provable status" for every existing reader while the verdict rides its own field. A store bridge records those ids in `hostAttestedAbsentPtyIds` regardless of whether a pane is mounted to hear it — during reconnect none is — and the gate stops counting a recorded id the host has disowned. Settled when a PTY answers to that id again, because a redeployed relay renumbers from pty-1. This narrows #17963, which pinned the same exit as unverified on the grounds that not-found cannot separate "verified the pid is dead" from "my session map never had this id". Everything #17963 protects is untouched: `-1` still fails isProvenProcessExit, so the tab is not closed, the pane's leaf binding is not dropped on exit, and markUnverifiedPtyLoss still fires. Only the reconnect respawn gate reads the new field, and only for an id whose sole channel — the relay that answered — has disowned it, which no client can reach again under any verdict. That is the reading ssh-pty-relay-absence-verdict.test.ts already pins for the spawn path; the reconnect path now agrees with it. Rejected: parsing the relay's mint epoch out of `pty2:<epoch>:<n>`. It needs the current epoch on the wire (a capability-negotiated relay change), it has no answer for legacy `pty-N` ids, and a relay that comes back with zero PTYs gives the client no epoch to compare against. Rejected: clearing the leaf record outright, because the remote workspace snapshot re-hydrates those ids after the clear and the gate would refuse again. * refactor(ssh): name the relay-disowned signal for disownership, not exit
…very (stablyai#7661) * Robustify SSH terminal reconnect and session restore recovery - Release the replay guard after a fallback timeout to prevent permanent keyboard input lockouts when an unmounted terminal never parses. - Verify backing process liveness on PTY attach and reap stale entries so dead shells cleanly trigger a fresh pane spawn. - Normalize connection IDs during restore to prevent spurious mismatch errors, and treat true mismatches as expired sessions instead of crashing. * Route disconnected SSH terminal panes through deferred connection gate Avoid spawning SSH terminal processes against disconnected targets, which otherwise throws "No PTY provider" and leaves panes stranded. - Intercept spawning via a new connect gate that triggers the deferred connection flow when the SSH target is disconnected. - Fall back to composite worktree IDs during cold-start hydration to ensure deferred SSH session IDs are properly stashed. - Retry spawning and remounting terminal panes upon SSH reconnect if they are stranded or failed to spawn.
…tablyai#17957) * fix(ssh): stop respawning panes on client-side-only absence evidence Three respawn gates acted on evidence weaker than host-attested exit. Per docs/reference/ssh-execution-boundary.md, loss of contact, a failed reattach, an identity mismatch and absence from a client map are all `unverifiable`, never `exited`. Gate 1 (ipc-pty-connect.ts): "belongs to SSH connection" is minted by the id router from a pure client-side string compare, before any relay is asked, and still returned `sessionExpired: true` -> fresh PTY + agent resume. After an SSH target re-adoption the "other" connection is the same machine, so that puts a second `claude --resume` on the transcript the surviving PTY still owns. Now returns undefined with no error, which routes the pane to recoverUnverifiableDirectSshReattach (remount + reattach, no shell restart) and keeps stablyai#7661's no-red-toast outcome. Gate 3 (ssh-reconnect-pane-retry.ts): `!tabPtyId` read `tab.ptyId`, which is only the single-pane fallback for legacy attach. It diverges from the real records deterministically: workspace-terminal-reconnect fills ptyIdsByTabId from the leaf map but writes tab.ptyId only when a tab-level id survives, and clearTransientTerminalState nulls tab.ptyId on every hydrated row. Both leave live leaf PTYs with a null fallback field, arming a generation bump onto the fresh-spawn path. Now consults ptyIdsByTabId and the layout leaf map too; a tab with no PTY in any record still retries. Gate 2 (recoverTerminalPane): an `expired` lease plus `!pty.connected` authorized createTerminal. Every writer of `expired` records that the CLIENT lost its route, not that the shell died. Now also requires the runtime's own liveness verdict to be neither `live` nor `unverifiable`, and ssh-relay-session records markPtyLivenessLive at the persistPtyBinding refusal, which is reached only after pty.attach succeeded. See the report for why this branch is currently unreachable for SSH panes. * fix(ssh): let the respawn gate see the relay's own absence answer Gate 3 refused to respawn a pane whose records still named a PTY, which is right for a transport drop and wrong for a killed relay: after the relay is SIGKILLed and comes back, the leaf map still holds `pty2:<dead-epoch>:1` while the new relay answers that it has no such id. stablyai#18017's "replaces the pane only when the host proves the session is gone" regressed on exactly that. The gap was not the predicate, it was its inputs. `handlePtyReattachFailure` already distinguishes the three reattach outcomes and only its not-found branch publishes anything — a lost link and an identity mismatch send nothing. But it published `pty:exit { code: -1 }`, and `-1` is the sentinel every reader resolves to `stop_unverified`, so the one branch holding positive host evidence of absence arrived looking exactly like loss of contact. The renderer had no host answer at all, which the gate's own comment conceded. The exit now carries `livenessVerdict: 'exited'` beside the unchanged `-1`, so the code keeps meaning "no provable status" for every existing reader while the verdict rides its own field. A store bridge records those ids in `hostAttestedAbsentPtyIds` regardless of whether a pane is mounted to hear it — during reconnect none is — and the gate stops counting a recorded id the host has disowned. Settled when a PTY answers to that id again, because a redeployed relay renumbers from pty-1. This narrows stablyai#17963, which pinned the same exit as unverified on the grounds that not-found cannot separate "verified the pid is dead" from "my session map never had this id". Everything stablyai#17963 protects is untouched: `-1` still fails isProvenProcessExit, so the tab is not closed, the pane's leaf binding is not dropped on exit, and markUnverifiedPtyLoss still fires. Only the reconnect respawn gate reads the new field, and only for an id whose sole channel — the relay that answered — has disowned it, which no client can reach again under any verdict. That is the reading ssh-pty-relay-absence-verdict.test.ts already pins for the spawn path; the reconnect path now agrees with it. Rejected: parsing the relay's mint epoch out of `pty2:<epoch>:<n>`. It needs the current epoch on the wire (a capability-negotiated relay change), it has no answer for legacy `pty-N` ids, and a relay that comes back with zero PTYs gives the client no epoch to compare against. Rejected: clearing the leaf record outright, because the remote workspace snapshot re-hydrates those ids after the clear and the gate would refuse again. * refactor(ssh): name the relay-disowned signal for disownership, not exit
Summary
This PR improves remote connection terminal error handling, robustifies SSH terminal reconnect workflows, and fixes terminal session restore recovery issues:
ssh-pane-connect-gate.ts), ensuring connection is established before spawning rather than throwing "No PTY provider" errors on cold starts.replay-guard.ts). If xterm's parse callback never fires (e.g., when offscreen or unmounted on cold restore), the guard auto-releases to prevent permanent keystroke lockouts ("can't type after reconnecting").worktreesByRepoby falling back to the repo ID embedded in the composite worktree ID.Screenshots
No visual change.
Testing
pnpm lintpnpm typecheckpnpm testpnpm buildAI Review Report
I performed a comprehensive review of the changes:
isProcessAlivelogic using signal 0. OnlyESRCH(No such process) is treated as dead, while error codes likeEPERMare treated as alive, minimizing false-positive reaps.onParsedfrom firing.process.kill(pid, 0)is supported across Unix and Windows environments for existence checks. The composite worktree and connection routing logic is platform-agnostic and does not introduce path separators issues.Security Audit
process.kill(pid, 0)does not execute arbitrary shell commands and contains no injection risks.belongs to SSH connection) instead of letting uncaught rejections crash the renderer process.Notes