Skip to content

Improve remote connection terminal error msg and session restore recovery - #7661

Merged
AmethystLiang merged 3 commits into
mainfrom
improve-remote-connection-terminal-error-msg
Jul 7, 2026
Merged

AmethystLiang merged 3 commits into
mainfrom
improve-remote-connection-terminal-error-msg

Conversation

@AmethystLiang

Copy link
Copy Markdown
Contributor

Summary

This PR improves remote connection terminal error handling, robustifies SSH terminal reconnect workflows, and fixes terminal session restore recovery issues:

  • Deferred Connection Routing: Disconnected SSH terminal panes now route through a deferred connection gate (ssh-pane-connect-gate.ts), ensuring connection is established before spawning rather than throwing "No PTY provider" errors on cold starts.
  • Input Lockout Prevention: Adds a 1000ms fallback timeout to the replay guard (replay-guard.ts). If xterm's parse callback never fires (e.g., when offscreen or unmounted on cold restore), the guard auto-releases to prevent permanent keystroke lockouts ("can't type after reconnecting").
  • Stale Connection Recovery: Handles cross-connection SSH reattaches as expired sessions rather than raising critical error toasts, allowing a fresh spawn recovery.
  • Cold-Start Hydration Fallback: Resolves cold-start restore failures where SSH worktrees are not yet available in worktreesByRepo by falling back to the repo ID embedded in the composite worktree ID.
  • PTY Liveness Verification: Proactively reaps dead backing shells from the PTY handler cache via a conservative signal 0 existence probe.

Screenshots

No visual change.

Testing

  • pnpm lint
  • pnpm typecheck
  • pnpm test
  • pnpm build
  • Added or updated high-quality tests that would catch regressions, or explained why tests were not needed

AI Review Report

I performed a comprehensive review of the changes:

  • Process Liveness Checks: Verified isProcessAlive logic using signal 0. Only ESRCH (No such process) is treated as dead, while error codes like EPERM are treated as alive, minimizing false-positive reaps.
  • Replay Guard Robustness: Analyzed the 1000ms fallback. It safely resolves potential offscreen/unmounted rendering delays preventing onParsed from firing.
  • Cross-Platform Compatibility: Checked macOS, Linux, and Windows compatibility. Node's process.kill(pid, 0) is supported across Unix and Windows environments for existence checks. The composite worktree and connection routing logic is platform-agnostic and does not introduce path separators issues.

Security Audit

  • Process Signaling: Verified process.kill(pid, 0) does not execute arbitrary shell commands and contains no injection risks.
  • Error Propagation: IPC transport gracefully catches and translates connection mismatch errors (belongs to SSH connection) instead of letting uncaught rejections crash the renderer process.
  • Data Leakage: Confirming no sensitive data or credentials from SSH configurations are printed in console warnings or logs.

Notes

  • The 1000ms replay guard fallback is chosen as a trade-off to ensure input is never permanently swallowed, which is strictly preferable to a permanent lockout.

- Release the replay guard after a fallback timeout to prevent permanent
  keyboard input lockouts when an unmounted terminal never parses.
- Verify backing process liveness on PTY attach and reap stale entries
  so dead shells cleanly trigger a fresh pane spawn.
- Normalize connection IDs during restore to prevent spurious mismatch
  errors, and treat true mismatches as expired sessions instead of crashing.
Avoid spawning SSH terminal processes against disconnected targets,
which otherwise throws "No PTY provider" and leaves panes stranded.

- Intercept spawning via a new connect gate that triggers the deferred
  connection flow when the SSH target is disconnected.
- Fall back to composite worktree IDs during cold-start hydration to
  ensure deferred SSH session IDs are properly stashed.
- Retry spawning and remounting terminal panes upon SSH reconnect if
  they are stranded or failed to spawn.
@AmethystLiang
AmethystLiang merged commit aa0993f into main Jul 7, 2026
2 checks passed
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR hardens SSH terminal reconnection and PTY reattachment. On the relay side, PtyHandler now verifies the backing process is alive via a new isProcessAlive helper before reattaching, reaping stale PTYs otherwise. A new resolveSshPaneConnectGate resolver and shouldRetryPaneSpawnOnSshReconnect predicate govern deferred SSH connect flows and pane remount retries, replacing prior ad-hoc checks in pty-connection.ts and useIpcEvents.ts. pty-transport.ts now recovers from cross-connection PTY mismatches as session-expired. replay-guard.ts adds a bounded fallback release to prevent stuck replay counters. ssh-pty-id.ts normalizes execution-host style connection ids, and terminals.ts adds a repoId fallback for SSH worktrees missing from state during hydration. Extensive tests accompany each change.

Changes

Cohort / Area Changes
Relay PTY liveness Added isProcessAlive; PtyHandler.attach reaps dead PTYs before reattach; new tests for dead/alive backing shells
SSH pty id normalization Added normalizeConnectionId; applied in toAppSshPtyId/toRelaySshPtyId; new routing tests
Replay guard Added engageReplayGuard with fallback timeout; rewired replayIntoTerminal/replayIntoTerminalAsync; new tests
SSH pane connect gate & retry Added resolveSshPaneConnectGate and shouldRetryPaneSpawnOnSshReconnect; wired into pty-connection.ts and useIpcEvents.ts; pty-transport.ts handles cross-connection mismatch as session-expired; terminals.ts adds repoId fallback for deferred SSH sessions; extensive supporting tests

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant PtyHandler
  participant isProcessAlive

  Client->>PtyHandler: attach(id, paneKey)
  PtyHandler->>isProcessAlive: check(managed.pty.pid)
  isProcessAlive-->>PtyHandler: false
  PtyHandler->>PtyHandler: notify exit listener + dispose + remove
  PtyHandler-->>Client: throw PTY not found
Loading
sequenceDiagram
  participant PtyConnection
  participant resolveSshPaneConnectGate
  participant SshConnect

  PtyConnection->>resolveSshPaneConnectGate: input(sshStatus, sessionIds, tabPtyId)
  resolveSshPaneConnectGate-->>PtyConnection: gate(pendingSessionId, enterDeferredFlow, sshConnected)
  alt gate.enterDeferredFlow
    PtyConnection->>SshConnect: enter deferred reconnect flow
  else
    PtyConnection->>PtyConnection: proceed to spawn
  end
Loading

Related PRs: None identified from the provided summaries.

Suggested labels: area: relay, area: terminal, area: ssh, tests

Suggested reviewers: No reviewer information available from provided context.

Poem
A rabbit checks each pty's pulse with care,
"Are you alive?" it asks the phantom pair.
If dead, it reaps; if breathing, lets it stay,
While gates and guards keep reconnects at bay.
Hop by hop, the SSH tunnels mend —
🐇 stable sessions, end to end.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the PR's main focus on remote terminal error handling and session recovery.
Description check ✅ Passed The description follows the required sections and includes summary, screenshots, testing, AI review, security audit, and notes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch improve-remote-connection-terminal-error-msg

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
src/relay/pty-handler.ts (1)

900-905: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

revive() liveness check uses different (inconsistent) semantics than the new isProcessAlive.

revive()'s inline process.kill(entry.pid, 0) treats any thrown error (including EPERM) as "not alive" and skips reviving the entry, whereas the new isProcessAlive explicitly treats non-ESRCH errors like EPERM as alive. A revived-but-unsignalable process would be silently dropped here while an attach-reaped one wouldn't. Consider reusing isProcessAlive in revive() for consistent liveness semantics across both code paths.

♻️ Proposed fix
-      // Only re-attach if the original process is still alive
-      try {
-        process.kill(entry.pid, 0)
-      } catch {
-        continue
-      }
+      // Only re-attach if the original process is still alive
+      if (!isProcessAlive(entry.pid)) {
+        continue
+      }
src/shared/ssh-pty-id.ts (1)

46-67: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider normalizing the embedded parsed.connectionId too, for symmetry.

Both functions normalize the incoming connectionId parameter before comparing, but compare it against parsed.connectionId — the connection id embedded inside an existing pty id — without normalizing that side. If any app pty id was ever created by encoding an execution-host-form id (e.g. before this fix landed, or via some other caller passing "ssh:<target>"), its embedded connectionId would be "ssh:<target>" rather than the bare target id, and a legitimate reattach from a normalized bare id would spuriously fail the equality check.

The blast radius is soft today — pty-transport.ts's new SSH_PTY_CONNECTION_MISMATCH_MARKER handling turns that spurious mismatch into a sessionExpired recovery rather than a crash — but it would still mean losing reattach/scrollback continuity unnecessarily for a session that is actually valid.

🛡️ Proposed fix for symmetric normalization
 export function toAppSshPtyId(connectionId: string, relayPtyId: string): string {
   const normalizedConnectionId = normalizeConnectionId(connectionId)
   const parsed = parseAppSshPtyId(relayPtyId)
   if (parsed) {
-    if (parsed.connectionId !== normalizedConnectionId) {
+    if (normalizeConnectionId(parsed.connectionId) !== normalizedConnectionId) {
       throw new Error(`PTY ${relayPtyId} belongs to SSH connection "${parsed.connectionId}"`)
     }
     return relayPtyId
   }
   return `${SSH_PTY_ID_PREFIX}${encodeURIComponent(normalizedConnectionId)}${SSH_PTY_ID_SEPARATOR}${relayPtyId}`
 }

 export function toRelaySshPtyId(connectionId: string, ptyId: string): string {
   const parsed = parseAppSshPtyId(ptyId)
   if (!parsed) {
     return ptyId
   }
-  if (parsed.connectionId !== normalizeConnectionId(connectionId)) {
+  if (normalizeConnectionId(parsed.connectionId) !== normalizeConnectionId(connectionId)) {
     throw new Error(`PTY ${ptyId} belongs to SSH connection "${parsed.connectionId}"`)
   }
   return parsed.relayPtyId
 }

Worth confirming whether any code path could have produced (or could still produce, from another caller not in this cohort) an app pty id with a host-id-form connection embedded, to gauge if this is a live concern or purely theoretical.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 479ff287-3682-46e5-9478-1c4bcb3f5fe2

📥 Commits

Reviewing files that changed from the base of the PR and between 6a4e6ce and ba4934c.

📒 Files selected for processing (19)
  • src/relay/pty-handler.test.ts
  • src/relay/pty-handler.ts
  • src/relay/pty-shell-utils.test.ts
  • src/relay/pty-shell-utils.ts
  • src/renderer/src/components/terminal-pane/pty-connection.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection.ts
  • src/renderer/src/components/terminal-pane/pty-transport.test.ts
  • src/renderer/src/components/terminal-pane/pty-transport.ts
  • src/renderer/src/components/terminal-pane/replay-guard.test.ts
  • src/renderer/src/components/terminal-pane/replay-guard.ts
  • src/renderer/src/components/terminal-pane/ssh-pane-connect-gate.test.ts
  • src/renderer/src/components/terminal-pane/ssh-pane-connect-gate.ts
  • src/renderer/src/hooks/ssh-reconnect-pane-retry.test.ts
  • src/renderer/src/hooks/ssh-reconnect-pane-retry.ts
  • src/renderer/src/hooks/useIpcEvents.ts
  • src/renderer/src/store/slices/terminals-hydration.test.ts
  • src/renderer/src/store/slices/terminals.ts
  • src/shared/ssh-pty-id.test.ts
  • src/shared/ssh-pty-id.ts

Comment on lines +33 to +39
const fallbackTabSessionId =
!sshConnected &&
!input.hasLeafSessionMap &&
input.tabPtyId &&
parseAppSshPtyId(input.tabPtyId)?.connectionId === input.connectionId
? input.tabPtyId
: null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect normalizeConnectionId's definition/usage and getConnectionId's normalization.
fd 'ssh-pty-id.ts$' src/shared
cat -n src/shared/ssh-pty-id.ts
rg -n 'normalizeConnectionId' -C3 src
rg -n 'function getConnectionId' -C10 src

Repository: stablyai/orca

Length of output: 10013


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the connectionId resolution path and the caller that feeds ssh-pane-connect-gate.
cat -n src/renderer/src/lib/connection-context.ts | sed -n '1,220p'
echo '--- folder-workspace-connection.ts ---'
cat -n src/renderer/src/lib/folder-workspace-connection.ts | sed -n '1,260p'
echo '--- pty-connection.ts ---'
cat -n src/renderer/src/components/terminal-pane/pty-connection.ts | sed -n '1,260p'

Repository: stablyai/orca

Length of output: 24034


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Trace where repo/folder-workspace/project-group connectionId values are written.
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
  'connectionId\s*[:=]' src shared
echo '--- repo type ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
  'type Repo|interface Repo|connectionId\??:' src/shared src/renderer/src src/main
echo '--- folder workspace type ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
  'type FolderWorkspace|interface FolderWorkspace|connectionId\??:' src/shared src/renderer/src src/main
echo '--- project group type ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
  'type ProjectGroup|interface ProjectGroup|connectionId\??:' src/shared src/renderer/src src/main

Repository: stablyai/orca

Length of output: 50370


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- ssh-pane-connect-gate.ts ---'
cat -n src/renderer/src/components/terminal-pane/ssh-pane-connect-gate.ts | sed -n '1,120p'

echo '--- shared/project-groups.ts ---'
cat -n src/shared/project-groups.ts | sed -n '1,140p'

echo '--- shared/folder-workspaces.ts ---'
cat -n src/shared/folder-workspaces.ts | sed -n '1,150p'

echo '--- shared/execution-host.ts ---'
cat -n src/shared/execution-host.ts | sed -n '130,190p'

Repository: stablyai/orca

Length of output: 16278


Compare normalized connection IDs here. parseAppSshPtyId(...) returns the decoded id, but input.connectionId comes from stored workspace/repo metadata without the same normalization, so an execution-host-style id can miss the fallback reattach and spawn a fresh shell.

Jinwoo-H added a commit that referenced this pull request Jul 7, 2026
46 commits from main. Conflict resolutions (branch structure wins, main
semantics grafted):

- pending-backlog cap (#7630): main's capPendingPtyData/droppedBacklog
  2MB tail-trim EXCISED — this branch's droppedOutput sentinel mechanism
  (O(1) drop + query salvage + settings-scaled cap + restore marker)
  already bounds the unsent backlog and strictly supersedes it; carrying
  both would race two cap policies over one buffer. Kept main's
  makePtyDataPayload helper and exit-time pending flush (adapted to the
  sentinel). Main's duplicate never-ACK cap test removed (covered by the
  existing sentinel cap tests); its reload-gate tests kept with the
  runtime mock extended.
- replay guard (#7661): main's blind REPLAY_GUARD_RELEASE_FALLBACK_MS
  release dropped — this branch's probe-certified stall release covers
  the same cold-restore reattach lockout without the auto-reply leak a
  timed release can cause in agent TUIs. Main's three regression tests
  KEPT, adapted to probe timing (explicit stallCheckMs).
- hidden-pressure scenario: kept this branch's STRICT 1500ms restore
  budget (the background keep-tail global budget earns it; main relaxed
  to 4s for CI drain overhead) + grafted main's timer-drift gate const.
- ssh-docker-relay-perf: kept this branch's ACK-stalled background PTY
  test AND grafted main's new file-streams+git-churn test (4 tests).
- preload/renderer: droppedBacklog threading excised with the mechanism;
  main's unrelated changes (ssh importConfig reAdopt, etc.) kept.

Gates: typecheck clean; ipc+daemon+terminal-pane+pane-manager suites
4,396 passed / 14 skipped.

Co-authored-by: Orca <help@stably.ai>
nwparker added a commit that referenced this pull request Sep 3, 2026
Three respawn gates acted on evidence weaker than host-attested exit.
Per docs/reference/ssh-execution-boundary.md, loss of contact, a failed
reattach, an identity mismatch and absence from a client map are all
`unverifiable`, never `exited`.

Gate 1 (ipc-pty-connect.ts): "belongs to SSH connection" is minted by the
id router from a pure client-side string compare, before any relay is
asked, and still returned `sessionExpired: true` -> fresh PTY + agent
resume. After an SSH target re-adoption the "other" connection is the
same machine, so that puts a second `claude --resume` on the transcript
the surviving PTY still owns. Now returns undefined with no error, which
routes the pane to recoverUnverifiableDirectSshReattach (remount +
reattach, no shell restart) and keeps #7661's no-red-toast outcome.

Gate 3 (ssh-reconnect-pane-retry.ts): `!tabPtyId` read `tab.ptyId`, which
is only the single-pane fallback for legacy attach. It diverges from the
real records deterministically: workspace-terminal-reconnect fills
ptyIdsByTabId from the leaf map but writes tab.ptyId only when a
tab-level id survives, and clearTransientTerminalState nulls tab.ptyId on
every hydrated row. Both leave live leaf PTYs with a null fallback field,
arming a generation bump onto the fresh-spawn path. Now consults
ptyIdsByTabId and the layout leaf map too; a tab with no PTY in any
record still retries.

Gate 2 (recoverTerminalPane): an `expired` lease plus `!pty.connected`
authorized createTerminal. Every writer of `expired` records that the
CLIENT lost its route, not that the shell died. Now also requires the
runtime's own liveness verdict to be neither `live` nor `unverifiable`,
and ssh-relay-session records markPtyLivenessLive at the persistPtyBinding
refusal, which is reached only after pty.attach succeeded. See the report
for why this branch is currently unreachable for SSH panes.
nwparker added a commit that referenced this pull request Sep 3, 2026
…17957)

* fix(ssh): stop respawning panes on client-side-only absence evidence

Three respawn gates acted on evidence weaker than host-attested exit.
Per docs/reference/ssh-execution-boundary.md, loss of contact, a failed
reattach, an identity mismatch and absence from a client map are all
`unverifiable`, never `exited`.

Gate 1 (ipc-pty-connect.ts): "belongs to SSH connection" is minted by the
id router from a pure client-side string compare, before any relay is
asked, and still returned `sessionExpired: true` -> fresh PTY + agent
resume. After an SSH target re-adoption the "other" connection is the
same machine, so that puts a second `claude --resume` on the transcript
the surviving PTY still owns. Now returns undefined with no error, which
routes the pane to recoverUnverifiableDirectSshReattach (remount +
reattach, no shell restart) and keeps #7661's no-red-toast outcome.

Gate 3 (ssh-reconnect-pane-retry.ts): `!tabPtyId` read `tab.ptyId`, which
is only the single-pane fallback for legacy attach. It diverges from the
real records deterministically: workspace-terminal-reconnect fills
ptyIdsByTabId from the leaf map but writes tab.ptyId only when a
tab-level id survives, and clearTransientTerminalState nulls tab.ptyId on
every hydrated row. Both leave live leaf PTYs with a null fallback field,
arming a generation bump onto the fresh-spawn path. Now consults
ptyIdsByTabId and the layout leaf map too; a tab with no PTY in any
record still retries.

Gate 2 (recoverTerminalPane): an `expired` lease plus `!pty.connected`
authorized createTerminal. Every writer of `expired` records that the
CLIENT lost its route, not that the shell died. Now also requires the
runtime's own liveness verdict to be neither `live` nor `unverifiable`,
and ssh-relay-session records markPtyLivenessLive at the persistPtyBinding
refusal, which is reached only after pty.attach succeeded. See the report
for why this branch is currently unreachable for SSH panes.

* fix(ssh): let the respawn gate see the relay's own absence answer

Gate 3 refused to respawn a pane whose records still named a PTY, which is
right for a transport drop and wrong for a killed relay: after the relay is
SIGKILLed and comes back, the leaf map still holds `pty2:<dead-epoch>:1` while
the new relay answers that it has no such id. #18017's "replaces the pane only
when the host proves the session is gone" regressed on exactly that.

The gap was not the predicate, it was its inputs. `handlePtyReattachFailure`
already distinguishes the three reattach outcomes and only its not-found branch
publishes anything — a lost link and an identity mismatch send nothing. But it
published `pty:exit { code: -1 }`, and `-1` is the sentinel every reader
resolves to `stop_unverified`, so the one branch holding positive host evidence
of absence arrived looking exactly like loss of contact. The renderer had no
host answer at all, which the gate's own comment conceded.

The exit now carries `livenessVerdict: 'exited'` beside the unchanged `-1`, so
the code keeps meaning "no provable status" for every existing reader while the
verdict rides its own field. A store bridge records those ids in
`hostAttestedAbsentPtyIds` regardless of whether a pane is mounted to hear it —
during reconnect none is — and the gate stops counting a recorded id the host
has disowned. Settled when a PTY answers to that id again, because a redeployed
relay renumbers from pty-1.

This narrows #17963, which pinned the same exit as unverified on the grounds
that not-found cannot separate "verified the pid is dead" from "my session map
never had this id". Everything #17963 protects is untouched: `-1` still fails
isProvenProcessExit, so the tab is not closed, the pane's leaf binding is not
dropped on exit, and markUnverifiedPtyLoss still fires. Only the reconnect
respawn gate reads the new field, and only for an id whose sole channel — the
relay that answered — has disowned it, which no client can reach again under
any verdict. That is the reading ssh-pty-relay-absence-verdict.test.ts already
pins for the spawn path; the reconnect path now agrees with it.

Rejected: parsing the relay's mint epoch out of `pty2:<epoch>:<n>`. It needs the
current epoch on the wire (a capability-negotiated relay change), it has no
answer for legacy `pty-N` ids, and a relay that comes back with zero PTYs gives
the client no epoch to compare against. Rejected: clearing the leaf record
outright, because the remote workspace snapshot re-hydrates those ids after the
clear and the gate would refuse again.

* refactor(ssh): name the relay-disowned signal for disownership, not exit
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…very (stablyai#7661)

* Robustify SSH terminal reconnect and session restore recovery

- Release the replay guard after a fallback timeout to prevent permanent
  keyboard input lockouts when an unmounted terminal never parses.
- Verify backing process liveness on PTY attach and reap stale entries
  so dead shells cleanly trigger a fresh pane spawn.
- Normalize connection IDs during restore to prevent spurious mismatch
  errors, and treat true mismatches as expired sessions instead of crashing.

* Route disconnected SSH terminal panes through deferred connection gate

Avoid spawning SSH terminal processes against disconnected targets,
which otherwise throws "No PTY provider" and leaves panes stranded.

- Intercept spawning via a new connect gate that triggers the deferred
  connection flow when the SSH target is disconnected.
- Fall back to composite worktree IDs during cold-start hydration to
  ensure deferred SSH session IDs are properly stashed.
- Retry spawning and remounting terminal panes upon SSH reconnect if
  they are stranded or failed to spawn.
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…tablyai#17957)

* fix(ssh): stop respawning panes on client-side-only absence evidence

Three respawn gates acted on evidence weaker than host-attested exit.
Per docs/reference/ssh-execution-boundary.md, loss of contact, a failed
reattach, an identity mismatch and absence from a client map are all
`unverifiable`, never `exited`.

Gate 1 (ipc-pty-connect.ts): "belongs to SSH connection" is minted by the
id router from a pure client-side string compare, before any relay is
asked, and still returned `sessionExpired: true` -> fresh PTY + agent
resume. After an SSH target re-adoption the "other" connection is the
same machine, so that puts a second `claude --resume` on the transcript
the surviving PTY still owns. Now returns undefined with no error, which
routes the pane to recoverUnverifiableDirectSshReattach (remount +
reattach, no shell restart) and keeps stablyai#7661's no-red-toast outcome.

Gate 3 (ssh-reconnect-pane-retry.ts): `!tabPtyId` read `tab.ptyId`, which
is only the single-pane fallback for legacy attach. It diverges from the
real records deterministically: workspace-terminal-reconnect fills
ptyIdsByTabId from the leaf map but writes tab.ptyId only when a
tab-level id survives, and clearTransientTerminalState nulls tab.ptyId on
every hydrated row. Both leave live leaf PTYs with a null fallback field,
arming a generation bump onto the fresh-spawn path. Now consults
ptyIdsByTabId and the layout leaf map too; a tab with no PTY in any
record still retries.

Gate 2 (recoverTerminalPane): an `expired` lease plus `!pty.connected`
authorized createTerminal. Every writer of `expired` records that the
CLIENT lost its route, not that the shell died. Now also requires the
runtime's own liveness verdict to be neither `live` nor `unverifiable`,
and ssh-relay-session records markPtyLivenessLive at the persistPtyBinding
refusal, which is reached only after pty.attach succeeded. See the report
for why this branch is currently unreachable for SSH panes.

* fix(ssh): let the respawn gate see the relay's own absence answer

Gate 3 refused to respawn a pane whose records still named a PTY, which is
right for a transport drop and wrong for a killed relay: after the relay is
SIGKILLed and comes back, the leaf map still holds `pty2:<dead-epoch>:1` while
the new relay answers that it has no such id. stablyai#18017's "replaces the pane only
when the host proves the session is gone" regressed on exactly that.

The gap was not the predicate, it was its inputs. `handlePtyReattachFailure`
already distinguishes the three reattach outcomes and only its not-found branch
publishes anything — a lost link and an identity mismatch send nothing. But it
published `pty:exit { code: -1 }`, and `-1` is the sentinel every reader
resolves to `stop_unverified`, so the one branch holding positive host evidence
of absence arrived looking exactly like loss of contact. The renderer had no
host answer at all, which the gate's own comment conceded.

The exit now carries `livenessVerdict: 'exited'` beside the unchanged `-1`, so
the code keeps meaning "no provable status" for every existing reader while the
verdict rides its own field. A store bridge records those ids in
`hostAttestedAbsentPtyIds` regardless of whether a pane is mounted to hear it —
during reconnect none is — and the gate stops counting a recorded id the host
has disowned. Settled when a PTY answers to that id again, because a redeployed
relay renumbers from pty-1.

This narrows stablyai#17963, which pinned the same exit as unverified on the grounds
that not-found cannot separate "verified the pid is dead" from "my session map
never had this id". Everything stablyai#17963 protects is untouched: `-1` still fails
isProvenProcessExit, so the tab is not closed, the pane's leaf binding is not
dropped on exit, and markUnverifiedPtyLoss still fires. Only the reconnect
respawn gate reads the new field, and only for an id whose sole channel — the
relay that answered — has disowned it, which no client can reach again under
any verdict. That is the reading ssh-pty-relay-absence-verdict.test.ts already
pins for the spawn path; the reconnect path now agrees with it.

Rejected: parsing the relay's mint epoch out of `pty2:<epoch>:<n>`. It needs the
current epoch on the wire (a capability-negotiated relay change), it has no
answer for legacy `pty-N` ids, and a relay that comes back with zero PTYs gives
the client no epoch to compare against. Rejected: clearing the leaf record
outright, because the remote workspace snapshot re-hydrates those ids after the
clear and the gate would refuse again.

* refactor(ssh): name the relay-disowned signal for disownership, not exit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant