Skip to content

fix(resume): a paired host resumes each pane's agent after its terminal is lost - #30

Merged
kiwidream merged 11 commits into
swaplabs/mainfrom
fix/host-resume-after-terminal-loss
Sep 30, 2026
Merged

kiwidream merged 11 commits into
swaplabs/mainfrom
fix/host-resume-after-terminal-loss

Conversation

@kiwidream

@kiwidream kiwidream commented Sep 30, 2026 •

Copy link
Copy Markdown
Files Added Deleted Net
Test 19 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​2119 $\color{#cf222e}{\Huge{\mathbf{−}}}$​16 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​2103
Prod 38 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​931 $\color{#cf222e}{\Huge{\mathbf{−}}}$​142 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​789

ELI5

When a paired Orca server loses its terminals, through a reboot or its terminal daemon being replaced, every pane comes back as an empty shell and you have to type claude --resume and pick the conversation yourself. The server already knows which conversation each pane was running. Now it uses that to start claude --resume <that conversation> in the same pane by itself.

What Changed

Before: after a host reboot, or after the terminal daemon was replaced while orca serve kept running, each pane on a paired host came back as a bare bash prompt, and mouse movement printed 35;12;40M… into it. None of the client-side resume paths apply to panes on a remote host, and the host re-created the pane with no command (hasCommand:false in daemon.log).

After: the host re-creates the pane with claude <your default args> --resume <the pane's session id>, once per conversation, and the pane's stale mouse, focus and paste modes are cleared.

How:

  1. The host keeps the resume identity through terminal loss (agent-hooks). Losing a terminal because the daemon or host went away no longer deletes the pane's providerSession: the boot sweep and PTY-exit cleanup keep a terminal-loss remnant. Only the agent ending itself (Claude SessionEnd with prompt_input_exit or logout), a different session in the pane, or closing the tab clears it.
  2. The host resumes on materialize (orca-runtime-perform-mobile-session-pty-records-refresh.ts, host-pane-agent-resume.ts). When session.tabs.activate re-creates a pane whose terminal is gone, it builds claude … --resume <id> with buildAgentResumeStartupPlan from the host's own agent settings, as ensureAgentSession does.
    • Right before spawning, it re-checks that the tab still exists, that the pane still has no live terminal, and that the session is the same one.
    • It claims the conversation so it is resumed once. Concurrent activations share a single spawn.
    • A failed resume falls back to a plain shell and keeps the identity for next time.
    • SSH workspaces are excluded.
  3. Request order no longer decides which tabs exist (orca-runtime-hydrate-headless-mobile-session-tabs-from-workspace-session.ts). On a freshly started host, a tab-list request that arrived before activate built a runtime-only snapshot, and dropped tabs the client had created (how Claude panes are made from the Mac). The client then retired those panes. The next full hydrate now merges in the persisted tabs it would have listed. This only affects snapshots the host built itself.
  4. Mode reset on a replacement process (renderer remote transport). When the transport resubscribes a pane to a replacement session, it writes the same POST_REPLAY_MODE_RESET the cold-restore path writes.

Why

The host owns the daemon, the transcripts and the agent-status store, so it is the only side that knows a pane's terminal is gone and which conversation it ran. That fits docs/reference/ssh-execution-boundary.md and agent-status-store.md. It needs no protocol change, and it works whatever version the desktop is, including mobile and web clients. The client-side resume paths (#1, #6, #9, #11) all bail out for remote panes by design: a remote pane arrives as a reattach, and there is no RPC for uploading resume records. A proof-of-concept patch at this host point made a reproduction resume (root cause: task_f306faff5eff).

launchAgent is deliberately not persisted on the durable tab row. Doing so would re-arm the fresh-launch path (stablyai#7587) after the host's 7-day status window, and start a new Claude where the old conversation can no longer be resumed. The agent icon returns with the resumed agent's own hook row.

Linked Issue

None (fork). User reports on 2026-09-24, 09-28 and 09-29: the pane comes back as a shell after a restart.

Visual Proof

N/A: the change shows up as which command a re-created pane runs, and that is asserted by argv capture in e2e (below), not by visible UI.

Testing

New e2e spec tests/e2e/paired-host-agent-resume-after-terminal-loss.spec.ts. It runs an isolated headless paired host and a hidden Electron client (Xvfb, ORCA_BACKGROUND_LAUNCH=1), with a fake claude that records argv and posts real Claude hooks. 10/10 pass:

Journey Result
Reboot, agent idle, activate first same pane relaunched with --resume <id>; mouse tracking off
Reboot, agent idle, tab-list first same
Reboot, agent working --resume <id>
Daemon replaced, serve up, idle and working --resume <id> on the pane's next activation; new daemon pid asserted
Serve restart, daemon survives reattach; one launch only, original process alive
User ended Claude (SessionEnd) before the reboot plain shell, no resume
Tab closed before the reboot nothing resumes
Two panes in one worktree each resumes its own id
Two clients reconnecting at once exactly one resume
  • Repeats: reboot/idle in both request orders, and two clients, each passed 3/3 with --repeat-each=3.

  • Negative control: the same journeys fail on the untouched base 64dea21a91; there is no --resume within 180 s.

  • Unit tests: added for the hook-store remnant and SessionEnd, host resume planning (argv pinned to claude '--dangerously-skip-permissions' '--model' 'opus' '--effort' 'max' '--resume' '<id>'), revalidation, single-flight, and both hydrate orders.

  • Gates: pnpm tc, oxlint and check:code-quality:changed are clean. The remaining pnpm test failures reproduce identically on base.

  • Not run: a real Mac client (the client was the same renderer under Xvfb), and the real TeamClaude claude wrapper (argv form unit-pinned). The host was Linux only; Windows and WSL hosts use the shared quoting helpers.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

AI Disclosure

Claude Opus 5.5 implemented and tested this as a delegated worker, and Claude Opus 5.5 coordinated and reviewed it. Root cause from a Fable 5.1 investigation.

Review

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Mixed versions: the WSL relay adds two optional agent.hook fields, and there is no new opcode. A new relay stamps reportsUserEndedSessions and forwards a user-ended SessionEnd as a userEndedSessionId notice, which an older host drops. An older relay's WSL rows are never auto-resumed. Old clients still get the host resume. A new client talking to an old host behaves as before.
  • When it takes effect: on a host, only after the terminal daemon is recycled, because the running daemon and main keep the old code until restarted.
  • Follow-up, not in this PR: tabs whose ptyId is in daemon form are still dropped by the runtime-only ownership predicate after a plain serve restart. The order fix covers the resume path. Root cause and the proposed predicate are in the worker report.

Deferred review items (follow-up)

  • Abort a host resume re-creation when pane revalidation fails during the workspace await. It should return a distinct stale result, not fall back to a plain-shell create (Codex P2, orca-runtime-perform-mobile-session-pty-records-refresh.ts:245).
  • Announce replacements created by terminal.recoverPane, so the input-mode reset also runs on the SSH expired-attachment path (Codex P2, remote-runtime-pty-transport.ts:1621).
  • After the daemon dies under a running host, tell subscribed clients so idle panes re-activate without the user. Today the pane resumes on its next activation (journey 4).
  • Ownership predicate for daemon-form tabs after a plain serve restart (see Notes).
  • Keep the resume identity when a pane's aliases are cleared before PTY-exit retention, so legacy or reminted panes keep it (Codex P2, server-tab-cleanup.ts:117).
  • Let a newer ended-session remnant in another pane block auto-resuming the same session in an older pane (Codex P2, server-retained-identity.ts:61).
  • Accept a same-handle replacement with a new incarnation during require-replacement recovery (Codex P2, remote-runtime-pty-transport.ts:1819). The terminal.recoverPane mode grounding (Tenki, medium) is the same area as the recoverPane item above.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

kiwidream and others added 5 commits September 30, 2026 14:38
A paired host re-creates a pane after a reboot or a daemon replacement,
but PTY teardown deleted the pane's status row and the agent's provider
session with it, so nothing could resume the agent there. Teardown now
keeps that identity as a hidden retained row marked resumable after
terminal loss, and the boot sweep does the same for a working row whose
PTY died while Orca was down. A Claude SessionEnd with a user reason
(prompt_input_exit, logout) keeps the identity for a manual resume only.
A retained row no longer lends its agent type, permission hold or
compact ownership to the next event in the pane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a paired client re-activates a pane whose terminal is gone, the
host re-created it as a bare shell: the only launch it resolved was a
fresh one from tab.launchAgent, which the persisted tab row lacks. The
host now asks its own agent-status store for the session that pane lost
and, when the agent never ended, launches it with --resume using its own
agent command, default args and env, as ensureAgentSession does. It
revalidates the pane, its binding and the session after every await,
claims the session so no other re-creation can resume it, shares one
re-creation between concurrent activations, and falls back to a plain
shell (keeping the identity) when the resume spawn is refused. The
fresh launch for a pending agent tab now applies only when the agent
never reported in that pane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sh host first

A fresh headless host builds its first snapshot from whichever request
arrives first. A tab list or subscribe hydrates runtime-only, which keeps
only serve and SSH ids, and the full hydrate that follows skipped the
non-empty snapshot. A pane a paired client created binds a daemon-form id,
so after a reboot or a serve restart it was never listed, its activation
failed with tab_not_found, and the client ended the pane instead of
resuming it. The next full hydrate now completes a snapshot a runtime-only
hydrate built first. Only headless-built snapshots are touched; renderer
publications and the ownership predicate are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its replacement

A remote pane that recovers onto a replacement process kept the modes
the dead TUI had armed, so mouse movement reached the new shell as SGR
reports. The transport's recovery now tells the pane when the process
incarnation behind its handle changed, before it subscribes to the
replacement, and the pane writes the same reset a cold restore writes.
A re-minted handle for the same live process keeps its modes, and a host
that publishes no incarnation changes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… end to end

Adds a paired headless host and hidden desktop client journey with a
fake claude that records every launch's argv and posts real Claude
hooks. It covers a host reboot (idle, mid-turn, and both first-request
orders), a replaced daemon under a running host, a serve restart over a
surviving daemon, an agent the user ended, a closed tab, two panes with
their own sessions, and two clients reconnecting at once, and checks the
resumed pane's xterm no longer reports the mouse. The host helper takes
spec-scoped env; the daemon pid reader is exported for reuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T20:33:33.450453Z 30b0706 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 366a505c9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ale SessionEnd

- The host's resume claim now uses the same provider identity as
  agentProviderSessionsEqual (shared agentProviderSessionIdentityKey), so two
  Pi-family sessions that share an id but not a transcript both resume. The
  renderer's claim key reuses the helper.
- A SessionEnd whose launch token differs from the pane's current row (a late
  hook from the process a resume replaced) no longer marks the resumed
  session as user-ended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99750fe138

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A WSL relay stamps its hook rows with connectionId "wsl:<distro>", so the
local-only check treated every WSL session as SSH and a lost WSL terminal
came back as a plain shell. Only a genuinely remote (SSH) route is excluded
now; the workspace check already reports WSL as local.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tenki-reviewer

tenki-reviewer Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review complete. No issues found — approved ✅.


Splits the agent-hooks server into focused modules and adds agent auto-resume after remote terminal loss, with a paired-host e2e journey.

Reviewed commit: abf8338

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: abf833809c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tenki-reviewer[bot]
tenki-reviewer Bot previously approved these changes Sep 30, 2026
kiwidream and others added 4 commits September 30, 2026 16:25
Brings the branch up to the fork tip 0727f8e so CI tests the merge result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream stablyai#23744 removed runtime.markWorkspaceTrustedForAgent, which the host
resume called while preparing its launch. On the merged tree the call threw
and every resume fell back to a plain shell. A resume re-opens a conversation
that already ran in that folder, so it drops its own trust write, the input
that carried it and the tests that pinned it. The spawn itself goes through
upstream's spawn builder like upstream's own ensureAgentSession resume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude in WSL posts its hooks to the WSL relay, which dropped SessionEnd because
it yields no status row. Exiting Claude and then losing the terminal therefore
brought back a session the user had quit.

The WSL relay now forwards a SessionEnd with reason prompt_input_exit or logout
as an agent.hook notice carrying an optional userEndedSessionId, and stamps
reportsUserEndedSessions on every envelope it sends. The host retires the
session as the local path does (same session, same route, the launch-token check
for a stale SessionEnd) and resumes a WSL row only when that stamp is on it, so
a row from an older relay is never auto-resumed. The notice's payload state is
one no host admits, so an older host drops it. The relay also drops the ended
process's cached row, which a later replay would otherwise resurrect. SSH relays
are unchanged and still never resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the e2e journey

After the terminal daemon dies under a running host, nothing re-activates an
idle pane: the host respawns the daemon lazily, never tells subscribed clients,
and relies on one client re-activation that is not guaranteed and never
retried. CI hit exactly that and waited 3 minutes on dead handles.

Journey 4 now waits for the old daemon to exit and asserts the resume on the
pane's next activation (the user selecting it), without the network-return
nudge, which a daemon dying does not raise. Making the host tell its clients is
a follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tenki-reviewer

tenki-reviewer Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review complete. 🟡 1 medium

📍 Outside the diff (1)

  • 🟡 Recover-pane path skips mode grounding on new incarnation (src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts) — The PR grounds terminal modes whenever a rebind lands on a different incarnationId, via announceProcessReplacement -> storedCallbacks.onProcessReplaced, but the terminal.recoverPane path never calls it: recoverExpiredHostPane adopts the resolved terminal with adoptExecutionMetadata and rebinds the handle while firing only onPtyRebind (src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts:1317-1370).
🧹 Nitpicks (1) — 🟢 1 low
  • 🟢 Declare the missing hostPaneAgentResumes field (orca-runtime-perform-mobile-session-pty-records-refresh.ts:222) — getHostPaneAgentResumes() assigns this.hostPaneAgentResumes ??= new HostPaneAgentResumes() (src/main/runtime/orca-runtime-perform-mobile-session-pty-records-refresh.ts:222-225) but the property is declared nowhere in the class chain — a repo-wide grep finds only the two usage sites.

Adds user-ended-session relay notices, retained provider-session identity for resume after terminal loss, and incarnation-aware terminal-mode grounding across the agent-hook server, relay, host runtime, and renderer transports.

Reviewed commit: 30b0706

@tenki-reviewer
tenki-reviewer Bot dismissed their stale review September 30, 2026 20:32

A newer Tenki review found issues on this pull request.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30b07068f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant