Skip to content

fix(ssh): decline an agent resume when the answering relay never minted the pane's PTY (STA-5698) - #18010

Closed
brennanb2025 wants to merge 4 commits into
mainfrom
brennanb2025/sta5698-epoch-gate
Closed

brennanb2025 wants to merge 4 commits into
mainfrom
brennanb2025/sta5698-epoch-gate

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​390 $\color{#cf222e}{\Huge{\mathbf{−}}}$​7 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​383
Prod 33 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​528 $\color{#cf222e}{\Huge{\mathbf{−}}}$​66 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​462

Fixes STA-5698.

In plain terms

You have an AI agent running in a terminal on a remote machine. The connection drops, or the helper process over there restarts. Orca reconnects, finds your terminal gone, opens a fresh one, and automatically restarts your agent where it left off — without ever checking whether the old one is still running.

If it is, you now have two agents editing the same files.

This makes Orca check. When the terminal's name shows it was created by a different helper lifetime than the one now answering, Orca still gives you a working shell but does not restart the agent, and says so.

What you'll notice: almost always nothing. Measured across five ordinary flows — idle, disconnect/reconnect, a 49s network pause, a 90s sleep, and a full app quit and relaunch — the helper was never replaced, so the resume path is untouched. It only declines when the helper genuinely was replaced, which in testing meant the old terminal was already gone anyway.

How it decides

Terminal ids now carry the minting helper's identity (pty2:<mintEpoch>:<seq>, from #16901). On a post-absence fresh spawn:

Situation Behaviour
epoch known and different decline the resume, spawn a plain shell, raise the existing resume-unavailable banner
epoch known and same resume exactly as before
legacy id, old relay, RPC failure, non-SSH no verdict — behave exactly as today

That last row is deliberate. Declining on "don't know" would break the common absence case (the PTY died while the helper stayed up, which must keep auto-resuming), and the residual risk needs a legacy id and a helper replacement and a surviving agent at once. The tradeoff is commented at the decision site rather than left implicit.

Polarity follows the repo's own rule, stated in src/relay/pty-shell-utils.ts:183-193: only positive proof authorises the automatic path.

Root cause of the first failed attempt

An earlier revision passed unit tests but failed end-to-end: the relay was replaced, the epoch demonstrably changed, and Orca still relaunched with --resume.

Instrumentation on a real SSH host proved why, and it was none of the obvious candidates. relay.status was called zero times — the read never failed or timed out. spawnForStablePane ran with owner === null, because relay recovery cleared the stable owner before the renderer requested its cold resume. The gate was skipped entirely rather than answered wrongly.

The fix retains the epoch-bearing owner across that retirement, keyed by connection and pane, and consumes it at the fresh-spawn point when the live owner is gone and there is explicit persisted resume identity.

Review

Reviewed to clean by two independent models. One finding was seeded from my own read and fixed here: the retired-owner map was module-level with its only deletion inside take(), so any retirement not followed by a fresh spawn — closed tab, removed connection — leaked an entry for the life of the process. It is now bounded at 256 with insertion-order eviction and delete-then-set recency, with regression tests.

Deployment review was explicit, after a sibling PR was closed for exactly this class of defect: this change depends only on the existing pty2 id string and the relay.status contract — no patched-only native symbols, nothing absent from the relay's remotely-installed dependencies.

Honest limits

  • It proves which relay incarnation minted the terminal, not whether the agent is alive. A setsid'd grandchild that outlives its shell is invisible to it.
  • The guarded seam runs promptly only with fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn #16763; without that PR this is correct but largely dormant. It should land alongside it.
  • Panes bound before this ships carry legacy ids and get no verdict until they respawn.

Related

Builds on #16901 (id format) and #16894 (relay reaps its PTYs on a fatal exit), both merged. #16843 was the first attempt and was closed — it inferred replacement from process age and would have refused a resume after an ordinary host reboot.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Epoch gate core — new relay-pty-mint-epoch.ts parses the pty2:<mintEpoch>:<seq> id, reads the answering relay's relay.status.ptyIdMintEpoch (per-provider cached, 2s timeout), compares, and strips resume options when a known-different epoch answers. The read ?? Promise.resolve(undefined) fallback is live, not dead — the trailing .then short-circuits to undefined when requestHostRpc is absent.
  • Retired-owner retention — SSH reconnect records the epoch-bearing owner (keyed connectionId \0 paneKey, bounded at 256) before clearing an absent PTY, and spawnForStablePane consumes it only when the live owner is gone and explicit resume identity (resumeProviderSession / agentSessionEnsure) is present.
  • Reply/state propagation — agentResumeUnavailable threads from PtySpawnResult through both IPC and runtime commit paths and relay.status; the renderer cancels the pending resume and shows the resume-unavailable banner.
  • Tests & validation — 46 targeted tests with exact (not truthiness) assertions plus an E2E validation report demonstrating real relay replacement declines the resume and a same-epoch relay resumes.

I traced the two invariants the whole mechanism depends on and both hold: the SshPtyProvider is recreated per reconnect (new SshPtyProvider + a fresh allocateSshPtyProviderGeneration), so the WeakMap epoch cache is never stale across a relay replacement; and the retired-owner key is produced by makePaneKey(tabId, leafId) on both the retire and spawn sides. The ...ctx.result widening in commitRuntimePtySpawn is benign because the final reply is field-whitelisted by toRuntimeSpawnReply.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The change adds relay PTY mint-epoch tracking and retired-owner storage. Stable-pane spawning compares relay epochs and declines agent resume for stale ownership. Spawn contracts propagate the unavailable-resume marker through IPC, runtime, terminal creation, and renderer transport. Commit paths suppress agent-launch side effects for declined resumes. Renderer flows clear pending startup state and launch metadata. Tests and validation records cover epoch matching, relay recovery, reconnects, and resume cleanup.

Merge Risk: 🟡 Moderate · up to 5a4e1

The PR adds relay-lifetime checks to prevent automatic agent duplication, but several current paths can still resume on a different relay, bind a fresh shell to stale session state, or retain authority for a surviving old agent. Those cases can cause concurrent agents to edit the same worktree or route status to the wrong pane, so merge should wait for the affected paths to be corrected or explicitly accepted by the owner.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 30 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the SSH agent-resume change and the relay PTY minting condition. It is specific and related to the main purpose of the pull request.
Description check ✅ Passed The description clearly explains the problem, solution, decision rules, root cause, linked issue, testing evidence, scope, and known limitations. It does not use every template heading and lacks an ex…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description clearly explains the problem, solution, decision rules, root cause, linked issue, testing evidence, scope, and known limitations. It does not use every template heading and lacks an explicit Visual Proof section and checklist completion, but the description is substantially complete and reviewable.

Full details: Docstring Coverage

Explanation

Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 30 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/main/ipc/pty/runtime/spawn-execute.ts (1)

63-84: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Apply the relay-epoch gate to agentSessionEnsure spawns.

When args.agentSessionEnsure is set, this branch bypasses spawnForStablePane and sends unmodified ctx.spawnOptions to the provider. It does not read the retired owner or compare its relay epoch. A runtime SSH resume can therefore continue on a newly answering relay that did not mint the prior PTY.

Derive the epoch-aware options before agentSessionOwners.ensure, use them for the provider spawn, and propagate agentResumeUnavailable when the gate declines the resume.

src/main/runtime/orca-runtime-create-terminal.ts (1)

213-217: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Gate structured-session binding for declined resumes.

When deriveStablePaneFreshSpawnOptions declines a resume, it removes the resume and agent-launch options, so the provider creates a fresh shell. ensureAgentSession still passes structuredAgentSessionId, and this block binds that fresh shell to the structured session. Later writes can target the replacement shell. Add !agentResumeDeclined to the condition.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 9ce2e61f-c108-4ea9-adb1-bf866c6f63a0

📥 Commits

Reviewing files that changed from the base of the PR and between 8fef582 and 5a4e175.

⛔ Files ignored due to path filters (5)
  • validation-artifacts/sta5698-task01-final-20260829T010120Z/test1-relay-replaced-pass.png is excluded by !**/*.png
  • validation-artifacts/sta5698-task01-final-20260829T010120Z/test1-stub-launches.log is excluded by !**/*.log
  • validation-artifacts/sta5698-task01-final-20260829T010120Z/test1-working-shell-pass.png is excluded by !**/*.png
  • validation-artifacts/sta5698-task01-final-20260829T010120Z/test2-same-relay-resume-pass.png is excluded by !**/*.png
  • validation-artifacts/sta5698-task01-final-20260829T010120Z/test2-stub-launches.log is excluded by !**/*.log
📒 Files selected for processing (31)
  • src/main/ipc/pty/ipc/spawn-commit-persist.ts
  • src/main/ipc/pty/ipc/spawn-commit.ts
  • src/main/ipc/pty/ipc/spawn-execute.ts
  • src/main/ipc/pty/ipc/spawn-options.ts
  • src/main/ipc/pty/pane/relay-pty-mint-epoch.ts
  • src/main/ipc/pty/pane/stable-owner-relay-epoch.test.ts
  • src/main/ipc/pty/pane/stable-owner.ts
  • src/main/ipc/pty/runtime/spawn-commit.ts
  • src/main/ipc/pty/runtime/spawn-execute.ts
  • src/main/ipc/pty/runtime/spawn-options.ts
  • src/main/ipc/pty/runtime/spawn.ts
  • src/main/providers/pty-provider-contract.ts
  • src/main/providers/pty-spawn-result.ts
  • src/main/runtime/orca-runtime-create-terminal.ts
  • src/main/runtime/runtime-pty-controller-contract.ts
  • src/main/ssh/ssh-relay-session.test.ts
  • src/main/ssh/ssh-relay-session.ts
  • src/relay/pty-handler.ts
  • src/relay/relay-daemon.ts
  • src/relay/subprocess.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection-cold-restore-agent-resume.test.ts
  • src/renderer/src/components/terminal-pane/pty-connection/apply-reattach-payload.ts
  • src/renderer/src/components/terminal-pane/pty-connection/connect-pane-pty-session.ts
  • src/renderer/src/components/terminal-pane/pty-connection/connect-pane-pty.ts
  • src/renderer/src/components/terminal-pane/pty-connection/declined-agent-resume.ts
  • src/renderer/src/components/terminal-pane/pty-connection/fresh-spawn-start.ts
  • src/renderer/src/components/terminal-pane/pty-connection/pane-serializer-settle.ts
  • src/renderer/src/components/terminal-pane/pty-connection/reattach-payload-session.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/shared/runtime-terminal-contracts.ts
  • validation-artifacts/sta5698-task01-final-20260829T010120Z/report.md

Included review availability: Your plan provides up to 10 included reviews per hour; 5 remain after this review.

Comment thread src/main/ipc/pty/pane/stable-owner.ts Outdated
}
}
const result = await args.provider.spawn(args.spawnOptions)
const retiredRelayOwnerPtyId = takeRetiredRelayEpochOwner(args.connectionId, args.paneKey)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Keep retired ownership when the fresh spawn fails.

takeRetiredRelayEpochOwner deletes the retired owner before provider.spawn() completes. If the spawn rejects, a retry has no owner PTY ID to compare. The retry can then retain automatic agent-resume options for a different relay epoch.

Remove this record only after a successful spawn, or restore it in the error path. Add a regression test for a failed fresh spawn followed by a retry.

@brennanb2025

Copy link
Copy Markdown
Contributor Author

STA-5698 runtime evidence rerun on the current PR head (5a4e175fa6, brennanb2025/sta5698-gate-review):

Test 1 (relay replaced → must not resume): FAIL. Initial pty2:f714605e-513e-4bcc-be40-846dcacaef0c:1; exact relay PID 1027126 SIGKILLed. Replacement PTY epochs changed (pty2:3b121812-a210-4fdd-8481-be17e75118b6:1, then pty2:cf10e220-78a8-4abe-b32f-a4ace9c4aa60:1), but the stub log gained a second launch carrying --resume sta5698-runtime-session; no previous session unavailable banner appeared.

Test 2 (same relay → must resume): FAIL, over-declining. Relay PID 1043262 remained alive; exact pane shell PID 1045141 was SIGKILLed. Replacement pty2:cf10e220-78a8-4abe-b32f-a4ace9c4aa60:2 kept the epoch, but no third stub launch or agent process appeared; the pane was only a shell prompt. This control failure blocks the gate.

Screenshots and full logs are in validation-artifacts/sta5698-runtime-20260901131755/ in the validation workspace: test1-relay-replaced.png, test1-shell.png, test2-same-relay-control.png, test1-stub.log, test2-stub.log, remote-evidence.txt, and report.md. SSH target was throwaway namespace /home/brennan/sta5698-runtime-20260901131755 on openclaw, cloned stablyai/demo-project; remote repo was clean. pnpm build:relay passed; cleanup completed and no production code changed.

@brennanb2025
brennanb2025 marked this pull request as draft September 1, 2026 22:30
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Marking draft. Both runtime tests fail on the current head, so this is not mergeable as it stands.

Note the earlier passing end-to-end run was against pre-rebase code. Between then and now the branch was rebased onto current main and the retired-owner map was bounded. One of those regressed it, and the passing evidence does not transfer.

Two things stand out for whoever picks this up:

  1. Test 1 saw the epoch change twice (3b121812… then cf10e220…) during a single reconnect, i.e. more than one relay incarnation. The retired-owner bookkeeping is keyed by connection + pane and consumed once; a second replacement mid-recovery may consume or overwrite the entry before the fresh spawn reads it.
  2. Test 2 produced no agent relaunch at all — not a stripped resume, but no third stub launch and no agent process, just a shell prompt. That is a different failure shape from over-declining via the gate, and worth separating before assuming a single cause.

Full evidence is in validation-artifacts/sta5698-runtime-20260901131755/ in the validation workspace.

@OrcaWin

OrcaWin commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

SSH still works: Orca connects to the host over SSH and routes workspace operations through the managed remote Orca server (orcad). These merges remove the legacy SSH relay, not SSH support.

Closing as obsolete after the merged SSH relay retirement in #27214 and #27215. This PR concerns the SSH relay PTY attach, recovery, replay, or lease lifecycle. #27214 removed the desktop relay connection/fallback path; #27215 deleted the SSH relay server, deployment assets, and relay-only tests. SSH hosts now use the managed Orca server (orcad), and unsupported hosts fail explicitly, so this relay-specific work is no longer planned. This closes the proposed relay implementation; it does not claim that any independent local, WSL, or managed-server behavior was fixed.

@OrcaWin

OrcaWin commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Thanks @brennanb2025 — the relay itself is gone, but the idea behind this (don't auto-resume an agent when the host's "terminal not found" was never observed as a real exit) still applies to managed hosts and paired runtimes. I've captured it on #13716 with credit to this PR; this one stays closed since its mechanism was relay-specific.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants