Repository navigation
fix(relay): stop advertising an agent session for a pane whose tab is gone (#12447) - #17012
Conversation
… gone (#12447) `pty.shutdown` was the only signal the relay ever got that a tab had closed, and it retired nothing: it requested a kill and returned. Every retirement path in the relay keys on proof of process death, so when the kill did not reap the pane shell the relay kept forwarding the orphaned agent's hook events as a live agent pane with no tab, and kept publishing its `agentSessionOwners` from `pty.listProcesses` with no liveness check at all. The relay now records the retirement the client stated, drops the pane's cached agent status with it, refuses to forward or replay posts from a retired pane, verifies the kill actually landed instead of assuming it, and reaps any PTY whose pid it can prove is gone before listing it. No wire change: no new RPC, field or stream opcode.
📝 WalkthroughWalkthroughThe relay now tracks pane surfaces retired by PTY shutdown. Merge Risk: 🟡 Moderate · up to The change suppresses updates from closed panes, but delayed retries, stale shutdown cleanup, and retirement-record eviction can still cause stale agent state or terminate a legitimately restored terminal. These bounded correctness and availability risks should be fixed or explicitly accepted before merging. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description is substantially complete. It explains the problem, implementation, rationale, compatibility impact, limitations, linked issue, and extensive test results. It does not explicitly complete every template heading or checklist item, but the missing sections are non-critical because the relevant information is provided elsewhere.
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 9500145e-564b-427c-a4d6-9f51f57675d8
📒 Files selected for processing (8)
src/relay/agent-hook-cached-pane-status.tssrc/relay/agent-hook-retired-pane-suppression.test.tssrc/relay/agent-hook-server.tssrc/relay/pty-handler-retired-pane-surface.test.tssrc/relay/pty-handler.tssrc/relay/relay-agent-hook-runtime.tssrc/relay/retired-pane-surfaces.test.tssrc/relay/retired-pane-surfaces.ts
Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review.
| if (this.isPaneSurfaceRetired(event.paneKey)) { | ||
| this.clearPaneState(event.paneKey) | ||
| return |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect whether retry scheduling can re-enter applyEvent after a retired post.
rg -n -C 10 '\bscheduleAssistantMessageRetry\s*\(|\bscheduleCodexSubagentPoll\s*\(|\bapplyEvent\s*\(' src/relayRepository: stablyai/orca
Length of output: 12088
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- repository conventions ---'
find /tmp/coderabbit-repo-knowledge/stablyai-orca-89dc44e4 -maxdepth 2 -type f -name '*.md' -print \
-exec sh -c 'echo "--- $1"; head -80 "$1"' _ {} \;
printf '%s\n' '--- agent hook server ---'
sed -n '300,355p' src/relay/agent-hook-server.ts
printf '%s\n' '--- retry scheduler ---'
sed -n '1,215p' src/relay/agent-hook-result-retry-scheduler.ts
printf '%s\n' '--- pane-state cleanup and scheduler clearing ---'
rg -n -C 8 '\bclearPaneState\b|\bclearAssistantMessageRetry\b|\bclearCodexSubagentPoll\b|new AgentHookResultRetryScheduler' src/relay srcRepository: stablyai/orca
Length of output: 50370
🏁 Script executed:
#!/bin/bash
set -euo pipefail
log=/tmp/coderabbit-shell-logs/shell-output-Q1J5hU
for marker in '--- agent hook server ---' '--- retry scheduler ---' '--- pane-state cleanup and scheduler clearing ---'; do
echo "$marker"
line=$(rg -n -F -- "$marker" "$log" | head -1 | cut -d: -f1 || true)
if [ -n "$line" ]; then
end=$((line + 230))
sed -n "${line},${end}p" "$log"
else
echo "marker not found"
fi
doneRepository: stablyai/orca
Length of output: 35310
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- cache cleanup ---'
rg -n -C 12 'function clearPaneCacheState|const clearPaneCacheState|export .*clearPaneCacheState|clearPaneCacheState' src/shared src/relay
printf '%s\n' '--- codex subagent predicate ---'
rg -n -C 16 'function hasCodexTranscriptSubagents|const hasCodexTranscriptSubagents|export .*hasCodexTranscriptSubagents|hasCodexTranscriptSubagents' src/shared src/relay
printf '%s\n' '--- request admission and scheduler calls ---'
sed -n '270,292p' src/relay/agent-hook-server.ts
sed -n '220,233p' src/relay/agent-hook-server.tsRepository: stablyai/orca
Length of output: 40887
Stop assistant retry scheduling for retired panes.
applyEvent() clears the pane state and returns, but handleRequest() still calls scheduleAssistantMessageRetry(). That method can arm a timer from the request body. If the pane key is restored before the callback, the callback can apply the orphan event. Skip this scheduler call when the pane is retired.
| if (attemptsRemaining <= 0) { | ||
| process.stderr.write( | ||
| `[pty-handler] retired pane PTY ${managed.id} still alive after force kill; ownership retained as unverifiable\n` | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use unverifiable, not “still alive”. isProcessAlive() treats errors other than ESRCH as alive, so this log can report an unverified remote process as live. As per coding guidelines: SSH verdicts must use live, unverifiable, or exited, with no synonyms.
Source: Coding guidelines
…base #17012 landed on main spying on ptyShellUtils.isProcessAlive while this branch moved the probe into pty-process-probes. The rebase merges textually clean but breaks 6 of the file's 9 tests and the node typecheck.
Resolves src/relay/agent-hook-server.ts against #17012, which landed on main after this branch extracted applyEvent/handleRequest into their own modules. Main's retired-pane-surface gate lived in the extracted body, so a plain 'keep ours' resolution would have dropped it: applyRelayEvent now takes isPaneSurfaceRetired and drops a retired pane before caching or forwarding, and agent-hook-event-application.test.ts pins it at its new home. Options types move to agent-hook-server-options.ts to keep the server file under max-lines.
… gone (stablyai#12447) (stablyai#17012) `pty.shutdown` was the only signal the relay ever got that a tab had closed, and it retired nothing: it requested a kill and returned. Every retirement path in the relay keys on proof of process death, so when the kill did not reap the pane shell the relay kept forwarding the orphaned agent's hook events as a live agent pane with no tab, and kept publishing its `agentSessionOwners` from `pty.listProcesses` with no liveness check at all. The relay now records the retirement the client stated, drops the pane's cached agent status with it, refuses to forward or replay posts from a retired pane, verifies the kill actually landed instead of assuming it, and reaps any PTY whose pid it can prove is gone before listing it. No wire change: no new RPC, field or stream opcode. (cherry picked from commit 94e7586)
… gone (stablyai#12447) (stablyai#17012) `pty.shutdown` was the only signal the relay ever got that a tab had closed, and it retired nothing: it requested a kill and returned. Every retirement path in the relay keys on proof of process death, so when the kill did not reap the pane shell the relay kept forwarding the orphaned agent's hook events as a live agent pane with no tab, and kept publishing its `agentSessionOwners` from `pty.listProcesses` with no liveness check at all. The relay now records the retirement the client stated, drops the pane's cached agent status with it, refuses to forward or replay posts from a retired pane, verifies the kill actually landed instead of assuming it, and reaps any PTY whose pid it can prove is gone before listing it. No wire change: no new RPC, field or stream opcode.
Closes the server-side half of #12447 — @chanmuzi's relay dataset. Relay-side only; no wire change.
ELI5
Orca runs a small helper — the relay — on your SSH host. It owns your remote terminals, and it also runs a tiny web server that your agent (
claude,codex, …) posts status updates to: "I'm working", "I'm waiting for you", "I'm done". The relay forwards those to every connected Orca so the sidebar can show a live agent.When you close an agent tab, Orca asks the relay to kill that terminal. Usually it dies. Sometimes it doesn't — an interactive shell ignores the polite signal, or the forceful one only reaches the shell and not the agent inside it. When that happens, the agent is still alive and still posting status updates, and the relay had no idea it should stop listening. It kept forwarding them. So every Orca on that host kept being told there was a live agent in a pane that no longer had a tab — and the next time you opened the project, Orca did what you'd want it to do for a live agent it had lost track of: it opened a terminal and typed
claude --resume <that session>.Now you have two
claudeprocesses writing into one conversation transcript. That is not recoverable.The relay's mistake was believing the kill worked. It never asked again, and it never wrote down the one thing it did know for certain: you closed the tab.
After this change the relay writes that down. The moment
pty.shutdownarrives, the pane is retired:And one deliberate non-action: if the relay cannot prove the process died, it keeps the agent-session ownership claim. That sounds backwards, but it is the safety property. That claim is what makes the next terminal you open adopt the surviving pane instead of starting a second agent on the same transcript. An honest "I can't verify this" keeps you at one agent; forgetting would take you to two.
Root cause
PtyHandler.shutdown()—src/relay/pty-handler.ts:1939(pre-change).pty.shutdownis the only authoritative statement the relay ever receives that a tab has closed, and it retired nothing. It requested a kill and returned. Every retirement path in the relay keys on proof of process death, and there are exactly three:pty-handler.ts:872(onExit)pty-handler.ts:1785(attach)isProcessAliveand found the pid gonepty-handler.ts:2406(disposePtyForRelayShutdown)Client disconnect retires nothing either (
relay-reconnect-listener.ts:120→dispatcher.ts:185only close the writer), and the shipped grace is unlimited (DEFAULT_SSH_RELAY_GRACE_PERIOD_SECONDS = 0,src/shared/ssh-types.ts:7), so with ≥1 live PTY nothing is ever armed.So when the kill did not actually reap the pane shell — SIGTERM ignored by an interactive shell, or
forceKillPosixPtyProcessGroupsdegrading to a root-pid kill whengetPosixPtyProcessGroupsreturnsnull(src/main/pty/posix-pty-process-groups.ts:99-108) — two advertisements survived the tab:src/relay/agent-hook-server.ts:301(applyEvent). The orphaned agent keeps POSTing to the relay's loopback hook listener with the closed pane'sORCA_PANE_KEY.applyEventcached it inlastStatusByPaneKeyandforwarded it to every connected client as anagent.hooknotification — with no check whatsoever that the relay still serves a PTY for that paneKey.replayCachedPayloadsForPanes(:200) then re-published it on the nextagent_hook.requestReplay, i.e. on every reconnect. The only eviction wasclearPaneState, reachable solely throughnotifyExitListener— the exit that never came.src/relay/pty-handler.ts:2114(listProcesses). This is what publishesagentSessionOwners, the host's "there is a live agent session here you can adopt". It iteratedthis.ptysand emitted the registry entry with no liveness verification at all — whileattach()fourteen hundred lines earlier does exactly that probe and reaps ("verify liveness because shells can exit without node-pty onExit",:1812).That is the loop @chanmuzi reported, and it needs no lost kill RPC to close: close → shell survives → orphan advertises → reopen resumes → two agents, one transcript.
The fix
All hunks are relay-side. Per SSH-v3 P2 the host is the authoritative liveness source, so it has to look, and it must not advertise what it has not verified.
1.
pty.shutdownretires the pane surface (pty-handler.ts,retired-pane-surfaces.ts). A newRetiredPaneSurfaceRegistryrecords paneKeys whose tab the client has said is gone. Retirement is deliberately not a liveness verdict — the vocabulary stayslive/unverifiable/exitedand this asserts none of them about the process. It records only that no client tab can own this pane any more, which is knowable without guessing because the client stated it. A newsetSurfaceRetiredListenerseam is kept separate fromPtyExitListenerfor exactly that reason: retirement must never read as an exit. A PTY that later binds the same paneKey — a reopened pane, a revive, or an acceptedpty.attach— clears the retirement.2. A retired pane cannot advertise (
agent-hook-server.ts,agent-hook-cached-pane-status.ts,relay-agent-hook-runtime.ts).applyEventdrops — does not cache, does not forward — a post from a retired pane, and drops the pane's stale cache with it.replayCachedPayloadsForPanesre-checks at replay time, because retirement can be recorded after the status was cached. The runtime routes the predicate toptyHandler.isPaneSurfaceRetiredand clears the cache eagerly on the retirement edge.3. The kill is verified, not assumed (
armShutdownReapSweep). A bounded, unref'd sweep (3 escalations, ~6s apart) re-probesisProcessAlive. Gone → run the full retirement even though node-pty never producedonExit. Still alive → re-issue the force kill, because a SIGKILL that returned success is not proof of death. POSIX only: on Windows ConPTY's kill is already force-final and re-closing the handle is the hazarddisposeManagedPtyguards against.4.
listProcessesverifies before it advertises. A PTY that is disposed, or whose pid probes dead, is reaped (sharedreapExitedPty, now used by all three probes so they cannot drift) instead of being published with itsagentSessionOwners.The deliberate non-fix, and why it is the safety property
When the sweep exhausts its attempts, the agent-session ownership claim is retained, not released. Releasing a claim the host cannot prove dead is precisely what authorises a reopened project to spawn a second agent:
reconcileAgentSessionOwnerListings(src/main/ipc/pty/pane/agent-session-owners.ts:105) treats the relay's listing as authoritative and releases any claim missing from it, after whichensure()creates rather than adopts. Keeping the claim makes the nextpty.spawnadopt this PTY — one process, one transcript.unverifiableis the honest verdict and it is also the safe one.This is why the fix is not "withhold
agentSessionOwnersfor a retired pane". That would have read as a fix and caused the corruption it was meant to prevent.Remote wire compatibility
Checked against
docs/reference/remote-wire-compatibility.md. No wire change: no new RPC method, no new request or response field, no new stream opcode, noRUNTIME_PROTOCOL_VERSIONbump.pty.shutdownparams and result are byte-identical;PtyProcessSummaryis unchanged.isPaneSurfaceRetiredis a host-internal constructor option.Two Rule-3 projections (content the host stops publishing), stated explicitly:
Old client → new relay.
agent.hooknotifications for a pane it asked the host to shut down. Previously those refreshed a live-agent record for a tab that no longer existed; now nothing arrives and the client's own record ages out viaAGENT_STATUS_STALE_AFTER_MS. No client requires these frames: the desktop already clears its own records for a user-closed tab (removeSleepingAgentSessionsForTab,src/renderer/src/store/terminals/terminal-tab-close.ts:154), so nothing is stranded waiting for a final host hook. Strictly fewer false live-agent records; no path hangs on absence.pty.listProcessesrows whose process is provably dead. Old clients treat listing absence as an authoritative release — which is the correct verdict for a dead process, and already the outcome the momentattach()reaped the same entry. A live pane is never withheld, so no old client loses a claim it still needs.New client → old relay. Identical to today. Nothing in this change asks the host for anything new or reads a new field, so a client on this build against a pre-fix relay gets exactly the pre-fix behaviour: orphan hook events still forwarded, dead rows still listed. The fix is a pure host-side narrowing with no client-side dependence, so there is no new-client-requires-new-host skew to negotiate.
Not on the enforcement harness.
tests/e2e/cross-version-wire/…covers the terminal stream only, not the agent-hook channel orpty.listProcesses. Run anyway and green (5 passed) — it proves nothing here, and this section is the reasoning the doc asks for in its place.Relationship to #16989 (STA-5698)
Adjacent, not overlapping, and not conflicting. I read it before starting.
src/main/ipc/pty/…); this PR is entirely relay-side (src/relay/). Zero file overlap.answeringRelayPredatesBindinggate never fires:uptimeMsis larger than the binding age, so it returnsfalseand the resume proceeds. It cannot cover this report.Fix evidence
Failing-first, full production revert
src/relay/{pty-handler,agent-hook-server,relay-agent-hook-runtime,agent-hook-cached-pane-status}.tsreverted, new tests kept:Honest note on that run. Several of those are
TypeError, not assertion failures — the retirement API does not exist on the unfixed relay, so the suite cannot express the pre-fix behaviour without it. A whole-file revert is therefore weak evidence. The ablation table below is the real proof, and one test — "shutting down a pane with no pane key retires nothing" — pins that behaviour is unchanged for a paneless PTY and by construction cannot fail before the change. I am not claiming failing-first for it.Post-fix: 19 passed (19).
Ablation table — every production hunk reverted individually
applyEventretirement gateshutdown()retires the surfaceshutdown()arms the reap sweeplistProcessesliveness verificationwireAndStorerestores a rebound paneattach()restores a reattached paneNo uncovered hunk. H7 was mutated to a permissive predicate (
() => false) rather than deleted, since deletion only restores the default — a weak mutation would have made the table lie.Requested coverage
wsl-agent-hook-relay.ts:67)tests/e2e/cross-version-wiregreenGates
pnpm tc— exit 0oxlint(incl. the react-doctor config, via the commit hook) on all changed files — clean. Nomax-linesdisable added:agent-hook-server.tscrossed 300 counted lines, soreplayCachedPayloadsForPanes's body and the cached-pane cap eviction were extracted intoagent-hook-cached-pane-status.tsrather than suppressed.pnpm exec oxfmt --writeon changed paths onlygit diff --check— cleanpnpm test src/relay— 147 files, 1519 passed, 2 skipped, 0 failedsrc/relay+src/main/ipc/pty+src/main/providers+src/main/daemon— 429 files, 4573 passed, 0 failedtests/e2e/cross-version-wire/cross-version-terminal-wire.unit.test.ts— 5 passedIdentity-space findings — verified, deliberately left
Three findings landed mid-review. This fix keys on neither incarnation identifier, so none of them apply to it; I verified all three in the code and am reporting rather than fixing, so they can be filed separately.
What this fix keys on, and why. Retirement is keyed on
paneKey(ORCA_PANE_KEY,pty-handler.ts:1623) and liveness onmanaged.pty.pid. Both are relay-local and host-owned:paneKeyis the identity the agent-hook payload already carries (it is howclearPaneStatehas always worked), and the pid is what the host can probe directly. NeitherptyIncarnationnorincarnationIdis read, compared, or used as a key anywhere in this diff.listProcessesstill emitsincarnationIdexactly as before.Consequently there is no no-semantic-incarnation degradation to handle here. A PTY with no semantic
incarnationId— an old relay, alegacy:pin, a paired runtime — retires and reaps identically, because retirement never consults an incarnation. The one place this diff touches an incarnation-adjacent contract islistProcessesomitting a row for a provably dead PTY, and that is unambiguous in both directions of skew.Verified and not fixed:
ptyIncarnation≠incarnationId.SshPtyExitCallback(src/main/providers/ssh-pty-provider-contract.ts:34-40) carries both —ptyIncarnation: stringalways present,incarnationId?: PtyIncarnationIdoptional — andresolvePtyIncarnation(src/main/providers/ssh-pty-provider-output-state.ts:131-139) synthesizeslegacy:${providerGeneration}:${serial}:${relayPtyId}first-write-wins perrelayPtyIdwhen the wire omits one. Alegacy:pin can never equal a realincarnationId.findExactPendingExitcompares across that boundary (src/main/ssh/ssh-relay-session.ts:2530-2542): it matchesexit.ptyIncarnation(transport) against callers'attachResult.incarnationId(semantic —:2433-2435and:2458). Under alegacy:pin the exact match can never succeed, falling through to the coarser "any pending exit for this ptyId" clauses. Real, in adjacent territory, untouched here.ptyIncarnationByIdhas two uncoordinated last-write-wins writers (src/main/ipc/pty/provider/ownership-state.ts:5), with no generation fencing: renderer-driven spawn/reattach and connection-level bulk reattach-on-reconnect. A stale last write makesisCurrentPtyExit(:7-10) silently reject a real exit. Untouched here.What I did NOT verify
claudealive underrelay.js --detached, and watch the advertisement stop. All evidence is unit-level againstPtyHandlerand a realRelayAgentHookServerover loopback HTTP.pty.revivewill not re-list it either.managed.pty.pidfor up to ~24s. If node-pty missed the child's exit and the host recycled the pid inside that window, the sweep would signal an unrelated process group. The identical shape already exists inarmForceKillFallback's retry (+5.25s); this widens it. Note that a recycled pid normally implies the child was reaped, which impliesonExitfired, which removes the entry and stops the sweep — so the exposure is confined to the missed-onExitcase the feature exists for. Not measured on a real host.SHUTDOWN_REAP_VERIFY_DELAY_MS = 6_000/SHUTDOWN_REAP_MAX_SWEEPS = 3budget is reasoned, not tuned. 6s sits just past the existing 5s armed fallback so the first sweep observes post-kill state; three escalations bound it at ~24s. I did not measure how often a real host needs more than one.pty.killwith no pending-kill journal (src/main/ipc/pty/runtime/kill.ts) — is untouched, as ispersistPtyBinding's unconditional tombstone delete. This PR closes the server-side advertisement loop, not the client-side lost kill.Refs #12447. Adjacent to #16989 (STA-5698); neither blocks the other.