Repository navigation
fix(relay): don't re-run an agent resume against a relay that never held the terminal (STA-5698) #16989
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
fix(relay): don't re-run an agent resume against a relay that never held the terminal (STA-5698) #16989
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
145 changes: 145 additions & 0 deletions
145
src/main/ipc/pty/pane/relay-replacement-resume-suppression.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| import type { Store } from '../../../persistence' | ||
| import type { IPtyProvider, PtySpawnOptions, PtySpawnResult } from '../../../providers/types' | ||
| import { parseAppSshPtyId, toRelaySshPtyId } from '../../../providers/ssh-pty-id' | ||
|
|
||
| /** Bounded so a slow relay leaves the fallback spawn on today's timing, not blocked behind a probe. */ | ||
| export const RELAY_AGE_STATUS_TIMEOUT_MS = 2_000 | ||
|
|
||
| /** | ||
| * Slack between two independently measured elapsed times — the client's clock over the binding and | ||
| * the host's own uptime counter. Both are durations, so no clock-offset correction is possible or | ||
| * needed; this only absorbs jitter and the gap between daemon start and the first binding write. | ||
| */ | ||
| export const RELAY_AGE_CLOCK_SLACK_MS = 30_000 | ||
|
|
||
| /** | ||
| * Startup intent replayed by the absence fallback. Mirrors the set `attachStablePaneOwner` already | ||
| * strips for an attach: everything that would run an agent, a resume argv, or a claim in the shell. | ||
| */ | ||
| export function stripStartupIntent(options: PtySpawnOptions): PtySpawnOptions { | ||
| return { | ||
| ...options, | ||
| command: undefined, | ||
| commandDelivery: undefined, | ||
| startupCommandDelivery: undefined, | ||
| launchAgent: undefined, | ||
| startupIngress: undefined, | ||
| agentSessionEnsure: undefined, | ||
| agentSessionCreateOperationId: undefined | ||
| } | ||
| } | ||
|
|
||
| export function carriesStartupIntent(options: PtySpawnOptions): boolean { | ||
| return Boolean( | ||
| options.command || | ||
| options.launchAgent || | ||
| options.startupIngress || | ||
| options.agentSessionEnsure || | ||
| options.agentSessionCreateOperationId | ||
| ) | ||
| } | ||
|
|
||
| /** Age of the lease that recorded this SSH binding, or null when nothing dates it. */ | ||
| function bindingAgeMs( | ||
| store: Store | undefined, | ||
| connectionId: string, | ||
| ptyId: string, | ||
| now: number | ||
| ): number | null { | ||
| if (!store || typeof store.getSshRemotePtyLeases !== 'function') { | ||
| return null | ||
| } | ||
| const parsed = parseAppSshPtyId(ptyId) | ||
| if (!parsed || parsed.connectionId !== connectionId) { | ||
| return null | ||
| } | ||
| let createdAt: number | null = null | ||
| for (const lease of store.getSshRemotePtyLeases(connectionId)) { | ||
| if ( | ||
| toRelaySshPtyId(connectionId, lease.ptyId) !== parsed.relayPtyId || | ||
| typeof lease.createdAt !== 'number' || | ||
| !Number.isFinite(lease.createdAt) | ||
| ) { | ||
| continue | ||
| } | ||
| createdAt = createdAt === null ? lease.createdAt : Math.min(createdAt, lease.createdAt) | ||
| } | ||
| if (createdAt === null) { | ||
| return null | ||
| } | ||
| const age = now - createdAt | ||
| return Number.isFinite(age) && age > 0 ? age : null | ||
| } | ||
|
|
||
| /** | ||
| * Does the relay that just answered "that PTY is absent" predate the binding it answered about? | ||
| * | ||
| * A daemon whose reported uptime is shorter than the binding's age cannot have owned the bound | ||
| * process, so its positive absence is `unverifiable` rather than `exited` | ||
| * (docs/reference/ssh-execution-boundary.md) — the shell it named may still be alive as an orphan | ||
| * of the daemon it replaced. Only an answer that ARRIVES and is demonstrably younger says so: | ||
| * a missing provider, a rejected or timed-out `relay.status`, an absent or unusable `uptimeMs`, | ||
| * and an undatable binding all return false, leaving the caller on today's behavior. Silence is | ||
| * never promoted to a verdict here. | ||
| * | ||
| * Reads only fields `relay.status` already publishes; adds no request field and no opcode. | ||
| */ | ||
| export async function answeringRelayPredatesBinding(args: { | ||
| provider: IPtyProvider | ||
| store?: Store | undefined | ||
| connectionId?: string | null | ||
| ptyId: string | ||
| now?: () => number | ||
| }): Promise<boolean> { | ||
| const { provider, connectionId, ptyId } = args | ||
| if (!connectionId) { | ||
| return false | ||
| } | ||
| const age = bindingAgeMs(args.store, connectionId, ptyId, (args.now ?? Date.now)()) | ||
| if (age === null) { | ||
| return false | ||
| } | ||
| let status: unknown | ||
| try { | ||
| // Optional call: a provider with no host RPC yields undefined, which is not an answer. | ||
| status = await provider.requestHostRpc?.( | ||
| 'relay.status', | ||
| {}, | ||
| { | ||
| timeoutMs: RELAY_AGE_STATUS_TIMEOUT_MS | ||
| } | ||
| ) | ||
| } catch { | ||
| return false | ||
| } | ||
| if (!status || typeof status !== 'object') { | ||
| return false | ||
| } | ||
| const uptimeMs = (status as { uptimeMs?: unknown }).uptimeMs | ||
| if (typeof uptimeMs !== 'number' || !Number.isFinite(uptimeMs) || uptimeMs < 0) { | ||
| return false | ||
| } | ||
| return age > uptimeMs + RELAY_AGE_CLOCK_SLACK_MS | ||
| } | ||
|
|
||
| /** | ||
| * The absence fallback's spawn, minus the startup intent a replacement relay never held. | ||
| * | ||
| * Returns null whenever nothing proves a replacement, leaving the caller on its unchanged path — | ||
| * the spawn itself is never gated here, only the intent replayed into it. | ||
| */ | ||
| export async function spawnWithoutReplayedIntent(args: { | ||
| provider: IPtyProvider | ||
| store?: Store | undefined | ||
| connectionId?: string | null | ||
| ptyId: string | ||
| spawnOptions: PtySpawnOptions | ||
| onFreshSpawn?: (result: PtySpawnResult) => void | ||
| }): Promise<PtySpawnResult | null> { | ||
| if (!carriesStartupIntent(args.spawnOptions) || !(await answeringRelayPredatesBinding(args))) { | ||
| return null | ||
| } | ||
| const result = await args.provider.spawn(stripStartupIntent(args.spawnOptions)) | ||
| args.onFreshSpawn?.(result) | ||
| return { ...result, agentResumeUnavailable: true as const } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🔎 Supported by static analysis
🏁 Script executed:
Repository: stablyai/orca
Length of output: 50370
🏁 Script executed:
Repository: stablyai/orca
Length of output: 50370
🏁 Script executed:
Repository: stablyai/orca
Length of output: 24421
🏁 Script executed:
Repository: stablyai/orca
Length of output: 50369
🏁 Script executed:
Repository: stablyai/orca
Length of output: 50370
🏁 Script executed:
Repository: stablyai/orca
Length of output: 50369
🏁 Script executed:
Repository: stablyai/orca
Length of output: 18101
Classify young replacement relays as
unverifiablebefore exit handling.attachStablePaneOwnercallsruntime.onPtyExit(..., 0, ...)beforespawnWithoutReplayedIntentchecksrelay.status.uptimeMs. This treats a possibly live SSH PTY as exited. Add a test that assertsonPtyExitis not called.Source: Coding guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Judged this from the code rather than the label, and I think it's real as a contract gap — but it is pre-existing, not introduced here, and the fix is bigger than the thread implies. Leaving it open for @brennanb2025 rather than resolving.
The ordering claim is accurate. At head
24614bd:stable-owner.ts:249-251— the new hook only recordsrelayAnsweredAbsent.stable-owner.ts:262—runtime?.onPtyExit(owner.ptyId, 0, owner.incarnationId)fires immediately after, inside the samecatch.stable-owner.ts:305-307—spawnWithoutReplayedIntent(and thereforeansweringRelayPredatesBinding→relay.status.uptimeMs) runs only afterattachStablePaneOwnerhas already returnednull.So the exit is published before anything has evaluated whether the answering relay could have owned the binding.
Why the exit code matters, and it isn't cosmetic. In
orca-runtime.ts::15992preservesAbnormalSshSurfacerequiresexitCode < 0. The call site passes0, so it is false.:16002is therefore skipped —markPtyLivenessUnverifiable(ptyId, SSH_EXIT_UNCONFIRMED_REASON)never runs.:16012processDeathCertified = exitCode >= 0 || …is true, soreconcileAgentStatusForEndedProcessFnruns and the agent status is reconciled as ended.onPtyExit(ptyId, 0, …)is a certified clean death. That isexited.Which is the one verdict this case is not allowed to be.
docs/reference/ssh-execution-boundary.md, "Distinguishingunverifiablefromexited": "A stale event, an event for a superseded incarnation, or one quiet terminal with no host evidence does not [establish exited]." A relay whose uptime is shorter than the binding's age is a superseded incarnation by construction — that is the PR's own premise, stated verbatim atrelay-replacement-resume-suppression.ts:76-83("its positive absence isunverifiablerather thanexited"). The code acts on that premise for the resume intent and not for the liveness verdict.Two things the thread gets wrong, though.
It isn't a regression from this PR.
:262is unchanged onmainand fires for everyisPtyAlreadyGoneError. This PR adds the knowledge that a subset of those absences is unverifiable; it does not make the verdict worse than it already was."Add a test that asserts
onPtyExitis not called" is not the right assertion, and the naive fix regresses something else. SkippingonPtyExitleaves the runtime holding a binding no relay will ever answer for. And unconditionally passingexitCode < 0for everySshPtyAbsentFromRelayErrorwould mark genuinely-exited SSH PTYsunverifiable, which feedsverifyUnstoppedPtys(unstopped-pty-verification.ts:17) and can block worktree removal. The correct shape is: still retire the surface, but chooseexitCode 0vs-1based on the age probe — which means hoistingansweringRelayPredatesBindingabove:262and awaiting up toRELAY_AGE_STATUS_TIMEOUT_MS(2s) inside the teardown path. That is a real latency and design change, and it is why the "heavy lift" tag is the accurate part of this comment.One thing worth acting on regardless of the above: every case in
stable-pane-relay-age-resume-suppression.test.tspassesruntime: undefined(:63,:217), soruntime?.onPtyExitis a no-op in all 12 tests. Whatever this PR decides the verdict should be, nothing currently pins it.Also unrelated to this thread but in the same file —
answeringRelayPredatesBindingis named backwards. It returnstruewhenage > uptimeMs + slack, i.e. when the relay started after the binding and therefore does not predate it. The behavior is correct and the test names use "predates" in the correct English sense (…test.ts:138pairs "predates the binding" witholderRelayand expects a replay), so the identifier is the sole outlier. Worth renaming before someone inverts the gate reading it.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@.wk-scratch/reply-16989-a.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Length of output: 415
@brennanb2025,.wk-scratch/reply-16989-a.mdis not present in this checkout. Please paste the draft content or provide an accessible file path.You are interacting with an AI system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This describes #16989's tree only. Nothing here survives the PR's closure — and main's
onPtyExit(…, 0, …)is correct as written. Recording the check so it is not rediscovered.The reported ordering does not exist on
main. Atorigin/main(94e7586665)stable-owner.tsis 293 lines, the exit is at:255, andspawnWithoutReplayedIntent/relay.status.uptimeMsappear nowhere in the repo —grep -rn answeringRelayPredatesBinding src/returns zero hits. The "publishes before the gate" relationship is created by this PR, so it dies with it.The ordering is real inside the PR.
attachStablePaneOwnerpublishes the exit and thenreturn nulls; the gate runs one frame later, inspawnForStablePane. So the PR suppresses the replayed startup intent but never retracts the exit it already published. Two aggravators, also PR-local:kill.ts:79-88publishes-1for the identicalisPtyAlreadyGoneErrorpredicate, andadopt-stable.ts:60callsattachStablePaneOwnerdirectly, so that caller never reaches the gate at all.On
mainthe0is not a coercion — it is a host-verified absence. By the time line 255 runs, the two non-evidence classes have already been diverted above it:Each remaining arm of
isPtyAlreadyGoneError(provider/liveness.ts:58-67) is positive evidence from the owning host on this path:SshPtyAbsentFromRelayError— raised only atssh-pty-session-reattach.ts:238, after:229diverts identity mismatch ("the id names a LIVE PTY owned by another pane"). Pinned both directions byssh-pty-relay-absence-verdict.test.ts./PTY ".+" not found/i— produced only bysrc/relay/pty-handler.ts, i.e. the relay answering. On the attach path it is converted to the class above before any caller sees it; the other four throw sites arecloseStartupQueryAuthority,sendSignal,getCwd,getInitialCwd, none of which are on this path./Session not found/i—SessionNotFoundError, reachable here only from a received daemon response (decodeDaemonResponseError) or fromresolution.kind === 'absent'inDaemonSessionOwnerResolver.spawnAttachOnly, which routeskind === 'unknown'toTerminalSessionOwnerUnverifiedErrorinstead.A transport failure cannot reach line 255:
stable-pane-relay-absence-respawn.test.ts:98-107pins that a lost link, a request timeout, and a bareSSH_SESSION_EXPIREDall rethrow and keep the binding.So the downstream composition is right.
resolveUnreportedExitCause(0)yields{ kind: 'unknown', reason: 'cause_unreported' }— "the reporting host did not say why", which is exactly what a relay-confirmed absence with no status tells us.processDeathCertified(orca-runtime.ts:16012) becomes true, correct for a verified absence, andpreservesAbnormalSshSurface(:15992) correctly does not mark the paneunverifiable. Switchingmainto-1would be the regression, not the fix: it would publishstop_unverifiedandmarkPtyLivenessUnverifiablefor a process the host positively reported gone.Leaving this thread open per the reviewer's request. No
mainPR was opened, because there is nothing onmainto change.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@brennanb2025, agreed on the scope correction.onPtyExit(..., 0, ...)is correct onmain. Onmain, the attach path has only host-verified absence cases at that point. The replacement-relay age gate does not exist there.The finding is limited to
#16989. This PR adds a new fact that can change the absence verdict: a relay younger than the binding could not have owned that binding. The current flow publishes the certified exit before it evaluates that fact.No
mainfollow-up is needed. The open thread should remain scoped to the PR-local ordering and its direct callers, includingadopt-stable.ts.🐇🔍
✏️ Learnings added
You are interacting with an AI system.