Skip to content

fix(runtime): keep a live host verdict when a status probe can't reach it (#19647, #16516) - #19873

Closed
nwparker wants to merge 3 commits into
mainfrom
nwparker/reconnect-churn-19647-writer
Closed

nwparker wants to merge 3 commits into
mainfrom
nwparker/reconnect-churn-19647-writer

Conversation

@nwparker

@nwparker nwparker commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​807 $\color{#cf222e}{\Huge{\mathbf{−}}}$​1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​806
Prod 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​9 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​9

Note

Rebase decision: resolved. No reviewer adjudication needed.

Both questions this section previously asked a reviewer to decide were answered by reading and testing main. The answers are below with their evidence. One open defect is recorded at the end — it is not this PR's and is not blocking it.

Rebased 2026-09-11 onto main at 5fa62feda7c264c6eda99ba4788ffebab9aae26c. Old head 1751433b2118fd15aaa1afc0e3b40fc0fb276674 → rebased head 200a98c19eafcb0c7f059fb0d139619e0a08c4d1 → current head 6ad1e48e4d61f36c73678bf5c0171987266685d5.

main landed #20003, fix(runtime): let connections own host status recovery (3b82d8de642), which deleted src/renderer/src/store/slices/runtime-status-recheck.ts and moved failed-status retries into a connection-owned RuntimeHostStatusOwner. The rebase accepted that deletion.

Q1 — does #20003's connection-owned retry cover the deleted recheck ladder? Yes, all three arms.

ladder behaviour this PR added covered on main where
re-ask a status: null host on the backoff ladder (#16516) yes, and more broadly. RuntimeHostStatusOwner.complete() calls scheduleRetry() on every failure, on the same [3s, 6s, 12s, 30s, 60s] ladder. The old shouldRecheck predicate required SHARED_CONTROL + remoteControl.state !== 'ready', which is precisely why a status: null host was never re-asked. There is no predicate now. src/shared/runtime-host-status-owner.ts (RETRY_DELAYS_MS, complete, scheduleRetry); pinned by runtime-host-status-owner.test.ts → "retries a failed status operation while retaining healthy transport and last good metadata" and by runtime-environment-status-recovery.test.ts → "recovers a saved host after its first status check fails, without another UI request"
cancel the ladder on runtime_manually_disconnected yes, at the source rather than by reacting to an error code. runtimeEnvironments:disconnect marks the environment and invalidates the transport → closeRemoteRuntimeRequestConnection → owner.dispose(), which aborts the in-flight request, clears the retry timer and sets verification: 'blocked'. startRequest and scheduleRetry both bail on blocked, and a re-created owner is disposed immediately while the environment is still manually disconnected. getRuntimeEnvironmentStatus also short-circuits before the owner is touched. runtime-environment-connectivity-handlers.ts, runtime-environment-request-connections.ts, owner dispose/startRequest/scheduleRetry; pinned by "disconnect settles readers and prevents late results and retry resurrection"
runtime_unavailable must not null a live verdict inside the ladder yes, structurally. The owner's complete() on a failure patches only checkedAt and verification; it never touches snapshot.status. A failed probe therefore cannot overwrite a verified verdict at all, rather than being talked out of it by a conditional. setRuntimeEnvironmentStatus then reads previous?.snapshot?.status ?? previous?.status, so recovery finds the same runtimeId and does not advance the connection generation. runtime-host-status-owner.ts complete(); runtime-status.ts previousVerifiedStatus; pinned by runtime-status-snapshot.test.ts → "represents failed verification honestly without manufacturing a session restart or toast"

Nothing was lost. The deletion was correct and the ladder must not be re-landed. #20003's mechanism is a strict improvement: one retry slot per connection shared by all readers, instead of a renderer-side timer keyed on a capability predicate.

Q2 — is the flipped assertion in runtime-status.test.ts correct, or was main right? The flip is correct; main's assertion pinned a defect.

main's records null and returns false when a runtime refresh fails stubs an API with no getStatusSnapshots and a getStatus that rejects, so it exercises refreshRuntimeEnvironmentStatus's catch branch — the one place with no snapshot and no owner behind it. Measured on main at 94c2f96ea46, that path over a recorded live verdict produces:

{ reachable: false, status: null, disconnectToastCalls: 1 }

So it does not merely record null: it fires showRuntimeDisconnectedToast and drops the environment out of getReachableRuntimeSessionMirrorTargets. That is a user-facing "host disconnected" claim and a mirror retirement, both derived from a call that threw. docs/reference/ssh-execution-boundary.md: "Absence from a client-side set, a lookup that threw, a socket that closed, a command that timed out — none of these observe the process. They are unverifiable by construction." main's own snapshot reader already refuses the same inference in the same file — applyRuntimeHostStatusSnapshot routes a non-verified snapshot around setRuntimeEnvironmentStatus with the comment "Lost contact or a failed method observes no runtime session ending." The catch branch was simply never revisited when that landed.

Mutation-proven both ways, on 200a98c19ea, vitest run --config config/vitest.config.ts src/renderer/src/store/slices/runtime-status.test.ts (43 tests):

mutation result
guard disabled (if (false && …)) 1 failed / 42 passed — exactly preserves a recorded live verdict when a refresh probe fails, AssertionError: expected null to be { runtimeId: 'rt', … }
guard widened to if (entry.status === null) return (drop the live-verdict condition) 2 failed / 41 passed — records null on a first-contact refresh failure, so host coverage completes and shares one full catalog and status sweep across overlapping hydrations

The guard is therefore pinned from both sides: it cannot be removed and it cannot be widened.

What this PR actually delivers on main — narrower than the body below claims

The catch branch is the only writer the guard now covers. On main a transport fault no longer throws: getRuntimeEnvironmentStatus resolves ok: false through the owner, and the renderer takes the snapshot path. Both production shells (src/preload/api/runtime-environments-bridge.ts, src/renderer/src/web/preload-api/web-runtime-environments-api.ts) expose getStatusSnapshots, so the guard fires only when getStatus itself rejects — an unknown selector, an IPC failure, or an unexpected throw. The #19647 invariant on the main path is now enforced by #20003's owner; this guard backstops the exception path. That is a real and correctly-pinned narrowing, not a no-op, but the "Proof of fix" measurements further down were taken against the pre-#20003 tree and describe a writer that no longer exists.

Follow-up commit 6ad1e48e4d6

  • runtime-status.ts: the guard's comment described a recheck ladder that no longer exists. Restated in fix(runtime): let connections own host status recovery #20003's vocabulary. The guard itself is byte-for-byte unchanged.
  • paired-remote-terminal-tab-switch-reconnect.spec.ts: the flap arm's expect(flapped.sawNullStatus).toBe(false) gate cannot hold on main and would fail. fix(runtime): let connections own host status recovery #20003 split the vocabulary — the store entry's status means "verified on the current socket" and the owner's snapshot.status holds the last verdict — so a dropped socket publishes verification: 'unavailable' and the renderer derives entry.status = null by design. That null is also what drives the session-tabs mirror's own rebuild once the socket returns (the mirror-target key changes, which reinstalls the subscriptions), so asserting it stays non-null would contradict the recovery the same arm measures. Demoted to a logged diagnostic with that reason recorded inline. The connection-generation and pane-mounted assertions — the actual [Bug] Orca has to reconnect every time I switch tabs #19647 contract — are unchanged.

Open defect found while answering this, NOT introduced here and NOT blocking

main retires a live host's session-tabs mirror on an unverifiable probe over a ready transport. Reproduced against 94c2f96ea46 by driving the store's own publication path:

verified snapshot            -> mirror targets: [{ environmentId: 'env-a', runtimeId: 'rt-1', … }]
snapshot(verification:'unavailable', transport:'ready')
  entry: { status: null, snapshot.status: { runtimeId: 'rt-1' }, verification: 'unavailable', transport: 'ready' }
  mirror targets: []

applyRuntimeHostStatusSnapshot nulls entry.status for any non-verified snapshot; getReachableRuntimeSessionMirrorTargets skips on !status?.status; that key feeds a useEffect in use-web-session-tabs-sync.ts whose cleanup unsubscribes session.tabs.subscribeAll and calls clearWebSessionTabsTrackingForEnvironment. So a status probe that goes unanswered while the shared-control socket is still ready cold-reloads the host's tab mirror — while main's own runtimeHostConnectionStateForEntry calls that same entry runtime-unavailable, which isConnectedRuntimeHostState counts as connected. Two derivations, opposite verdicts, and the destructive one wins.

This is #20003's reader, not this PR's writer, and it is deliberately not fixed here: holding the target through an outage without a resubscribe-on-stream-death path would strand the mirror, because today's teardown is what restores it (installWindowVisibilitySubscriptionParking retries subscribe failures only; a mid-stream { type: 'end' } triggers no resubscribe). That needs its own change.

Conflict ledger

file resolution
runtime-host-connection-state.ts / .test.ts Took main. Its copy is a strict superset — same derivation plus a snapshot branch in runtimeHostConnectionStateForEntry, plus one extra test.
runtime-status-recheck.ts / .test.ts Accepted main's deletion. Confirmed correct — see Q1.
runtime-status.ts Took main for the removed reconcileRuntimeStatusRecheck call. The #19647 guard survives and is mutation-proven.
runtime-status.test.ts The assertion flip. Confirmed correct — see Q2.

Gates on 6ad1e48e4d6

  • node config/scripts/run-typecheck-projects-in-parallel.mjs — exit 0, empty log. Proven failable first by injecting const __gateProof: number = "not a number" → TS2322, exit 1.
  • vitest run --config config/vitest.config.ts src/renderer/src/store/slices/runtime-status.test.ts — 43 passed, exit 0. Proven failable by the two mutations above.
  • oxlint and oxfmt --check on both changed files — clean.
  • The one tsc -p config/tsconfig.e2e.json error in the spec (TS2345 at line 406, setActiveView('changes')) is pre-existing on 200a98c19ea and untouched; that project is red repo-wide and is not a gate.

#20059 carries the same commits

#20059 carries these commits and hit the identical conflict. Both verdicts apply there unchanged, and 6ad1e48e4d6 was cherry-picked onto it as 51ebc047ce1 — the two contested files are byte-identical across both branches (blobs 1dd012cb317 and 1b798747a27 before, identical trees after).

ELI5

When you switch tabs against a remote Orca server over a flaky link (Tailscale, VPN), the client would suddenly decide the server was gone and rebuild everything — tearing down its live session-tab mirror and demanding a reconnect — even while your agent kept happily running on that server. It also dimmed the sidebar and showed a red "disconnected" glyph for a server it had never actually failed to reach.

The reason: a background status probe (status.get) dials its own brand-new socket. On a half-open link, that fresh dial hangs and fails while your established terminal/agent flows keep delivering. The client treated that "I couldn't even ask" failure as "the host is gone" and wrote status: null over a perfectly live verdict. That one write dropped the environment out of the session-tabs mirror targets (teardown #1), and when the next probe succeeded it advanced the connection generation and rebuilt the mirror again (teardown #2). One transient probe failure, two full mirror teardowns.

This makes the writer honest: loss of contact is unverifiable, never "exited" (per docs/reference/ssh-execution-boundary.md). A status.get that fails with runtime_unavailable — the code the client's own main process mints for a transport failure it never got an answer to — no longer overwrites a recorded live verdict. The live status stays, the mirror is never retired, the connection generation never bumps, and the status recheck ladder keeps probing until it gets a real answer. First-contact failures (no prior live verdict) still record null so host coverage completes.

It also carries forward the reader-side fixes from #19163 (against #16516): the recheck ladder now re-asks a status: null host on its backoff, and the sidebar/glyph use the shared runtimeHostConnectionState derivation so a host that was never probed is no longer painted the same destructive red as one a probe found unreachable.

Root cause

fireRuntimeStatusRecheck (src/renderer/src/store/slices/runtime-status-recheck.ts) and the refreshRuntimeEnvironmentStatus wrapper (src/renderer/src/store/slices/runtime-status.ts) caught a failed status.get and published { status: null } unconditionally — while the sibling caller runtime-client-ipc-bridge.ts:200 already documented that a failed probe here is unverifiable and must publish nothing. status.get is exactly the call that dials a fresh socket (sendRemoteRuntimeRequest), so a flaky path kills it while established flows keep delivering.

null was carrying three meanings — "never asked", "asked and the socket failed", "host is gone, retire its mirror". The mirror read meaning two as three (#19647); the sidebar read meaning one as three (#16516).

The discriminator (verified)

Main synthesizes code: 'runtime_unavailable' for a transport failure in getRuntimeEnvironmentStatus (src/main/ipc/runtime-environment-transport-routing.ts). I verified the host status.get handler (src/main/runtime/orca-runtime-get-status.ts, rpc/methods/status.ts) never mints that code — it is minted only by the client's own main process and never crosses the wire. So keying on it is safe under docs/reference/remote-wire-compatibility.md Rules 1 and 3 (no wire field changes; nothing new is published to old peers). Verdict on the discriminator: sound and confirmed.

Proof of fix (measured, not inspected)

New e2e rig tests/e2e/paired-remote-terminal-tab-switch-reconnect.spec.ts runs a real paired desktop client through a TCP hop whose stall-new mode lets established flows deliver while fresh dials hang (the reporter's Tailscale shape).

  • Baseline (unfixed): on a link flap under stall-new, nullStatusEnvironments went 0 -> 1 (status nulled) and the connection generation advanced 0 -> 1 (mirror rebuilt) — the double teardown, reproduced.
  • Fixed: nullStatus=false throughout the flap and connection generation stays 0 -> 0. The live verdict survives; recovery is not a second connection.

Arm A (healthy tab switching) showed transportSubscribeCount delta 0 across 3 park/reveal rounds — plain tab switching does not by itself churn the control channel; the churn is the writer nulling a live verdict on a transport fault.

Out of scope — filed as issues

Tests

Co-authored-by: Omar Shahine 10343873+omarshahine@users.noreply.github.com
Co-authored-by: ASDFGHoney 86656987+ASDFGHoney@users.noreply.github.com

@coderabbitai

coderabbitai Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

📝 Walkthrough

Walkthrough

The runtime status refresh now preserves a recorded live verdict when a later status request fails. First-contact failures can still publish a null status according to policy. New end-to-end helpers inject TCP link faults for reconnect testing. New Playwright coverage tests terminal reconnection and runtime-host sidebar glyph states.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: 🔵 Low · up to 6ad1e

Runtime status handling now keeps a previously confirmed live host verdict when a status probe fails over a broken link, instead of showing the host as unknown. This behavior is narrow and covered by unit tests, so user-facing risk is low. The main remaining gap is in the new reconnect end-to-end test, whose key assertion can pass without actually exercising the behavior it is meant to protect, so the regression guarantee is weaker than it appears.

🚥 Pre-merge checks | ✅ 1 | ❌ 4

❌ Failed checks (4 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning Issue #20059 requires an outage guard in host-mirror-handle-gap-wait.ts. The guard must classify an expired parked-pane handle-gap wait as link loss by using runtimeHostConnectionStateForEntry, so… Add the #20059 guard to host-mirror-handle-gap-wait.ts. Add tests that cover an outage during the wait, subsequent bounded waiting, TypeScript validation, and mutation coverage for the outage branch.
Out of Scope Changes check ⚠️ Warning The live-verdict preservation change in runtime-status.ts and the paired-client tab-switch end-to-end test target the status.get transport failure and issue #19647 behavior. The sidebar disconnect… Remove the unrelated runtime-status, tab-switch, and sidebar-glyph changes from this issue's change set, or link them to the issue that defines those requirements. Keep only changes that implement or support the #20059 handle-gap safeguard.
Docstring Coverage ⚠️ Warning Docstring coverage is 21.88% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 32 functions across 13 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description provides detailed technical context, rationale, issue references, validation results, and test coverage. However, it omits the required template sections and information, including an … Reformat the description using the repository template. Add an explicit linked issue entry, attach before/after visual proof or state N/A with a reason, document manual testing and platforms, complete the Testing and checklist items, and …
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: preserving a live runtime host verdict when a status probe cannot reach the host.
Full details: Linked Issues check

Explanation

Issue #20059 requires an outage guard in host-mirror-handle-gap-wait.ts. The guard must classify an expired parked-pane handle-gap wait as link loss by using runtimeHostConnectionStateForEntry, so later handle gaps still use the bounded wait. The change summary lists runtime-status.ts, its tests, link-fault helpers, and unrelated end-to-end specs. It does not show the required handle-gap implementation or its tests. The reviewed head also does not provide reviewable evidence of that file. The required combined host-mirror tests and mutation coverage are therefore not present in the available change evidence.

Full details: Out of Scope Changes check

Explanation

The live-verdict preservation change in runtime-status.ts and the paired-client tab-switch end-to-end test target the status.get transport failure and issue #19647 behavior. The sidebar disconnected-glyph test also does not exercise parked-pane handle-gap expiry. These changes do not implement or verify issue #20059's handle-gap safeguard. The shared runtimeHostConnectionStateForEntry definition is relevant dependency context, but it does not connect the listed changes to the required handle-gap behavior.

Full details: Description check

Explanation

The description provides detailed technical context, rationale, issue references, validation results, and test coverage. However, it omits the required template sections and information, including an explicit Fixes #... entry, Visual Proof or an N/A explanation, the Testing checklist, and the remaining required checklist sections.

Resolution

Reformat the description using the repository template. Add an explicit linked issue entry, attach before/after visual proof or state N/A with a reason, document manual testing and platforms, complete the Testing and checklist items, and include the required Notes and Review sections.

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 007e08d1-985e-435a-976f-54670c47981f

📥 Commits

Reviewing files that changed from the base of the PR and between e74c22a and 1751433.

📒 Files selected for processing (13)
  • src/renderer/src/components/sidebar/NoticeHostGlyph.test.tsx
  • src/renderer/src/components/sidebar/NoticeHostGlyph.tsx
  • src/renderer/src/components/sidebar/WorktreeCard.ssh-reconnect-prompt.test.tsx
  • src/renderer/src/components/sidebar/use-worktree-card-foundation.ts
  • src/renderer/src/runtime/runtime-host-connection-state.test.ts
  • src/renderer/src/runtime/runtime-host-connection-state.ts
  • src/renderer/src/store/slices/runtime-status-recheck.test.ts
  • src/renderer/src/store/slices/runtime-status-recheck.ts
  • src/renderer/src/store/slices/runtime-status.test.ts
  • src/renderer/src/store/slices/runtime-status.ts
  • tests/e2e/helpers/runtime-endpoint-link-fault.ts
  • tests/e2e/paired-remote-terminal-tab-switch-reconnect.spec.ts
  • tests/e2e/sidebar-runtime-host-disconnected-glyph.spec.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 5 remain after this review.

Comment on lines +243 to +249
const statuses = window.__store?.getState().runtimeStatusByEnvironmentId
let nullStatusEnvironments = 0
for (const entry of statuses?.values() ?? []) {
if (entry.status === null) {
nullStatusEnvironments += 1
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Scope the null-status count to the target environment.

The loop counts status === null across every entry in runtimeStatusByEnvironmentId. The hard gate at Line 505 then fails when any unrelated environment carries a null status. Per the PR objectives, a first-contact failure still records status: null, so an unrelated or never-probed host produces a false #19647 failure. Count the null verdict for environmentId only.

🐛 Proposed fix
-      const statuses = window.__store?.getState().runtimeStatusByEnvironmentId
-      let nullStatusEnvironments = 0
-      for (const entry of statuses?.values() ?? []) {
-        if (entry.status === null) {
-          nullStatusEnvironments += 1
-        }
-      }
+      const statuses = window.__store?.getState().runtimeStatusByEnvironmentId
+      const targetEntry = statuses?.get(environmentId)
+      const nullStatusEnvironments = targetEntry && targetEntry.status === null ? 1 : 0
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const statuses = window.__store?.getState().runtimeStatusByEnvironmentId
let nullStatusEnvironments = 0
for (const entry of statuses?.values() ?? []) {
if (entry.status === null) {
nullStatusEnvironments += 1
}
}
const statuses = window.__store?.getState().runtimeStatusByEnvironmentId
const targetEntry = statuses?.get(environmentId)
const nullStatusEnvironments = targetEntry && targetEntry.status === null ? 1 : 0

Comment on lines +509 to +512
expect(
generationAfterRecovery,
'recovery advanced the connection generation, so the session-tabs mirror was rebuilt (#19647)'
).toBe(generationBeforeFlap)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Prove the generation was recorded before comparing it.

connectionGeneration falls back to null when the store holds no entry for environmentId. If that happens, both reads are null and this gate passes without testing the writer. Assert the pre-flap generation is a number first.

🐛 Proposed fix
+    expect(
+      generationBeforeFlap,
+      'the client never recorded a connection generation for the paired environment'
+    ).not.toBeNull()
     expect(
       generationAfterRecovery,
       'recovery advanced the connection generation, so the session-tabs mirror was rebuilt (`#19647`)'
     ).toBe(generationBeforeFlap)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
expect(
generationAfterRecovery,
'recovery advanced the connection generation, so the session-tabs mirror was rebuilt (#19647)'
).toBe(generationBeforeFlap)
expect(
generationBeforeFlap,
'the client never recorded a connection generation for the paired environment'
).not.toBeNull()
expect(
generationAfterRecovery,
'recovery advanced the connection generation, so the session-tabs mirror was rebuilt (#19647)'
).toBe(generationBeforeFlap)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Writer fix (the crux) — fireRuntimeStatusRecheck now preserves a recorded live verdict when a status.get fails with runtime_unavailable (minted only by the client's own main process on a transport failure it never got an answer to), refreshing just the remoteControl diagnostics so the recheck ladder keeps probing without a teardown; refreshRuntimeEnvironmentStatus does the same for the default refresh path.
  • Reader-side carry-forward (#16516) — shouldRecheck(null) now returns true so a status: null host is re-asked on backoff, and the sidebar/glyph switch from raw !entry?.status truthiness to the shared runtimeHostConnectionStateForEntry derivation (absent entry → checking, not destructive).
  • Manual-disconnect short-circuit — a runtime_manually_disconnected probe failure cancels the recheck instead of retrying against a user decision.
  • Proof of fix — new unit tests plus a real paired-client e2e rig (stall-new TCP hop) whose hard gates (sawNullStatus === false, connection generation unchanged) both fail on the unfixed writer.

I traced the discriminator end to end: the host status.get handler (src/main/runtime/rpc/methods/status.ts) returns getStatus() only, and runtime_unavailable is synthesized solely in getRuntimeEnvironmentStatus (src/main/ipc/runtime-environment-transport-routing.ts:80), so keying on it is sound with no wire change. Generation stability follows because a preserved live status keeps runtimeSessionStarted false in setRuntimeEnvironmentStatus. Tests assert exact outcomes that the pre-fix code would fail.

ℹ️ Preserve is broader than the shared-control hosts the rationale targets

The preserve guards in both runtime-status.ts and runtime-status-recheck.ts apply to any recorded live verdict, not just hosts advertising REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY. A legacy (non-shared-control) host has no recheck ladder (shouldRecheck returns false without the capability) and no shared-control diagnostics to settle the verdict, so a genuinely dead legacy host now keeps a stale "connected" verdict where it previously flipped to "disconnected" on a failed refresh. That is consistent with ssh-execution-boundary (loss of contact is unverifiable), and the mirror-teardown rationale is genuinely shared-control-specific — just flagging that the guard is intentionally broad and there is no automatic backstop to clear a stale verdict for that host class.

Technical details
# Preserve applies to legacy non-shared-control hosts

## Affected sites
- src/renderer/src/store/slices/runtime-status.ts:311 — default refresh drops `status: null` whenever a live status exists, no capability check
- src/renderer/src/store/slices/runtime-status-recheck.ts:183 — recheck preserve branches on `current?.status != null`, no capability check

## Open question for the human
- Is "stale connected" the intended outcome for legacy hosts with no shared-control diagnostics backstop, or should the preserve be scoped to shared-control-capable environments?

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

nwparker and others added 2 commits September 11, 2026 01:27
…h it (#19647)

A status.get dials its own fresh socket (sendRemoteRuntimeRequest), so on a
half-open link it hangs and fails with the client-minted `runtime_unavailable`
while established terminal/agent flows keep delivering. The status recheck ladder
and the refresh wrapper caught that failure and published `status: null` over a
recorded live verdict. That single write dropped the environment out of
getReachableRuntimeSessionMirrorTargets (mirror teardown #1), and the next
successful probe advanced the connection generation and rebuilt it (teardown #2)
— one transient probe failure, two full session-tabs mirror teardowns, and the
client demanding a reconnect while the agent kept working.

Treat that failure as unverifiable, per docs/reference/ssh-execution-boundary.md:
loss of contact is never evidence the host exited. A status.get that fails with
`runtime_unavailable` (client-minted, never on the wire — safe under
remote-wire-compatibility Rules 1 and 3) no longer overwrites a recorded live
verdict. The live status stays, the mirror is never retired, the connection
generation never bumps, and the recheck ladder keeps probing until a real answer.
First-contact failures (no prior live verdict) still record null so host coverage
completes.

Carries forward the reader-side fixes from #19163 (filed against #16516): the
recheck ladder re-asks a status: null host on its backoff and cancels on manual
disconnect, and NoticeHostGlyph / use-worktree-card-foundation use the shared
runtimeHostConnectionState derivation so a never-probed host is not painted the
destructive red of one a probe found unreachable.

Measured with a new paired-client e2e rig (stall-new TCP hop): baseline nulls the
status and bumps the generation 0->1 on a link flap; fixed keeps nullStatus=false
and the generation stable. Two adjacent defects found while measuring are filed
separately: #19871 (multiplexer socket dropped on last-stream close) and #19872
(un-park banner claims auto-retry while parkRetry arms no timer).

Co-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com>
Co-authored-by: ASDFGHoney <86656987+ASDFGHoney@users.noreply.github.com>
@nwparker
nwparker force-pushed the nwparker/reconnect-churn-19647-writer branch from 1751433 to 200a98c Compare September 11, 2026 08:34
#20003 moved failed-status retries into the connection's status owner and split
the vocabulary this PR's guard was written against: the store entry's `status`
now means "verified on the current socket", while the owner's
`snapshot.status` keeps the last verdict across an unverifiable probe. The
guard itself is unchanged and still correct — it is the only writer for a
`status.get` that threw, where there is no snapshot and no retry ladder — but
its comment described a recheck ladder that no longer exists.

The e2e flap arm's `sawNullStatus` gate does not survive that split. Under
#20003 a dropped socket publishes `verification: 'unavailable'` and the
renderer derives `entry.status = null` from it by design, and that null is
what drives the session-tabs mirror's own rebuild once the socket returns.
Asserting it stays non-null would contradict the recovery the same arm
measures, so it is logged rather than gated. The connection-generation and
pane-mounted assertions, which are the actual #19647 contract, are unchanged.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
tests/e2e/paired-remote-terminal-tab-switch-reconnect.spec.ts (1)

503-506: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert that a connection generation was recorded before comparing it.

observePane returns connectionGeneration: null when the store holds no entry for environmentId (Line 255). If the client never records an entry, generationBeforeFlap and generationAfterRecovery are both null, and this toBe passes without exercising the writer. This is the only hard gate in the test, so a vacuous pass reports the #19647 contract as covered when it is not.

💚 Proposed fix
+    expect(
+      generationBeforeFlap,
+      'the client never recorded a connection generation for the paired environment'
+    ).not.toBeNull()
     expect(
       generationAfterRecovery,
       'recovery advanced the connection generation, so the session-tabs mirror was rebuilt (`#19647`)'
     ).toBe(generationBeforeFlap)

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 17517486-2986-4a98-8d42-a923648ade55

📥 Commits

Reviewing files that changed from the base of the PR and between 1751433 and 6ad1e48.

📒 Files selected for processing (3)
  • src/renderer/src/store/slices/runtime-status.test.ts
  • src/renderer/src/store/slices/runtime-status.ts
  • tests/e2e/paired-remote-terminal-tab-switch-reconnect.spec.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 2 remain after this review.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since the prior Pullfrog review, the PR was rebased onto main, which had landed #20003 (let connections own host status recovery) — that deleted runtime-status-recheck.ts and moved failed-status retries into the connection-owned RuntimeHostStatusOwner, and this rebase accepted that deletion. Most of the originally-reviewed production surface is gone; the net delta is now a single exception-path guard plus test restructuring.

  • Narrowed the writer fix to an exception-path guard — refreshRuntimeEnvironmentStatus now skips publishing a { status: null } entry (no snapshot) when a recorded live verdict exists, so a status.get that threw no longer retires the session-tabs mirror. The main-path #19647 invariant is enforced by #20003's owner; this guard backstops the throw case.
  • Restated the guard's comment in #20003's vocabulary (the connection's status owner holds the last verdict and retries), with the guard itself byte-for-byte unchanged.
  • Flipped runtime-status.test.ts to pin "preserves a recorded live verdict when a refresh probe fails" and split out "records null on a first-contact refresh failure" — both exact assertions, and the guard is mutation-pinned from both sides.
  • Demoted the e2e sawNullStatus gate to a diagnostic in paired-remote-terminal-tab-switch-reconnect.spec.ts, keeping the connection-generation and pane-mounted hard gates as the #19647 contract.

The guard is correct: it only fires for no-snapshot { status: null } entries (the catch branch of runtime-status-refresh.ts), and snapshot-owned entries are handled upstream by applyRuntimeHostStatusSnapshot / setRuntimeEnvironmentStatus's snapshot guard, so the two paths can't shadow each other.

ℹ️ PR description below the rebase-record is stale

The ## ELI5, ## Root cause, ## Proof of fix, and ## Tests sections still describe the pre-rebase implementation — the fireRuntimeStatusRecheck recheck ladder, shouldRecheck(null), the sidebar/glyph reader changes, the publishUnreachable option, and the runtime-status-recheck.test.ts / runtime-host-connection-state.test.ts files — none of which are in the current diff. The rebase-record note flags this ("narrower than the body below claims") but leaves the stale sections in place, so a reviewer or future maintainer who skips the note gets the wrong story. Recommend rewriting those sections (or folding them into the rebase-record) to describe the rebased deliverable: a single exception-path guard backed by #20003's owner.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

nwparker added a commit that referenced this pull request Sep 17, 2026
…ration

`connectionGeneration` carried two meanings and one reader was always wrong.
holding the session mirror through an outage leaves its subscriptions stranded
and that edge was the only thing left to revive them. But the same value is the
mirror's cache key -- use-runtime-session-mirror-environment-key.ts keys the
subscription effect on it, every published frame is stamped with it, and
web-session-terminal-retirement-proof-ledger.ts drops retained proofs when it
moves. So the bump #20085 needed as a resubscribe signal re-keyed and rebuilt
the mirror after any brief flap, which is the #19647 symptom #19873/#20059 fix.

Measured first: with the reconnect bump deleted, an ended stream followed by
recovery issues zero resubscribes, and the mirror's subscribe call registers no
`onClose`, so main's terminal close is dropped. #20085's claim is true -- the
subscription really is dead after recovery -- so the trigger has to exist. It
just must not be the cache key.

Give each meaning its own value:

- `connectionGeneration` returns to identity only: a new runtime session, a
  re-pair, an explicit clear. A same-runtime return no longer moves it, so no
  stamp, fence or retained proof is invalidated by a flap.
- `hostContactEpoch` counts "the host answered again after we lost contact". It
  lives on the store entry and is read only as a dependency of the two
  subscription effects in use-web-session-tabs-sync.ts -- never passed to an
  installer, never part of `environmentKey`, so it cannot become a stamp.

`useRuntimeSessionMirrorEnvironmentKey` becomes
`useRuntimeSessionMirrorEnvironmentKeys`, returning `environmentKey` (identity)
and `resubscribeSignal` (the epoch edge) from the one target scan, so the hot
ownership scan is not doubled.

Each direction is pinned by its own test: removing the resubscribe dependency
fails only 'reinstalls both session-tabs subscriptions when the host answers
again'; restoring the reconnect bump fails only the two key-stability tests.
nwparker added a commit that referenced this pull request Sep 17, 2026
…ration

`connectionGeneration` carried two meanings and one reader was always wrong.
holding the session mirror through an outage leaves its subscriptions stranded
and that edge was the only thing left to revive them. But the same value is the
mirror's cache key -- use-runtime-session-mirror-environment-key.ts keys the
subscription effect on it, every published frame is stamped with it, and
web-session-terminal-retirement-proof-ledger.ts drops retained proofs when it
moves. So the bump #20085 needed as a resubscribe signal re-keyed and rebuilt
the mirror after any brief flap, which is the #19647 symptom #19873/#20059 fix.

Measured first: with the reconnect bump deleted, an ended stream followed by
recovery issues zero resubscribes, and the mirror's subscribe call registers no
`onClose`, so main's terminal close is dropped. #20085's claim is true -- the
subscription really is dead after recovery -- so the trigger has to exist. It
just must not be the cache key.

Give each meaning its own value:

- `connectionGeneration` returns to identity only: a new runtime session, a
  re-pair, an explicit clear. A same-runtime return no longer moves it, so no
  stamp, fence or retained proof is invalidated by a flap.
- `hostContactEpoch` counts "the host answered again after we lost contact". It
  lives on the store entry and is read only as a dependency of the two
  subscription effects in use-web-session-tabs-sync.ts -- never passed to an
  installer, never part of `environmentKey`, so it cannot become a stamp.

`useRuntimeSessionMirrorEnvironmentKey` becomes
`useRuntimeSessionMirrorEnvironmentKeys`, returning `environmentKey` (identity)
and `resubscribeSignal` (the epoch edge) from the one target scan, so the hot
ownership scan is not doubled.

Each direction is pinned by its own test: removing the resubscribe dependency
fails only 'reinstalls both session-tabs subscriptions when the host answers
again'; restoring the reconnect bump fails only the two key-stability tests.
nwparker added a commit that referenced this pull request Sep 17, 2026
…ration

`connectionGeneration` carried two meanings and one reader was always wrong.
holding the session mirror through an outage leaves its subscriptions stranded
and that edge was the only thing left to revive them. But the same value is the
mirror's cache key -- use-runtime-session-mirror-environment-key.ts keys the
subscription effect on it, every published frame is stamped with it, and
web-session-terminal-retirement-proof-ledger.ts drops retained proofs when it
moves. So the bump #20085 needed as a resubscribe signal re-keyed and rebuilt
the mirror after any brief flap, which is the #19647 symptom #19873/#20059 fix.

Measured first: with the reconnect bump deleted, an ended stream followed by
recovery issues zero resubscribes, and the mirror's subscribe call registers no
`onClose`, so main's terminal close is dropped. #20085's claim is true -- the
subscription really is dead after recovery -- so the trigger has to exist. It
just must not be the cache key.

Give each meaning its own value:

- `connectionGeneration` returns to identity only: a new runtime session, a
  re-pair, an explicit clear. A same-runtime return no longer moves it, so no
  stamp, fence or retained proof is invalidated by a flap.
- `hostContactEpoch` counts "the host answered again after we lost contact". It
  lives on the store entry and is read only as a dependency of the two
  subscription effects in use-web-session-tabs-sync.ts -- never passed to an
  installer, never part of `environmentKey`, so it cannot become a stamp.

`useRuntimeSessionMirrorEnvironmentKey` becomes
`useRuntimeSessionMirrorEnvironmentKeys`, returning `environmentKey` (identity)
and `resubscribeSignal` (the epoch edge) from the one target scan, so the hot
ownership scan is not doubled.

Each direction is pinned by its own test: removing the resubscribe dependency
fails only 'reinstalls both session-tabs subscriptions when the host answers
again'; restoring the reconnect bump fails only the two key-stability tests.
nwparker added a commit that referenced this pull request Sep 17, 2026
…ration (#20359)

* fix(runtime): split the host-contact epoch out of the connection generation

`connectionGeneration` carried two meanings and one reader was always wrong.
holding the session mirror through an outage leaves its subscriptions stranded
and that edge was the only thing left to revive them. But the same value is the
mirror's cache key -- use-runtime-session-mirror-environment-key.ts keys the
subscription effect on it, every published frame is stamped with it, and
web-session-terminal-retirement-proof-ledger.ts drops retained proofs when it
moves. So the bump #20085 needed as a resubscribe signal re-keyed and rebuilt
the mirror after any brief flap, which is the #19647 symptom #19873/#20059 fix.

Measured first: with the reconnect bump deleted, an ended stream followed by
recovery issues zero resubscribes, and the mirror's subscribe call registers no
`onClose`, so main's terminal close is dropped. #20085's claim is true -- the
subscription really is dead after recovery -- so the trigger has to exist. It
just must not be the cache key.

Give each meaning its own value:

- `connectionGeneration` returns to identity only: a new runtime session, a
  re-pair, an explicit clear. A same-runtime return no longer moves it, so no
  stamp, fence or retained proof is invalidated by a flap.
- `hostContactEpoch` counts "the host answered again after we lost contact". It
  lives on the store entry and is read only as a dependency of the two
  subscription effects in use-web-session-tabs-sync.ts -- never passed to an
  installer, never part of `environmentKey`, so it cannot become a stamp.

`useRuntimeSessionMirrorEnvironmentKey` becomes
`useRuntimeSessionMirrorEnvironmentKeys`, returning `environmentKey` (identity)
and `resubscribeSignal` (the epoch edge) from the one target scan, so the hot
ownership scan is not doubled.

Each direction is pinned by its own test: removing the resubscribe dependency
fails only 'reinstalls both session-tabs subscriptions when the host answers
again'; restoring the reconnect bump fails only the two key-stability tests.

* test(runtime): pin the mirror hydration verdict across a host flap

The generation tests assert the key string; this asserts what the user feels.
The mirror's hydration verdict is stamped with the connection generation, so any
bump discards it and every mirrored pane re-parks -- the tab-list rebuild. Held
across an unverifiable probe, still discarded when the runtime id actually moved.

* test(runtime): build real host statuses instead of casting partials
nwparker added a commit that referenced this pull request Sep 17, 2026
…ration

`connectionGeneration` carried two meanings and one reader was always wrong.
holding the session mirror through an outage leaves its subscriptions stranded
and that edge was the only thing left to revive them. But the same value is the
mirror's cache key -- use-runtime-session-mirror-environment-key.ts keys the
subscription effect on it, every published frame is stamped with it, and
web-session-terminal-retirement-proof-ledger.ts drops retained proofs when it
moves. So the bump #20085 needed as a resubscribe signal re-keyed and rebuilt
the mirror after any brief flap, which is the #19647 symptom #19873/#20059 fix.

Measured first: with the reconnect bump deleted, an ended stream followed by
recovery issues zero resubscribes, and the mirror's subscribe call registers no
`onClose`, so main's terminal close is dropped. #20085's claim is true -- the
subscription really is dead after recovery -- so the trigger has to exist. It
just must not be the cache key.

Give each meaning its own value:

- `connectionGeneration` returns to identity only: a new runtime session, a
  re-pair, an explicit clear. A same-runtime return no longer moves it, so no
  stamp, fence or retained proof is invalidated by a flap.
- `hostContactEpoch` counts "the host answered again after we lost contact". It
  lives on the store entry and is read only as a dependency of the two
  subscription effects in use-web-session-tabs-sync.ts -- never passed to an
  installer, never part of `environmentKey`, so it cannot become a stamp.

`useRuntimeSessionMirrorEnvironmentKey` becomes
`useRuntimeSessionMirrorEnvironmentKeys`, returning `environmentKey` (identity)
and `resubscribeSignal` (the epoch edge) from the one target scan, so the hot
ownership scan is not doubled.

Each direction is pinned by its own test: removing the resubscribe dependency
fails only 'reinstalls both session-tabs subscriptions when the host answers
again'; restoring the reconnect bump fails only the two key-stability tests.
ivanvykopal pushed a commit to ivanvykopal/orca that referenced this pull request Sep 20, 2026
…ration (stablyai#20359)

* fix(runtime): split the host-contact epoch out of the connection generation

`connectionGeneration` carried two meanings and one reader was always wrong.
holding the session mirror through an outage leaves its subscriptions stranded
and that edge was the only thing left to revive them. But the same value is the
mirror's cache key -- use-runtime-session-mirror-environment-key.ts keys the
subscription effect on it, every published frame is stamped with it, and
web-session-terminal-retirement-proof-ledger.ts drops retained proofs when it
moves. So the bump stablyai#20085 needed as a resubscribe signal re-keyed and rebuilt
the mirror after any brief flap, which is the stablyai#19647 symptom stablyai#19873/stablyai#20059 fix.

Measured first: with the reconnect bump deleted, an ended stream followed by
recovery issues zero resubscribes, and the mirror's subscribe call registers no
`onClose`, so main's terminal close is dropped. stablyai#20085's claim is true -- the
subscription really is dead after recovery -- so the trigger has to exist. It
just must not be the cache key.

Give each meaning its own value:

- `connectionGeneration` returns to identity only: a new runtime session, a
  re-pair, an explicit clear. A same-runtime return no longer moves it, so no
  stamp, fence or retained proof is invalidated by a flap.
- `hostContactEpoch` counts "the host answered again after we lost contact". It
  lives on the store entry and is read only as a dependency of the two
  subscription effects in use-web-session-tabs-sync.ts -- never passed to an
  installer, never part of `environmentKey`, so it cannot become a stamp.

`useRuntimeSessionMirrorEnvironmentKey` becomes
`useRuntimeSessionMirrorEnvironmentKeys`, returning `environmentKey` (identity)
and `resubscribeSignal` (the epoch edge) from the one target scan, so the hot
ownership scan is not doubled.

Each direction is pinned by its own test: removing the resubscribe dependency
fails only 'reinstalls both session-tabs subscriptions when the host answers
again'; restoring the reconnect bump fails only the two key-stability tests.

* test(runtime): pin the mirror hydration verdict across a host flap

The generation tests assert the key string; this asserts what the user feels.
The mirror's hydration verdict is stamped with the connection generation, so any
bump discards it and every mirrored pane re-parks -- the tab-list rebuild. Held
across an unverifiable probe, still discarded when the runtime id actually moved.

* test(runtime): build real host statuses instead of casting partials
@nwparker

Copy link
Copy Markdown
Contributor Author

Closing as obsolete — the defect this fixes is no longer live on main.

Why. The prod change here is a 9-line guard in the renderer refresh callback: if getStatus throws and we already hold a live status, don't publish null. On main that null publish is already dropped one layer down, at src/renderer/src/store/slices/runtime-status.ts:172:

if (previous?.snapshot && !status.snapshot) {

Every live entry in the shipping app carries a snapshot — desktop preload, web preload and the test harness all expose getStatusSnapshots, so the refresh path always takes the applySnapshot branch on success, and main-process pushes land through applyRuntimeHostStatusSnapshot. The IPC handler resolves transport failures as {ok: false}, so the catch branch only fires on an unknown selector or an IPC throw — and even then the guard above covers it once a snapshot exists.

That leaves this PR's guard reachable only for a snapshot-less live entry, a shape only its own unit test manufactures.

Measured on main: a live entry seeded the way the app seeds it, with getStatus rejecting, keeps its status and connection generation and shows no disconnect — it passes without this PR. Disabling the guard at line 172 turns that same test red. So the existing guard is the load-bearing protection, not anything added here.

The rest of the family has landed separately: mirror retirement and parking/transport reads on an unverifiable probe; the connection generation not moving on a same-runtime return; a never-probed host not painting red; and the contact vocabulary rework.

The direction was right — an unreachable probe must never downgrade a live verdict, and first contact still records null. It is redundant, not wrong.

One asset worth salvaging separately: the stall-new TCP hop helper in tests/e2e/helpers/runtime-endpoint-link-fault.ts. The existing link-fault e2e is on/off, whereas this models "established flows still deliver while fresh dials hang", which nothing else covers. If picked up it needs the wip commit cleaned, a setActiveView('changes') type error fixed, and it should be framed as a rig addition rather than a fix — the gate it asserts is already unit-pinned elsewhere.

Also worth a small follow-up: there is no unit test pinning the line-172 guard directly. That is the protection the app actually relies on, and it is currently unpinned.

@nwparker nwparker closed this Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant