Skip to content

fix(runtime): hold four more host reads through an unverifiable probe - #20095

Merged
nwparker merged 4 commits into
mainfrom
nwparker/status-reads-hold-unverifiable-host
Sep 17, 2026
Merged

nwparker merged 4 commits into
mainfrom
nwparker/status-reads-hold-unverifiable-host

Conversation

@nwparker

@nwparker nwparker commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 3 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​441 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​441
Prod 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​48 $\color{#cf222e}{\Huge{\mathbf{−}}}$​9 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​39

Stacked on #20085 — it introduces lastVerifiedRuntimeStatus(), which this branch imports. Rebase onto main once #20085 lands.

runtime-status-snapshot.ts:29 nulls entry.status for any non-verified probe, while the snapshot separately retains the host's runtime identity. A host that is genuinely reachable — transport ready, flows still delivering — therefore has entry.status === null whenever its last probe was merely unverifiable. Every reader gating on entry.status != null misreads that live host as gone. #20085 fixed one instance; these are its siblings.

Governing rule: docs/reference/ssh-execution-boundary.md — verdicts are live / unverifiable / exited, no synonyms, and loss of contact is never evidence of process death.

The two findings that matter most

The client-event stream case is the worst of the family, because the failure is self-concealing. The stream is closed deliberately by the subscription reconciler, so the gap-replay path that normally recovers a dropped stream never runs — there is no gap to detect. The user loses worktree-appearance, sshStateChanged, automationsChanged and Linear events from a host whose terminals keep streaming the whole time, with nothing anywhere indicating why. It recovers only if a later probe happens to verify.

The wrong-explanation cases are worse than the silent ones, not better. client-creation-action-policy.ts hides "New browser tab" on the paired-web surface with the reason "this web client's paired runtime does not support browser streaming", and assertRuntimeManagedBrowserCreationAvailable throws the same text mid-action — for a host that is merely unprobed and fully capable. That sends the next person to debug a capability negotiation that is not the problem. A reviewer will instinctively rank these below the destructive ones; they should not be. (Not fixed here — see Left alone.)

Fixed

Five call sites across four files, each with a failing test written first.

File Change
hooks/ipc-events/runtime-environment-subscription-selection.ts one file-local isRuntimeHostStillInContact, shared by both selectors
store/slices/runtime-environment-ssh-selectors.ts isEnvironmentReachable -> same composition
components/landing-preflight-runtime.ts reachability -> same composition
runtime/web-session-tabs-sync/use-web-session-tabs-sync.ts ?.status?.runtimeId -> lastVerifiedRuntimeStatus(entry)?.runtimeId

No new predicate. The reachability sites compose two primitives that already exist and are already the shared vocabulary — isConnectedRuntimeHostState(runtimeHostConnectionStateForEntry(entry)). The web-tabs site is not a reachability read at all: it wants the last identity the host answered with, which is exactly what lastVerifiedRuntimeStatus() is for.

User-facing consequence of each

Site What a user experienced
getRuntimeClientEventEnvironmentIds Host drops out of the desired set -> runtime-client-events-sync.ts:138-144 unsubscribes its client-event stream. See above.
getReachableRuntimeEnvironmentIds Fires the "just went unreachable" edge -> markEnvironmentSshStateStale -> advances the SSH generation, which is a component of buildRuntimeClientEventEnvironmentKey. So even the active host — otherwise pinned into the desired set unconditionally — gets its subscription torn down and cold-rebuilt. Spurious project refresh on recovery.
use-web-session-tabs-sync.ts:42 The web twin of #20085. activeWorktreeRuntimeId flips to null; it is an effect dep, so installActiveSessionTabsSubscription tears down and cold-rebuilds. The paired web client's active tab list reloads through a fresh snapshot round-trip — twice per blip, out and back.
landing-preflight-runtime.ts:22 invalidatePreflightStatus() wipes preflightStatus to null and drops the visibility/focus listeners. The landing page's integration banner vanishes until a probe verifies again.
runtime-environment-ssh-selectors.ts:15 selectRuntimeAwareSshStatus/Error return null — mirrored SSH target rows blank out while their sessions are live. Conservative: hides, tears nothing down.

Mutation table — the four verdict-based sites

Entry shape Verdict Before After Changed
no entry checking out out —
verified + transport ready connected in in —
verified + workspace window closed workspace-window-closed in in —
unavailable + transport ready runtime-unavailable out in the fix
unavailable + transport disconnected reconnecting out out —
checking, never answered checking out out —
blocked disconnected out out —
retired disconnected out out —
unavailable + transport connecting/unknown, no ready control socket disconnected (fall-through gap) out out —
unavailable + transport connecting/unknown, ready control socket runtime-unavailable out in yes — independent transport evidence
legacy snapshot-less entry, status set / null connected / disconnected in / out in / out —

One intended cell flips. Every "genuinely gone" cell is byte-identical, so no host gains a permanent retry loop — that was the trap in the obvious fix (see Got wrong).

Mutation table — use-web-session-tabs-sync

Entry Dep value before After
verified runtime-a runtime-a runtime-a
unavailable, snapshot retains runtime-a null -> tears down runtime-a, held
verified runtime-b (host restart) runtime-b, restarts runtime-b, restarts
no entry null null

Both directions asserted, so the test distinguishes "held through a blip" from "never restarts".

Full inventory

Every reader of the nulled field, classified. The three sites originally handed over are marked with an asterisk.

Destructive — tears state down on a live host. All four fixed here: runtime-environment-subscription-selection.ts:24* and :36, use-web-session-tabs-sync.ts:42, landing-preflight-runtime.ts:22.

Conservative — under-reports, hides something.

  • runtime-environment-ssh-selectors.ts:15* — fixed. selectRuntimeAwareSshTargetRemoved returns false, which was already safe.
  • browser-pane/restored-client-hosted-recovery-window.ts:42* — the 45s clock never starts, and a mid-wait blip restarts it (the effect cleanup clears the timer). A restored client-hosted page spins indefinitely instead of offering the unavailable/reopen notice. Not fixed — see below.
  • settings/runtime-server-row.tsx:62,67 — the Settings host row flips to 'error'. The same component computes connectionState through the snapshot-aware path, so the two disagree within one component.
  • client-creation-action-policy.ts:72,98 — the wrong-explanation case described above.

Capability / metadata reads that go null by accident. Same root cause, different shape: these read host identity (capabilities, hostPlatform, pairedDeviceId), which never changes without the host answering, so they should all read lastVerifiedRuntimeStatus() and can never legitimately go null on a blip.

  • Real, with a demonstrable consequence: client-creation-action-policy.ts:72,98 (wrong-reason hide + throw); web-runtime-browser-creation-context.ts:78,90 and workspace-browser-tab-open.ts:52 (browser creation loses the host's capability set and device scope mid-blip); terminal-input-host-platform.ts:75,98 (host platform falls back, so keybinding/path behaviour silently switches platform under the user).
  • Real but consequence unproven: paired-runtime-parking-capabilities.ts:37 feeds canWatcherCoverParkedTerminalTab, and pty-connection/paired-parked-terminal-restore.ts:17 feeds deferred-session-reattach-choice.ts:96. Both plausibly affect whether a paired terminal parks or reattaches, but the reattach decision tree there has several interacting branches and the user-facing outcome could not be proven without more work than the evidence justified. Flagged, not claimed.
  • Incidental — reads the field but is already safe: sidebar/workspace-creator-visibility.ts:19, which falls back to environment.pairedDeviceId from the environment record.

Not this bug — already correct. SshStatusSegment.tsx:116, NoticeHostGlyph.tsx:40, use-worktree-card-foundation.ts:191, RepositoryHostSetupsSection.tsx:230 all route through runtimeHostConnectionStateForEntry. shared/execution-host-registry.ts:162-166 does status ?? snapshot?.status plus snapshot-aware health — a third inline implementation of the concept, but a correct one.

Beyond the three sites originally handed over

Two findings nobody had flagged:

  1. use-web-session-tabs-sync.ts:42 is the web twin of fix(runtime): hold the session mirror through an unverifiable probe #20085 — the same defect, the same consequence, on the paired-web path instead of the desktop path. Fixed here.
  2. There are two writers of the nulling rule, and they diverge. runtime-status-snapshot.ts:29 uses verification === 'verified' && !snapshot.retired and merges remoteControl into the status. runtime-status-refresh.ts:25 uses only verification === 'verified' and does neither. A retired host therefore publishes a non-null entry.status through the manual-refresh path and null through the event path. Not touched — it is writer-side, adjacent to fix(runtime): hold the session mirror through an unverifiable probe #20085's territory.

Left alone, deliberately

  • The runtimeHostConnectionStateForEntry fall-through gap. transport: 'connecting' with a failed probe yields disconnected; the right mapping under the boundary rule is checking. Reported, not fixed: five sites in this PR now route through that function, and so does fix(runtime): hold the session mirror through an unverifiable probe #20085. Changing it flips isDisconnectedRuntimeHostState to false for a connecting host, which changes fix(runtime): hold the session mirror through an unverifiable probe #20085's mirror-retention decision inside its own file. That is its author's call. (isConnectedRuntimeHostState stays false either way, so nothing in this PR moves.)
  • The capability-read cluster. The fix is mechanical, but the parking/reattach members need their own proof.
  • runtime-server-row.tsx — a product decision. "Show error when the probe failed" is defensible; that one component derives connectionState snapshot-aware and status not is the actual inconsistency, and picking the winner is a design call.
  • restored-client-hosted-recovery-window.ts — correct under the rule, but fixing it changes when a user-facing notice appears. Notice timing is a product decision.
  • The writer-side divergence in runtime-status-refresh.ts.

Got wrong, and how it was caught

  • The obvious predicate lastVerifiedRuntimeStatus(entry) != null was the first choice at the subscription site. Walking the mutation table before writing code showed that keeps a genuinely-down host (transport: 'disconnected') desired forever, adding a permanent 30s-cap subscribe retry loop against a dead host. Switched to isConnectedRuntimeHostState, which leaves every disconnected cell untouched.
  • The subscription defect was initially scoped to non-active hosts, since the active id is added unconditionally at :20-22. Tracing markEnvironmentSshStateStale -> SSH generation -> buildRuntimeClientEventEnvironmentKey showed the active host's subscription is rebuilt too.

Verification

Each gate was proven able to fail before its silence was trusted: a deliberate TS2322 in a touched file produced errors from config/tsconfig.tc.web.json (clean after revert); an unused var made oxlint exit 1 (these files exit 0); the three new tests failed 3/3, 1/2 and 1/2 respectively before the fixes, with their negative controls passing throughout.

Full renderer suite after the fixes: 3723 files / 31189 tests passed, 2+6 skipped, zero FAIL lines in the grepped log. tsconfig.tc.web.json and tsconfig.node.json both clean. oxfmt applied.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The web-tabs fix drops the disconnected guard its sibling mirror code keeps, so a blocked/retired host now holds its active session.tabs.subscribe instead of tearing it down — the inverse of the bug being fixed (see inline).

Reviewed changes

  • runtime-environment-subscription-selection.ts — getRuntimeClientEventEnvironmentIds and getReachableRuntimeEnvironmentIds now derive membership from the shared isConnectedRuntimeHostState(runtimeHostConnectionStateForEntry(...)) verdict via a file-local isRuntimeHostStillInContact, so an unverifiable-but-ready host stays in both sets.
  • runtime-environment-ssh-selectors.ts — isEnvironmentReachable composes the same verdict, keeping mirrored SSH rows for a ready host whose probe went unverifiable.
  • landing-preflight-runtime.ts — reachability now derives from the shared verdict instead of status === null.
  • use-web-session-tabs-sync.ts — activeWorktreeRuntimeId reads lastVerifiedRuntimeStatus(entry)?.runtimeId instead of entry.status?.runtimeId.
  • Three new test files covering held-through-blip and still-drops transport-down / retired / no-entry cases for the verdict-based sites, and hold-vs-restart for the web-tabs site.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

const environmentId = getExplicitRuntimeEnvironmentIdForWorktree(state, state.activeWorktreeId)
return environmentId
? (state.runtimeStatusByEnvironmentId.get(environmentId)?.status?.runtimeId ?? null)
? (lastVerifiedRuntimeStatus(state.runtimeStatusByEnvironmentId.get(environmentId))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lastVerifiedRuntimeStatus drops the disconnected guard the sibling mirror code keeps (lib/runtime-session-mirror-targets.ts:55). A blocked or retired host retains its last snapshot.status, so this returns the stale runtimeId and keeps the active session.tabs.subscribe alive for a host that is gone — the inverse of the bug this PR fixes, where the old entry.status?.runtimeId read (nulled on any non-verified probe) tore it down.

Technical details
# web-tabs site loses the disconnected guard

## Affected sites
- src/renderer/src/runtime/web-session-tabs-sync/use-web-session-tabs-sync.ts:45 — reads `lastVerifiedRuntimeStatus(entry)?.runtimeId` with no `disconnected` verdict guard.
- src/renderer/src/lib/runtime-session-mirror-targets.ts:55 — the #20085 sibling, which correctly guards with `isDisconnectedRuntimeHostState(runtimeHostConnectionStateForEntry(entry))` before reading the same helper.

## Why it regresses
- `runtimeHostConnectionStateForEntry` maps `retired` and `verification === 'blocked'` to `'disconnected'` (runtime-host-connection-state.ts:123), which the four verdict-based sites in this PR drop correctly.
- `lastVerifiedRuntimeStatus` = `snapshot.status ?? status ?? null` with no retired/blocked guard. `RuntimeHostStatusOwner.dispose()` and `authenticationRejected()` both merge patches that keep `snapshot.status` (the last verified identity), so a previously-verified host that becomes `blocked`/`retired` still yields a non-null `runtimeId`.
- Result: `activeWorktreeRuntimeId` stays `'runtime-a'`, so the effect does not tear down `installActiveSessionTabsSubscription`; the stale tab mirror persists and the subscribe errors are only logged (active-session-subscription.ts:236-238, 274-278), never uninstalled.

## Required outcome
- A `blocked`/`retired` host must tear down the active session-tabs subscription exactly as the old `entry.status?.runtimeId` read did, while an `unverifiable`-but-ready host (and `reconnecting`/`checking`) keeps it.

## Suggested approach
- Mirror the sibling: derive the verdict, return `null` when `isDisconnectedRuntimeHostState(runtimeHostConnectionStateForEntry(entry))`, and only then read `lastVerifiedRuntimeStatus(entry)?.runtimeId`. This keeps `runtime-unavailable`, `reconnecting`, and `checking` holding, while `blocked`/`retired` (→ `disconnected`) drop.
- Add a `blocked`/`retired` case to `web-session-tabs-sync-unverifiable-host.test.tsx` asserting the subscription is unsubscribed/not re-held, since the current mutation table omits it.

Two derivations read the same host state and reached opposite verdicts, and
the destructive one won. When a status probe came back unverifiable over a
still-ready transport, runtimeHostConnectionStateForEntry called the host
'runtime-unavailable' (connected) while getReachableRuntimeSessionMirrorTargets
dropped it, tearing down and cold-rebuilding the session-tab mirror while the
host's flows were still delivering.

The root cause is that applyRuntimeHostStatusSnapshot nulls entry.status for any
non-verified probe while the snapshot retains the runtime identity. The
connection-state reader consults the snapshot; the mirror-target reader did not.

Give both readers one answer:

- lastVerifiedRuntimeStatus() in shared/runtime-host-status.ts is now the single
  definition of "the last identity the host answered with". runtime-status.ts
  already had this inline as previousVerifiedStatus and now calls it.
- The mirror-target reader asks the shared connection verdict instead of
  entry.status, gated on isDisconnectedRuntimeHostState -- only the one exit
  verdict earns a destructive read, per docs/reference/ssh-execution-boundary.md.
  'checking' and 'reconnecting' are unverifiable, not evidence of an exit.

Holding the target through the outage would strand the mirror on its own: the
subscription is installed by the effect in use-web-session-tabs-sync.ts keyed on
useRuntimeSessionMirrorEnvironmentKey(), a stream 'end' frame is dropped without
resubscribing, and the parking layer retries only a rejected subscribe call. The
teardown was the recovery. So regaining contact now advances the connection
epoch, giving recovery its own "the host is back" trigger rather than leaving the
mirror to be restored as a side effect of having been destroyed.

The connection epoch is not the runtime session: a same-runtime return fires no
restart hook, no provider session bump, and no toast.
Siblings of the session-mirror defect fixed in #20085. runtime-status-snapshot
nulls `entry.status` for any non-verified probe while the snapshot retains the
host's identity, so a host with a ready transport that is still delivering reads
as gone to anything gating on `entry.status != null`.

- client-event subscription selection dropped the stream for such a host, and
  its disconnect edge bumped the SSH generation, rebuilding even the active
  host's subscription
- the web client's active session-tabs stream tore down and cold-rebuilt, twice
  per blip
- landing preflight discarded its whole result
- runtime-aware SSH selectors blanked mirrored target rows

Each now reads the shared verdict, isConnectedRuntimeHostState of
runtimeHostConnectionStateForEntry, or lastVerifiedRuntimeStatus where the read
is host identity rather than reachability. No new predicate: every
"genuinely gone" case is byte-identical, so nothing gains a retry loop.
@nwparker
nwparker force-pushed the nwparker/mirror-holds-unverifiable-host branch from ef82ede to e7046b1 Compare September 17, 2026 07:23
@nwparker
nwparker force-pushed the nwparker/status-reads-hold-unverifiable-host branch from ae5e9f7 to e17ac8e Compare September 17, 2026 07:23
@nwparker
nwparker changed the base branch from nwparker/mirror-holds-unverifiable-host to main September 17, 2026 09:02
@nwparker
nwparker merged commit 182ff17 into main Sep 17, 2026
34 checks passed
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: af9b90c5-25fa-4668-8dcb-8ece494814f4

📥 Commits

Reviewing files that changed from the base of the PR and between 67dda9a and e17ac8e.

📒 Files selected for processing (14)
  • src/renderer/src/components/landing-preflight-runtime.ts
  • src/renderer/src/components/landing-preflight-unverifiable-host.test.tsx
  • src/renderer/src/hooks/ipc-events/runtime-environment-subscription-selection.ts
  • src/renderer/src/lib/runtime-host-unverifiable-probe-readers.test.ts
  • src/renderer/src/lib/runtime-session-mirror-targets.test.ts
  • src/renderer/src/lib/runtime-session-mirror-targets.ts
  • src/renderer/src/lib/runtime-session-mirror-unverifiable-host.test.ts
  • src/renderer/src/runtime/web-session-tabs-sync-unverifiable-host.test.tsx
  • src/renderer/src/runtime/web-session-tabs-sync/use-web-session-tabs-sync.ts
  • src/renderer/src/store/slices/runtime-environment-ssh-selectors.ts
  • src/renderer/src/store/slices/runtime-status-reconnect-connection-generation.test.ts
  • src/renderer/src/store/slices/runtime-status-snapshot.test.ts
  • src/renderer/src/store/slices/runtime-status.ts
  • src/shared/runtime-host-status.ts

📝 Walkthrough

Walkthrough

The change adds a shared helper for retrieving the last verified runtime status. Runtime reachability now uses shared host connection verdicts across preflight, subscriptions, SSH selectors, and mirror target selection. Session tab synchronization preserves the last verified runtime identity during unverifiable probes. Connection generation now advances when contact returns after an outage. New tests cover ready transports, disconnected transports, retired hosts, checking states, remote-control closure, reconnection, and runtime replacement.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

ivanvykopal pushed a commit to ivanvykopal/orca that referenced this pull request Sep 20, 2026
…stablyai#20095)

* fix(runtime): hold the session mirror through an unverifiable probe

Two derivations read the same host state and reached opposite verdicts, and
the destructive one won. When a status probe came back unverifiable over a
still-ready transport, runtimeHostConnectionStateForEntry called the host
'runtime-unavailable' (connected) while getReachableRuntimeSessionMirrorTargets
dropped it, tearing down and cold-rebuilding the session-tab mirror while the
host's flows were still delivering.

The root cause is that applyRuntimeHostStatusSnapshot nulls entry.status for any
non-verified probe while the snapshot retains the runtime identity. The
connection-state reader consults the snapshot; the mirror-target reader did not.

Give both readers one answer:

- lastVerifiedRuntimeStatus() in shared/runtime-host-status.ts is now the single
  definition of "the last identity the host answered with". runtime-status.ts
  already had this inline as previousVerifiedStatus and now calls it.
- The mirror-target reader asks the shared connection verdict instead of
  entry.status, gated on isDisconnectedRuntimeHostState -- only the one exit
  verdict earns a destructive read, per docs/reference/ssh-execution-boundary.md.
  'checking' and 'reconnecting' are unverifiable, not evidence of an exit.

Holding the target through the outage would strand the mirror on its own: the
subscription is installed by the effect in use-web-session-tabs-sync.ts keyed on
useRuntimeSessionMirrorEnvironmentKey(), a stream 'end' frame is dropped without
resubscribing, and the parking layer retries only a rejected subscribe call. The
teardown was the recovery. So regaining contact now advances the connection
epoch, giving recovery its own "the host is back" trigger rather than leaving the
mirror to be restored as a side effect of having been destroyed.

The connection epoch is not the runtime session: a same-runtime return fires no
restart hook, no provider session bump, and no toast.

* test(runtime): drop the redundant status casts the new casting gate rejects

* fix(runtime): hold four more host reads through an unverifiable probe

Siblings of the session-mirror defect fixed in stablyai#20085. runtime-status-snapshot
nulls `entry.status` for any non-verified probe while the snapshot retains the
host's identity, so a host with a ready transport that is still delivering reads
as gone to anything gating on `entry.status != null`.

- client-event subscription selection dropped the stream for such a host, and
  its disconnect edge bumped the SSH generation, rebuilding even the active
  host's subscription
- the web client's active session-tabs stream tore down and cold-rebuilt, twice
  per blip
- landing preflight discarded its whole result
- runtime-aware SSH selectors blanked mirrored target rows

Each now reads the shared verdict, isConnectedRuntimeHostState of
runtimeHostConnectionStateForEntry, or lastVerifiedRuntimeStatus where the read
is host identity rather than reachability. No new predicate: every
"genuinely gone" case is byte-identical, so nothing gains a retry loop.

* test(runtime): build real host statuses instead of casting partials
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant