Skip to content

fix(ssh): a managed server restart no longer marks still-running agents done - #29604

Merged
nwparker merged 5 commits into
mainfrom
nwparker/managed-restart-agent-status
Oct 16, 2026
Merged

nwparker merged 5 commits into
mainfrom
nwparker/managed-restart-agent-status

Conversation

@nwparker

@nwparker nwparker commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​231 $\color{#cf222e}{\Huge{\mathbf{−}}}$​3 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​228
Prod 10 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​119 $\color{#cf222e}{\Huge{\mathbf{−}}}$​18 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​101

ELI5

When Orca's server on an SSH host restarts (a crash, or an update while terminals are open), the terminals keep running. Any coding agent that was still working in one of them was suddenly shown as finished, with a green check, on every connected client. This change shows "agent here, state unknown" instead, until the agent reports again.

What Changed

Before: an agent is working in a terminal on an SSH host. The host's managed Orca server restarts. Its separate terminal daemon keeps the shells and the agent alive, and output keeps streaming. Even so, every client (desktop, a second desktop, a phone) flips that agent to done, with a green check and an empty prompt. A freshly connected client sees done as well.

After: the agent keeps its identity and its last known state, marked unconfirmed. The sidebar and the tab show the agent without a spinner or a done check, exactly as a locally restored row already looks after a desktop restart. The next hook the agent sends (or a shell title painted after the turn) settles it as usual.

Mechanism: on startup the host's hook server reloads last-status.json and marks every nonterminal row restoredUnconfirmed, which is correct. But the session.tabs projection (buildRuntimeMobileAgentStatus) only knew two kinds of hook evidence, "live" and "identity". A restored row is neither, so it fell through to the last-resort projection. That projection synthesised state: 'done' stamped with Date.now(). The fresh timestamp then beat the client's existing working row in the renderer's merge (agent-status-patch.ts).

  • selectRuntimeHookAgentRowForPane now also returns the newest fresh, nonterminal restored row, unless a confirmed row at least as new supersedes it.
  • session.tabs projection gates it behind a new client capability (below); mobile native chat no longer offers Stop for an unconfirmed turn.
  • buildRuntimeMobileAgentStatus publishes that row's state and prompt with restoredUnconfirmed: true, dated by its own receivedAt, and without question fields (interactivePrompt, toolName). It does this before the done fallback. A shell title painted after the turn still retires it to done. A restarted server's default "Terminal N" title does not.

Why

The execution host owns agent status, and losing contact with a turn is not evidence that it ended. The verdict here is "unverifiable", not "exited". The renderer already renders restoredUnconfirmed rows as identity-only: no spinner, no attention badge, no question card. So reusing that flag gives the honest display with no new wire field, opcode or renderer code.

Alternatives considered:

  • Omit agentStatus for these panes. This drops the provider-session identity that native chat and resume need.
  • Keep the existing renderer row on the client. This leaves fresh and second clients, which have no prior row, still showing done.

Dating the row by its own evidence means it never outranks a client's newer status. Stripping the question fields keeps #12346 fixed: no question card is resurrected after a restart.

Mixed versions: an older client cannot read the flag, so it would take the last known state (working) as live, show a spinner, and offer Stop. The new publication is therefore capability-gated. Clients advertising agent-status.restored-unconfirmed.v1 receive it: the desktop and web renderers, and mobile, whose native chat now withholds Stop for an unconfirmed turn. Every other client keeps exactly the legacy done projection it received before.

Linked Issue

No pre-existing issue. Found in an SSH end-to-end soak.

Visual Proof

Before: after killing the managed server, both still-running agents show green "done" checks (tab bar and sidebar), and their titles reset.
After: a second client connecting after a forced managed-server update shows the agents without a spinner or a done check (unverifiable). The agents were still printing output.

(Screenshots from the isolated soak profile are in the PR comments.)

Testing

  • Live, end to end: an isolated desktop built from this branch and two real Linux SSH hosts behind a private sshd with a throwaway HOME. A fake hook-emitting agent kept running while the managed server was force-updated, which restarts orcad while the daemon keeps the PTYs. A fresh second client then saw working + restoredUnconfirmed instead of done.
  • pnpm test src/main/runtime/orca-runtime-hook-agent-status-projection.test.ts:
  • Also: runtime-mobile-agent-status-builder.test.ts (synced-leaf retirement, shell vs non-shell titles), session-tab-agent-status-projection.test.ts (capability gate), web-session-tabs-sync-remote-status-title-flap.test.ts (a restored block cannot pierce live client status). Each new behavioural test fails on main.
  • Full local replay of the pr.yml static-analysis job, including the ratchets and the orcad smoke. Typecheck ran via pnpm tc because the local bun is too old for the CI compiler.

Verified live vs only in tests: the end-to-end restart and fresh-client display were verified live on Linux (omarchy). These were verified only in unit tests: the capability gate, mobile Stop suppression, synced-leaf retirement and the client-authority fence.

  • I manually tested these changes locally
  • Automated tests added/updated, or explained why not below

Review

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Remote SSH: this is the case being fixed.
  • Mobile: gets the same publication. Old mobile builds ignore the flag and show the last known state.
  • Local desktop: the projection is host-side only, and local panes already restore this way.
  • Cost: no extra work on the hot path beyond one more slot in an existing loop.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1997effb-8d7c-4ff8-afee-770f4db0cde4
📥 Commits

Reviewing files that changed from the base of the PR and between 067ddf3 and 183e1ba.

📒 Files selected for processing: 5
  • src/main/runtime/orca-runtime-hook-agent-status-projection.test.ts
  • src/main/runtime/runtime-mobile-agent-status-builder.test.ts
  • src/main/runtime/runtime-mobile-agent-status-builder.ts
  • src/renderer/src/runtime/web-session-tabs-sync-remote-status-title-flap.test.ts
  • src/renderer/src/runtime/web-session-tabs-sync/agent-status-primitives.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

📝 Walkthrough

The runtime now selects eligible restored hook rows and projects them into mobile agent status while retaining their state, prompt, and timestamps. A sufficiently recent non-agent PTY title observation retires the restored status. Client capabilities determine whether the runtime preserves that status or projects it as legacy done. The mobile active-working resolution excludes statuses marked restoredUnconfirmed.

Priority: ⬇️ Low

Merge Risk: 🟡 Moderate · up to 183e1

An SSH agent that is still running can appear done after its restored evidence ages out or when an older confirmed status takes precedence. Resolve these status errors before merging.

Pre-merge checks: Passed 3 · Failed 2

Failed Failed checks: 2 warnings

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 14 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Warning The description is detailed and covers the user impact, mechanism, rationale, testing, visual proof, compatibility, and checklist. However, the required Linked Issue section does not contain an actual… Add a valid linked issue reference in the Linked Issue section, such as Fixes #12345, or provide the repository-approved issue link format if this change is exempt from the requirement.
Passed Passed checks: 3 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly identifies the SSH server restart issue and the primary behavior change: still-running agents no longer appear as done.
Full details: Description check

Explanation

The description is detailed and covers the user impact, mechanism, rationale, testing, visual proof, compatibility, and checklist. However, the required Linked Issue section does not contain an actual issue reference; it states that no pre-existing issue exists.

Resolution

Add a valid linked issue reference in the Linked Issue section, such as Fixes #12345, or provide the repository-approved issue link format if this change is exempt from the requirement.

  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6976d26e-ae8c-44e8-b2ae-0d2bd7c66061
📥 Commits

Reviewing files that changed from the base of the PR and between bb441d1 and bf7ed62.

📒 Files selected for processing: 6
  • mobile/src/session/use-mobile-native-chat-active-resolution.ts
  • src/main/runtime/orca-runtime-hook-agent-status-projection.test.ts
  • src/main/runtime/runtime-mobile-agent-status-builder.test.ts
  • src/main/runtime/runtime-mobile-agent-status-builder.ts
  • src/main/runtime/runtime-mobile-agent-status-projection.ts
  • src/main/runtime/session-lifecycle/projection/prune-mobile-session-tab-group-layout.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

Comment on lines +140 to +141
const restored = !pty?.lastAgentStatus && !shellTitleRetiredRestored ? hookRow.restored : null
if (restored) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
# Inspect the renewal contract and the two status branches; do not run repository code.
ast-grep outline src/main/runtime/runtime-mobile-agent-status-builder.ts --items all --match 'buildRuntimeMobileAgentStatus|renewRuntimeMobileAgentStatusFromPtyTitle'
rg -n -C 8 'renewRuntimeMobileAgentStatusFromPtyTitle|const liveRow|const restored =' src/main/runtime

Repository: stablyai/orca

Length of output: 41760


🏁 Script executed:

nl -ba src/main/runtime/runtime-mobile-agent-status-builder.ts | sed -n '70,175p'
printf '\n--- projection helper ---\n'
nl -ba src/main/runtime/runtime-mobile-agent-status-projection.ts | sed -n '1,115p'

Repository: stablyai/orca

Length of output: 11283


🏁 Script executed:

nl -ba src/main/runtime/runtime-mobile-agent-status-builder.ts | sed -n '1,75p'
printf '\n--- row selection and resolution ---\n'
rg -n -C 12 'function selectRuntimeHookAgentRowForPane|export function selectRuntimeHookAgentRowForPane|function resolveRuntimeHookLiveAgentRow|export function resolveRuntimeHookLiveAgentRow' src/main/runtime/runtime-mobile-agent-status-projection.ts
nl -ba src/main/runtime/runtime-mobile-agent-status-projection.ts | sed -n '115,245p'

Repository: stablyai/orca

Length of output: 10873


🏁 Script executed:

rg -n -C 12 'buildRuntimeMobileAgentStatus\(' src/main/runtime
rg -n -C 8 'retained.*RuntimeAgentRowSnapshot|RuntimeAgentRowSnapshot.*retained|retained:' src/main/runtime/runtime-mobile-session-projection.ts src/main/runtime

Repository: stablyai/orca

Length of output: 41230


Give a newer restored row precedence over an older confirmed hook row.

selectRuntimeHookAgentRowForPane keeps both rows when the restored row is newer. The live branch runs first, and the renewal helper can return the older done status when no PTY agent status exists. Skip that branch when the restored hook row is newer. Keep an explicitly retained client row unchanged.

🐛 Suggested fix
   const liveRow = retained ?? resolveRuntimeHookLiveAgentRow(hookRow.live, pty, nonAgentTitle)
-  if (liveRow) {
+  const restoredRowIsNewer =
+    !retained &&
+    hookRow.restored !== null &&
+    liveRow !== null &&
+    hookRow.restored.updatedAt > liveRow.updatedAt
+  if (liveRow && !restoredRowIsNewer) {

entry.providerSessionOnly !== true &&
entry.restoredUnconfirmed === true &&
entry.state !== 'done' &&
(entry.evidenceObservedAt ?? entry.receivedAt) >= freshAfter &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not expire an unconfirmed turn into done based only on age.

If an SSH agent runs without another hook event past AGENT_STATUS_STALE_AFTER_MS, this check removes its restored row. The builder can then publish done from the remaining provider-session identity, although the agent may still be running. Keep the turn unconfirmed until a confirmed row or a later shell-title observation resolves it.

Based on learnings: “Timers cannot reliably distinguish a stalled task from one still legitimately running elsewhere.”

Source: Learnings

@nwparker
nwparker merged commit 98064b2 into main Oct 16, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant