Skip to content

fix(terminal): allow independent hidden account restarts to progress - #31994

Merged
nwparker merged 3 commits into
mainfrom
nwparker/account-restart-batch-progress
Oct 23, 2026
Merged

nwparker merged 3 commits into
mainfrom
nwparker/account-restart-batch-progress

Conversation

@nwparker

@nwparker nwparker commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​282 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​282
Prod 5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​176 $\color{#cf222e}{\Huge{\mathbf{−}}}$​60 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​116

ELI5

When you switch Codex/Claude accounts, Orca asks whether to restart your running agents. If you say yes, agents in tabs you are not looking at get restarted in the background.

Before: those background restarts ran strictly one at a time. If one restart got stuck (its spawn never answered), every restart after it waited forever: those tabs stayed on the old account with typing blocked and no prompt explaining why.

After:

  • Up to 4 background restarts run at the same time. A free slot is reused as soon as any one finishes, so one stuck restart no longer blocks the others, and a batch finishes about twice as fast.
  • If a restart is still stuck after 90 seconds, its slot is freed and that tab shows the restart question again instead of staying silently blocked. If the stuck spawn finishes later anyway, Orca still adopts the new process and clears the question.

Addresses the "independent progress" and "visible, recoverable outcome" parts of #22243 (reported by @Tauri-EPO; the regression tests reproduce the stuck-spawn scenario from that issue, so they are credited as co-author). It does not diagnose why a real Windows spawn stalls, so it does not close the issue.

What changed

  • codex-detached-pane-restart.ts: the sweep starts claims concurrently instead of awaiting each pane in turn. Each claim reports when it finishes so the scheduler can fill the free slot.
  • detached-pane-restart-slots.ts (new): the shared cap of 4 slots, plus a 90s timer per claim. When the timer fires it frees the slot and reopens that pane's prompt. The per-PTY claim (inFlightPtyIds) stays until main settles, so a pane can't be claimed twice and a late result still goes through the existing adopt-or-reap checks.
  • detached-pane-restart-pty-release.ts (new): the existing reap/release helpers, moved out unchanged to keep the executor under the 300-line lint cap.
  • codex-detached-pane-restart-scheduler.ts: removes the "one sweep at a time" lock. Claims are still guarded per PTY.
  • Unchanged: all the existing per-pane safety checks (mounted-pane ownership, Claude waits until its turn ends, tab generation and leaf checks, adopting or reaping late spawn results, skipping foreign-machine PTYs).
  • An experimental reliability gate, agent-session.detached-account-restart-progress, updated to cover the stall deadline.

Verification

  • Rebased onto current main (28c726855e1d+, which fixes the unrelated headless-terminal-tab-close.ts max-lines failure that had turned the previous CI run red and skipped the test job). No conflicts.
  • Unit (real store and scheduler, synthetic PTY IPC): 76/76 pass across the queue-progress, Codex/Claude detached executor and restart-startup suites. The new test holds 4 spawns, lets 90s pass, and checks three things: the 5th pane restarts, the stalled pane is asking again, and resolving the stalled spawn late still rebinds the tab with nothing killed. Red/green: with the timer disabled, the new test fails. The 3 earlier progress tests fail on main's serial executor.
  • Real app, real PTYs (earlier run, macOS, folder workspace): 6 hidden real Codex PTYs, accepted account-switch restart, one PTY ignoring HUP/TERM. On main, panes finished at 2.4s steps up to 14.5s. On this branch, all 6 finished in 5.0–8.4s. In every run all leaves were rebound and no notices or pending entries were left behind. Revealing a restarted tab attached it to the replacement Codex process (screenshot below).
  • Concurrent Codex startups and the token-refresh race (checked against the real binary):
    • Source (cloned openai/codex main, codex-rs/login/src/auth/manager.rs and storage.rs): ChatGPT token refresh is guarded only within one process (refresh_lock semaphore). Before refreshing it re-reads auth.json from disk, and on a 401 it re-reads auth.json again. There is no lock across processes, and auth.json is written with truncate+write.
    • Exercise: a node-pty harness ran 4 real codex 0.163.0 processes at once against a temp CODEX_HOME. The access token was about to expire, and a local fake OAuth server accepted each refresh token only once. Nothing touched real credentials or the network. Results:
      • codex exec: 3 of 4 processes lost the race (refresh_token_reused). All 4 then picked up the winner's tokens from disk; no request used the old token after 2s.
      • TUI (codex, 3 runs): no error ever reached the screen, and every backend request used the refreshed token.
      • auth.json was left valid every time.
    • Claude Code (installed 2.1.309 native binary, read-only string inspection): refreshes under a cross-process .oauth_refresh.lock (stale 60s), with race_recovered / dead_holder_takeover handling. Concurrent Claude restarts are serialized by Claude itself.
  • oxlint clean. check:code-quality:changed and check:reliability-gates pass. pnpm tc passes on the rebased head. oxfmt --check clean on changed files.

revealed restarted codex pane

Scope and compatibility

  • Renderer-only. No IPC or wire changes, so mixed client/server versions are unaffected. Main already accepts concurrent spawns per worktree, and it serializes spawns for the same pane (reserveIdlePaneSpawn).
  • Remote/SSH PTYs and mobile: unchanged. Foreign-machine PTYs are still left for the mounted pane path.
  • Folder workspaces: verified live, as described above.

Remaining tradeoffs (honest)

  • Codex refresh race: up to 4 hidden Codex restarts can start together. If the new account's token needs a refresh at that moment, the Codex processes race and the losers get refresh_token_reused. In the harness they recovered silently from disk. Two limits on that evidence:
    • It does not prove what OpenAI's real server does on reuse. If the server revoked the whole token family, re-login would be needed.
    • The same race already exists on main whenever mounted panes restart together, which is uncapped, and when Orca restores several sessions at startup.
  • A stall is surfaced, not fixed: a stalled spawn is never cancelled. After 90s the pane asks again, but input stays blocked until the user answers. Accepting again waits behind the stuck spawn in main. A spawn that is merely slow (over 90s) shows the prompt briefly before its late result clears it.
  • Not exercised: Windows/ConPTY, WSL, Linux, real account credentials, and the rendered restart prompt.

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

📝 Walkthrough

The restart sweep now starts pending restarts concurrently and limits in-flight claims to four. It calls a settlement callback after removing each completed claim. After a claim stalls for 90 seconds, the sweep releases its slot and reopens the prompt. The scheduler uses the callback instead of tracking running sweeps and deferred requests. Tests cover queue progress, ownership checks, executor reinstallation, and late-result adoption. An experimental reliability gate records the invariant, test coverage, evidence, and promotion criteria.

Priority: ➖ Normal

Merge Risk: 🟡 Moderate · up to 44f3f

If a hidden account restart spawn never completes, the re-offered restart prompt can leave the affected pane's input blocked with no recovery. This is confined to a stalled-spawn edge case, but the retry flow should be fixed or the prompt left ambiguous before merging.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 5 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: allowing independent hidden account restarts to progress.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is detailed and on-topic. It includes the user impact, implementation changes, rationale, testing evidence, compatibility scope, and known tradeoffs. It does not fully follow the templ…
Full details: Description check

Explanation

The description is detailed and on-topic. It includes the user impact, implementation changes, rationale, testing evidence, compatibility scope, and known tradeoffs. It does not fully follow the template because it lacks an actual linked issue URL, explicit visual-proof compliance, AI disclosure or applicability, review/checklist sections, and the agent-skill boundary section. These omissions do not prevent review because the critical technical information is present.

Full details: Docstring Coverage

Explanation

Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 5 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 2
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔇 Additional comments: 4
src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts: 1

35-35: LGTM!

Also applies to: 41-49, 52-52, 113-113

src/renderer/src/components/terminal-pane/codex-detached-pane-restart-scheduler.ts: 1

72-74: LGTM!

src/renderer/src/components/terminal-pane/account-restart-queue-progress.test.ts: 1

1-245: LGTM!

config/reliability-gates.jsonc-88-88: 1

88-88: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Align blocking promotion with the manifest policy.

If this criterion permits blocking promotion, “or” allows promotion after only one threshold. policy.blockingPromotion requires both 100 runs and 14 days. Change “or” to “and,” or state that this criterion applies only to an earlier maturity level.


ℹ️ Review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 76ab9d88-cd85-4c7a-a67f-fbaa33266486
📥 Commits

Reviewing files that changed from the base of the PR and between c30fa62 and 62e2eeb.

📒 Files selected for processing: 4
  • config/reliability-gates.jsonc
  • src/renderer/src/components/terminal-pane/account-restart-queue-progress.test.ts
  • src/renderer/src/components/terminal-pane/codex-detached-pane-restart-scheduler.ts
  • src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

@nwparker
nwparker force-pushed the nwparker/account-restart-batch-progress branch from 41f0f6f to 44f3f7f Compare October 23, 2026 03:44

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 91e923e3-868d-4c9d-be3a-bd1d499426b0
📥 Commits

Reviewing files that changed from the base of the PR and between 62e2eeb and 44f3f7f2d33866ec5b499b976d76a344a511c558.

📒 Files selected for processing: 5
  • config/reliability-gates.jsonc
  • src/renderer/src/components/terminal-pane/account-restart-queue-progress.test.ts
  • src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts
  • src/renderer/src/components/terminal-pane/detached-pane-restart-pty-release.ts
  • src/renderer/src/components/terminal-pane/detached-pane-restart-slots.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

Comment on lines +108 to +112
releaseSlot = holdDetachedRestartSlot(ptyId, () => {
console.warn('[codex-restart] detached restart spawn stalled; asking again')
// Why: a late result still adopts and clears this prompt, since main may have stopped the old PTY.
reopenCurrentCodexRestartPrompt(claimedPane, ptyId)
onSettled?.()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🔎 Supported by static analysis

🏁 Script executed:

set -eu
file='src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts'
printf '%s\n' '--- restart file: relevant symbols and numbered source ---'
rg -n -F -- 'inFlightPtyIds' -e 'restartUnclaimedPane' -e 'sweepUnclaimedCodexPaneRestarts' -e 'holdDetachedRestartSlot' -e 'reopenCurrentCodexRestartPrompt' -e 'executeDetachedCodexPaneRestart' "$file" || true
nl -ba "$file" | sed -n '1,190p'

printf '%s\n' '--- scheduler callers and definitions ---'
rg -n -F --glob '*.ts' -e 'sweepUnclaimedCodexPaneRestarts' -e 'restartUnclaimedPane' -e 'holdDetachedRestartSlot' -e 'inFlightPtyIds' src/renderer/src || true

printf '%s\n' '--- stalled-spawn tests ---'
rg -n -F --glob '*.test.ts' -e 'stalled' -e 'inFlightPtyIds' -e 'reopenCurrentCodexRestartPrompt' -e 'sweepUnclaimedCodexPaneRestarts' src/renderer/src || true
test_files=$(rg -l -F --glob '*.test.ts' -e 'stalled' -e 'sweepUnclaimedCodexPaneRestarts' src/renderer/src || true)
for t in $test_files; do
  printf '\n--- %s ---\n' "$t"
  nl -ba "$t" | sed -n '1,260p'
done

Repository: stablyai/orca

Length of output: 45609


🏁 Script executed:

set -eu
printf '%s\n' '--- scheduler ---'
nl -ba src/renderer/src/components/terminal-pane/codex-detached-pane-restart-scheduler.ts | sed -n '1,150p'

printf '%s\n' '--- slot helper ---'
nl -ba src/renderer/src/components/terminal-pane/detached-pane-restart-slots.ts | sed -n '1,180p'

printf '%s\n' '--- exact restart-test matches ---'
rg -n -F --glob 'codex-detached-pane-restart.test.ts' \
  -e 'stalled' -e 'stall' -e 'asking again' -e 'in-flight' -e 'reopen' -e 'settled' \
  src/renderer/src/components/terminal-pane/codex-detached-pane-restart.test.ts || true

printf '%s\n' '--- restart-test focused ranges ---'
nl -ba src/renderer/src/components/terminal-pane/codex-detached-pane-restart.test.ts | sed -n '380,610p'

printf '%s\n' '--- execute tail and prompt helper ---'
nl -ba src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts | sed -n '163,390p'

Repository: stablyai/orca

Length of output: 22674


🏁 Script executed:

set -eu
printf '%s\n' '--- dedicated slot/stall tests and references ---'
rg --files src/renderer/src | rg 'detached-pane-restart|account-restart-queue-progress|restart.*slot' || true
rg -n -F --glob '*.ts' \
  -e 'STALLED_DETACHED_RESTART_MS' \
  -e 'holdDetachedRestartSlot(' \
  -e 'asking again' \
  -e 'restartRequested' \
  -e 'reopenCodexRestartPrompt' \
  -e 'queuePaneAccountRestarts' \
  src/renderer/src/components/terminal-pane src/renderer/src/store src/renderer/src/lib || true

printf '%s\n' '--- restart action declarations ---'
rg -n -F --glob '*.ts' \
  -e 'reopenCodexRestartPrompt:' \
  -e 'reopenCodexRestartPrompt =' \
  -e 'queuePaneAccountRestarts:' \
  -e 'queuePaneAccountRestarts =' \
  src/renderer/src/store src/renderer/src || true

printf '%s\n' '--- test files containing stall or slot behavior ---'
for t in $(rg -l -F --glob '*.test.ts' \
  -e 'STALLED_DETACHED_RESTART_MS' \
  -e 'holdDetachedRestartSlot' \
  -e 'asking again' \
  -e 'detached restart spawn stalled' \
  src/renderer/src || true); do
  printf '\n--- %s ---\n' "$t"
  nl -ba "$t" | sed -n '1,260p'
done

Repository: stablyai/orca

Length of output: 35957


🏁 Script executed:

set -eu
printf '%s\n' '--- stalled-spawn test remainder ---'
nl -ba src/renderer/src/components/terminal-pane/account-restart-queue-progress.test.ts | sed -n '246,330p'

printf '%s\n' '--- terminal restart state ---'
nl -ba src/renderer/src/store/terminals/terminal-restart-state.ts | sed -n '1,220p'

Repository: stablyai/orca

Length of output: 11800


🏁 Script executed:

set -eu
printf '%s\n' '--- restart notice state ---'
nl -ba src/renderer/src/components/terminal-pane/codex-restart-notice-state.ts | sed -n '1,180p'

printf '%s\n' '--- focused helper definitions in queue-progress test ---'
nl -ba src/renderer/src/components/terminal-pane/account-restart-queue-progress.test.ts | sed -n '1,115p'

Repository: stablyai/orca

Length of output: 5037


🏁 Script executed:

set -eu
file='src/renderer/src/components/codex-restart-notice-state.ts'
nl -ba "$file" | sed -n '1,180p'

Repository: stablyai/orca

Length of output: 1107


Do not reopen the prompt while the original spawn is still in flight.

The stall timer releases the slot, but inFlightPtyIds remains set until executeDetachedCodexPaneRestart settles. A second acceptance queues the same PTY ID, and the guard returns before the restart is consumed or spawned. restartRequested then keeps pane input blocked. If the original spawn never settles, this state can persist indefinitely.

Keep the prompt ambiguous until settlement. If retry is required, add cancellation or claim-generation handling that prevents the late original result from adopting after a new claim starts.

Suggested fix
--- "a/src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts"
+++ "b/src/renderer/src/components/terminal-pane/codex-detached-pane-restart.ts"
@@ -105,11 +105,11 @@
     }
     inFlightPtyIds.add(ptyId)
     const claimedPane = located
     releaseSlot = holdDetachedRestartSlot(ptyId, () => {
       console.warn('[codex-restart] detached restart spawn stalled; asking again')
-      // Why: a late result still adopts and clears this prompt, since main may have stopped the old PTY.
-      reopenCurrentCodexRestartPrompt(claimedPane, ptyId)
+      // Keep the prompt blocked until the original spawn settles. A retry would
+      // be skipped by the in-flight claim guard.
       onSettled?.()
     })
     await executeDetachedCodexPaneRestart(located, ptyId, agent)
   } catch (err) {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
releaseSlot = holdDetachedRestartSlot(ptyId, () => {
console.warn('[codex-restart] detached restart spawn stalled; asking again')
// Why: a late result still adopts and clears this prompt, since main may have stopped the old PTY.
reopenCurrentCodexRestartPrompt(claimedPane, ptyId)
onSettled?.()
releaseSlot = holdDetachedRestartSlot(ptyId, () => {
console.warn('[codex-restart] detached restart spawn stalled; asking again')
// Keep the prompt blocked until the original spawn settles. A retry would
// be skipped by the in-flight claim guard.
onSettled?.()

nwparker and others added 3 commits October 22, 2026 21:02
The regression tests reproduce the stalled-spawn scenario reported in #22243.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Tauri-EPO <176725071+Tauri-EPO@users.noreply.github.com>
… stalls

A spawn that never settles no longer holds one of the four restart slots or
leaves its pane silently input-blocked: after 90s the pane's restart prompt
reopens and the slot is released, while the claim stays until main settles
so a late result is still adopted or reaped.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant