Skip to content

fix(server): Stop always ends the background work a thread shows - #14636

Merged
juliusmarminge merged 2 commits into
t3code/codex-turn-mappingfrom
v2/stop-background-work-always
Oct 2, 2026
Merged

juliusmarminge merged 2 commits into
t3code/codex-turn-mappingfrom
v2/stop-background-work-always

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Pressing Stop on the composer's "Waiting on …" strip could do nothing. The orchestrator accepted the Stop, but ProviderTurnControlService dropped it for any settled turn whose adapter has no per-thread pending-work probe (every adapter except Claude and Codex). Codex's probe also missed work its process no longer tracked, and with no live provider session the Stop was rejected outright. The strip kept showing work nothing would ever end.

Fix

  • Stop always reaches the adapter. ProviderTurnControlService.interrupt now forwards a settled turn's Stop with requestRuntimeRestart: true instead of returning early. The adapter stops whatever it still runs, or reports there's nothing left. It still returns early in one case: no live session, so no adapter to call.
  • Adapters accept a settled Stop. A settled-turn Stop with nothing live is a no-op success in every adapter, so the settle step always runs.
    • Claude: with no CLI process left for the thread (it exited after the turn), Stop succeeds and drops that gone process's roster. It used to fail with "has no live query" before anything was settled.
    • Cursor and Pi: no-op success. Turn finalization already ended their tools, and Pi runs nothing between prompts.
    • OpenCode: aborts the session's descendants, since a background task child can outlive its turn.
    • Codex: a settled Stop it tracks nothing for returns instead of failing. Terminating a background terminal whose app-server already exited counts as terminated.
  • Leftover work is settled after the adapter returns. The effect worker then dispatches a new internal command, thread.background-work.settle. It marks interrupted any background item (command_execution, dynamic_tool, subagent) the thread still shows on that provider thread, or on provider threads with no live session. It also clears the pending-work roster of a dead process. It does nothing if a new run has started since the Stop.
    • Bound to the stopped run. The command carries the stopped provider turn and settles only that run's work and older runs'. An item with no run counts as the stopped run's. A later run's work survives an earlier Stop.
    • Idempotent. Its command id derives from the Stop command. A settle that ends nothing still records its receipt, so a replayed Stop effect (crash, failed outbox completion) hits the receipt instead of settling work that appeared since.
  • No live session is no longer an error. A Stop on a settled turn with no live session settles directly, in the Stop command itself.
  • Shared selection. pendingBackgroundTurnItems (shared) is the selection the strip and the settle now both use, so they can't disagree. ProjectionStore gains a turnItemStatuses record filter so Stop reads only active items, not a thread's whole history.

Provider specifics stay in the adapters. The orchestrator only settles projected items that no live process will report on.

What Stop now does, per provider

Provider Background work after the turn settles Stop on the strip
Claude Background Bash and subagents in the CLI process Unchanged: the adapter closes the query, killing the tasks and clearing the roster. With the CLI process already gone, no-op plus roster reset, then leftovers are settled
Codex Background terminals (unified exec) Adapter terminates tracked terminals; work it no longer tracks (process gone, lost item/completed) is settled interrupted
Grok Held runs stay running, so the regular Stop applies Settled turn: ACP contains the orphan runtime (process-group kill), then leftovers are settled
ACP registry, Antigravity Tools left open by a failed/cancelled prompt Adapter no-op, then leftovers are settled interrupted
Cursor None (finalization ends every tool) No-op, anything shown is settled
OpenCode Background task child sessions Aborts descendants, then leftovers are settled
Pi Tools left open by transport death or a rejected prompt No-op, then leftovers are settled
Any, provider session released or restarted Whatever the projection still shows Settled directly by the Stop command

Codex: is the stale item an adapter bug?

I checked against Codex 0.156.1 source. Codex always sends item/completed for a background terminal, with the original turnId, including after turn/completed:

  • Exit watcher: core/src/unified_exec/async_watcher.rs:165-247.
  • Kill paths also complete the item (as failed): terminate, clean, prune and timeout, in process_manager.rs:1725,1798,1835 and :649-657.
  • Turn end never kills terminals: core/src/tasks/mod.rs:782.
  • The adapter handles a late completion: via resolveItemEventContext / settledTurns.

So there is no adapter bug to fix with a fixture. A projected item outlives its command only when the notification never reaches T3:

  • the app-server exits or is released;
  • the thread is unsubscribed on detach;
  • a notification fails schema decode, which the client drops silently (effect-codex-app-server/src/client.ts:165-170).

The maintainer's thread probably ran on a Mac, so I couldn't inspect it here.

Verification

All server tests ran inside unshare -U --map-current-user -p -f --mount-proc. The branch contains the pid-1 group-kill guard (6562235).

  • New tests: each fails on the base branch and passes with the fix.
    • CodexAdapterV2.test.ts, "Stop ends a background command no Codex process tracks any more". Through the orchestrator: a Codex background command is still running when the app-server exits. On base the command stays running.
    • stop_background_work_after_failed_turn/acpRegistry replay fixture, for an adapter without the probe. An ACP prompt fails while an execute tool is in progress; the frames are hand-ported from the registry tool frames plus the recorded grok_prompt_error error frame. On base the strip still lists dev-server after Stop.
    • stop_background_work_after_release/acpRegistry replay fixture. The same thread after the idle timeout released its session. On base Stop is rejected with "Provider session … is not active".
  • Review follow-up tests: each fails on the previous head (813da49) and passes now.
    • ClaudeAdapterV2.test.ts, "a settled Stop with no CLI process left succeeds and clears the roster". The CLI exits after the turn settled with a background task on the roster. On 813da49 Stop fails with "has no live query".
    • Orchestrator.control-reads.test.ts, "settles only the stopped run's background work, once". Two settled runs each have a running background command. The settle for run 1's Stop leaves run 2's command running. A settle that ended nothing replays as a no-op after new work appears. On 813da49 run 2's command is interrupted.
  • Existing background Stop tests still pass:
    • claude_background_task_interrupt;
    • the Codex background terminal Stop tests ("stops a command after root completion…", "interrupts a completed run's background command through orchestration");
    • Claude "stops background work after the turn settled" and its replaced-process sibling.
  • Full files run: OrchestratorReplayFixtures.integration.test.ts, CodexReplayFixtures.integration.test.ts, ProviderTurnControlService.test.ts, EffectWorker.test.ts, ProjectionControlReads.test.ts, and the Codex, Claude, Cursor, Pi and OpenCode adapter tests. 470 passed.
    • ProjectionControlReads.test.ts asserted the old drop and was updated.
    • packages/shared orchestrationV2PendingBackgroundWork.test.ts: 31 passed.
  • Review follow-up run (head 4743813), inside the PID namespace: Orchestrator.control-reads.test.ts, EffectWorker.test.ts, ProjectionControlReads.test.ts, ProviderTurnControlService.test.ts, OrchestratorReplayFixtures.integration.test.ts and .contract.test.ts, ClaudeReplayFixtures.integration.test.ts, CodexReplayFixtures.integration.test.ts, CodexAdapterV2.test.ts, ClaudeAdapterV2.test.ts. 10 files, 395 passed.
  • AcpAdapterV2.test.ts, outside the namespace (its process-tree tests need real process groups), 5 runs: 5 of 5 runs, 114/114 each (the 5 failures the first run reported were from running it inside the namespace). CI's "carries a live subagent lineage across an interrupt" failure on 813da49 is a timing flake, not this change. This PR touches no ACP adapter, runtime, test or mock-agent code, and that test calls interruptTurn without requestRuntimeRestart, directly on the adapter, so the always-forward change in ProviderTurnControlService is not on its path.
  • Static checks:
    • tsc --noEmit exits 0 with 0 errors for apps/server and packages/contracts (re-run on 4743813); packages/shared unchanged since.
    • vp lint on the touched files shows only pre-existing warnings.
    • knip reports nothing in the files this PR touches.
  • CI on 4743813: all six server shards, Check, Typecheck, Lint, Build and Macroscope Correctness are green.
  • Not run: repo-wide checks, web/mobile clients, live providers.

Model: Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code


Devin Review

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Oct 1, 2026
The Waiting strip's Stop was accepted but could be dropped: the control
service skipped any settled turn whose adapter has no per-thread pending-work
probe, the Codex probe missed work its process no longer tracked, and with no
live session the Stop was rejected. The strip kept showing the work.

A settled Stop now always reaches the adapter, which stops what it still runs
or reports nothing is left. Afterwards a settle command marks interrupted
whatever the thread still shows on that provider thread, or on provider
threads with no live session; with no live session the Stop settles directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 1, 2026
Comment thread apps/server/src/orchestration-v2/EffectWorker.ts
Comment thread apps/server/src/orchestration-v2/ProviderTurnControlService.ts
@macroscopeapp

macroscopeapp Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR substantially changes Stop behavior across multiple production providers, including aborting descendants, terminating background processes, and durably settling orphaned work through a new orchestration path. The cross-provider runtime and process-lifecycle impact is broader than a small self-contained fix.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 4.9 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.7 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 20.4 KiB — 29.3 KiB ✅
Codex Live turn messages — 2 — 8 ✅
Claude Total thread wire — 4.9 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.7 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 20.8 KiB — 29.3 KiB ✅
Claude Live turn messages — 2 — 8 ✅

Baseline: unavailable · PR result: 4743813 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 106.1 KiB
  • Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@juliusmarminge
juliusmarminge force-pushed the v2/stop-background-work-always branch from 4d058d4 to 813da49 Compare October 1, 2026 18:48
…ss left

A settled-turn Stop on a Claude session whose CLI process had already
exited failed with "has no live query", so the settle follow-up never ran
and the Waiting strip kept the work. A settled-turn Stop with nothing live
is now a no-op success in the Claude adapter too, as in Codex, Cursor, Pi,
OpenCode and ACP; it also drops the gone process's roster.

The settle follow-up now carries the stopped provider turn and ends only
that run's and older runs' background work, so a later run's work survives
an earlier Stop. A settle that ends nothing records its receipt, so a
replayed Stop effect hits it instead of settling work that appeared since.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added size:XL 500-999 changed lines (additions + deletions). and removed size:L 100-499 changed lines (additions + deletions). labels Oct 1, 2026
@juliusmarminge
juliusmarminge merged commit 66a9107 into t3code/codex-turn-mapping Oct 2, 2026
31 checks passed
@juliusmarminge
juliusmarminge deleted the v2/stop-background-work-always branch October 2, 2026 05:02
tarunravi added a commit to tarunravi/t3code that referenced this pull request Oct 3, 2026
When the latest run was interrupted before its provider turn started, it
has no provider turn, so Stop found nothing to interrupt and failed with
"Run ... is not interruptible". The orchestrator now falls back to the
provider thread's latest turn when background work is pending.

Upstream pingdotgg#14636 already reads background work thread-wide and lets a
settled Stop reach the adapter, so only this fallback is carried. The
integration test covers Claude and Codex scenarios from one harness. The
"Use T3 subagents only" setting notes that it applies to Claude sessions
started after the change.

Squashes fork commits 99b07da and 85d5723.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tarunravi added a commit to tarunravi/t3code that referenced this pull request Oct 5, 2026
When the latest run was interrupted before its provider turn started, it
has no provider turn, so Stop found nothing to interrupt and failed with
"Run ... is not interruptible". The orchestrator now falls back to the
provider thread's latest turn when background work is pending.

Upstream pingdotgg#14636 already reads background work thread-wide and lets a
settled Stop reach the adapter, so only this fallback is carried. The
integration test covers Claude and Codex scenarios from one harness. The
"Use T3 subagents only" setting notes that it applies to Claude sessions
started after the change.

Squashes fork commits 99b07da and 85d5723.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tarunravi added a commit to tarunravi/t3code that referenced this pull request Oct 5, 2026
When the latest run was interrupted before its provider turn started, it
has no provider turn, so Stop found nothing to interrupt and failed with
"Run ... is not interruptible". The orchestrator now falls back to the
provider thread's latest turn when background work is pending.

Upstream pingdotgg#14636 already reads background work thread-wide and lets a
settled Stop reach the adapter, so only this fallback is carried. The
integration test covers Claude and Codex scenarios from one harness. The
"Use T3 subagents only" setting notes that it applies to Claude sessions
started after the change.

Squashes fork commits 99b07da and 85d5723.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tarunravi added a commit to tarunravi/t3code that referenced this pull request Oct 5, 2026
When the latest run was interrupted before its provider turn started, it
has no provider turn, so Stop found nothing to interrupt and failed with
"Run ... is not interruptible". The orchestrator now falls back to the
provider thread's latest turn when background work is pending.

Upstream pingdotgg#14636 already reads background work thread-wide and lets a
settled Stop reach the adapter, so only this fallback is carried. The
integration test covers Claude and Codex scenarios from one harness. The
"Use T3 subagents only" setting notes that it applies to Claude sessions
started after the change.

Squashes fork commits 99b07da and 85d5723.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:XL 500-999 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant