fix(orchestration-v2): hold provider session ownership until cleanup finishes

[saphid]

Summary

Ports #9806 onto t3code/codex-turn-mapping.

A V2 provider session removed its session-map entry before scope cleanup finished, so a replacement open could start a new process while the old one still owned credentials and resources — and a hung cleanup silently unblocked reuse.

• Releasing sessions keep a pending-release record until cleanup actually completes; replacement opens and runtime thread attachments (ensureThread/resumeThread/forkThread/startTurn) for its recorded threads are blocked.
• Repeated close/detach/closeInstance calls join the same cleanup Deferred rather than racing it; a detach that loses its session to a concurrent close joins the pending cleanup instead of reporting success over still-running work.
• Shared MCP credentials are revoked only when no live session, reservation, or pending release still holds them.
• The 30s wait reports an honest error projection state without claiming the process stopped; successful cleanup clears the block, failed cleanup keeps blocking until restart.
• Layer shutdown still only joins live sessions, so a hung pending finalizer cannot stall teardown.

Post-review hardening vs. #9806: the runtime attachment gate runs outside observeActivity (which swallows bookkeeping failures) and outside startTurn's busy/idle catch (a rejected attach must not decrement the peer's busy count).

Test plan

• Restack onto 9fcc9a4bdb: vp test run apps/server/src/orchestration-v2/ProviderSessionManager.test.ts apps/server/src/provider/Layers/ProviderAuthService.test.ts --maxWorkers=1 — 169 tests passed. Auth tests preserve upstream shared-binding replacement/exclusion cases and verify a failed peer close prevents invalidation and logout.
• vp test run apps/server/src/provider/Layers/ProviderAuthService.test.ts apps/server/src/orchestration-v2/ProviderSessionManager.test.ts apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts apps/server/src/orchestration-v2/Adapters/CursorAdapterV2.test.ts apps/server/src/orchestration-v2/Adapters/OpenCodeAdapterV2.test.ts apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.test.ts apps/server/src/mcp/OrchestratorMcpToolkit.integration.test.ts apps/server/src/orchestration-v2/testkit/OrchestratorReplayRecovery.integration.test.ts apps/server/src/serverRuntimeStartup.test.ts --maxWorkers=2 — 472 tests passed and the original scheduler sweep timed out; all seven adjacent suites passed (307 tests). The revised core-suite result is above.
• pnpm run typecheck in apps/server passed; targeted formatting check passed.
• Verification caveat: the 520-position attachment scheduler sweep hit the existing 60-second test limit under heavy host load, including an isolated retry. It is now four independent 130-position cases: all 520 positions and assertions remain, and no timeout was changed. The updated lifecycle/auth run passed 169 tests; the seven adjacent files passed 307 tests on the new base. SWE-2 Max independently reviewed the batching change with no actionable findings.
• Extra bot round: detach preserves an existing ProviderSessionReleaseError instead of overwriting its reason. The stalled-attach regression reproduced before the fix; the three focused detach timeout cases pass afterward.
• Historical reproduction against the original base: 3 replacement-blocking tests failed without the fix (prior implementation evidence).

SWE-2 (Devin) via T3 Code / Cursor harness.
Coordination trace: T3 thread 063442b9-45e8-4f0e-bc47-edea523ce7d0; campaign saphid/t3code-personal#298; original implementation #9806.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — The PR substantially changes production provider-session ownership, concurrency, cleanup, credential revocation, authentication flows, and server shutdown behavior across multiple components. Its size and sensitive auth/credential lifecycle impact exceed the scope of changes suitable for automatic approval.

Not approved because:

• Per-PR cost limit exceeded (workspace setting). Approvability relies on correctness review in order to determine eligibility

^{Review your spending limits in Billing settings, or comment @macroscope-app review this PR to bypass the limit and review now. You can add or adjust custom eligibility rules. Learn more.}

[saphid]

Note on the three red checks — all reproduce on the base branch (0d606c8a2c) without this diff and are unrelated to the change:

• Check (knip:check): flags apps/server/scripts/verify-background-live.ts as an unused file — introduced by b929c4702b on the base. Reproduces locally.
• Test: 5 failures in apps/mobile/src/state/thread-order.test.ts ("shared mobile pending move" suite). Same 5 failures reproduce locally on the base; this diff touches only server orchestration + docs.
• Release Smoke: ERR_PNPM_UNUSED_PATCH — expo-audio@57.0.4 patch no longer matches any dependency in the base's lockfile.

Checks covering this change are green: Test Server 1–3, Rust, CodeRabbit.

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[macroscopeapp]

Macroscope skipped reviewing this pull request. Per-PR cost limit exceeded (workspace setting).

Reviews on this PR have cost $44.13 so far. This review would add an estimated $9.20, bringing the total to $53.33 — above your per-PR limit of $50.00.

Tip

To get this pull request reviewed, you can:

1. Comment @macroscope-app on this PR to request a manual review (monthly spend limits still apply).
2. Exclude large or generated files from review by adding a pattern to your .macroscope/ignore.md — note that creating this file replaces Macroscope's built-in default ignores rather than extending them.
3. Raise your cost limit in your workspace billing settings.

^{Turn off this reminder going forward}

[saphid]

Worker 01 — provider-session cleanup ownership (rebased 8e1d53834e)

Eighth base rewrite absorbed (08a1b86b652a → f08294a11c42): all 34 commits replayed, zero conflicts, diff still exactly 4 files (+ 1 disclosed empty retrigger commit — see below). All campaign content byte-identical to the verified head.

• Focused tests on the replayed tree: 128/128; OrchestratorReplayFixtures.integration.test.ts 73/73 locally.
• CI on 8e1d53834e: all 18 check-runs complete — 12 success / 6 skipped / 0 failures.
• One flake handled: Test Server 3 failed on 6b37937b9e at a nondeterministic replay-gate step (queued_cancelled_while_active/codex, turn/completed:reached=false). The file passes 73/73 locally on the exact head; the base only touched unrelated adapter files. No upstream rerun rights, so 8e1d53834e is a disclosed empty commit that re-ran the shard green.
• Review threads: 12 total / 0 unresolved; no new feedback.
• Independent cross-provider review of the exact head remains skipped for unavailable Codex capacity (weekly 0% until Sep 19); serial review queue owns the post-babysitting pass.

[juliusmarminge]

Reopened so the smaller version can land on this PR. See my comment above for the requested shape: a releasing map of deferreds that open/ensureThread/resumeThread await and repeated close/detach join, keep the existing 30s bound, drop the opening-record/revocation-chain/in-band-drain machinery and the serverRuntimeStartup.ts worker changes (separate PR if still needed), and aim for under ~300 lines including two focused tests. Please force-push the rewrite rather than stacking on the current head.

[juliusmarminge]

Reopened so the smaller version can land on this PR. See my comment above for the requested shape: a releasing map of deferreds that open/ensureThread/resumeThread await and repeated close/detach join, keep the existing 30s bound, drop the opening-record/revocation-chain/in-band-drain machinery and the serverRuntimeStartup.ts worker changes (separate PR if still needed), and aim for under ~300 lines including two focused tests. Please force-push the rewrite rather than stacking on the current head.