You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12
Part of #5. Foundation — ships first, standalone (real bug). Supersedes the original tasks-only run_id proposal, which adversarial review proved insufficient.
Observed
Two coordinators clashed; one reported another working on the same thing. A module-scope activeCoordinator (orchestration-gates.ts:10) + the getActiveCoordinatorRun check (:49) are supposed to prevent two concurrent runs — so the clash came through a gap (likely a leftover status='running' zombie row, or two runtimes/SSH). Reproduce and confirm the exact trigger first.
Why tasks-only scoping fails (review)
run_id on tasks alone leaves these GLOBAL and still leaking:
dispatch_contexts family: getStaleDispatches (db.ts:692), countActiveDispatches (db.ts:879), the per-assignee uniqueness guard (db.ts:580 — the likely source of the observed "already working" throw).
messages routed by handle only: shared default 'coordinator' handle (orchestration-gates.ts:54) → one run markAsReads another's worker_done (db.ts:316, coordinator.ts:244) → silent hang (a data-loss clash, worse than the throw).
getIdleTerminals (db.ts:890) derives terminals from global message history; promoteReadyTasks (db.ts:548) scans all pending tasks.
Fix-B "check active run then start" is TOCTOU (no transaction/unique index).
Proper fix (decide A vs B)
A — per-run DB file/namespace (orchestration-<runId>.db): structural isolation; no query can leak; inboxes naturally per-run. Cost: supervisor queries must fan out across run DBs. getOrchestrationDb(runId?) overload (orca-runtime.ts:2448). Review's recommendation.
B — coordinator_run_id on ALL tables (tasks, dispatch_contexts, messages, decision_gates) + thread through ~25 queries.
Either way, ALSO required:
Per-run coordinator handle (not the 'coordinator' default).
Atomic run-start: partial unique index ON coordinator_runs(status) WHERE status='running' (per target) or BEGIN IMMEDIATE around check+insert.
Lean A — enforces the invariant once at open time instead of in every query author forever. Coordinates with F3 (#14) reconciling orphaned running rows.
changed the title [-]Guardrail: prevent coordinator clash — run-scope the orchestration DAG[/-][+]Bridge F1: per-run isolation of the orchestration DB (proper clash fix)[/+]on Jun 23, 2026
F1 implemented — PR #17 (ready for review, not merged).
Worker chose Approach B (coordinator_run_id column scoping) over the briefed default A (per-run DB files), with a documented rationale in DECISION.md. The argument is sound and code-grounded:
Tasks exist before a run does (taskCreate → run; decompose() requires pre-created tasks). Per-run files would have to physically migrate rows into orchestration-<runId>.db at run-start; a column lets run-start adopt unowned rows with one UPDATE … WHERE coordinator_run_id IS NULL.
The worker/runtime surface is handle-addressed, no runId in scope (push-on-idle delivery, exit attribution, agent-status). Per-run files would need a handle→run→DB resolver = a new shared registry (lost on restart → orphaned runs), re-introducing the shared routing state A was meant to remove.
Atomic single-active-run + cross-run supervision both need one shared coordinator_runs table. The partial unique index only gives cross-process mutual exclusion if all runs share that table — so A is really "per-run files + a shared registry," more complex than B for identical isolation.
What B delivers: nullable coordinator_run_id on tasks/dispatch_contexts/decision_gates (schema v5→v6, run-start adopts unowned tasks); per-run coordinator handle (coordinator-<hex> instead of literal 'coordinator') as the message-inbox isolation mechanism (messages can't carry a runId — the worker never learns one — so unique handles are the correct fix, matching the review's finding); atomic run-start via partial unique index on coordinator_runs(status) WHERE status='running'.
226 orchestration tests + new per-run-isolation regression suite green; full suite passes (1 pre-existing flaky relay test, unrelated); typecheck + build clean. F3 orphan-run reconcile noted as follow-up, not built (correct scope). 9 files, +635/-80.
Process note: brief said pause-and-confirm before switching A→B; worker documented thoroughly and proceeded to PR instead of pausing. Net acceptable — it honored "don't silently switch."
🔴 Global single-run guard → per-target. Added coordinator_runs.target_key (schema v6→v7); startCoordinatorRun rejects only a status='running' row for the same target_key, keeping BEGIN IMMEDIATE. Concurrent Orcastrators across repos now run in parallel (the intent the UI's N>1 activity loop already assumed).
🟠 Stale "partial unique index" docs removed (mechanism is BEGIN IMMEDIATE).
🟠 Replaced the hollow atomicity test with two OrchestrationDb instances over one on-disk file (same target → one wins; different targets → both start).
🟡 Manual dispatch --inject resolves a real per-run handle or refuses (no black-holed liveness); dead getIdleTerminals removed; taskList annotated deliberate; handle entropy 24→64 bits.
Bonus:failActiveDispatchOnExit rerouted to the dispatch's own run (was getActiveCoordinatorRun() — a misroute the old global guard had masked).
233 orchestration tests green; typecheck + build clean. Deferred to F4: per-run runStop (today stops latest run only). Merge-readiness pending a re-review of the round-2 delta.
Round-2 re-review — BLOCKER found; not merge-ready.
Enabling per-target concurrency (round 2) opened a real hole the round-1 review's "adopt race is safe" verdict had only been safe because of the now-removed global guard:
🔴 Cross-target task poaching.target_key is only on coordinator_runs, never on tasks. adoptUnownedTasks (target-blind) lets the first-starting run swallow allcoordinator_run_id IS NULL tasks — including another target's. taskCreate stamps via global getActiveCoordinatorRun(), so tasks land under the wrong run. = Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12 reintroduced at the task-ownership layer for concurrent runs.
🟠 New tests cover run-start mutual exclusion but not adoption-under-concurrency (the "different targets" test creates no tasks), which is why the blocker slipped through.
Confirmed fixed: handle entropy, dead code, manual-dispatch black-hole, taskList doc, v6→v7 migration, real cross-process race test.
Round 3 fix direction: extend the target boundary to task ownership — stamp target on tasks at taskCreate (resolve from caller's worktree/handle), scope adoptUnownedTasks by target, make taskCreate's run lookup target-scoped; add the adoption-under-concurrency test; make the target key a single stable identity.
Round 3 — blocker closed; F1 merge-ready (pending merge decision).
🔴 Cross-target poaching closed: target_key extended to tasks (schema v7→v8); adoptUnownedTasks filters AND target_key IS ?; taskCreate uses getActiveCoordinatorRunForTarget (not global latest). Adoption-under-concurrency test added first, proven red→green (run A no longer adopts run B's tasks).
🟠 resolveOrchestrationTargetKey now fails closed — resolves worktree:<id> or throws; no silent selector: fallback that split the key across processes.
🟡 failActiveDispatchOnExit NULL fallback + null-target shared slot documented as known limitations.
237 orchestration tests green; typecheck + build clean. Three adversarial review passes converged — each found a real defect (global guard → start-only scoping → task-ownership poaching), round 3 closes the last with a proving test. Remaining lifecycle items (per-run runStop, orphaned-run reconcile) are explicitly F3/F4.
Activity
[-]Guardrail: prevent coordinator clash — run-scope the orchestration DAG[/-][+]Bridge F1: per-run isolation of the orchestration DB (proper clash fix)[/+]zaridan commentedon Jun 23, 2026
F1 implemented — PR #17 (ready for review, not merged).
Worker chose Approach B (
coordinator_run_idcolumn scoping) over the briefed default A (per-run DB files), with a documented rationale inDECISION.md. The argument is sound and code-grounded:taskCreate→run;decompose()requires pre-created tasks). Per-run files would have to physically migrate rows intoorchestration-<runId>.dbat run-start; a column lets run-start adopt unowned rows with oneUPDATE … WHERE coordinator_run_id IS NULL.coordinator_runstable. The partial unique index only gives cross-process mutual exclusion if all runs share that table — so A is really "per-run files + a shared registry," more complex than B for identical isolation.What B delivers: nullable
coordinator_run_idontasks/dispatch_contexts/decision_gates(schema v5→v6, run-start adopts unowned tasks); per-run coordinator handle (coordinator-<hex>instead of literal'coordinator') as the message-inbox isolation mechanism (messages can't carry a runId — the worker never learns one — so unique handles are the correct fix, matching the review's finding); atomic run-start via partial unique index oncoordinator_runs(status) WHERE status='running'.226 orchestration tests + new per-run-isolation regression suite green; full suite passes (1 pre-existing flaky relay test, unrelated); typecheck + build clean. F3 orphan-run reconcile noted as follow-up, not built (correct scope). 9 files, +635/-80.
Process note: brief said pause-and-confirm before switching A→B; worker documented thoroughly and proceeded to PR instead of pausing. Net acceptable — it honored "don't silently switch."
fix(orchestration): scope run-start guard per-target, not global (#12)
fix(orchestration): resolve a real coordinator handle for manual disp…
test(orchestration): per-target + cross-connection run-start coverage (…
zaridan commentedon Jun 23, 2026
Round 2 done — review blocker resolved, PR #17 updated (not merged).
coordinator_runs.target_key(schema v6→v7);startCoordinatorRunrejects only astatus='running'row for the same target_key, keepingBEGIN IMMEDIATE. Concurrent Orcastrators across repos now run in parallel (the intent the UI's N>1 activity loop already assumed).BEGIN IMMEDIATE).OrchestrationDbinstances over one on-disk file (same target → one wins; different targets → both start).dispatch --injectresolves a real per-run handle or refuses (no black-holed liveness); deadgetIdleTerminalsremoved;taskListannotated deliberate; handle entropy 24→64 bits.failActiveDispatchOnExitrerouted to the dispatch's own run (wasgetActiveCoordinatorRun()— a misroute the old global guard had masked).233 orchestration tests green; typecheck + build clean. Deferred to F4: per-run
runStop(today stops latest run only). Merge-readiness pending a re-review of the round-2 delta.zaridan commentedon Jun 23, 2026
Round-2 re-review — BLOCKER found; not merge-ready.
Enabling per-target concurrency (round 2) opened a real hole the round-1 review's "adopt race is safe" verdict had only been safe because of the now-removed global guard:
target_keyis only oncoordinator_runs, never ontasks.adoptUnownedTasks(target-blind) lets the first-starting run swallow allcoordinator_run_id IS NULLtasks — including another target's.taskCreatestamps via globalgetActiveCoordinatorRun(), so tasks land under the wrong run. = Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12 reintroduced at the task-ownership layer for concurrent runs.resolveOrchestrationTargetKeyinstability. Same target →worktree:<id>vsselector:<raw>depending on resolution success (SSH/headless) → guard fails to reject a duplicate → silent Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12 clash.Confirmed fixed: handle entropy, dead code, manual-dispatch black-hole, taskList doc, v6→v7 migration, real cross-process race test.
Round 3 fix direction: extend the target boundary to task ownership — stamp target on tasks at
taskCreate(resolve from caller's worktree/handle), scopeadoptUnownedTasksby target, maketaskCreate's run lookup target-scoped; add the adoption-under-concurrency test; make the target key a single stable identity.fix(orchestration): scope task ownership per-target, closing cross-ta…
fix(orchestration): stable target identity (fail closed) + target-awa…
test(orchestration): adoption-under-concurrency + fail-closed coverage (
zaridan commentedon Jun 23, 2026
Round 3 — blocker closed; F1 merge-ready (pending merge decision).
target_keyextended totasks(schema v7→v8);adoptUnownedTasksfiltersAND target_key IS ?;taskCreateusesgetActiveCoordinatorRunForTarget(not global latest). Adoption-under-concurrency test added first, proven red→green (run A no longer adopts run B's tasks).resolveOrchestrationTargetKeynow fails closed — resolvesworktree:<id>or throws; no silentselector:fallback that split the key across processes.failActiveDispatchOnExitNULL fallback + null-target shared slot documented as known limitations.237 orchestration tests green; typecheck + build clean. Three adversarial review passes converged — each found a real defect (global guard → start-only scoping → task-ownership poaching), round 3 closes the last with a proving test. Remaining lifecycle items (per-run
runStop, orphaned-run reconcile) are explicitly F3/F4.feat(orchestration): coordinator creates lineage-visible worktrees (F…