Skip to content

Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12

Closed

Description

@zaridan

Part of #5. Foundation — ships first, standalone (real bug). Supersedes the original tasks-only run_id proposal, which adversarial review proved insufficient.

Observed

Two coordinators clashed; one reported another working on the same thing. A module-scope activeCoordinator (orchestration-gates.ts:10) + the getActiveCoordinatorRun check (:49) are supposed to prevent two concurrent runs — so the clash came through a gap (likely a leftover status='running' zombie row, or two runtimes/SSH). Reproduce and confirm the exact trigger first.

Why tasks-only scoping fails (review)

run_id on tasks alone leaves these GLOBAL and still leaking:

  • dispatch_contexts family: getStaleDispatches (db.ts:692), countActiveDispatches (db.ts:879), the per-assignee uniqueness guard (db.ts:580 — the likely source of the observed "already working" throw).
  • messages routed by handle only: shared default 'coordinator' handle (orchestration-gates.ts:54) → one run markAsReads another's worker_done (db.ts:316, coordinator.ts:244) → silent hang (a data-loss clash, worse than the throw).
  • decision_gates: listGates({status}) (db.ts:783) unscoped → cross-run gate touching.
  • getIdleTerminals (db.ts:890) derives terminals from global message history; promoteReadyTasks (db.ts:548) scans all pending tasks.
  • Fix-B "check active run then start" is TOCTOU (no transaction/unique index).

Proper fix (decide A vs B)

  • A — per-run DB file/namespace (orchestration-<runId>.db): structural isolation; no query can leak; inboxes naturally per-run. Cost: supervisor queries must fan out across run DBs. getOrchestrationDb(runId?) overload (orca-runtime.ts:2448). Review's recommendation.
  • B — coordinator_run_id on ALL tables (tasks, dispatch_contexts, messages, decision_gates) + thread through ~25 queries.

Either way, ALSO required:

  • Per-run coordinator handle (not the 'coordinator' default).
  • Atomic run-start: partial unique index ON coordinator_runs(status) WHERE status='running' (per target) or BEGIN IMMEDIATE around check+insert.

Lean A — enforces the invariant once at open time instead of in every query author forever. Coordinates with F3 (#14) reconciling orphaned running rows.

Activity

  1. changed the title [-]Guardrail: prevent coordinator clash — run-scope the orchestration DAG[/-] [+]Bridge F1: per-run isolation of the orchestration DB (proper clash fix)[/+] on Jun 23, 2026
  2. zaridan commented on Jun 23, 2026

    @zaridan
    OwnerAuthor

    F1 implemented — PR #17 (ready for review, not merged).

    Worker chose Approach B (coordinator_run_id column scoping) over the briefed default A (per-run DB files), with a documented rationale in DECISION.md. The argument is sound and code-grounded:

    1. Tasks exist before a run does (taskCreate → run; decompose() requires pre-created tasks). Per-run files would have to physically migrate rows into orchestration-<runId>.db at run-start; a column lets run-start adopt unowned rows with one UPDATE … WHERE coordinator_run_id IS NULL.
    2. The worker/runtime surface is handle-addressed, no runId in scope (push-on-idle delivery, exit attribution, agent-status). Per-run files would need a handle→run→DB resolver = a new shared registry (lost on restart → orphaned runs), re-introducing the shared routing state A was meant to remove.
    3. Atomic single-active-run + cross-run supervision both need one shared coordinator_runs table. The partial unique index only gives cross-process mutual exclusion if all runs share that table — so A is really "per-run files + a shared registry," more complex than B for identical isolation.

    What B delivers: nullable coordinator_run_id on tasks/dispatch_contexts/decision_gates (schema v5→v6, run-start adopts unowned tasks); per-run coordinator handle (coordinator-<hex> instead of literal 'coordinator') as the message-inbox isolation mechanism (messages can't carry a runId — the worker never learns one — so unique handles are the correct fix, matching the review's finding); atomic run-start via partial unique index on coordinator_runs(status) WHERE status='running'.

    226 orchestration tests + new per-run-isolation regression suite green; full suite passes (1 pre-existing flaky relay test, unrelated); typecheck + build clean. F3 orphan-run reconcile noted as follow-up, not built (correct scope). 9 files, +635/-80.

    Process note: brief said pause-and-confirm before switching A→B; worker documented thoroughly and proceeded to PR instead of pausing. Net acceptable — it honored "don't silently switch."

  3. zaridan commented on Jun 23, 2026

    @zaridan
    OwnerAuthor

    Round 2 done — review blocker resolved, PR #17 updated (not merged).

    • 🔴 Global single-run guard → per-target. Added coordinator_runs.target_key (schema v6→v7); startCoordinatorRun rejects only a status='running' row for the same target_key, keeping BEGIN IMMEDIATE. Concurrent Orcastrators across repos now run in parallel (the intent the UI's N>1 activity loop already assumed).
    • 🟠 Stale "partial unique index" docs removed (mechanism is BEGIN IMMEDIATE).
    • 🟠 Replaced the hollow atomicity test with two OrchestrationDb instances over one on-disk file (same target → one wins; different targets → both start).
    • 🟡 Manual dispatch --inject resolves a real per-run handle or refuses (no black-holed liveness); dead getIdleTerminals removed; taskList annotated deliberate; handle entropy 24→64 bits.
    • Bonus: failActiveDispatchOnExit rerouted to the dispatch's own run (was getActiveCoordinatorRun() — a misroute the old global guard had masked).

    233 orchestration tests green; typecheck + build clean. Deferred to F4: per-run runStop (today stops latest run only). Merge-readiness pending a re-review of the round-2 delta.

  4. zaridan commented on Jun 23, 2026

    @zaridan
    OwnerAuthor

    Round-2 re-review — BLOCKER found; not merge-ready.

    Enabling per-target concurrency (round 2) opened a real hole the round-1 review's "adopt race is safe" verdict had only been safe because of the now-removed global guard:

    • 🔴 Cross-target task poaching. target_key is only on coordinator_runs, never on tasks. adoptUnownedTasks (target-blind) lets the first-starting run swallow all coordinator_run_id IS NULL tasks — including another target's. taskCreate stamps via global getActiveCoordinatorRun(), so tasks land under the wrong run. = Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12 reintroduced at the task-ownership layer for concurrent runs.
    • 🟠 resolveOrchestrationTargetKey instability. Same target → worktree:<id> vs selector:<raw> depending on resolution success (SSH/headless) → guard fails to reject a duplicate → silent Bridge F1: per-run isolation of the orchestration DB (proper clash fix) #12 clash.
    • 🟠 New tests cover run-start mutual exclusion but not adoption-under-concurrency (the "different targets" test creates no tasks), which is why the blocker slipped through.

    Confirmed fixed: handle entropy, dead code, manual-dispatch black-hole, taskList doc, v6→v7 migration, real cross-process race test.

    Round 3 fix direction: extend the target boundary to task ownership — stamp target on tasks at taskCreate (resolve from caller's worktree/handle), scope adoptUnownedTasks by target, make taskCreate's run lookup target-scoped; add the adoption-under-concurrency test; make the target key a single stable identity.

  5. zaridan commented on Jun 23, 2026

    @zaridan
    OwnerAuthor

    Round 3 — blocker closed; F1 merge-ready (pending merge decision).

    • 🔴 Cross-target poaching closed: target_key extended to tasks (schema v7→v8); adoptUnownedTasks filters AND target_key IS ?; taskCreate uses getActiveCoordinatorRunForTarget (not global latest). Adoption-under-concurrency test added first, proven red→green (run A no longer adopts run B's tasks).
    • 🟠 resolveOrchestrationTargetKey now fails closed — resolves worktree:<id> or throws; no silent selector: fallback that split the key across processes.
    • 🟡 failActiveDispatchOnExit NULL fallback + null-target shared slot documented as known limitations.

    237 orchestration tests green; typecheck + build clean. Three adversarial review passes converged — each found a real defect (global guard → start-only scoping → task-ownership poaching), round 3 closes the last with a proving test. Remaining lifecycle items (per-run runStop, orphaned-run reconcile) are explicitly F3/F4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions