feat: per-agent dispatch circuit breaker to prevent backlog poisoning (#526)

[AndriiPasternak31]

Summary

Implements the per-agent dispatch circuit breaker (RELIABILITY-007). When an agent is auth-dead — reachable but answering HTTP 503 (execution error_code == AUTH) — this producer-side breaker fast-fails new executions at the dispatch layer instead of poisoning the persistent backlog with doomed tasks, then self-heals through a half-open probe. Distinct from, and namespace-isolated from, the existing transport-reachability breaker (#631).

Backend

• services/dispatch_breaker.py — consecutive-failure state machine (closed → open → half-open(probe) → closed) in Redis via atomic Lua. AUTH-only counting (threshold 3), exponential backoff (30s→300s cap), one-probe-at-a-time SET NX EX lock. Fail-open, never raises.
• redis_breaker_util.py — shared fail-open Redis client + Lua ScriptCache + decode helpers, extracted from agent_client so both breakers reuse one plumbing layer (top-level leaf module → no circular import).
• services/circuit_breaker_view.py — single source for the unified {dispatch, transport, open, config} block.
• task_execution_service — single outcome-recording path records every terminal; on →open it backgrounds fail_queued_for_agent + clear in-memory queue + audit via _spawn_bg (strong ref against GC). CircuitOpen → FAILED row, never enqueues (no-enqueue invariant).
• capacity_manager — dispatch gate at the top of acquire() (gated on per-agent circuit_breaker_enabled and global DISPATCH_BREAKER_ENABLED); half-open probe admitted only into a free slot; 60s breaker-aware backstop re-fails queued backlog for still-open breakers if an inline drain is lost.
• DB — fail_queued_for_agent (QUEUED→FAILED, not CANCELLED), circuit_breaker_enabled opt-in column (migration + schema, default OFF), bulk accessors.
• routers/agents — GET/PUT /api/agents/{name}/circuit-breaker (read = authorized, config = owner-only + audit), reset (admin) now clears both breakers; unified block embedded in GET /api/agents/{name}.

Frontend

• Danger-styled "⚡ circuit open" badge in AgentHeader and AgentNode, derived from the embedded block / bulk slots map; clears on recovery. Playwright spec added.

Default OFF — both the per-agent opt-in and the global master switch must be on to engage (opt-in canary).

Test Coverage

Run on an isolated sibling Docker stack (never the running instance): Redis-only docker-compose.sibling.yml on :6390 for the Lua state machine, plus a throwaway sibling backend on :8100 for the api-client auth fixture.

• Unit — 32 tests (test_dispatch_breaker.py, test_dispatch_breaker_wiring.py): transition value-object, AUTH-only routing, fail-open (Redis down / no-Lua), acquire-gate + no-enqueue, half-open-probe-into-free-slot, drain wiring, maintenance backstop. fakeredis.
• Integration — 19 tests (test_dispatch_breaker.py, test_fail_queued_for_agent.py) against real Redis Lua: consecutive-AUTH threshold, success reset, ignored outcomes, retry_after, half-open probe close, probe reopen with grown backoff, missed-heartbeat seam, record_success no-op, operator hooks; and fail_queued_for_agent sets FAILED not CANCELLED.
• Full unit suite green across all 3 CI seeds (12345/67890/99999): 1843 passed. lint_sys_modules passes; schema/migration parity intact.

Tests: +4 files (2 unit, 2 integration), +1 dep (fakeredis).

Pre-Landing Review

• CRITICAL: none. Parameterized SQL; correct auth boundaries (read/owner/admin); sound atomic Lua; fail-open/fail-safe on every hot path; follows the three-layer + mixin + centralized-models + leaf-import invariants; schema↔migration parity.
• Fixed during review: the 4 new unit tests (TestMaintenanceBackstop, TestDrainWiring) were victims of a pre-existing test-isolation leak — test_904_agent_call_limiter installs a method-less database.db stub it never restores, so under randomized order setattr(database.db, …) raised AttributeError. Fix: the tests now swap the whole db object in the namespace the code resolves. Verified green across all 3 CI seeds.
• Informational: the half-open probe-lock TTL (10s) is shorter than a typical execution, so a few extra probes can leak at cooldown boundaries during a long auth-dead window — identical to the proven transport breaker and bounded by capacity slots. Acceptable.

Pre-existing failures (NOT this branch)

test_orphaned_execution_recovery::{test_in_registry_left_alone, test_multiple_agents_mixed} fail under randomized order on dev as well (confirmed by running the suite without this branch's new files: 1811 passed, 2 failed). Caused by an unrelated agent-client mock-clobber polluter; cleanup_service is untouched by this PR. CI's new-failing-test diff against base will not flag them.

Test plan

• 32 unit + 19 integration dispatch-breaker tests pass on the sibling stack
• Full tests/unit green across CI seeds 12345 / 67890 / 99999 (1843 passed)
• lint_sys_modules passes; migration applies on a fresh DB (sibling backend booted clean)
• Frontend Playwright badge spec — runs in the ui-gated frontend-e2e CI job (not run locally; needs the frontend stack)

Closes #526

🤖 Generated with Claude Code

[github-actions]

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

[vybe]

✅ Validated via /validate-pr (RELIABILITY-007, P1).

• Auth boundaries correct (read=authorized / config=owner+audit / reset=admin), SQL parameterized, schema↔migration parity intact.
• No-enqueue invariant preserved end-to-end including across the half-open probe window; AUTH-only outcome recording respects the caller contract; fail-open throughout.
• CI green across base+head × seeds 12345/67890/99999, sys.modules lint, schema-parity, regression diff.
• Ships default-OFF dual-gated opt-in canary.

Merge conflicts with dev (feature-flows.md, requirements.md §10.10 collision → renumbered dispatch breaker to §10.11, migrations.py MIGRATIONS list → all four migrations preserved) resolved in the merge commit; CI re-running.