perf: cache per-agent container stats + bulk /sync-health lookup (#73)

[AndriiPasternak31]

What & why

Fixes the addressable backend CPU sink in #73: a 4-vCPU instance running ~30 agents with 2-3 browser users sits at ~45% backend CPU. Backend-only changes.

1. TTL cache + single-flight on GET /api/agents/{name}/stats (the dominant cost)
container.stats(stream=False) costs ~1-2s of Docker double-sampling, was uncached, had no request coalescing, and funnels through a 4-worker thread pool. N agents × multiple tabs polling every 10s pinned the CPU.

• Per-agent TTL cache (default 12s, env-overridable via AGENT_STATS_CACHE_TTL_SECONDS, defensively parsed — a bad value never crashes startup; 0 disables).
• Per-agent asyncio.Lock single-flight: concurrent same-agent misses share one Docker call instead of N.
• Generation counter discards an in-flight leader's stale write if a lifecycle invalidation races past it. Errors (404/400/500) are never cached, so a transient stop never gets pinned for the TTL.
• Invalidation lives at the docker_utils lifecycle-primitive boundary (container_stop/start/remove/rename) — the enforced chokepoint for every container op — so all state-changing paths invalidate (UI start/stop/delete, ops restart, deploy_local, subscription re-assign, system restart, rename), not just the API routers. (Earlier revisions hooked only the routers; see the review note below.)

2. Bulk the /api/agents/sync-health auto-sync lookup (removes an N+1)
Replaced the per-agent get_git_auto_sync_enabled(name) loop with a single scoped get_all_git_auto_sync_enabled(accessible) query, chunked at 900 IN-params to stay under SQLite's host-parameter cap. Access scope unchanged (still get_accessible_agents).

Mirrors the PERF-269 context-stats cache pattern (in-process, per-worker). No schema change, no new endpoints, no frontend change. Payload shapes byte-for-byte identical.

Testing

21 unit tests, no live server (Docker layer monkeypatched):

• tests/unit/test_73_stats_cache.py — single-flight coalescing (10 concurrent → 1 Docker call), per-agent lock isolation, TTL hit/expiry, invalidation, in-flight invalidation race (generation guard), 404/400/500 not-cached, defensive env parse, TTL=0 disable, payload parity, and primitive-boundary invalidation (stop/start/remove/rename + label fallback + non-agent no-op).
• tests/unit/test_73_sync_health_bulk.py — scoped/unscoped/empty-set maps, per-agent parity, multi-chunk IN() merge.

pytest tests/unit/ -q
1831 passed, 7 skipped

Pre-landing review (gstack /ship)

• Test coverage: 80% of changed paths (target met); core cache + bulk-query logic 100%. Remaining gaps are carried-over branches needing a live server.
• Specialists (testing, maintainability, security, performance, api-contract): security + api-contract clean (SQL params bound, /sync-health scoping correct, payloads byte-identical); performance/maintainability findings were documented design trade-offs. Testing flagged the missing 500-not-cached path → test added.
• Adversarial (Claude + Codex, cross-model agreement; Codex rated P2): the stats cache was only invalidated by the start/stop/delete router handlers, but internal lifecycle paths (deploy_local, ops restart, subscription re-assign, system restart, rename) change containers via the container_stop/start/remove/rename primitives directly — so /stats could serve stale "running" data for ≤ the 12s TTL after an out-of-band restart, and a renamed-away/reused name could briefly serve a different agent's stats. Resolved by moving invalidation to the primitive chokepoint (commit 831dff9f). Re-verified: circular import broken via lazy import, coverage strictly expands (no path lost invalidation), rename evicts the OLD name, helper is best-effort and can't break a lifecycle op.

ruff on the touched modules adds no new errors (pre-existing repo-wide debt unchanged).

Not in this PR (tracked)

• Frontend WS-reconnect backoff — the other documented half of perf: Backend CPU saturation at 45% with ~30 agents due to WebSocket churn and polling storms #73 (utils/websocket.js flat 5s retry, no backoff/jitter). Frontend-only, separate blast radius → follow-up under perf: Backend CPU saturation at 45% with ~30 agents due to WebSocket churn and polling storms #73. This PR is backend-only by approved scope.
• Docker stats executor-pool sizing (Codex F7) — low real-world impact (per-detail-page poll, not fanned out); deferred.

Closes the backend half of #73.

🤖 Generated with Claude Code

[vybe]

LGTM — read the concurrency logic in full. Single-flight (atomic setdefault), generation-guard (captured before Docker await, write gated on unchanged gen), and double-checked locking are all correct; errors never cached. The Codex-P2 fix moving invalidation to the docker_utils primitive chokepoint (container_stop/start/remove/rename) is the right call — all out-of-band lifecycle paths invalidate, and rename evicts the OLD name first. Bulk /sync-health SQL is parameterized (f-string is placeholders only) and chunked at 900 under SQLite's param cap. Payload byte-identical, CI green incl schema-parity, 21 targeted tests. Backend half only — #73 stays open for the frontend WS-backoff follow-up. Validated via /validate-pr.