fix(selfhost): stop the queue drain from claiming new work during shutdown; add shutdown headroom to compose (#9007)

[JSONbored]

Problem

Every "PR looks stuck" incident this session (orphaned ai-review-lock, a disposition stranded between publish and execute, etc.) traced back to one root cause: the process cannot shut down cleanly, so a deploy during active review traffic SIGKILLs a pass mid-flight.

Two compounding defects:

1. pump()'s while (await processOne()) loop (both src/selfhost/pg-queue.ts and src/selfhost/sqlite-queue.ts) never checked shutdown state — once a pump claimed the first due job it kept claiming every subsequent due job for as long as more existed, entirely blind to a concurrent stop(). Graceful shutdown was therefore bounded by "drain the whole backlog", not "finish what's already in flight".
2. docker-compose.yml had no stop_grace_period for the app service — Docker's 10s default is far shorter than a review pass (AI calls + GitHub round-trips routinely take tens of seconds to minutes).

Combined: docker compose restart/stop/a redeploy during active traffic sends SIGKILL before the (buggy, unbounded) drain ever completes, severing whatever pass is in flight and leaving its Redis locks/DB state exactly where it stood.

Fix

A shuttingDown flag, deliberately distinct from running. The obvious first attempt — gating the drain loop on running itself — is wrong and I caught it via a full-file test run before pushing: running also gates two other legitimate call shapes — binding.send()'s fire-and-forget kickOne(), and drain()'s own direct pump() call — both of which run a pump loop before start() has ever been called (running starts false and isn't set until the first start()). Gating on running makes every pre-start() drain()/send() a silent no-op, which broke ~150 pre-existing tests relying on exactly that pattern in each file.

shuttingDown instead means only "stop() has been called and hasn't been superseded by a later start()": false until stop() sets it (before its own drain-wait), reset by the next start(). Applied identically to both queue backends.

Also adds stop_grace_period: 300s to the loopover service so the now-correct drain actually gets time to run before Docker escalates.

Tests

Two regressions per backend (4 total), plus an invariant guarding the exact mistake caught while building the fix:

1. Reproduces the bug directly: preload 5 jobs at concurrency:1, start(), let the lone pump claim job 1, stop() mid-consume — must return once job 1 finishes, not after 2-5 also drain.
2. Guards the running-vs-shuttingDown mistake: a pump kicked via send()/drain() before start() is ever called must be unaffected.

Validation

• npx vitest run test/unit/selfhost-pg-queue.test.ts test/unit/selfhost-sqlite-queue.test.ts — 280/280 passed (both full files, not just the new tests).
• npx vitest run test/unit/selfhost-compose-db-health.test.ts — 4/4 passed (existing compose structural checks).
• docker compose config --quiet — valid.
• tsc --noEmit --incremental false — clean.
• Prettier flags all 5 touched files identically on main (pre-existing repo-wide state), so left unreformatted rather than dragging in an unrelated tree-wide diff.

Closes #9007

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.66%. Comparing base (3c1173e) to head (1bc3974).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9098      +/-   ##
==========================================
- Coverage   93.88%   92.66%   -1.23%     
==========================================
  Files         810      810              
  Lines       80600    80603       +3     
  Branches    24454    24455       +1     
==========================================
- Hits        75675    74689     -986     
- Misses       3560     4840    +1280     
+ Partials     1365     1074     -291     

Flag	Coverage Δ
backend	93.47% <100.00%> (-1.71%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/selfhost/sqlite-queue.ts	99.61% <100.00%> (+<0.01%)	⬆️

... and 3 files with indirect coverage changes

[loopover-orb]

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 LoopOver is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing}