selfhost: unbounded graceful shutdown — a redeploy SIGKILL parks the in-flight review for up to 30 minutes

[JSONbored]

Summary

queue.stop() waits for in-flight jobs with no deadline, so a redeploy that lands during a multi-minute AI review is SIGKILLed by Docker's stop grace. Because the killed job's lease was heartbeated right up to the kill, neither runtime reclaim nor boot recovery picks it up for up to the full 30-minute lease timeout — so the interrupted review resumes 20–30 minutes later, with duplicated AI spend, against a 1–5 minute latency target and automated redeploys.

Mechanism (verified at HEAD, 776c414)

src/server.ts:1413-1435 — shutdown releases held locks, then calls backend.shutdown().

src/selfhost/pg-queue.ts:1789-1799 — stop() is effectively while (active > 0) await sleep(10), unbounded. The sqlite twin is identical (src/selfhost/sqlite-queue.ts:1712-1736), and its own comment at :1306-1307 already concedes this is "the root cause of every review pass severed mid-flight by a deploy".

The lease heartbeat runs at processingTimeoutMs / 3 ≈ 10 minutes (pg-queue.ts:1323), with DEFAULT_PROCESSING_TIMEOUT_MS = 30 minutes (src/selfhost/queue-common.ts:23). reclaimExpiredProcessingJobs (pg-queue.ts:1840-1849) only recovers rows older than now − 30 min. A job killed at the last heartbeat is therefore invisible to recovery for nearly the full window.

On a declared single-host deployment, that conservative multi-instance lease semantics buys nothing.

Deliverables

• Give stop() a deadline. On expiry, explicitly re-pend this process's own activeJobIds (the queue already knows exactly which they are) before exiting — converting a SIGKILL race into an immediate retry.
• Set stop_grace_period on the loopover service to something ≥ the longest legitimate job, or accept the bounded-stop behaviour above as the answer. Document which.
• Coordinate with locks: boot flush, SIGTERM ordering, registry keying and Redis LRU each free a live holder's lock #9468: locks are currently released before the drain wait, so a sibling can duplicate during it. The correct combined ordering is "stop claiming → drain with deadline → re-pend the remainder → release locks".
• Consider gating the conservative lease reclaim on an explicit single-instance signal (or a startup instance epoch), so a single-host deploy can recover its own jobs immediately at boot.

Tests

• SIGTERM during a long job ⇒ the job is re-pended within the deadline and claimable immediately, not after 30 minutes.
• Both queue backends behave identically.
• A job that finishes inside the deadline is not re-pended (no duplicate execution).

[linear-code]

JSO-6549