Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments #3138

Follow-up from #2544 (boot-time recovery via ensureWorldStarted()).

Problem

start()'s boot recovery re-enqueues every pending/running run in storage — it does not know which instance is responsible for a run. On a horizontally-scaled or rolling deployment of the Postgres World, every booting instance therefore performs a full paginated scan of the active runs and re-enqueues all of them: N replicas ⇒ N scans + N duplicate enqueues per in-flight run on every deploy. Replay idempotency, the inline-ownership lease, and job-key dedup make this correct, but at scale it is a thundering herd of redundant scans, enqueues, and wake replays.

Current mitigation

WORKFLOW_SKIP_BOOT_RECOVERY=1 (added in #2544) lets operators run recovery from a single designated instance and opt the rest out. That is manual designation, not a mechanism.

Possible directions

• Single-flight: an advisory lock (e.g. pg_advisory_lock) or a recovery-epoch row so only one booting instance per deploy performs the scan.
• Owner-aware: attribute runs to instances (lease/heartbeat) so a booting instance only re-drives runs whose owner is gone; this would also shorten the mid-step recovery delay currently bounded by the inline-ownership lease (WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS, default 860s), since a dead owner could be detected affirmatively instead of waiting out the lease.

Related: #2978 (shared-DB namespace isolation for recovery), #2780 (inline-ownership lease).

🤖 Generated with Claude Code