world-postgres: startup recovery re-enqueues runs a LIVE peer process is executing — concurrent replay corrupts the event log #3758

Summary

reenqueueActiveRuns() re-enqueues running runs on every world start, with no way to tell whether a different, live process is executing them right now. On a horizontally-scaled deployment that is not a restart — it is a scale-out, and the previous owner is still running. Two processes then replay the same run concurrently, both append to the event log, and the run dies REPLAY_DIVERGENCE ×4 → CORRUPTED_EVENT_LOG.

This is separate from the two open issues on the same function:

• world-postgres: startup recovery re-enqueues parked runs and accumulates duplicate jobs #3119 — over-selection of durably parked runs, and duplicate job accumulation across repeated restarts. Its stated impact is backlog, startup delay, memory.
• Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs #2978 — recovery crossing application/namespace boundaries.

Both are about which runs get enqueued and how many jobs accumulate. This one is about a concurrency invariant: even a perfectly-selected, perfectly-deduplicated recovery enqueue is unsafe if the run is executing elsewhere at that moment. The three are additive — fixing the recovery predicate narrows the window, it does not close it, because the same duplicate arises from any other path that enqueues a run whose owner is mid-replay.

The invariant is already yours

packages/world-postgres/src/queue.ts serializes replays of one run, with this comment:

// Preserve step fan-out while preventing two workflow replays from
// mutating the same run's event log at the same time.
const previous = inflightWorkflowRuns.get(workflowRunSerializationKey);

inflightWorkflowRuns is a module-level Map. The invariant holds within one process and is unenforced across processes — while reenqueueActiveRuns is the thing that manufactures the cross-process case, and does so on a completely routine event (a pod boot).

Production failure

Azure Container Apps, HTTP-autoscaled 1↔N, one Postgres world shared by all replicas. Versions: workflow@4.8.3, @workflow/world-postgres@4.3.3, @workflow/world@4.3.1, graphile-worker@0.16.6, Node 24. (Checked against the latest stable line before filing: world@4.4.0's recovery.ts and world-postgres@4.3.4's start() are unchanged.)

A scheduled pipeline run — two LLM turns, ~12 minutes — was 30 seconds from its finalize step when the autoscaler added a replica.

12:45:00.060  pod A  run wrun_…SWTZ8W created; first step step_…SWTZ9G at 12:45:00.080
12:45–12:56   pod A  both turns complete successfully
12:56:53      ── SuccessfulRescale: scaled to 2 by http-scaler ──
12:57:04      ── pod B container started ──
12:57:09.151  pod B  WDK worker started
12:57:09.???  pod B  [world-postgres] Re-enqueued 4 active run(s) on startup
12:57:10.492  pod B  step_created for wrun_…SWTZ8W  ← 1.3s after the re-enqueue
12:57:1x      pod B  [Workflow] Workflow replay diverged … workflowRunId: 'wrun_…SWTZ8W'
12:57:1x      pod A  [Workflow] Workflow replay diverged … workflowRunId: 'wrun_…SWTZ8W'   ← same run, same second

The divergent event is step_created whose correlationId is the run's first step, created 12 minutes earlier — pod B replaying from the top of a run pod A owns. (Timestamps decoded from the ULID prefixes of the ids in the error message; pod A's wall clock was ~45s behind, which is why the raw log timestamps do not line up. Unrelated host issue, mentioned only so the ordering above reads correctly.)

Error, identical on all three attempts:

Workflow replay diverged 4 times after 3 recovery replays; latest divergent event was wevt_….
Last divergence: Replay could not consume event: eventType=step_created, correlationId=step_…, eventId=wevt_….
errorName: CorruptedEventLogError   errorCode: CORRUPTED_EVENT_LOG

Impact was data loss, not delay. Our business-retry fires on the next reconcile tick, so attempts 1 and 2 were created inside the same two-minute window while both replicas were still racing, and hit the same duplicate. The retry budget was gone in 120 seconds. Two completed LLM turns were paid for and discarded, the run produced nothing, and the user-facing artifact for that session simply did not exist.

Background rate on the same service: 308 replay diverged log lines in five days, most absorbed by the three recovery replays. 27 of 68 one-minute buckets containing a divergence fall within three minutes of a Re-enqueued … active run(s) on startup.

What we would want

Any of these closes it; they are listed cheapest-first, not as alternatives we prefer between:

1. A recoverActiveRuns: false option on createWorld(). Lets a deployment that has its own reconciler opt out entirely. One boolean, no semantics to design.
2. Skip runs whose queue job is currently locked. graphile_worker's locked_at/locked_by already says "a worker is executing this"; a dead worker's lock expires, so crash recovery is preserved. Confined to world-postgres, needs no cross-world contract.
3. Cross-process serialization of workflow replays — the durable equivalent of inflightWorkflowRuns. A Postgres advisory lock keyed by run id is enough for world-postgres. This is the general fix: it also covers duplicates that arrive from paths other than startup recovery.

At minimum, the doc comment on reenqueueActiveRuns should stop asserting that duplicate enqueues are safe:

 * Re-enqueue all active (pending/running) workflow runs so they resume
 * processing after a world restart. The workflow handler is idempotent
 * (event-log replay), so duplicate enqueues are safe.

Idempotent-under-replay is not the same as safe-under-concurrent-replay, and inflightWorkflowRuns exists because you already know that. As written, the comment reads as a guarantee to anyone deciding whether the package is safe to run on more than one instance.

What we shipped meanwhile

Option 3, outside the library: a Postgres session advisory lock keyed by run id, taken by our own HTTP wrapper around the /flow handler and held for the dispatch. A dispatch that loses it re-parks (200 + {timeoutSeconds}) instead of replaying. /step is untouched — the fan-out there is deliberate. Fail-open: no DB, connect failure, or query error proceeds unlocked, since a lock we cannot take must not become an outage.

Happy to test a patch against production traffic — this reproduces for us naturally, several times a day, whenever a scale-out lands on a long run.