Repository navigation
[world-postgres] Skip parked runs in startup recovery and key recovery jobs by run - #5407
div-cowboy wants to merge 1 commit into
Conversation
Startup recovery listed every pending or running run and enqueued each with a fresh Graphile job key, so a run parked on a hook or wait was replayed on every boot and repeated boots stacked duplicate jobs. The recovery listing now leaves out a running run whose only live state is an open hook or a wait that is not yet due, with no unfinished step and no wake owed. A new workflow_runs.wake_pending flag records an owed wake: it is set in the transaction of run_started, hook_received, hook_disposed, wait_completed, step_completed and step_failed (taking the run row lock before the hook, wait or step row, the order other writers use), and cleared when an orchestrator delivery completes. A wake whose job was never queued therefore still recovers. Rows that predate the migration start with the flag set, so the first boot after upgrading recovers them as before. Each recovery job is keyed startup-recovery:<runId>, as a Graphile job key only (not a message idempotency key, which would skip per-run serialization), so a boot replaces the job an earlier boot left queued. Fixes vercel#3119 Signed-off-by: div-cowboy <6291624+div-cowboy@users.noreply.github.com>
🦋 Changeset detectedLatest commit: 202e88c The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@div-cowboy is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| headers: opts?.headers, | ||
| delaySeconds: opts?.delaySeconds, | ||
| jobKey: opts?.idempotencyKey ?? messageId, | ||
| jobKey: opts?.jobKey ?? opts?.idempotencyKey ?? messageId, |
There was a problem hiding this comment.
When a startup-recovery delivery returns { timeoutSeconds }, its replacement loses the per-run job key, so subsequent boots enqueue additional recovery jobs for the same active run.
| const completedRunId = orchestratorRunIdOf(body); | ||
| if (completedRunId) { |
There was a problem hiding this comment.
A successful delivery to another registered queue prefix clears wake_pending for a matching runId in its payload, allowing an unrelated consumer to suppress workflow startup recovery.
|
@vercel/workflow this is ready for review. The changeset is in and the DCO sign-off passes. |
Description
Fixes #3119.
Startup recovery lists every
pending/runningrun and enqueues each one with a fresh Graphile job key. A run parked on a hook or a wait is replayed on every boot, and repeated boots stack duplicate jobs for the same run.This change:
runningrun out of the recovery listing when all of these hold: its only live state is an open hook or a wait that isn't due yet, it has no unfinished step, and no wake is owed. Everything else still recovers, including legacy runs.workflow_runs.wake_pending(migration0031) to record an owed wake. The hook row alone can't distinguish a parked run from a storedhook_receivedwhose job was never queued, which is the problem raised on fix(world-postgres): skip parked runs and dedupe recovery jobs on startup #3162.run_started,hook_received,hook_disposed,wait_completed,step_completed, andstep_failed, and cleared when an orchestrator delivery completes.hook_received.startup-recovery:<runId>as a Graphile job key only. It is not a messageidempotencyKey, so the delivery keeps per-run serialization. A boot replaces the job an earlier boot left queued instead of adding another one.Not covered:
preserve_run_atmode still resets its attempt count (discussed on the issue).How did you test your changes?
test/startup-recovery.test.ts(Postgres via testcontainers):hook_receivedwithout its job is recovered, and so is a first delivery that died after creating its hook.step_completedandhook_receivedon one run don't deadlock.queue-http,queue-lease,queue-lease-default).PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete