Skip to content

fix(world-postgres): skip parked runs during startup recovery - #1

Closed
div-cowboy wants to merge 1 commit into
mainfrom
cursor/fix-postgres-recovery-parked-runs-4f24
Closed

div-cowboy wants to merge 1 commit into
mainfrom
cursor/fix-postgres-recovery-parked-runs-4f24

Conversation

@div-cowboy

Copy link
Copy Markdown
Owner

Description

Fixes vercel#3119.

world-postgres startup recovery listed every pending/running run and enqueued each one with a fresh Graphile job key. Parked hook/wait sessions therefore replayed on every process restart, and repeated boots accumulated duplicate jobs.

A query over event history is not enough to tell those parks from the crash window after a wake is stored: production sessions parked in a for await still have hook_received in the log. vercel#3162 skipped any run that still had a hook row, which loses that crash window, and stamped the recovery key as a message idempotencyKey, which skips per-run serialization.

This change:

  • Adds workflow_runs.wake_pending, set in the same transaction as an orchestrator wake (hook_received, wait_completed, step_completed, step_failed, hook_disposed) and cleared when an orchestrator delivery completes.
  • Skips a running run that has open hooks and/or non-due waits, no pending wake, and no interrupted steps. Pending runs, due waits, interrupted steps, wake_pending, and runs with no suspension state still recover (fail-open). Existing parked rows default to wake_pending=false, so they skip immediately after upgrade.
  • Enqueues recovery with Graphile jobKey startup-recovery:<runId> and not a message idempotency key, so repeated boots replace the outstanding job instead of stacking new ones, without clobbering a later legitimate wake.

I did not change Graphile preserve_run_at retry-budget reset (the follow-up discussed on the issue). Concurrent live-peer replay (vercel#3758) is also out of scope.

How did you test your changes?

  • Unit tests in recovery.test.ts / reenqueue.test.ts / queue.test.ts for skip vs recover, the crash window, fail-open, stable job keys, and clearing wake_pending only on orchestrator completion.
  • Integration test test/startup-recovery.test.ts against a real Postgres (testcontainers): parked hook/wait skipped, runnable work recovered, wake_pending set by hook_received, five boots leave one job per runnable run.

Those tests fail on current main (every active run is enqueued, jobs accumulate) and pass with this change.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
    • Use the correct semver bump type: patch for bug fixes, minor for new features, major for breaking changes.
    • Use pnpm changeset --empty if you are changing documentation or workbench apps
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
Open in Web Open in Cursor 

Startup recovery listed every pending/running run and enqueued each
with a fresh Graphile job key, so parked hook/wait sessions replayed
on every boot and jobs accumulated.

Classify runnable vs parked from persisted state, using a wake_pending
checkpoint for the crash window after a wake is stored, and enqueue
recovery jobs with a stable Graphile job key that is not a message
idempotency key.

Fixes vercel#3119

Signed-off-by: div-cowboy <6291624+div-cowboy@users.noreply.github.com>
@div-cowboy

Copy link
Copy Markdown
Owner Author

Superseded by vercel#5407.

@div-cowboy div-cowboy closed this Nov 13, 2026
@div-cowboy
div-cowboy deleted the cursor/fix-postgres-recovery-parked-runs-4f24 branch November 13, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

world-postgres: startup recovery re-enqueues parked runs and accumulates duplicate jobs

1 participant