Repository navigation
fix(world-postgres): skip parked runs during startup recovery - #1
Closed
div-cowboy wants to merge 1 commit into
Closed
div-cowboy wants to merge 1 commit into
div-cowboy wants to merge 1 commit into
Conversation
Startup recovery listed every pending/running run and enqueued each with a fresh Graphile job key, so parked hook/wait sessions replayed on every boot and jobs accumulated. Classify runnable vs parked from persisted state, using a wake_pending checkpoint for the crash window after a wake is stored, and enqueue recovery jobs with a stable Graphile job key that is not a message idempotency key. Fixes vercel#3119 Signed-off-by: div-cowboy <6291624+div-cowboy@users.noreply.github.com>
Owner
Author
|
Superseded by vercel#5407. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes vercel#3119.
world-postgresstartup recovery listed everypending/runningrun and enqueued each one with a fresh Graphile job key. Parked hook/wait sessions therefore replayed on every process restart, and repeated boots accumulated duplicate jobs.A query over event history is not enough to tell those parks from the crash window after a wake is stored: production sessions parked in a
for awaitstill havehook_receivedin the log. vercel#3162 skipped any run that still had a hook row, which loses that crash window, and stamped the recovery key as a messageidempotencyKey, which skips per-run serialization.This change:
workflow_runs.wake_pending, set in the same transaction as an orchestrator wake (hook_received,wait_completed,step_completed,step_failed,hook_disposed) and cleared when an orchestrator delivery completes.wake_pending, and runs with no suspension state still recover (fail-open). Existing parked rows default towake_pending=false, so they skip immediately after upgrade.jobKeystartup-recovery:<runId>and not a message idempotency key, so repeated boots replace the outstanding job instead of stacking new ones, without clobbering a later legitimate wake.I did not change Graphile
preserve_run_atretry-budget reset (the follow-up discussed on the issue). Concurrent live-peer replay (vercel#3758) is also out of scope.How did you test your changes?
recovery.test.ts/reenqueue.test.ts/queue.test.tsfor skip vs recover, the crash window, fail-open, stable job keys, and clearingwake_pendingonly on orchestrator completion.test/startup-recovery.test.tsagainst a real Postgres (testcontainers): parked hook/wait skipped, runnable work recovered,wake_pendingset byhook_received, five boots leave one job per runnable run.Those tests fail on current
main(every active run is enqueued, jobs accumulate) and pass with this change.PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRpatchfor bug fixes,minorfor new features,majorfor breaking changes.pnpm changeset --emptyif you are changing documentation or workbench appsgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete