Skip to content

feat(world-postgres): pause and resume claiming without closing the world - #4501

Open
himself65 wants to merge 1 commit into
vercel:mainfrom
himself65:world-postgres-pause-claims
Open

himself65 wants to merge 1 commit into
vercel:mainfrom
himself65:world-postgres-pause-claims

Conversation

@himself65

@himself65 himself65 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

Closes #4500.

Every process running the Postgres World against one database claims from the same Graphile Worker queue. A process that should stop taking new work while it stays up had no way to do so. The typical case is a blue-green revision that no longer receives traffic but is still draining its streams and steps:

  • world.close() is final: it closes the streamer and releases the worker utils.
  • A runner stopped any other way comes back on the next enqueue, because queue() calls start().

This adds pauseClaims() and resumeClaims() to the queue, and through it to the world:

  • pauseClaims(). Graphile Worker's runner stops gracefully, so jobs already running finish and are recorded as usual. While paused, neither start() nor queue() starts another runner. Enqueueing, streams and storage keep working. It resolves once the stop has begun, not once the stopped runner's jobs are done.
  • resumeClaims(). It starts only the runner. It does not repeat the reenqueueActiveRuns() recovery that world.start() performs, which would re-enqueue every active run again. Before start() has run, it only lifts the pause.
  • close() also waits for a runner that a pause is still stopping, so a job finishing after the pause can still enqueue its reschedule before the worker utils are released.
  • Ordering. Pause and resume apply in call order: each waits for the previous change, so the last call decides.

The world-postgres README and the v5 Postgres World docs get a short "Pausing claims" section.

How did you test your changes?

  • Unit tests, src/queue.test.ts (mocked Graphile Worker, same setup as the existing queue tests):
    • a paused queue starts no runner, even from its own enqueue;
    • resumeClaims() starts a runner without repeating start();
    • resuming before start() only lifts the pause;
    • a pause stops the runner and resolves while it is still stopping, and close() waits for it before releasing the worker utils;
    • pause, resume, pause applies in call order;
    • a runner that Graphile Worker already stopped is tolerated;
    • resuming after close() starts nothing.
  • Unit test, src/reenqueue.test.ts. A paused world.start() re-enqueues active runs once, and world.resumeClaims() starts the runner without re-enqueueing them again.
  • Integration test, test/queue-claims.test.ts (real Graphile Worker and Postgres via testcontainers, with a loopback executor):
    • a paused queue leaves its own enqueued job unclaimed for longer than the 500ms poll interval, and delivers it once resumed;
    • a job running when the pause lands finishes and its row is deleted as completed; close() waits for it; a job enqueued after the pause stays unclaimed.
  • Mutation checks. Removing the pause check in startRunnerWhenExecutorIsReady() fails the unit and integration cases. So does removing the wait for stopping runners in close().
  • Full package, rebased onto main at d0f6b91. pnpm --filter @workflow/world-postgres test, with Docker available and the package built: 20 files, 331 tests. The tests this PR adds or changes passed in all four local runs. Two existing integration tests flaked locally on macOS:
    • storage.test.ts › "should release a retained hook after its deadline" failed in 2 of 4 runs here and 1 of 3 runs on main. It compares a host-clock deadline with Postgres now().
    • An unhandled 57P01 (terminating connection due to administrator command) was reported in 2 of 4 runs here and in none of 3 on main. Vitest attributed it to stream-offset.test.ts and listen-reconnect.test.ts, which do not use the queue. It did not reproduce in isolation: queue-claims.test.ts alone passed 5 of 5 runs, and those two files together passed 8 of 8, all without it.
  • Lint and types. tsc --noEmit and biome check are clean for the touched files. The only warnings are the same ones main reports, in functions this PR does not change.

The same change, as a patch on @workflow/world-postgres@4.3.6, is how our service stops a drained blue-green revision from claiming. A local two-process simulation of a cut is summarized in #4500.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR (minor)
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@himself65
himself65 requested a review from a team as a code owner September 29, 2026 23:57
@changeset-bot

changeset-bot Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 97dd188

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@workflow/world-postgres Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@himself65 is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

…orld

Every process running the Postgres World against one database claims from
the same Graphile Worker queue, and a process that should stop taking work
while it stays up, such as a blue-green revision that no longer receives
traffic but is still draining, had no way to: close() is final, and a runner
stopped any other way comes back on the next enqueue, because queue() calls
start().

Add pauseClaims() and resumeClaims() to the queue and the world. Pausing
stops the runner gracefully, so jobs already running finish; while paused,
neither start() nor queue() starts another, and enqueueing, streams and
storage keep working. Resuming starts only the runner, without repeating the
active-run recovery world.start() performs. close() waits for a runner a
pause is still stopping. Pause and resume apply in call order.

Closes vercel#4500

Signed-off-by: Alex Yang <himself65@outlook.com>
@DarylThayil

Copy link
Copy Markdown

upvote on this
@himself65 still working on this?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

world-postgres: pause and resume claiming without closing the world

2 participants