[world-postgres] Renew running deliveries' job locks and redeliver abandoned jobs after a configurable stale window (opt-in) by pranaygp · Pull Request #4759 · vercel/workflow

pranaygp · 2026-10-08T05:50:39Z

Description

Summary. Adds an opt-in lease for @workflow/world-postgres deliveries. With jobLockStaleSeconds / WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS set, a running delivery renews its Graphile job lock, and every process that runs workers releases this World's jobs whose locks have gone unrenewed for that many seconds, so another process redelivers them. Today a job whose worker died stays locked for Graphile Worker's fixed 4 hours. The default is 0 (off), so upgrading changes nothing until a deployment opts in. 30 is the recommended value for deployments with more than one process.

This is the lease model #3992 and #3993 ask for ("a lease/visibility-timeout model where a delivery that stops being renewed is simply redelivered"), for crashed deliveries. It doesn't change retired jobs.

Fixes

The fix is opt-in: it takes effect once a deployment sets jobLockStaleSeconds (WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS; 30 is recommended for more than one process), on the v5 line (no backport).

Closes world-postgres: crash recovery runs only at startup, so a stranded run waits for a deploy #3993. A run whose process died mid-delivery no longer waits for the next deploy: a live peer redelivers its job within about 1.5 windows. The release only touches jobs whose locks went unrenewed, so unlike a periodic reenqueueActiveRuns it leaves runs a healthy peer is executing alone. That is the "lease, or a liveness check" @zachares's follow-up comment asks for. A job that had already used its last attempt still isn't redelivered (73 attempts on main since fix(world-postgres): leave job attempts past core's max-deliveries ceiling #4428; see world-postgres: step jobs are permanently retired by lost workers, not by failures #3992).
Closes sleep() is not durable across worker crashes in Postgres World #679. The restart case already recovers through start()'s re-enqueue of active runs ([world] Re-enqueue active runs on world restart #1534) when the app calls world.start() at server startup, as the Postgres World docs describe (feat(recovery): start the World at server boot to recover in-flight runs #2544 automates that call). What the thread left open is a worker that dies without a restart, whose job stays locked for Graphile's 4 hours: @jcourson-bg's analysis there, and @jiaruini's production evidence (66 jobs locked by 66 different workers after 71 container restarts). This fixes that. @jcourson-bg proposed a per-worker heartbeat table with forceUnlockWorkers; this renews each running job's own lock instead, which needs no new table and also keeps Graphile's 4 h reset from redelivering a healthy delivery that runs longer than that.

world-postgres: step jobs are permanently retired by lost workers, not by failures #3992 (@zachares), partly. A lost worker's job is redelivered after the window instead of 4 h, but its claim still spends an attempt and records no reason, and changing that accounting is what the issue asks for. fix(world-postgres): leave job attempts past core's max-deliveries ceiling #4428 raised the budget to 73 attempts.
world-postgres: each failed job release costs the queue a Graphile Worker worker for good #4750 / [world-postgres] Replace a runner that loses a worker to a failed job release #4751 (@himself65), partly. With the option on, a job whose release failed (and, in invoke mode, its executor queue) is released after the window instead of after Graphile's 4 h reset, which [world-postgres] Replace a runner that loses a worker to a failed job release #4751 documents as unchanged. Replacing the lost worker is [world-postgres] Replace a runner that loses a worker to a failed job release #4751's fix; the two are complementary.
Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments #3138, partly. @lunareed720's comment there reports a killed worker's job staying locked for 4 h on main; this releases it once its lock goes stale, on deployments with one consumer or several. Single-flight or owner-aware boot recovery is unchanged.
fix(world-postgres): abort stalled HTTP delivery on shutdown #3064 (@joeyhotz). It releases a stalled delivery's lock on graceful shutdown; this covers the hard kill (SIGKILL, OOM, a lost host) that it leaves to Graphile's 4 h reset. Complementary, not superseded.

Why

graphile-worker 0.16.6, which this package pins, sets locked_at once, when a worker claims a job (dist/sql/getJob.js). It resets only locks older than a hard-coded interval '4 hours' (dist/sql/resetLockedAt.js, every 8-10 min), and is_available requires locked_at is null. 0.17.3 and 0.18.0 still hard-code the 4 hours and have no renewal hook, so upgrading Graphile wouldn't help (and 0.17 is a breaking worker-to-pool change that raises the Node floor to 22.18); this PR stays on 0.16.6. The result:

A crash on a multi-host deployment waits 4 h or a restart. When a process dies mid-delivery, no live peer picks its job up. Only a restart's reenqueueActiveRuns recovers the run sooner. Repro against a real Postgres: 20 s after SIGKILLing the claiming worker, a live peer had made 0 deliveries and the job was still locked, attempts 1. The comment above DEFAULT_DELIVERY_HEADERS_TIMEOUT_MS said "Crash recovery is covered by Graphile releasing the job when the worker dies", which isn't what Graphile does. This PR corrects it.
A healthy delivery that runs longer than 4 h is redelivered while it still runs (a long inline step, a parent polling a child, an invoke executor). Graphile resets its lock and another worker claims the job.

This matters for durable agent harnesses that use the queue lease as their only crash backstop, with no heartbeats of their own. On world-vercel, an unacknowledged message is redelivered after its 300 s visibility window. On a multi-host world-postgres deployment, the same crash is recovered after 4 h or a restart. With this PR and WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS=30, it's recovered after about 1.5 windows, about 45 s.

What changed

New src/job-lease.ts. It's owned by the queue, not by a Graphile runner, so a runner replacement like [world-postgres] Replace a runner that loses a worker to a failed job release #4751 doesn't affect it.
- Renewal. Each task in the runner's taskList is wrapped. While a delivery runs, one unref'd timer per queue renews every held job in a single UPDATE of _private_jobs.locked_at, fenced on locked_by = <claiming worker>. The same statement renews the job's named queue in _private_job_queues, which covers invoke mode's per-run …flows:<runId>:executor lock. It runs every quarter of the window, and at least every 10 s.
- Stale-lock release ("sweep"). Every process with a runner runs Graphile's own resetLockedAt update with the configured window instead of 4 hours, every half window with jitter. It's scoped to this World's task identifiers (${jobPrefix}flows, ${jobPrefix}flows_executor), so other apps sharing the graphile_worker schema keep their locks. It uses FOR UPDATE … SKIP LOCKED, and both statements use the database clock.
- Lost-lock guard. A renewal that no longer finds its (job, worker) pair means the job was released. Matching on the pair matters because after a release, another worker of the same process can hold the same job id while the stale holder is still running. When that delivery returns, the wrapper throws JobLockLostError instead of resolving. Graphile then calls failJob, which is fenced on locked_by and changes nothing, rather than completeJob, which deletes by id only and would delete a successor's row. If no renewal has shown the lock to be held within half the window (a starved pool or a DB blip), the delivery renews once more before it's acknowledged.
- Outage guard. After this process's own renew or sweep fails, its next sweep only runs SELECT 1 to re-arm, so holders get a renewal in after a database outage before peers release their jobs.
- It uses the schema Graphile itself resolves (GRAPHILE_WORKER_SCHEMA or graphile_worker).
- With 0 (the default), wrap returns the task unchanged and no timers start.
queue.ts. Wraps the two task handlers, starts the sweeper after run(), and stops renewals after the runner has drained in close(). The handlers themselves are unchanged.
config.ts. Adds jobLockStaleSeconds. The env var is resolved in the queue, like the delivery timeouts. An invalid env value warns and falls back to 0; an invalid option throws.
Docs. README (option, env table, a "Crash recovery" section), HOW_IT_WORKS.md, and the v5 docs (Postgres World env var and "How it works", worlds configuration reference). Each says the option is off by default because of rolling upgrades, and how to turn it on.
Changeset. @workflow/world-postgres minor.

Turning it on

Set WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS=30 (or createWorld({ jobLockStaleSeconds: 30 })) on every process that shares the database, once every process runs a version that includes this PR. Use the same value everywhere, and keep it well above the longest a process can block its event loop or wait for a pool connection.

Compatibility and rollout

Default off. Upgrading is a no-op until a deployment sets the option. It's off because of rolling upgrades: a process on the previous version neither renews nor fences its acknowledgement, so if an upgraded process released one of its deliveries that runs longer than the window, the old process's unfenced completeJob would delete the successor's row when it finished. That loses the redelivery's retry, and the run stalls until a restart. The README ("Crash recovery"), HOW_IT_WORKS.md, the Postgres World page and the worlds configuration reference say this too, along with how to turn it on.
Writes to Graphile's private tables (_private_jobs, _private_job_queues; no new functions or indexes). No public function refreshes locked_at. The package pins graphile-worker exactly, and the integration test fails if a Graphile upgrade changes the schema or semantics. The existing invoke admission check still hard-codes graphile_worker; that's unchanged here.
False takeover. If a holder's renewals stall past the window, its job is redelivered while it still runs. Causes are synchronous code blocking the event loop, a starved pg pool (renewal shares the pool with deliveries), or a long DB outage. The lost-lock guard keeps the stale holder from deleting the successor's row, the outage guard covers DB-wide outages, and delivery was already at-least-once.
Load, when on. One batched UPDATE per busy process every ≤10 s, and one sweep per runner process every half window. The sweep scans locked rows of _private_jobs, as Graphile's own 8-10 min reset already does.
Attempts. A released crash still uses up the attempt the dead worker's claim spent (world-postgres: step jobs are permanently retired by lost workers, not by failures #3992), as Graphile's reset does today.
Not covered. A process that is alive but hung keeps renewing. The existing WORKFLOW_POSTGRES_HEADERS_TIMEOUT_MS / WORKFLOW_POSTGRES_BODY_TIMEOUT_MS deadlines still cover that.
Backport. Not intended: this is a new option.

How did you test your changes?

The new tests fail on main and pass here. I checked by running them against main's src/queue.ts and src/config.ts.

test/queue-lease.test.ts (new; real Postgres and real Graphile Workers). Each test opts in with a 2 s window.
- Another host redelivers the job of a worker that died mid-delivery. A child process (test/fixtures/crashing-queue.mjs, which uses dist) claims a delivery and is SIGKILLed. A live queue in the test process gets it again as attempt 2 and completes it. On main: expected [] to deeply equal [ { attempt: '2' } ] after 15 s.
- A delivery held for 3× the window keeps its lock while a second "host" sweeps. locked_at advances, attempts stays 1, and it's delivered once. On main, locked_at never moves.
- A delivery whose lock was taken over leaves the successor's job alone. The job's locked_by is moved to another worker mid-delivery. When the delivery finishes, the row is still there, locked by the successor. On main, Graphile's unfenced completeJob deletes it.
- Invoke mode, and other apps' jobs. An executor job and its workflow_flows:<runId>:executor queue lock, claimed by a dead worker an hour ago (inside Graphile's 4 h window), are released and redelivered with the executor provenance headers. A stale other_app_task lock is left alone.
src/job-lease.test.ts (new, fake timers). The 0 default, option and env resolution; renewal cadence, the 10 s cap and batching; lost locks, failed renewals, and two holders of one job id told apart by worker; a late renewal after the delivery finished; the final check when renewals stall; jobs without an id or worker; 0 doing nothing; sweep cadence, scoping and stop; the outage guard; GRAPHILE_WORKER_SCHEMA; and a very long window staying within setTimeout's 2^31 - 1 ms limit.
src/queue.test.ts. The wiring: the runner's task renews with [jobId], [workerId], the sweeper runs with [stale, ['workflow_flows', 'workflow_flows_executor']] (and the jobPrefix variant from the env var), and nothing runs after close().
Commands, on the current head (after merging main and turning the default off):
- TZ=UTC pnpm vitest run src in packages/world-postgres: 9 files, 94/94 pass.
- test/queue-lease.test.ts against a throwaway local PostgreSQL 14 (the testcontainers setup swapped for its connection string): 4/4 pass. On the earlier head, the whole package suite passed with testcontainers (347/348; the one failure, test/storage.test.ts > should release a retained hook after its deadline, compares the host clock with the container's and passed on rerun; this PR doesn't touch storage).
- pnpm typecheck is clean. Biome is clean for the changed files apart from three complexity warnings in queue.ts that already exist on main.
- node scripts/check-changesets.mjs and node scripts/check-no-unrun-tests.mjs pass.
- The latest commit only changes the docs and the changeset (the default-off rationale and how to turn it on).

Docs Preview

Page	v5
Worlds configuration: `jobLockStaleSeconds`	https://workflow-docs-git-pgp-world-postgres-lease-renewal.vercel.sh/docs/configuration/worlds#joblockstaleseconds
Postgres World: `WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS`	https://workflow-docs-git-pgp-world-postgres-lease-renewal.vercel.sh/worlds/postgres#workflow_postgres_job_lock_stale_seconds
Postgres World: How it works (crash recovery)	https://workflow-docs-git-pgp-world-postgres-lease-renewal.vercel.sh/worlds/postgres#how-it-works

Open questions for reviewers

Default off (0) or on? I chose off because of the rolling-upgrade case above. On would fix crash recovery for every multi-host deployment without configuration. Should a later release turn it on (for example 30 or 300 s) once processes without renewal are unlikely to share a database?
Writing to Graphile's private tables. Is that acceptable while the version is pinned, or would you rather upstream a renewal hook / configurable stale interval to graphile-worker first?
Shared pool. Renewal and sweep share the pg pool with deliveries, so pool starvation longer than the window causes a false takeover. A dedicated connection would avoid that, at the cost of another reconnect path (cf. [world-postgres] Fix process crash when the database drops the LISTEN connection #4658). Worth adding now?

Merge order

[world] State what a queue handler can rely on, and free world-local's slot while a delivery waits #4763 (the queue handler contract) documents the Postgres crash row; once this PR lands, that row can say a dead worker's delivery comes back after the stale window when jobLockStaleSeconds is set, instead of after Graphile Worker's 4 h reset.
[world-postgres] Replace a runner that loses a worker to a failed job release #4751, feat(world-postgres): pause and resume claiming without closing the world #4501, [world-postgres] Scope startup recovery to the World's jobPrefix #4761 and fix(world-postgres): store timestamps as timestamp with time zone #4757 also touch queue.ts, config.ts, the README or the Postgres World docs. The overlap here is a few lines in setupListeners / close() and new doc sections, so whichever lands second needs a small merge. The lost-lock path doesn't change the delivery's abort signal, which [world-postgres] Replace a runner that loses a worker to a failed job release #4751 rewires.

PR Checklist - Required to merge

📦 pnpm changeset was run to create a changelog for this PR (@workflow/world-postgres minor)
🔒 DCO sign-off passes (run git commit --signoff on your commits)
📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

🤖 Generated with Claude Code

…e ones after a configurable window graphile-worker 0.16 sets a job's locked_at once, at claim, and only resets locks older than a fixed 4 hours. A worker that dies mid-delivery therefore keeps its job locked for 4 hours: on a multi-host deployment nothing redelivers it until some process restarts and re-enqueues active runs. A healthy delivery that runs longer than 4 hours is reset while it runs and delivered twice. The queue now renews the job lock (and the named queue lock, for invoke mode's per-run executor queue) of every delivery it runs, in one batched UPDATE fenced on the claiming worker, every quarter of jobLockStaleSeconds (at most every 10 s). Every process with a runner releases this World's jobs whose locks went unrenewed for jobLockStaleSeconds, using Graphile Worker's own reset statement with a configurable window, scoped to this World's task identifiers. A delivery whose lock was lost fails instead of completing, because Graphile completes with an unfenced delete that could remove a successor's row, while failing is fenced. jobLockStaleSeconds / WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS defaults to 300; 0 restores the previous behavior. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>

Add the option and WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS to the README, HOW_IT_WORKS.md and the v5 docs (Postgres World page and the worlds configuration reference), with the recovery bound, the same-value-everywhere rule, the stalled-renewal caveat and the rolling-upgrade note. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>

vercel · 2026-10-08T05:50:40Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
example-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-astro-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-express-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-fastify-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-hono-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-nestjs-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-nitro-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-nuxt-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-python-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workbench-vite-workflow	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workflow-docs	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workflow-swc-playground	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workflow-tarballs	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC
workflow-web	Ready	Preview, v0	Oct 27, 2026 9:49pm UTC

changeset-bot · 2026-10-08T05:50:44Z

🦋 Changeset detected

Latest commit: 1cc7994

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-10-08T05:50:57Z

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

addTenWorkflow via pages router (nextjs-webpack · local-dev / local / node / stable)
hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running (python · vercel-prod / vercel / node / production)
promiseAllWorkflow via pages router (nextjs-webpack · local-dev / local / node / stable)
sleepingWorkflow via pages router (nextjs-webpack · local-dev / local / node / stable)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 21:50:27Z · abandoned wrun_01M5ZNF95XS136FCE4E0Y6KF8H
cold-start-warmup · suite warmup (tanstack-start) · at 21:50:29Z · abandoned wrun_01M5ZNF967TPJZ4G21E89T5DNZ
run-pickup-stall · basic step error preserves message and stack trace (nextjs-webpack) · at 21:58:38Z · abandoned wrun_01M5ZNY8B8C7KJE61TD5QJM594
run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:58:42Z · abandoned wrun_01M5ZNYCSMRJ6DX9PCGXKR4WX2

E2E Test Summary

Summary

	Passed	Skipped	Total
✅ ▲ Vercel Production	3982	878	4860
✅ 💻 Local Development	4669	551	5220
✅ 📦 Local Production	4669	551	5220
✅ 🐘 Local Postgres	4669	551	5220
✅ 🪟 Windows	348	12	360
✅ 🌐 Cross-language Conformance	68	87	155
✅ dynamic-runs	0	0	0
✅ vercel-http-transport	897	183	1080
✅ vercel-multi-region	27	0	27
✅ vercel-ws-transport	607	113	720
Total	19936	2926	22862

Details by Category

✅ ▲ Vercel Production

App	Passed	Skipped
✅ astro-node	145	35
✅ astro-quickjs	145	35
✅ example-node	145	35
✅ example-quickjs	145	35
✅ express-node	145	35
✅ express-quickjs	145	35
✅ fastify-node	145	35
✅ fastify-quickjs	145	35
✅ hono-node	145	35
✅ hono-quickjs	145	35
✅ nest-node	145	35
✅ nest-quickjs	145	35
✅ nextjs-turbopack-node	172	8
✅ nextjs-turbopack-quickjs	172	8
✅ nextjs-webpack-node	172	8
✅ nextjs-webpack-quickjs	172	8
✅ nitro-node	145	35
✅ nitro-quickjs	145	35
✅ nuxt-node	145	35
✅ nuxt-quickjs	145	35
✅ python-node	66	114
✅ sveltekit-node	164	16
✅ sveltekit-quickjs	164	16
✅ tanstack-start-node	145	35
✅ tanstack-start-quickjs	145	35
✅ vite-node	145	35
✅ vite-quickjs	145	35

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	151	29
✅ astro-stable-quickjs	151	29
✅ express-stable-node	151	29
✅ express-stable-quickjs	151	29
✅ fastify-stable-node	151	29
✅ fastify-stable-quickjs	151	29
✅ hono-stable-node	151	29
✅ hono-stable-quickjs	151	29
✅ nest-stable-node	151	29
✅ nest-stable-quickjs	151	29
✅ nextjs-turbopack-canary-node	179	1
✅ nextjs-turbopack-canary-quickjs	179	1
✅ nextjs-turbopack-quickjs-snapshot	179	1
✅ nextjs-turbopack-stable-node	179	1
✅ nextjs-turbopack-stable-quickjs	179	1
✅ nextjs-webpack-canary-node	179	1
✅ nextjs-webpack-canary-quickjs	179	1
✅ nextjs-webpack-stable-node	179	1
✅ nextjs-webpack-stable-quickjs	179	1
✅ nitro-stable-node	151	29
✅ nitro-stable-quickjs	151	29
✅ nuxt-stable-node	151	29
✅ nuxt-stable-quickjs	151	29
✅ sveltekit-stable-node	170	10
✅ sveltekit-stable-quickjs	170	10
✅ tanstack-start-node	151	29
✅ tanstack-start-quickjs	151	29
✅ vite-stable-node	151	29
✅ vite-stable-quickjs	151	29

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	151	29
✅ astro-stable-quickjs	151	29
✅ express-stable-node	151	29
✅ express-stable-quickjs	151	29
✅ fastify-stable-node	151	29
✅ fastify-stable-quickjs	151	29
✅ hono-stable-node	151	29
✅ hono-stable-quickjs	151	29
✅ nest-stable-node	151	29
✅ nest-stable-quickjs	151	29
✅ nextjs-turbopack-canary-node	179	1
✅ nextjs-turbopack-canary-quickjs	179	1
✅ nextjs-turbopack-quickjs-snapshot	179	1
✅ nextjs-turbopack-stable-node	179	1
✅ nextjs-turbopack-stable-quickjs	179	1
✅ nextjs-webpack-canary-node	179	1
✅ nextjs-webpack-canary-quickjs	179	1
✅ nextjs-webpack-stable-node	179	1
✅ nextjs-webpack-stable-quickjs	179	1
✅ nitro-stable-node	151	29
✅ nitro-stable-quickjs	151	29
✅ nuxt-stable-node	151	29
✅ nuxt-stable-quickjs	151	29
✅ sveltekit-stable-node	170	10
✅ sveltekit-stable-quickjs	170	10
✅ tanstack-start-node	151	29
✅ tanstack-start-quickjs	151	29
✅ vite-stable-node	151	29
✅ vite-stable-quickjs	151	29

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	151	29
✅ astro-stable-quickjs	151	29
✅ express-stable-node	151	29
✅ express-stable-quickjs	151	29
✅ fastify-stable-node	151	29
✅ fastify-stable-quickjs	151	29
✅ hono-stable-node	151	29
✅ hono-stable-quickjs	151	29
✅ nest-stable-node	151	29
✅ nest-stable-quickjs	151	29
✅ nextjs-turbopack-canary-node	179	1
✅ nextjs-turbopack-canary-quickjs	179	1
✅ nextjs-turbopack-quickjs-snapshot	179	1
✅ nextjs-turbopack-stable-node	179	1
✅ nextjs-turbopack-stable-quickjs	179	1
✅ nextjs-webpack-canary-node	179	1
✅ nextjs-webpack-canary-quickjs	179	1
✅ nextjs-webpack-stable-node	179	1
✅ nextjs-webpack-stable-quickjs	179	1
✅ nitro-stable-node	151	29
✅ nitro-stable-quickjs	151	29
✅ nuxt-stable-node	151	29
✅ nuxt-stable-quickjs	151	29
✅ sveltekit-stable-node	170	10
✅ sveltekit-stable-quickjs	170	10
✅ tanstack-start-node	151	29
✅ tanstack-start-quickjs	151	29
✅ vite-stable-node	151	29
✅ vite-stable-quickjs	151	29

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	174	0	6
✅ nextjs-turbopack-quickjs	174	0	6

✅ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
✅ python	68	0	87

✅ dynamic-runs

App	Passed	Failed	Skipped
✅ astro-vercel	0	0	0
✅ example-vercel	0	0	0
✅ express-vercel	0	0	0
✅ fastify-vercel	0	0	0
✅ hono-vercel	0	0	0
✅ nest-vercel	0	0	0
✅ nextjs-turbopack-vercel	0	0	0
✅ nextjs-webpack-vercel	0	0	0
✅ nitro-vercel	0	0	0
✅ nuxt-vercel	0	0	0
✅ sveltekit-vercel	0	0	0
✅ tanstack-start-vercel	0	0	0
✅ vite-vercel	0	0	0

✅ vercel-http-transport

App	Passed	Skipped
✅ example	145	35
✅ express	145	35
✅ hono	145	35
✅ nextjs-turbopack	172	8
✅ nitro	145	35
✅ vite	145	35

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Skipped
✅ example	145	35
✅ express	145	35
✅ nextjs-turbopack	172	8
✅ vite	145	35

📋 View full workflow run

github-actions · 2026-10-08T05:51:01Z

📊 Workflow Benchmarks

commit 1cc7994 · Tue, 27 Oct 2026 21:56:37 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1802 (+54%) 🔻	1910 🔴 (+37%) 🔻	1943 🔴 (+34%) 🔻	2007 🔴 (+11%)	30
TTFS	stream	1837 (+73%) 🔻	1937 🔴 (+33%) 🔻	1976 🔴 (+35%) 🔻	2136 🔴 (+40%) 🔻	30
TTFS	hook + stream	2097 (+32%) 🔻	2261 🔴 (+28%) 🔻	5109 🔴 (+179%) 🔻	5351 🔴 (+150%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	575 (+12%)	846 (+8.5%)	2214 (+175%) 🔻	2804 (+65%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2497 (+69%) 🔻	4189 (+1.7%)	4302 (+4.3%)	7459 (+24%) 🔻	10
STSO	1020 steps (inline)	81 (-1.2%)	98 (-10%)	114 (-12%)	155 (-46%) 💚	1019
WO	1020 steps	98984 (-12%)	98984 (-12%)	98984 (-12%)	98984 (-12%)	1

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 112293ms → this run 98718ms (Δ -13575ms, -12%)

 50-100 ms  ███████████████░░░░░░░░┃  main 499  this 792  +293
100-150 ms  █████┃████████            main 474  this 212  -262
150-200 ms  ┃                         main  24  this  10   -14
200-250 ms  ┃                         main   9  this   3    -6
250-300 ms  ┃                         main   3  this   2    -1
300-350 ms  ┃                         main   1  this   0    -1
350-400 ms  ┃                         main   1  this   0    -1
400-450 ms  ┃                         main   1  this   0    -1
450-500 ms  ┃                         main   1  this   0    -1
500-550 ms  ┃                         main   1  this   0    -1
550-600 ms  ┃                         main   4  this   0    -4
600-650 ms  ┃                         main   1  this   0    -1

📜 Previous results (3)

`76d4d49`

Thu, 08 Oct 2026 18:14:44 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	365 (+1.1%)	2168 🔴 (+48%) 🔻	2199 🔴 (+40%) 🔻	2271 🔴 (+12%)	30
TTFS	stream	1827 (+32%) 🔻	2132 🔴 (+47%) 🔻	2171 🔴 (+49%) 🔻	2285 🔴 (+47%) 🔻	30
TTFS	hook + stream	589 (-64%) 💚	2438 🔴 (+40%) 🔻	2507 🔴 (+36%) 🔻	5207 🔴 (+15%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	553 (+29%) 🔻	715 (-9.0%)	731 (-11%)	2241 (+8.6%)	10
Fan-out TTLS	Promise.all(100 steps)	1467 (+3.4%)	9005 (+129%) 🔻	9106 (+108%) 🔻	9211 (+42%) 🔻	10
STSO	1020 steps (inline)	91 (+5.8%)	109 (+1.9%)	126 (+0.8%)	154 (-12%)	1019
WO	1020 steps	109618 (±0%)	109618 (±0%)	109618 (±0%)	109618 (±0%)	1

`05af544`

Thu, 08 Oct 2026 16:08:44 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1861 (+23%) 🔻	2068 🔴 (+9.3%)	2126 🔴 (+7.4%)	2625 🔴 (+31%) 🔻	30
TTFS	stream	1849 (+81%) 🔻	1942 🔴 (+31%) 🔻	1955 🔴 (+31%) 🔻	1996 🔴 (+19%) 🔻	30
TTFS	hook + stream	1819 (+44%) 🔻	2278 🔴 (+29%) 🔻	2387 🔴 (+32%) 🔻	2685 🔴 (+38%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	557 (+3.7%)	893 (+8.0%)	2373 (+164%) 🔻	2789 (+53%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	1418 (-9.2%)	4012 (-38%) 💚	5197 (-25%) 💚	9220 (+22%) 🔻	10
STSO	1020 steps (inline)	88 (+1.1%)	118 (+9.3%)	138 (+7.8%)	243 (+41%) 🔻	1019
WO	1020 steps	116412 (+6.4%)	116412 (+6.4%)	116412 (+6.4%)	116412 (+6.4%)	1

`dd63d2d`

Thu, 08 Oct 2026 06:20:11 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	288 (-72%) 💚	1909 🔴 (+31%) 🔻	1931 🔴 (+25%) 🔻	2261 🔴 (+22%) 🔻	30
TTFS	stream	322 (-77%) 💚	1873 🔴 (+30%) 🔻	1938 🔴 (+33%) 🔻	2204 🔴 (+36%) 🔻	30
TTFS	hook + stream	644 (-60%) 💚	2295 🔴 (+33%) 🔻	2542 🔴 (+39%) 🔻	2685 🔴 (-39%) 💚	30
Fan-out TTFS	Promise.all(100 steps)	517 (+5.7%)	759 (-57%) 💚	2371 (+25%) 🔻	2457 (+20%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	5273 (+334%) 🔻	8324 (+31%) 🔻	8521 (+16%) 🔻	10361 (+33%) 🔻	10
STSO	1020 steps (inline)	60 (-15%) 💚	119 (+8.2%)	144 (+11%)	262 (+56%) 🔻	1019
WO	1020 steps	117568 (+4.0%)	117568 (+4.0%)	117568 (+4.0%)	117568 (+4.0%)	1

ℹ️ Metric definitions & methodology

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies. Streaming delivery performance is measured in durabench.}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

github-actions · 2026-10-08T05:53:50Z

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 44 total

fence=per-spec

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	6	0ms	ok
✅ `hook-at-step-started`	completed	11	0ms	ok
✅ `hook-at-step-completed`	completed	11	0ms	ok
✅ `hook-at-hook-created`	completed	11	0ms	ok
✅ `claim-hook-at-registration`	completed	11	0ms	ok
✅ `claim-hook-at-step-started`	completed	11	0ms	ok
✅ `deadline-hook-wins`	completed	6	1.0h	ok
✅ `deadline-expires`	completed	6	1.0h	ok
✅ `step-vs-timer-early-settlement`	completed	8	1.0h	ok
✅ `long-sleep`	completed	11	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	10	2.0s	ok
✅ `parallel-steps`	completed	9	0ms	ok
✅ `hook-on-execution-state`	completed	11	0ms	ok
✅ `peek-hook-before-branch`	completed	11	0ms	ok
✅ `peek-hook-after-branch`	completed	11	0ms	ok
✅ `peek-hook-at-registration`	completed	11	0ms	ok
✅ `race-hook-before-probe`	completed	11	0ms	ok
✅ `race-hook-after-probe`	completed	11	0ms	ok
✅ `race-duplicate-delivery`	completed	12	0ms	ok
✅ `attr-hook-before-step`	completed	10	0ms	ok
✅ `attr-hook-after-step`	completed	10	0ms	ok
✅ `attr-from-step-body`	completed	12	0ms	ok
✅ `fork-hook-after-timeout`	completed	13	1.0m	ok
✅ `fork-hook-before-timeout`	completed	13	1.0m	ok
✅ `count-hook-after-timeout`	completed	16	1.0m	ok
✅ `count-hook-before-timeout`	completed	19	1.0m	ok
✅ `stale-read-step-count-fork`	completed	19	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	13	1.0m	ok
✅ `step-vs-step-fork`	completed	12	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	12	0ms	ok
✅ `fence-catches-benign-direction`	completed	12	5ms	ok
✅ `in-flight-before-decision`	completed	16	1.0m	ok
✅ `in-flight-before-decision-counted`	completed	16	1.0m	ok
✅ `in-flight-after-decision`	completed	18	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	19	1.0m	ok
✅ `fork-hook-wins`	completed	12	1.0m	ok
✅ `fork-timeout-wins`	completed	12	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	16	1.0m	ok
✅ `claimed-payload-under-fork`	completed	16	1.0m	ok
✅ `writers-independent-step-bodies`	completed	12	0ms	ok
✅ `writers-scripted-tempo`	completed	11	0ms	ok
✅ `cancel-mid-step`	cancelled	7	0ms	skipped

Full trace: world-sim.txt

github-actions · 2026-10-08T05:54:07Z

Framework	Flow route	Step reg.	Framework output
hono	267.6 KiB (±0)	95.7 KiB (±0)	2.00 MiB (±0)
nextjs-turbopack	275.1 KiB (±0)	426 B (±0)	975.9 KiB (±0)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

1cc7994 · run

…he sweep timer A renewal batch can hold one job id twice: a stale holder whose renewals stalled past the window, and another worker of the same process that claimed the job again after it was released. The renewal statement only matches the current holder, but its result was keyed by job id alone, so the stale holder was treated as renewed and acknowledged, deleting the successor's job row. Key renewed locks by (job id, worker). Cap the sweep delay at setTimeout's 2^31 - 1 ms limit. Node runs longer timers after 1 ms, so a jobLockStaleSeconds above about 41 days turned the sweep into a busy loop against the database. Add an integration test that a delivery whose lock was taken over leaves the successor's job alone when it finishes (fails on main, where Graphile's unfenced completeJob deletes it). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * origin/main: [deps] Upgrade astro to 7.2.10 for GHSA-26w7-cxv4-gfx2 (#4738) [deps] Upgrade next to 16.3.8 for CVE-2026-94483 (#4737)

jobLockStaleSeconds / WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS now defaults to 0, so upgrading changes nothing until a deployment opts in. With renewal on by default, a rolling upgrade could lose a job: a process on the previous version neither renews its locks nor fences its acknowledgement, so an upgraded process releases any of its deliveries that run longer than the window, and when the old process finishes, Graphile Worker's unfenced completeJob deletes the successor's row. Turning it on is a deliberate step taken once every process runs a version that renews; the docs recommend 30 seconds for deployments with more than one process. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Say on every page the option appears (config reference, Postgres World page, README, HOW_IT_WORKS, changeset) that `jobLockStaleSeconds` is off by default because of rolling upgrades, and how to turn it on: the same value on every process that shares the database, once all of them run a version that includes the option. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

) ### Summary `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. ### Why - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. ### What changed - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. ### Compatibility and rollout - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. ### How did you test your changes? New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. ### Docs Preview | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | ### Open questions for reviewers 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. ### Related PRs and merge order - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. ### PR Checklist - Required to merge - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

) `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>

DarylThayil · 2026-10-27T19:32:06Z

Bump on this one!
Would prefer to absorb this than implement our own solution

VaguelySerious

(AI) Preliminary review: AI review: no blocking issues

VaguelySerious · 2026-10-27T21:45:11Z

+    if (jobs.length === 0) return;
+    const sentAt = now();
+    try {
+      const { rows } = await pool.query<RenewedRow>(renewQuery, [


AI Review: Note

Renewal (and confirm) go through the World's shared pg.Pool, so a process whose pool is saturated stops renewing, and its peers release deliveries that are still healthy. The sweeper's sweepArmed guard only reacts to this process's own failed queries; a renewal that's just waiting for a connection doesn't disarm anything, and peers can't tell anyway.

Repro against a real Postgres (scratch test, not committed): holder with maxPoolSize: 3, jobLockStaleSeconds: 2, delivery held open, then 3 × SELECT pg_sleep(6) on the holder's pool to occupy every connection for 3 windows, plus a peer with its own pool. Result: deliveries ["1","2"], so the peer redelivered the healthy delivery and the holder later hit JobLockLostError.

With the defaults (queueConcurrency 50, pool max 10, storage queries on the same pool), waits like this are plausible under load, and a redelivery adds more load. Options: give renewal/confirm a dedicated connection (one pg.Client per process, reconnected on error), or treat a renewal that hasn't come back within, say, staleMs / 4 as failing. At minimum, the docs could say pool waits count against the window, next to the event-loop caveat (they partly do in the config JSDoc, but not in the README or docs pages).

VaguelySerious · 2026-10-27T21:45:11Z

+  return {
+    enabled,
+    wrap(task) {
+      if (!enabled) return task;


AI Review: Note

Renewal and the sweep share one switch, so a process with 0 (the default) never renews. That leaves two things:

Partial opt-in: if the env var is set on one service but not another that shares the database (e.g. a web tier and a separate worker deployment), the opted-in side redelivers the other side's deliveries that run longer than the window. Repro (scratch): holder with 0, peer with 2, delivery held for 3 windows, deliveries ["1","2"].

Changing the default later brings back the rolling-upgrade hazard the docs describe, because every version up to that point has the default off and doesn't renew.

You could renew (and fence the ack) unconditionally at a fixed cadence, e.g. every 10s (the current cap), and gate only the sweep on jobLockStaleSeconds. Renewal is one batched UPDATE per tick per process, only while deliveries are running, and it also stops Graphile's 4 h reset from redelivering long healthy deliveries. Then any process on this version or later is safe next to a sweeper whose window is at least ~3–4× the cadence, and turning it on stops being a fleet-wide decision. If you keep the current coupling, it'd help to log a warning at start() when the sweep is on, saying every process must use the same setting.

VaguelySerious · 2026-10-27T21:45:11Z

+  SELECT jobs.id, jobs.job_queue_id, jobs.locked_by
+  FROM ${schema}._private_jobs AS jobs
+  JOIN ${schema}._private_tasks AS tasks ON tasks.id = jobs.task_id
+  WHERE jobs.locked_at < now() - make_interval(secs => $1::double precision)


AI Review: Note

There's no index on locked_at, so each sweep is a sequential scan of _private_jobs. EXPLAIN ANALYZE (postgres:15, warm cache): 10k waiting jobs ≈ 1 ms; 200k waiting jobs (e.g. many runs in sleep()) ≈ 19 ms, Seq Scan … Rows Removed by Filter: 200000. Every process sweeps about every stale/2 (≈15 s at 30), so the cost grows with jobs × processes. That's fine at the recommended settings and still far cheaper than the claim path. Maybe add a sentence to the crash-recovery docs so operators with large backlogs and many replicas aren't surprised by it; no change needed otherwise.

VaguelySerious · 2026-10-27T21:45:11Z

+      expect(deliveries).toEqual([{ attempt: '1' }]);
+    }, 30_000);
+
+    test("a delivery that lost its lock leaves the successor's job alone when it finishes", async () => {


AI Review: Note

Missing coverage: the { timeoutSeconds } reschedule path when the lock is lost mid-delivery. That path calls add_job with the same job_key before confirm throws JobLockLostError, so the outcome depends on Graphile's keyed-upsert semantics. I checked both variants in a scratch test, and both behave correctly:

released, not yet reclaimed: add_job replaces the row in place (attempts 0, run_at +30 s, payload attempt: 2), and the fenced failJob is a no-op;

reclaimed by another worker: the successor's row is neutralised (key null, attempts = max_attempts), and a separate follow-up row is inserted.

A test that pins this would protect against future changes to addGraphileJob (e.g. job_key_mode). A smaller gap: the confirm slow path (the extra renewal before ack) is only unit-tested against a mocked pool.

VaguelySerious · 2026-10-27T21:45:11Z

+  // before peers release their jobs.
+  let sweepArmed = true;
+
+  function markLost(job: HeldJob) {


AI Review: Nit

A lost lock is an expected outcome, but it shows up twice: this warn, then Graphile's Failed task … with error 'JobLockLostError…' at error level, with the full job row (payload included) and stack. Anyone alerting on Graphile error logs will get paged on a normal redelivery. Releases and lost locks also have no counter or metric, only console.warn, so operators can't easily see how often healthy deliveries get redelivered (see the pool-starvation note). Could the Graphile logger wrapper downgrade JobLockLostError failures, or could a counter be added?

VaguelySerious · 2026-10-27T21:45:11Z

+  // The schema Graphile Worker itself resolves when no `schema` option is
+  // passed, which this World never passes.
+  const schema = escapeIdentifier(
+    process.env.GRAPHILE_WORKER_SCHEMA || 'graphile_worker'


AI Review: Nit

This respects GRAPHILE_WORKER_SCHEMA, but the executor admission check (SELECT id FROM graphile_worker.jobs …) and invoke's graphile_worker.add_job(...) in queue.ts hard-code graphile_worker. That's not this PR's doing, but a deployment with a custom schema gets leases that work and an invoke mode that doesn't. Either thread one resolved schema through queue.ts as well, or drop the env lookup here so the World behaves the same everywhere.

Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

github-actions · 2026-10-28T17:07:08Z

Backport to stable failed for c07a6a3 due to a workflow error (backport job run).

This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.

Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

c07a6a395fef897602f9d99cf5aff46affd1ae7a

Takes peter/single-orchestrator's side of the two hunks that conflict with #4759, in packages/world-postgres/src/queue.ts and docs/content/worlds/v5/postgres.mdx, so that merging this branch into it is conflict-free. The branch reconciles both after the merge. Not for main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings in #4759 and #5000. Conflicts: - world-postgres/src/queue.ts (the delivery-timeout comment) and docs/content/worlds/v5/postgres.mdx (the "How it works" list): main's job-lock lease (#4759) rewrote the same lines the branch's spec 9 named-queue and dedupe notes edit. The merge takes the branch side of those hunks; the next commit combines both texts and states that, with jobLockStaleSeconds set, the lease also releases a hard-killed spec 9 run's named queue (renewal and release key on the job's job_queue_id, and the sweeper covers both task identifiers), so the "up to 4 hours" stall holds only without it. Spec 9 decisions: - #4759 job-lock lease: no runtime port. It is World-side and engine- agnostic: it wraps every task the queue registers, so spec 8 and spec 9 deliveries, and spec 9's per-run named queue, are covered as they are. Docs that said "invoke mode" for the per-run queue now name spec 9 too. - #5000 world-local fence lock: no port. It replaces the lock under world-local's in-band fence, which both specs' writes share; the branch's events-storage.ts merged cleanly onto withFenceFileLock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…run named queue Combines the branch's spec 9 named-queue and dedupe notes with #4759's job-lock lease in the delivery-timeout comment and the "How it works" list, and says where the per-run named queue is covered: with jobLockStaleSeconds set, the lease releases a hard-killed spec 9 run's queue lock too, so the "up to 4 hours" stall holds only without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Records what was done for each finding of the 2026-10-28 review, updates the world-postgres named-queue follow-up for main's opt-in job-lock lease (#4759), and adds follow-ups for a required QuickJS host contract, shared timer sends between the engines, and a QuickJS deadline handoff, which the new adapter contract test shows is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…n for spec 9 Main's opt-in job-lock lease (#4759) releases a hard-killed spec 9 run's named queue, but only with jobLockStaleSeconds set. Follow-up 7 now says to default the stale window on for spec 9 runs before spec 9 is default-on on world-postgres, and why it is not defaulted on here: the sweep cannot tell spec 9 jobs from spec 8's, earlier-version processes claim and complete spec 9 jobs without renewing or fencing, and a short window turns an event-loop stall into a second owner of an inline step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…and redeliver abandoned jobs after a configurable stale window (opt-in) (#5107) Manual backport of #4759 to `stable`. `stable` has no invoke mode (no executor task or per-run Graphile queues), keeps separate `workflow_flows` and `workflow_steps` tasks, and lacks `pollInterval`, `applicationManagedShutdown` and forwarding of Graphile's task `abortSignal`. `job-lease.ts` and `job-lease.test.ts` port unchanged (`stable` uses the same `graphile-worker` 0.16.6). ## Conflicts and how they were resolved - **`packages/world-postgres/src/config.ts`**: added only `jobLockStaleSeconds`. Left out `pollInterval`, `enableInvoke` and `applicationManagedShutdown`, which exist only on `main`. - **`packages/world-postgres/src/queue.ts`**: - Imports: added only `job-lease.js`, without `main`'s executor and invocation imports. - Kept `stable`'s `createQueueHandler`, and created `jobLeases` next to it. - Wrapped both of `stable`'s task handlers (`getJobQueueName(workflowPrefix)` and `getJobQueueName(stepPrefix)`) in `jobLeases.wrap`, instead of `main`'s flows and executor tasks. - The sweeper covers those same two task identifiers. - Kept `stable`'s `run()` options and `close()` (`runner.stop()`), adding `await jobLeases.stop()` after the runner stops, as in the original. - **`packages/world-postgres/src/queue.test.ts`**: - Dropped the ~290 lines of `main`-only tests in the conflict block and kept only the two new tests. - Expected sweep identifiers changed to `[..._flows, ..._steps]`. - `stable`'s `startHangingWorkflowHttpServer()` takes no argument. - `stable` doesn't forward Graphile's `abortSignal`, so the renewal test ends its held delivery with `WORKFLOW_POSTGRES_HEADERS_TIMEOUT_MS=1500` and expects `ETIMEDOUT`, instead of aborting and expecting `AbortError`. - **`packages/world-postgres/README.md`**: - Added the env example, the `jobLockStaleSeconds` option row and the `WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS` env row on top of `stable`'s tables. Left out `main`'s other new rows. - Kept `stable`'s queue-behavior bullet about the `/flow` and `/step` routes and added the new bullet after it. - Placed `### Crash recovery` after `stable`'s concurrent-deliveries paragraph, so that paragraph stays under Queue Behavior. - Dropped the invoke-mode sentence and said "flow and step jobs". - **`packages/world-postgres/HOW_IT_WORKS.md`**: - Added the `## Crash recovery` section. - Didn't add `main`'s `## Shutdown` section, which belongs to an earlier `main`-only change. - Replaced "(invoke mode's per-run executor queue)" with "if it has one". - **`docs/content/docs/v5/configuration/worlds.mdx`** and **`docs/content/worlds/v5/postgres.mdx`** (deleted on `stable`): - Kept them deleted. `stable` documents this World in a single page, `docs/content/docs/deploying/world/postgres-world.mdx`. - Ported the `WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS` section into that page's Configuration section, and added the Crash Recovery bullet to its How It Works list. ## Files that applied cleanly but needed changes for `stable` - **`test/queue-lease.test.ts`**: - Removed `applicationManagedShutdown` from `startQueue` and the executor-job header capture. - Replaced the invoke-mode test ("releases a dead executor's job and its per-run queue lock") with a `stable` equivalent. It checks that a dead worker's `workflow_steps` job is released and redelivered, and that another application's job is left locked. - **`test/fixtures/crashing-queue.mjs`**: removed `applicationManagedShutdown`. ## Testing - `pnpm --filter @workflow/world-postgres build` passes. - `vitest run src/job-lease.test.ts src/queue.test.ts test/queue-lease.test.ts` passes, including the four real-Postgres and Graphile lease tests. - `pnpm test` in `packages/world-postgres`: 212 tests pass. `test/spec.test.ts` fails at import in the `world-testing` harness (`compile is not a function`), which needs a workbench build and isn't touched by this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

@pranaygp

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to stable, this PR will be updated. # Releases ## @workflow/world-postgres@4.4.0 ### Minor Changes - [#4759](#4759) [`3ff458e`](3ff458e) Thanks [@pranaygp](https://github.com/pranaygp)! - Add `jobLockStaleSeconds` (`WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS`, default `0`, off). When set, a running delivery renews its Graphile job lock, and a job whose process died is redelivered by another process once its lock has gone unrenewed for that many seconds, instead of after Graphile Worker's fixed 4 hours or the next restart. `30` is a good value when more than one process shares the database. It's off by default because of rolling upgrades: processes on an earlier version don't renew their locks, so turn it on once every process that shares the database runs this version. ### Patch Changes - [#5136](#5136) [`5c9b65d`](5c9b65d) Thanks [@alangenfeld](https://github.com/alangenfeld)! - A `run_started` that races the run ending no longer moves the run back to `running`; it's refused with `RunExpiredError`. - Updated dependencies [[`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/world-local@4.4.5 ## @workflow/astro@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 - @workflow/rollup@4.0.31 - @workflow/vite@4.0.31 ## @workflow/builders@4.1.21 ### Patch Changes - [#5063](#5063) [`e4dbc29`](e4dbc29) Thanks [@joelfickson](https://github.com/joelfickson)! - Export `workflowSerdeComputedPropertyPattern` alongside the other workflow pattern regexes. - Updated dependencies []: - @workflow/core@4.8.16 ## @workflow/cli@4.3.20 ### Patch Changes - [#5182](#5182) [`f3d6142`](f3d6142) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Respect `--project` and `--team` over the inferred `.vercel` link, and prefer the nearest `.vercel/project.json` in monorepos - Updated dependencies [[`e4dbc29`](e4dbc29), [`dbfd1ba`](dbfd1ba), [`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/builders@4.1.21 - @workflow/web@4.1.32 - @workflow/world-local@4.4.5 - @workflow/core@4.8.16 ## @workflow/core@4.8.16 ### Patch Changes - Updated dependencies [[`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/world-local@4.4.5 ## @workflow/nest@4.0.32 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 ## @workflow/next@4.1.20 ### Patch Changes - [#5063](#5063) [`e4dbc29`](e4dbc29) Thanks [@joelfickson](https://github.com/joelfickson)! - Run the Turbopack workflow loader on files that use `static [WORKFLOW_SERIALIZE]` computed keys (such as `@ai-sdk/openai` models), fixing "Class … not found" errors in steps. - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 - @workflow/core@4.8.16 ## @workflow/nitro@4.1.22 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29), [`dbfd1ba`](dbfd1ba)]: - @workflow/builders@4.1.21 - @workflow/web@4.1.32 - @workflow/rollup@4.0.31 - @workflow/vite@4.0.31 - @workflow/core@4.8.16 ## @workflow/nuxt@4.0.32 ### Patch Changes - [#5211](#5211) [`d30f07e`](d30f07e) Thanks [@atinux](https://github.com/atinux)! - Upgrade `@nuxt/kit` to 4.6.3 to pick up the fix for CVE-2026-71314. - Updated dependencies []: - @workflow/nitro@4.1.22 ## @workflow/rollup@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 ## @workflow/sveltekit@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 - @workflow/rollup@4.0.31 - @workflow/vite@4.0.31 ## @workflow/vite@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 ## @workflow/vitest@4.0.32 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29), [`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/builders@4.1.21 - @workflow/world-local@4.4.5 - @workflow/rollup@4.0.31 - @workflow/core@4.8.16 ## @workflow/web@4.1.32 ### Patch Changes - [#5079](#5079) [`dbfd1ba`](dbfd1ba) Thanks [@karthikscale3](https://github.com/karthikscale3)! - Recognize unframed serialized abort packets in the Streams viewer so encrypted packets can be decrypted. ## @workflow/web-shared@4.1.32 ### Patch Changes - Updated dependencies []: - @workflow/core@4.8.16 ## workflow@4.8.16 ### Patch Changes - Updated dependencies [[`f3d6142`](f3d6142), [`d30f07e`](d30f07e), [`e4dbc29`](e4dbc29)]: - @workflow/cli@4.3.20 - @workflow/nuxt@4.0.32 - @workflow/next@4.1.20 - @workflow/astro@4.0.31 - @workflow/nest@4.0.32 - @workflow/nitro@4.1.22 - @workflow/rollup@4.0.31 - @workflow/sveltekit@4.0.31 - @workflow/core@4.8.16 - @workflow/typescript-plugin@4.0.3 ## @workflow/world-local@4.4.5 ### Patch Changes - [#5069](#5069) [`ceaaa5c`](ceaaa5c) Thanks [@AndrewBarba](https://github.com/AndrewBarba)! - Deliver stream chunks in the order they were written. A reader that opened, or polled, while chunks were being written could get a later chunk before an earlier one, and could get a chunk twice. Writes to one stream now land in order, including a `writeToStreamMulti` group and concurrent writes, and the reader holds live chunks while it reads from disk. - [#4909](#4909) [`b065b0b`](b065b0b) Thanks [@fazxes](https://github.com/fazxes)! - Name each stream chunk as it lands, after every chunk already in the stream, so a write that stalled before landing no longer lands ahead of chunks a reader already read. ## @workflow/world-testing@4.1.31 ### Patch Changes - Updated dependencies [[`f3d6142`](f3d6142)]: - @workflow/cli@4.3.20 - workflow@4.8.16 - @workflow/core@4.8.16 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

pranaygp and others added 2 commits October 7, 2026 22:49

vercel Bot deployed to Preview – workflow-docs October 8, 2026 05:51

View deployment

vercel Bot temporarily deployed to Preview – workflow-docs October 8, 2026 06:05 Inactive

pranaygp mentioned this pull request Oct 8, 2026

[world] State what a queue handler can rely on, and free world-local's slot while a delivery waits #4763

Draft

2 of 3 tasks

pranaygp and others added 2 commits October 8, 2026 08:49

vercel Bot deployed to Preview – workflow-docs October 8, 2026 15:55

View deployment

pranaygp mentioned this pull request Oct 8, 2026

[world-postgres] Scope startup recovery to the World's jobPrefix #4761

Merged

2 of 3 tasks

pranaygp changed the title ~~[world-postgres] Renew running deliveries' job locks and redeliver abandoned jobs after a configurable stale window~~ [world-postgres] Renew running deliveries' job locks and redeliver abandoned jobs after a configurable stale window (opt-in) Oct 8, 2026

This was referenced Oct 8, 2026

[world-vercel] Let a queue handler choose its visibility timeout #4755

Open

fix(world-postgres): store timestamps as timestamp with time zone #4757

Merged

pranaygp marked this pull request as ready for review October 8, 2026 16:07

pranaygp requested a review from a team as a code owner October 8, 2026 16:07

Copilot AI balanced review requested due to automatic review settings October 8, 2026 16:07

Copilot AI reviewed Oct 8, 2026

vercel Bot deployed to Preview – workflow-docs October 8, 2026 18:04

View deployment

VaguelySerious mentioned this pull request Oct 16, 2026

[core] Start the World at server boot so self-hosted Worlds recover in-flight runs #4906

Draft

VaguelySerious reviewed Oct 27, 2026

View reviewed changes

Merge branch 'main' into pgp/world-postgres-lease-renewal

1cc7994

Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

vercel Bot deployed to Preview – workflow-docs October 27, 2026 21:47

View deployment

VaguelySerious approved these changes Oct 27, 2026

View reviewed changes

VaguelySerious merged commit c07a6a3 into main Oct 28, 2026
186 of 187 checks passed

VaguelySerious deleted the pgp/world-postgres-lease-renewal branch October 28, 2026 17:06

github-actions Bot mentioned this pull request Oct 28, 2026

Version Packages #4961

Merged

pranaygp mentioned this pull request Oct 28, 2026

[docs] Document abort packet framing, World queue/stream contract, and resilient-start public key #5102

Open

pranaygp mentioned this pull request Oct 28, 2026

Backport #4759: [world-postgres] Renew running deliveries' job locks and redeliver abandoned jobs after a configurable stale window (opt-in) #5107

Merged

pranaygp mentioned this pull request Oct 28, 2026

[core] Single-orchestrator runtime (spec 9) #4661

Merged

github-actions Bot mentioned this pull request Oct 29, 2026

Version Packages #5087

Merged

pranaygp mentioned this pull request Oct 29, 2026

[docs] Document stream encryption key lookup failures for writes to another run #5130

Open

VaguelySerious mentioned this pull request Oct 30, 2026

[spec 9] world-postgres stale named-queue lock after a hard kill #5164

Closed

Conversation

pranaygp commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fixes

Related

Why

What changed

Turning it on

Compatibility and rollout

How did you test your changes?

Docs Preview

Open questions for reviewers

Merge order

PR Checklist - Required to merge

Uh oh!

vercel Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sim World

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

DarylThayil commented Oct 27, 2026

Uh oh!

VaguelySerious left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pranaygp commented Oct 8, 2026 •

edited

Loading

vercel Bot commented Oct 8, 2026 •

edited

Loading

changeset-bot Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading