Repository navigation
[world] In-band writer fence for world-local and world-postgres - #4821
Merged
VaguelySerious merged 7 commits intoOct 10, 2026
Merged
Conversation
…rld-postgres
Optional World capability `inBandFence`: a per-run in-band count, a
`{ seq, seqInBand }` snapshot on `events.list`, and in-band creates
refused with InBandSupersededError (412) unless `expectedSeqInBand`
matches, writing and allocating nothing. world-postgres checks the count
under a row lock on the run's slots row in the same transaction as its
entity updates and insert (migration 0026 adds `seq_in_band`). world-sim
implements the same fence. A shared conformance suite in
@workflow/world/test-support runs against all three, and world-testing
checks the fence end to end when a World declares it. Inert until a
caller sends `inBand: true`; the runtime does not yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Optional `deliveryCount` and `createdAt` on the queue handler's meta.
world-local reports both; world-postgres carries `createdAt` in the job
payload across `{ timeoutSeconds }` reschedules and delivers through
world-local's handler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
🦋 Changeset detectedLatest commit: dfba650 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 114726ms → this run 118706ms (Δ +3980ms, +3%) 📜 Previous results (2)7291660Fri, 09 Oct 2026 23:48:15 GMT · run logs
a9bd8eaFri, 09 Oct 2026 21:35:25 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (179 failed)astro-node (18 failed):
astro-quickjs (17 failed):
example-quickjs (18 failed):
express-quickjs (19 failed):
fastify-quickjs (18 failed):
hono-quickjs (18 failed):
nest-quickjs (17 failed):
nextjs-webpack-node (17 failed):
sveltekit-quickjs (18 failed):
vite-node (19 failed):
vercel-http-transport (51 failed)example (17 failed):
express (17 failed):
vite (17 failed):
vercel-multi-region (1 failed)nextjs-turbopack (1 failed):
vercel-ws-transport (32 failed)example (18 failed):
nextjs-turbopack (14 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3803 | 179 | 878 | 4860 |
| ✅ 💻 Local Development | 4669 | 0 | 551 | 5220 |
| ✅ 📦 Local Production | 4669 | 0 | 551 | 5220 |
| ✅ 🐘 Local Postgres | 4669 | 0 | 551 | 5220 |
| ✅ 🪟 Windows | 348 | 0 | 12 | 360 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 87 | 155 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ❌ vercel-http-transport | 846 | 51 | 183 | 1080 |
| ❌ vercel-multi-region | 26 | 1 | 0 | 27 |
| ❌ vercel-ws-transport | 575 | 32 | 113 | 720 |
| Total | 19673 | 263 | 2926 | 22862 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 127 | 18 | 35 |
| ❌ astro-quickjs | 128 | 17 | 35 |
| ✅ example-node | 145 | 0 | 35 |
| ❌ example-quickjs | 127 | 18 | 35 |
| ✅ express-node | 145 | 0 | 35 |
| ❌ express-quickjs | 126 | 19 | 35 |
| ✅ fastify-node | 145 | 0 | 35 |
| ❌ fastify-quickjs | 127 | 18 | 35 |
| ✅ hono-node | 145 | 0 | 35 |
| ❌ hono-quickjs | 127 | 18 | 35 |
| ✅ nest-node | 145 | 0 | 35 |
| ❌ nest-quickjs | 128 | 17 | 35 |
| ✅ nextjs-turbopack-node | 172 | 0 | 8 |
| ✅ nextjs-turbopack-quickjs | 172 | 0 | 8 |
| ❌ nextjs-webpack-node | 155 | 17 | 8 |
| ✅ nextjs-webpack-quickjs | 172 | 0 | 8 |
| ✅ nitro-node | 145 | 0 | 35 |
| ✅ nitro-quickjs | 145 | 0 | 35 |
| ✅ nuxt-node | 145 | 0 | 35 |
| ✅ nuxt-quickjs | 145 | 0 | 35 |
| ✅ python-node | 66 | 0 | 114 |
| ✅ sveltekit-node | 164 | 0 | 16 |
| ❌ sveltekit-quickjs | 146 | 18 | 16 |
| ✅ tanstack-start-node | 145 | 0 | 35 |
| ✅ tanstack-start-quickjs | 145 | 0 | 35 |
| ❌ vite-node | 126 | 19 | 35 |
| ✅ vite-quickjs | 145 | 0 | 35 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 174 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 174 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 87 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
❌ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ example | 128 | 17 | 35 |
| ❌ express | 128 | 17 | 35 |
| ✅ hono | 145 | 0 | 35 |
| ✅ nextjs-turbopack | 172 | 0 | 8 |
| ✅ nitro | 145 | 0 | 35 |
| ❌ vite | 128 | 17 | 35 |
❌ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ nextjs-turbopack | 26 | 1 | 0 |
❌ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ example | 127 | 18 | 35 |
| ✅ express | 145 | 0 | 35 |
| ❌ nextjs-turbopack | 158 | 14 | 8 |
| ✅ vite | 145 | 0 | 35 |
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟢 world-sim scenario book — 0 fail of 42 total
Full trace: |
Contributor
About these numbersSizes are gzip; parentheses show the change against
|
VaguelySerious
marked this pull request as ready for review
October 9, 2026 22:05
karthikscale3
left a comment
karthikscale3
left a comment
Contributor
There was a problem hiding this comment.
AI Review: Three reproducible issues in the new fence and queue metadata behavior are described inline. The existing local, Postgres, core, simulator, and live workflow checks passed; additional targeted probes reproduced these findings.
| // it. A restart starts every run back at `IN_BAND_SEQ_AT_RUN_CREATION`, | ||
| // which is safe: the old process's writers are gone, and the next | ||
| // delivery adopts the restarted value from its own snapshot. | ||
| const inBandSeq = new Map<string, number>(); |
Contributor
There was a problem hiding this comment.
AI Review: [P2] Share the fence at the storage boundary. These counters and locks are private to each storage instance, so two instances sharing one data directory do not exclude each other. I reproduced this with two createStorage(dir) instances: both read seqInBand=1, and both in-band attr_set calls with expectedSeqInBand=1 succeeded, taking slots 2 and 3. The advertised inBandFence contract requires exactly one writer to succeed and the other to receive InBandSupersededError before changing any entity. Please coordinate the counter/check/write across instances sharing the directory, or restrict the capability so it cannot promise exclusion that this configuration does not provide.
Member
Author
There was a problem hiding this comment.
(AI) Reproduced with a new conformance test (two storage instances over the same data race one in-band write each at the same count): on the old code both were accepted. Fixed in 15bc453: world-local's fence counters and locks now live in a process-global globalSingleton keyed by resolved data directory and run, so every createStorage(dir) in a process (and every bundled copy of the module) shares one fence. The test now gets exactly one success and one InBandSupersededError, and it also runs against world-postgres (a second createEventsStorage on the same pool), which passed before and after. Two OS processes on one directory are still not fenced against each other; world-local's run-file locks are per-process as well, so I documented that limit at the fence rather than adding on-disk claims.
Contributor
There was a problem hiding this comment.
AI Review: Verified on 7291660 with the original reproduction and the updated conformance suite: two createStorage(dir) instances now yield exactly one accepted write and one InBandSupersededError, with both snapshots at seqInBand 2. This fixes the reproduced same-process bug. The separate-OS-process limitation remains documented; this validation does not cover or establish a cross-process fence.
| // The same message: its id and creation time carry over, and | ||
| // the handler sees the next deliveryCount. | ||
| messageId: messageData.messageId, | ||
| createdAt: messageData.createdAt, |
Contributor
There was a problem hiding this comment.
AI Review: [P2] Preserve the cumulative delivery count across replacement jobs. With a real Graphile Worker and an HTTP handler returning timeoutSeconds twice, the same message was delivered three times with deliveryCount=[1,1,1], rather than [1,2,3]; messageId and createdAt remained stable. The replacement payload increments attempt, but createTaskHandler uses job.attempts + attemptOffset, and each replacement job resets job.attempts while retaining the old offset. The existing attempt reset predates this PR, but the new deliveryCount inherits it and violates its documented redelivery contract. Please carry the cumulative offset/count into replacement jobs and add a test that observes successive real deliveries; asserting only the addJob payload misses this.
Member
Author
There was a problem hiding this comment.
(AI) Reproduced with a real Graphile Worker test in test/queue-http.test.ts: a handler that answers { timeoutSeconds: 0 } twice saw attempts ['1','1','1']. Fixed in 86a5ed2: the replacement job's attemptOffset is now the deliveries made so far (attempt), since its own job.attempts starts again at 1. The test now observes ['1','2','3'], with the same messageId. Side effect, worth knowing: { timeoutSeconds } redeliveries on world-postgres now count toward core's MAX_QUEUE_DELIVERIES, which matches world-local (whose queue already counts them in x-vqs-message-attempt) and Vercel Queues' deliveryCount. Added a patch changeset.
Contributor
There was a problem hiding this comment.
AI Review: Verified on 7291660 using the original real-Graphile reproduction: timeoutSeconds redeliveries now report deliveryCount [1, 2, 3], with the same messageId and createdAt throughout. The new queue HTTP regression test also passes. The reproduced count-reset bug is fixed.
Member
Author
There was a problem hiding this comment.
(AI) Follow-up, revised in dfba650. Making attempt cumulative had a side effect: core counts attempt against its 48-delivery cap on every message, including a step retried in place via { timeoutSeconds } on its own message. I checked it end to end on world-postgres (nextjs-turbopack prod build, Postgres 15): a step with maxRetries: 100 that fails 55 times with RetryableError({ retryAfter: '1s' }) failed the run with MAX_DELIVERIES_EXCEEDED (49/48) under 86a5ed2, and completed at attempt 56 before it. So attempt is back to restarting with each replacement job, and the cumulative count is carried separately (deliveryOffset in the job payload, sent as x-vqs-message-delivery-count), which world-local's handler, the one world-postgres delegates to, reports as meta.deliveryCount. The Graphile test now asserts deliveryCount 1, 2, 3 and attempt 1, 1, 1 for one message. With this commit the 55-retry step completes again (attempt 56), and a 60-iteration sleep('1s') loop completes (60 wait_completed). Core reads only attempt, never deliveryCount, so the cap behaves as before.
Contributor
There was a problem hiding this comment.
AI Review: Confirmed on dfba650 with real Postgres/Graphile and the built core runtime: a step with maxRetries: 100 that throws RetryableError 55 times now completes on attempt 56, and a 60-iteration one-second sleep loop completes with all 60 wait_completed events. The metadata reproduction observes deliveryCount [1, 2, 3] separately from attempt [1, 1, 1], with stable messageId and createdAt. This corrects the retry-cap regression introduced by the earlier cumulative-attempt fix. The local queue, Postgres queue/HTTP and invocation suites also passed (91 tests).
| if (!isDefinitiveRefusal(error)) inBandSeq.set(key, current + 1); | ||
| throw error; | ||
| } | ||
| inBandSeq.set(key, current + 1); |
Contributor
There was a problem hiding this comment.
AI Review: [P2] Advance seqInBand only for positions actually allocated. create() can succeed without appending an event, for example run_started on an already-running run, but this unconditional increment still advances the fence. The simulator has the same issue at store.ts:747-749 and additionally returns allocated:1. A fresh simulator run followed by two fenced run_started calls reports snapshot={seq:2,seqInBand:3} despite having only two events; Postgres stays at {seq:2,seqInBand:2} for the same sequence. This contradicts the new allocation-count contract and can unnecessarily supersede another writer after a no-op. Please leave the count unchanged and report allocated:0 for idempotent creates that append nothing, with matching conformance coverage across backends.
Member
Author
There was a problem hiding this comment.
(AI) Reproduced with a new conformance test (a fenced run_started on a run that is already running): world-local and world-sim advanced seqInBand (and world-sim reported allocated: 1); world-postgres stayed put. Fixed in 15bc453: world-local and world-sim advance by the positions the create allocated (1 when it returned an event, 0 otherwise) and report allocated; the conformance test asserts the snapshot is unchanged, allocated is 0 where reported, and the same writer can keep writing at the unchanged count. It passes on world-local, world-postgres and world-sim.
Contributor
There was a problem hiding this comment.
AI Review: Verified on 7291660 with the original local/simulator probes and Postgres probe: an idempotent run_started leaves seq and seqInBand unchanged; local and simulator return allocated: 0. The updated fence conformance tests pass for all three backends. The reproduced no-op allocation bug is fixed.
…ence-and-run-serialization # Conflicts: # packages/world-local/src/storage/events-storage.ts
…ount only appended positions world-local's fence counters and locks were private to each storage instance, so two createStorage(dir) instances both accepted a write at the same count. They now live in a process-global singleton keyed by data directory and run. world-local and world-sim advanced the count on a create that appended nothing (run_started on a running run); they now advance by the positions allocated and report `allocated`. Conformance covers both, across world-local, world-postgres and world-sim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eschedules A rescheduled message is a new Graphile job whose job.attempts starts again at 1, but the replacement kept the old attemptOffset, so every redelivery reported attempt (and deliveryCount) 1. The replacement's offset is now the deliveries made so far. A real Graphile Worker test observes 1, 2, 3 across two reschedules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y as deliveryCount
Making `attempt` cumulative across { timeoutSeconds } reschedules counted
in-place step retries against core's max-deliveries cap: a step with
maxRetries: 100 failing 55 times failed the run with
MAX_DELIVERIES_EXCEEDED (49/48) on world-postgres, where it used to
succeed. `attempt` is back to restarting with each replacement job. The
cumulative count rides along as `deliveryOffset` in the job payload and
reaches the handler as `x-vqs-message-delivery-count`, which
world-local's handler reports as `deliveryCount`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
karthikscale3
approved these changes
Oct 10, 2026
karthikscale3
left a comment
karthikscale3
left a comment
Contributor
There was a problem hiding this comment.
AI Review: All three reproduced findings and the subsequent high-retry regression are fixed and independently verified on dfba650. Queue metadata now keeps cumulative deliveryCount separate from attempt; real Postgres workflows completed after 55 step retries and all 60 sleep iterations. The local queue, Postgres queue/HTTP and invocation suites passed, and the original fence/no-op probes remain fixed. No remaining actionable code-review blockers. CI is still running and should finish successfully before merging.
VaguelySerious
deleted the
peter/world-in-band-fence-and-run-serialization
branch
October 10, 2026 00:40
Contributor
|
No backport to This is additive feature work: a new optional To override, re-run the Backport to stable workflow manually via |
Merged
This was referenced Oct 12, 2026
Merged
pranaygp
added a commit
that referenced
this pull request
Oct 12, 2026
world-local kept a run's in-band count in memory, per process (#4821: "Not shared across OS processes"). Two processes on one data directory each fenced only their own writers: a stale in-band write from one was accepted after the other wrote in-band. world-local's event slots are already arbitrated across processes (exclusive publishes), so the fence was the one part that was not. - **The count lives on disk**, at `.locks/runs/<runId>.in-band[.tag]` next to the run's other lock files, written atomically (temp file and rename). - **Check and advance under a lock file** on it (proper-lockfile, the same settings as the hook token claim), taken after the in-process lock every storage instance in a process already shares, so writers in one process still take turns without polling the lock file. - **A writer that dies mid-write is counted.** Between its check and its publish an accepted write marks the count `pending`. A writer that died there leaves the mark (its lock goes stale), and the next writer through the lock counts that write, since it may have landed. As before, counting a write that did not land only costs a superseded redelivery; missing one that did would let a stale writer in. A snapshot reports the count without a pending write. - **Restarts keep the count** (before, a restart reset every run to `IN_BAND_SEQ_AT_RUN_CREATION`). - A refusal reports `seq` from the run's directory, not only from this instance's slot watermark. - A lock lost while a write held it (a process stalled past the lock's staleness) fails that write with a non-4xx `WorkflowWorldError`, so its writer treats the outcome as unknown. **Before / after**, two OS processes on one data directory (A loads, B in the other process writes in-band at that count, then A writes in-band at it): | | B | A (stale) | log | |---|---|---|---| | main | accepted | **accepted** | B's event, then A's | | this PR | accepted | refused, `InBandSupersededError` (`seqInBand: 2`) | B's event only | **Tests**: `in-band-fence.test.ts` forks a second process on the data directory (`in-band-fence.child.mjs`, which runs world-local's TypeScript source with Node's type stripping): a stale writer in the test process is refused after the other process wrote in-band, and of writers in four processes holding one count exactly one is accepted (both fail on main). Also: a pending mark is counted and cleared, a snapshot leaves it out, and a new storage instance reads the same count. world-local unit tests (661) pass, including the shared fence conformance suite; typecheck and biome clean. **Cost**: an in-band create now takes the lock file and writes the count file twice (pending, then the new count); out-of-band writes are unchanged, and a list reads one small file. Measured on a busy laptop (300 sequential `attr_set` writes on one run, two runs each): in-band p50 2.1-2.7 ms on main, 2.4-2.7 ms with this PR (out-of-band 1.2-1.3 ms on main), a list 2.5-3.3 ms on main, 2.7-2.8 ms with this PR. The runtime does not send `inBand` yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>
pranaygp
added a commit
that referenced
this pull request
Oct 12, 2026
Catches the docs up with SDK changes that shipped in the last day without (complete) docs updates. | Change | Source PR | Docs edit | | --- | --- | --- | | `world-postgres` honors `pagination.sortOrder` in `runs.list` and `steps.list`, so all first-party Worlds now do | #4756 | Added a `params.pagination.sortOrder` row (default `"desc"`, newest first) to `runs.list()` and `steps.list()` in the Storage reference | | `world-postgres` adds migration 0026, and every `events.list()` now reads its column | #4821 | The Postgres World setup callout now says to run `bootstrap` again when upgrading `@workflow/world-postgres` | | `world-postgres` handles idle-connection errors on the pool it creates, but not on a caller-owned `pg.Pool` | #4817 | Programmatic configuration notes that a pool you pass needs its own `error` listener, or a database restart crashes the process. Also moved the `pool` sentence above the `.env` snippet it was splitting from its lead-in | I reviewed the other PRs from the last day and they need no further docs: the in-band writer fence and queue handler meta (#4821) shipped their own Building a World docs and the fence is not used by the runtime yet, the lifecycle hook follow-ups (#4439) only touched tests and READMEs whose stream guidance the Lifecycle Hooks pages already cover, and the world-local stream listing fix (#4754), test pool teardown (#4785), and the Geistdocs bump aren't documented behavior. The Workflows pages in `vercel/front` were also checked and need no changes for these. ## Docs Preview | Page | Preview | | --- | --- | | Storage | [runs.list()](https://workflow-docs-git-workflow-docs-cleanup-mon-12-oct.vercel.sh/docs/api-reference/workflow-runtime/world/storage#runslist), [steps.list()](https://workflow-docs-git-workflow-docs-cleanup-mon-12-oct.vercel.sh/docs/api-reference/workflow-runtime/world/storage#stepslist) | | Postgres World | [Installation](https://workflow-docs-git-workflow-docs-cleanup-mon-12-oct.vercel.sh/worlds/postgres#installation), [Programmatic configuration](https://workflow-docs-git-workflow-docs-cleanup-mon-12-oct.vercel.sh/worlds/postgres#programmatic-configuration) | 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
pranaygp
added a commit
that referenced
this pull request
Oct 12, 2026
…ol-error-listener * commit 'c15581fe9422ac2ff9a07045b306eea755a2f2f0': [docs] Catch up docs with recently shipped SDK changes (#4837) [world-local] Fence in-band writes across processes (#4834) [changesets] Release pending minor changesets as patches (#4839) [world] In-band writer fence for world-local and world-postgres (#4821) # Conflicts: # docs/content/worlds/v5/postgres.mdx
pranaygp
added a commit
that referenced
this pull request
Oct 12, 2026
…ts the backend reports (#4833) The backend already refuses a stale in-band write on every run with an event sequencer: 412 `in-band-superseded`, carrying the run's `seq` and `seqInBand`, which #4733 maps to `InBandSupersededError`. The adapter declared no `inBandFence`, so a client had no way to know it may send the fence, and #4821's capability check skipped world-vercel. - **`capabilities.inBandFence: true`** on world-vercel. - **Counts, where the backend reports them.** `events.list` passes on `snapshot` when the list response carries it (the end frame's `snapshot`, else the `x-wf-snapshot-seq` / `x-wf-snapshot-seq-in-band` headers), and `events.create` passes on `allocated` from the response body. This is the wire shape #4661 reads, extracted from it. Today's backend sends neither on these runs (only the refusal carries counters), so both are inert until it does. - **Older runs.** On a run below spec 7 (no event sequencer) the backend answers an in-band write with a 400 `in-band-fence-unsupported`, never an unfenced write, so a client fences by the run's spec version. Documented, and pinned in a test. - **Contract: the counters, from a list or a refusal.** `inBandFence` no longer requires a snapshot on every page: a World may list none if every refusal carries both counters, read together. The capability doc, `EventListResponse` and Building a World say so, and Building a World gives the recipe for a writer that finds no snapshot: get the counters (`IN_BAND_SEQ_AT_RUN_CREATION` on a fresh run, otherwise a refusal of an in-band write at a count no run holds), list until the log holds position `seq`, write at `seqInBand`. `@workflow/world-testing` reads the counters the same way, so its fence suite holds for world-vercel. - **Strict counters.** Snapshot headers must be decimal digits, `seqInBand <= seq` is required, and a malformed end-frame snapshot lists none (falling back to the headers) instead of failing the page. `allocated` is passed on for single creates only; the batch count comes with #4661's batch report. **Inert for the runtime.** Nothing in core reads `inBandFence` or sends `inBand` yet, so this changes nothing for Workflow runs. A client that does (a durable agent harness writing its own session log on a World) can now fence its writes on Vercel. **Checked against production** (a run created through the adapter, in iad1), the sequence a stale writer meets: | step | result | |---|---| | list after `run_created` + `run_started` | no `snapshot` on the page | | in-band write at `MAX_SAFE_INTEGER` (a count no run holds) | 412 `InBandSupersededError`, `seq: 2, seqInBand: 1`, nothing written | | writer B: in-band `attr_set` at 1 | accepted, slot 3 | | writer A (loaded before B): in-band `attr_set` at 1 | 412 `InBandSupersededError`, `seqInBand: 2`, nothing written | | A and B at once, both at the same count | one accepted, one refused | For contrast, the same stale write made out-of-band with `eventCount` (bump and report) is accepted at the next slot. **Tests**: world-vercel unit tests (957), including new ones for the capability, the 400 on a run without the fence, the snapshot from the end frame and from the headers, no snapshot when the backend sends none or a malformed one (empty, hex, exponent, fractional, `seqInBand > seq`), and `allocated` on a create. Typecheck and biome clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
pranaygp
pushed a commit
that referenced
this pull request
Oct 12, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @workflow/ai@5.0.3 ### Patch Changes - Updated dependencies [[`fcb554b`](fcb554b)]: - @workflow/utils@5.0.2 ## @workflow/astro@5.0.4 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.4 - @workflow/rollup@5.0.4 - @workflow/vite@5.0.4 ## @workflow/builders@5.0.4 ### Patch Changes - Updated dependencies [[`854242e`](854242e), [`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b)]: - @workflow/core@5.2.1 - @workflow/errors@5.0.4 - @workflow/utils@5.0.2 ## @workflow/cli@5.0.4 ### Patch Changes - [#4760](#4760) [`1342a96`](1342a96) Thanks [@pranaygp](https://github.com/pranaygp)! - Improve `workflow inspect` for long runs and scoped lookups: add `--all` for complete steps, events, and sleeps listings, print reusable cursor hints, and keep pagination on the read path that issued the cursor. Support event ID and short workflow-name lookups, require `--runId` for individual steps, events, and streams, fix `st` to mean streams, include run IDs in stream hints, and exit non-zero when a stream read fails. - Updated dependencies [[`854242e`](854242e), [`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`b69fabe`](b69fabe), [`77ccb69`](77ccb69), [`c57820d`](c57820d), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/core@5.2.1 - @workflow/errors@5.0.4 - @workflow/world@5.0.4 - @workflow/world-vercel@5.2.1 - @workflow/world-local@5.1.1 - @workflow/utils@5.0.2 - @workflow/builders@5.0.4 - @workflow/web@5.0.4 ## @workflow/core@5.2.1 ### Patch Changes - [#4669](#4669) [`854242e`](854242e) Thanks [@http-samc](https://github.com/http-samc)! - Compress workflow arguments on the libuv threadpool when starting a run, so large inputs no longer block the caller's event loop during zstd compression. - [#4670](#4670) [`73a6ca3`](73a6ca3) Thanks [@http-samc](https://github.com/http-samc)! - Add a `compressArguments` option to `start()`; pass `false` to store the run's arguments uncompressed when the inputs are known to be incompressible. - [#4835](#4835) [`fcb554b`](fcb554b) Thanks [@pranaygp](https://github.com/pranaygp)! - Record the queue namespace on a run's `executionContext.queueNamespace` in `start()`, so restart recovery re-enqueues the run to its own namespace. - Updated dependencies [[`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`b69fabe`](b69fabe), [`77ccb69`](77ccb69), [`c57820d`](c57820d), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/errors@5.0.4 - @workflow/world@5.0.4 - @workflow/world-vercel@5.2.1 - @workflow/world-local@5.1.1 - @workflow/utils@5.0.2 ## @workflow/errors@5.0.4 ### Patch Changes - [#4733](#4733) [`e6bd692`](e6bd692) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Add `InBandSupersededError` and optional `inBand`/`expectedSeqInBand` event-create params for the in-band writer fence. `@workflow/world-vercel` forwards them, does not retry a fenced write in-process, and records the fence on its event spans. The runtime does not use them yet. - Updated dependencies [[`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/world@5.0.4 - @workflow/utils@5.0.2 ## @workflow/nest@5.1.2 ### Patch Changes - Updated dependencies [[`fcb554b`](fcb554b)]: - @workflow/utils@5.0.2 - @workflow/builders@5.0.4 ## @workflow/next@5.0.4 ### Patch Changes - Updated dependencies [[`854242e`](854242e), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b)]: - @workflow/core@5.2.1 - @workflow/builders@5.0.4 ## @workflow/nitro@5.0.4 ### Patch Changes - Updated dependencies [[`854242e`](854242e), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b)]: - @workflow/core@5.2.1 - @workflow/builders@5.0.4 - @workflow/web@5.0.4 - @workflow/rollup@5.0.4 - @workflow/vite@5.0.4 ## @workflow/nuxt@5.0.4 ### Patch Changes - Updated dependencies []: - @workflow/nitro@5.0.4 ## @workflow/rollup@5.0.4 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.4 ## @workflow/sveltekit@5.0.4 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.4 - @workflow/rollup@5.0.4 - @workflow/vite@5.0.4 ## @workflow/utils@5.0.2 ### Patch Changes - [#4835](#4835) [`fcb554b`](fcb554b) Thanks [@pranaygp](https://github.com/pranaygp)! - Add optional `Queue.registerRoute()` and `Queue.registerHandler()`, implemented by `@workflow/world-local` and `@workflow/world-postgres`, so a queue consumer other than the Workflow SDK receives its own prefix's messages on its own route or in process. `reenqueueActiveRuns()` re-enqueues a run to the queue namespace it recorded, and `@workflow/world-postgres` keeps a job's queue prefix. ## @workflow/vite@5.0.4 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.4 ## @workflow/vitest@5.0.4 ### Patch Changes - Updated dependencies [[`854242e`](854242e), [`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`b69fabe`](b69fabe), [`77ccb69`](77ccb69), [`c57820d`](c57820d), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/core@5.2.1 - @workflow/world@5.0.4 - @workflow/world-local@5.1.1 - @workflow/builders@5.0.4 - @workflow/rollup@5.0.4 ## @workflow/web@5.0.4 ### Patch Changes - Updated dependencies [[`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`b69fabe`](b69fabe), [`77ccb69`](77ccb69), [`c57820d`](c57820d), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7)]: - @workflow/world-local@5.1.1 ## @workflow/web-shared@5.0.4 ### Patch Changes - Updated dependencies [[`854242e`](854242e), [`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/core@5.2.1 - @workflow/world@5.0.4 - @workflow/utils@5.0.2 ## workflow@5.2.1 ### Patch Changes - [#4733](#4733) [`e6bd692`](e6bd692) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Add `InBandSupersededError` and optional `inBand`/`expectedSeqInBand` event-create params for the in-band writer fence. `@workflow/world-vercel` forwards them, does not retry a fenced write in-process, and records the fence on its event spans. The runtime does not use them yet. - Updated dependencies [[`1342a96`](1342a96), [`854242e`](854242e), [`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b)]: - @workflow/cli@5.0.4 - @workflow/core@5.2.1 - @workflow/errors@5.0.4 - @workflow/utils@5.0.2 - @workflow/next@5.0.4 - @workflow/nitro@5.0.4 - @workflow/typescript-plugin@5.0.0 - @workflow/nest@5.1.2 - @workflow/astro@5.0.4 - @workflow/rollup@5.0.4 - @workflow/sveltekit@5.0.4 - @workflow/nuxt@5.0.4 ## @workflow/world@5.0.4 ### Patch Changes - [#4733](#4733) [`e6bd692`](e6bd692) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Add `InBandSupersededError` and optional `inBand`/`expectedSeqInBand` event-create params for the in-band writer fence. `@workflow/world-vercel` forwards them, does not retry a fenced write in-process, and records the fence on its event spans. The runtime does not use them yet. - [#4835](#4835) [`fcb554b`](fcb554b) Thanks [@pranaygp](https://github.com/pranaygp)! - Add optional `Queue.registerRoute()` and `Queue.registerHandler()`, implemented by `@workflow/world-local` and `@workflow/world-postgres`, so a queue consumer other than the Workflow SDK receives its own prefix's messages on its own route or in process. `reenqueueActiveRuns()` re-enqueues a run to the queue namespace it recorded, and `@workflow/world-postgres` keeps a job's queue prefix. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Implement the optional in-band writer fence and list snapshot in `@workflow/world-local` and `@workflow/world-postgres`, declared with the new `inBandFence` capability. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Report `deliveryCount` and `createdAt` in queue handler meta on `@workflow/world-local` and `@workflow/world-postgres`. - [#4833](#4833) [`3e93033`](3e93033) Thanks [@pranaygp](https://github.com/pranaygp)! - `@workflow/world-vercel` declares the `inBandFence` capability (its backend refuses a stale in-band write on every run with an event sequencer, and answers an in-band write on an older run with a 400), and passes on the in-band counts its backend reports: `snapshot` on `events.list` and `allocated` on a single `events.create`. The `inBandFence` contract lets a World list no snapshot when its refusals carry both counters, and `@workflow/world-testing` takes the counters from a refusal for such a World. ## @workflow/world-local@5.1.1 ### Patch Changes - [#4835](#4835) [`fcb554b`](fcb554b) Thanks [@pranaygp](https://github.com/pranaygp)! - Add optional `Queue.registerRoute()` and `Queue.registerHandler()`, implemented by `@workflow/world-local` and `@workflow/world-postgres`, so a queue consumer other than the Workflow SDK receives its own prefix's messages on its own route or in process. `reenqueueActiveRuns()` re-enqueues a run to the queue namespace it recorded, and `@workflow/world-postgres` keeps a job's queue prefix. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Implement the optional in-band writer fence and list snapshot in `@workflow/world-local` and `@workflow/world-postgres`, declared with the new `inBandFence` capability. - [#4838](#4838) [`b69fabe`](b69fabe) Thanks [@fazxes](https://github.com/fazxes)! - Fix `initDataDir()` crashing on Bun for a new data directory: Bun's `opendir()` reports a missing directory on the first read, not when it is opened. - [#4834](#4834) [`77ccb69`](77ccb69) Thanks [@pranaygp](https://github.com/pranaygp)! - Fix in-band event count conflicts when multiple processes write events, and fix count not surviving world restarts - [#4754](#4754) [`c57820d`](c57820d) Thanks [@pranaygp](https://github.com/pranaygp)! - Register a run's stream names under a lock so `streams.list` no longer drops a stream when several of the run's streams first write at once. A stream written again after `clear()` is listed again. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Report a message's cumulative `deliveryCount` across `{ timeoutSeconds }` redeliveries on `@workflow/world-postgres`. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Report `deliveryCount` and `createdAt` in queue handler meta on `@workflow/world-local` and `@workflow/world-postgres`. - Updated dependencies [[`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/errors@5.0.4 - @workflow/world@5.0.4 - @workflow/utils@5.0.2 ## @workflow/world-postgres@5.0.4 ### Patch Changes - [#4817](#4817) [`026741b`](026741b) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Fix a process crash when the database closes an idle connection - [#4835](#4835) [`fcb554b`](fcb554b) Thanks [@pranaygp](https://github.com/pranaygp)! - Add optional `Queue.registerRoute()` and `Queue.registerHandler()`, implemented by `@workflow/world-local` and `@workflow/world-postgres`, so a queue consumer other than the Workflow SDK receives its own prefix's messages on its own route or in process. `reenqueueActiveRuns()` re-enqueues a run to the queue namespace it recorded, and `@workflow/world-postgres` keeps a job's queue prefix. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Implement the optional in-band writer fence and list snapshot in `@workflow/world-local` and `@workflow/world-postgres`, declared with the new `inBandFence` capability. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Report a message's cumulative `deliveryCount` across `{ timeoutSeconds }` redeliveries on `@workflow/world-postgres`. - [#4756](#4756) [`e631e78`](e631e78) Thanks [@pranaygp](https://github.com/pranaygp)! - Honor `pagination.sortOrder` in `runs.list` and `steps.list`, which always returned newest first. - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Report `deliveryCount` and `createdAt` in queue handler meta on `@workflow/world-local` and `@workflow/world-postgres`. - Updated dependencies [[`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`b69fabe`](b69fabe), [`77ccb69`](77ccb69), [`c57820d`](c57820d), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/errors@5.0.4 - @workflow/world@5.0.4 - @workflow/world-local@5.1.1 - @workflow/utils@5.0.2 ## @workflow/world-testing@5.0.4 ### Patch Changes - [#4821](#4821) [`ebaa5b7`](ebaa5b7) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Implement the optional in-band writer fence and list snapshot in `@workflow/world-local` and `@workflow/world-postgres`, declared with the new `inBandFence` capability. - [#4833](#4833) [`3e93033`](3e93033) Thanks [@pranaygp](https://github.com/pranaygp)! - `@workflow/world-vercel` declares the `inBandFence` capability (its backend refuses a stale in-band write on every run with an event sequencer, and answers an in-band write on an older run with a 400), and passes on the in-band counts its backend reports: `snapshot` on `events.list` and `allocated` on a single `events.create`. The `inBandFence` contract lets a World list no snapshot when its refusals carry both counters, and `@workflow/world-testing` takes the counters from a refusal for such a World. - Updated dependencies [[`1342a96`](1342a96), [`854242e`](854242e), [`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`73a6ca3`](73a6ca3), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/cli@5.0.4 - @workflow/core@5.2.1 - @workflow/world@5.0.4 - workflow@5.2.1 ## @workflow/world-vercel@5.2.1 ### Patch Changes - [#4733](#4733) [`e6bd692`](e6bd692) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Add `InBandSupersededError` and optional `inBand`/`expectedSeqInBand` event-create params for the in-band writer fence. `@workflow/world-vercel` forwards them, does not retry a fenced write in-process, and records the fence on its event spans. The runtime does not use them yet. - [#4833](#4833) [`3e93033`](3e93033) Thanks [@pranaygp](https://github.com/pranaygp)! - `@workflow/world-vercel` declares the `inBandFence` capability (its backend refuses a stale in-band write on every run with an event sequencer, and answers an in-band write on an older run with a 400), and passes on the in-band counts its backend reports: `snapshot` on `events.list` and `allocated` on a single `events.create`. The `inBandFence` contract lets a World list no snapshot when its refusals carry both counters, and `@workflow/world-testing` takes the counters from a refusal for such a World. - Updated dependencies [[`e6bd692`](e6bd692), [`fcb554b`](fcb554b), [`ebaa5b7`](ebaa5b7), [`ebaa5b7`](ebaa5b7), [`3e93033`](3e93033)]: - @workflow/errors@5.0.4 - @workflow/world@5.0.4 - @workflow/utils@5.0.2 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
VaguelySerious
added a commit
that referenced
this pull request
Oct 13, 2026
Main now carries the World-side extractions from this branch (#4817, #4818, #4821, #4833) plus #4835, #4669/#4670, #4828 and #4838. Main's versions are the base; this branch keeps only its spec-9 additions: - World fence code is main's (#4821 review fixes; world-local's on-disk count under a lock file; world-vercel's lenient snapshot parsing from #4833). This branch adds occurredAt-as-createdAt, the step bookkeeping conformance test, and a required inBandFence (spec 9 requires a list snapshot; earlier runs may report counters on a refusal). - Per-run orchestrator serialization stays on top of #4821's queue code and #4835's routing, for the SDK's own prefix only (world-postgres gates on it, world-local keys its gate by prefix). - world-postgres keeps attempt per Graphile job (core's 48-delivery cap) with deliveryCount cumulative; tests use main's TestPool. - Main's "a same-run wake can release an inline step" queue test is replaced by a runtime test: an outside resume aborting a running inline step is committed by the blocked delivery through its log poll. - Resilient-dispatch leftovers in the QuickJS entrypoint are removed; stepInput carries the input for the step handler. - Main's loopback stream tests skip while WORKFLOW_SERVER_URL_OVERRIDE is set, like the branch's other loopback tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 of 3 tasks
Merged
2 of 3 tasks
pranaygp
added a commit
that referenced
this pull request
Oct 15, 2026
) ### Summary `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. ### Why - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. ### What changed - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. ### Compatibility and rollout - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. ### How did you test your changes? New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. ### Docs Preview | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | ### Open questions for reviewers 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. ### Related PRs and merge order - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. ### PR Checklist - Required to merge - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
2 of 3 tasks
github-actions Bot
added a commit
that referenced
this pull request
Oct 15, 2026
) `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(AI) Extracted from #4661 to shrink it: the World side of the in-band writer fence for world-local and world-postgres (and world-sim), plus additive queue handler meta.
Inert on main. The fence only acts on a create that sends
inBand: true, which the runtime does not do yet (same as #4733 for world-vercel). Every type and capability added here is optional.@workflow/world: optionalWorldCapabilities.inBandFence,EventListResponsewith optionalsnapshot: { seq, seqInBand }(now the return type ofevents.list, a superset of the old one),EventLogSnapshot,IN_BAND_SEQ_AT_RUN_CREATION = 1, optionalEventResult.allocated, optionaldeliveryCount/createdAton queue handler meta. A shared conformance suite insrc/test-support/in-band-fence-conformance.ts. No spec version change;step_startedresults unchanged.listreturns the snapshot for slot-numbered runs; queue meta reportsdeliveryCountandcreatedAt.0026_add_seq_in_band(seq_in_band integer default 1). A fenced create locks the run'sworkflow_event_slotsrow (FOR UPDATE) and checks the count before any entity update, in one transaction with the insert; the insert's CTE advances the count. This closes the race where two writers holding the same count both pass a pre-check and the loser's entity-row update sticks: the loser now waits on the lock, sees the moved count, and gets 412 without touching any row (tested). Out-of-band writes never take that lock. Queue jobs carrycreatedAtacross reschedules, anddeliveryCountkeeps counting across{ timeoutSeconds }reschedules (sent asx-vqs-message-delivery-count).attempt, which core counts against its 48-delivery cap, still restarts with each replacement job, so a step retried in place more than 47 times on one message still completes on world-postgres (verified: 55 failures withmaxRetries: 100completes at attempt 56).createTestSuite()checks the snapshot and a stale in-band write end to end, skipped for a World that does not declareinBandFence.Fence scope. As on world-vercel's model:
run_createdis exempt and counts as the first in-band position, a refusal writes and allocates nothing, out-of-band writes move onlyseq, an in-band write without a count is a 400. The fence applies to any run a caller sendsinBandfor; the snapshot is returned only for slot-numbered runs (world-postgres: runs with aworkflow_event_slotsrow), since older runs have no position counter. No spec gating is added here; the caller decides when to fence.Left out of this PR: one orchestrator delivery per run (#4661's
orchestratorRunIdOf+ world-local run gate + world-postgres Graphile named queue). On main it would revert #4219 and break shipped behavior. Main's runtime relies on a second flow delivery of a run while the first is blocked in an inline step: the delayed wait continuation for a racingsleep(), and the workflow-sideAbortController.abort()that interrupts a running inline step through the abort stream. Verified on world-local (nextjs-turbopack prod build, run started from a route in the app) with a workflow that awaits a quick step and then races a step againstsleep():sleep('1s')wait_completedat +1s, sleep winswait_completedafterstep_completedat +10s, step winssleep('2s')+abort()A step that only ends when aborted would hold the run's gate indefinitely. The existing e2e suite does not catch this: the test runner process enqueues each run's first delivery through its own world-local queue, so the app process's per-process gate never sees it, and the e2e race/abort workflows race inside that first delivery. #4661 can serialize because there the blocked orchestrator reads the log itself. It stays in #4661.
Tested locally: unit tests for world (236), world-local (646), world-postgres incl. Docker (343, incl. the 17 fence tests), world-sim (79), world-testing (19, both fence tests run against world-local), core (3074). Typecheck and biome clean.
🤖 Generated with Claude Code