Repository navigation
[world-postgres] Don't move an ended run back to running on a late run_started - #5136
Conversation
…n_started run_started's UPDATE was unguarded, so a run that ended between the validation read and the UPDATE was set back to running, leaving a row with a completion and a live status that runs.get then fails to parse. The UPDATE now excludes terminal runs and the event is refused with RunExpiredError, matching the pre-validation answer. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
🦋 Changeset detectedLatest commit: fe4a659 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 121252ms → this run 138616ms (Δ +17364ms, +14%) ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3982 | 0 | 878 | 4860 |
| ✅ 💻 Local Development | 4669 | 0 | 551 | 5220 |
| ✅ 📦 Local Production | 4669 | 0 | 551 | 5220 |
| ✅ 🐘 Local Postgres | 4669 | 0 | 551 | 5220 |
| ✅ 🪟 Windows | 348 | 0 | 12 | 360 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 87 | 155 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ✅ vercel-http-transport | 897 | 0 | 183 | 1080 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 607 | 0 | 113 | 720 |
| Total | 19936 | 0 | 2926 | 22862 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 145 | 0 | 35 |
| ✅ astro-quickjs | 145 | 0 | 35 |
| ✅ example-node | 145 | 0 | 35 |
| ✅ example-quickjs | 145 | 0 | 35 |
| ✅ express-node | 145 | 0 | 35 |
| ✅ express-quickjs | 145 | 0 | 35 |
| ✅ fastify-node | 145 | 0 | 35 |
| ✅ fastify-quickjs | 145 | 0 | 35 |
| ✅ hono-node | 145 | 0 | 35 |
| ✅ hono-quickjs | 145 | 0 | 35 |
| ✅ nest-node | 145 | 0 | 35 |
| ✅ nest-quickjs | 145 | 0 | 35 |
| ✅ nextjs-turbopack-node | 172 | 0 | 8 |
| ✅ nextjs-turbopack-quickjs | 172 | 0 | 8 |
| ✅ nextjs-webpack-node | 172 | 0 | 8 |
| ✅ nextjs-webpack-quickjs | 172 | 0 | 8 |
| ✅ nitro-node | 145 | 0 | 35 |
| ✅ nitro-quickjs | 145 | 0 | 35 |
| ✅ nuxt-node | 145 | 0 | 35 |
| ✅ nuxt-quickjs | 145 | 0 | 35 |
| ✅ python-node | 66 | 0 | 114 |
| ✅ sveltekit-node | 164 | 0 | 16 |
| ✅ sveltekit-quickjs | 164 | 0 | 16 |
| ✅ tanstack-start-node | 145 | 0 | 35 |
| ✅ tanstack-start-quickjs | 145 | 0 | 35 |
| ✅ vite-node | 145 | 0 | 35 |
| ✅ vite-quickjs | 145 | 0 | 35 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 174 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 174 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 87 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 145 | 0 | 35 |
| ✅ express | 145 | 0 | 35 |
| ✅ hono | 145 | 0 | 35 |
| ✅ nextjs-turbopack | 172 | 0 | 8 |
| ✅ nitro | 145 | 0 | 35 |
| ✅ vite | 145 | 0 | 35 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 145 | 0 | 35 |
| ✅ express | 145 | 0 | 35 |
| ✅ nextjs-turbopack | 172 | 0 | 8 |
| ✅ vite | 145 | 0 | 35 |
Sim WorldSimulated world deterministic testing for races. Traces 🟢 world-sim scenario book — 0 fail of 44 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
…rked on the row The test polled for any ungranted lock and committed when its deadline passed even without one, and then the validation read refuses the run_started on its own. It now waits for this run_started's UPDATE to be waiting on a lock before ending the run. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…n_started (#5136) A `run_started` that arrives just as the run ends used to set the run back to running. The row was left with its output and completion time still set, and reading the run then failed. Now the late `run_started` leaves the ended run alone and is refused, the same answer it would have gotten had the run already ended when it was checked. - The UPDATE that marks the run running now skips runs in a terminal status. If it matches nothing because the run has ended, the event fails with `RunExpiredError` and no `run_started` is written. This showed up as `ZodError` failures from `runs.get` in the world-postgres e2e lanes, in the `start({ hook })` "start or resume" test: https://github.com/vercel/workflow/actions/runs/40935700151 Added a test that reproduces the race every time. A second connection ends the run and holds its row lock while `run_started` waits on that lock. Against a local Postgres 17 the test fails without the fix (the run comes back `running`) and passes with it. The world-postgres storage and spec tests pass locally. --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
|
Backport PR opened against |
…ing on a late run_started (#5138) Automated backport of #5136 to `stable` ([backport job run](https://github.com/vercel/workflow/actions/runs/40941356751)). Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to stable, this PR will be updated. # Releases ## @workflow/world-postgres@4.4.0 ### Minor Changes - [#4759](#4759) [`3ff458e`](3ff458e) Thanks [@pranaygp](https://github.com/pranaygp)! - Add `jobLockStaleSeconds` (`WORKFLOW_POSTGRES_JOB_LOCK_STALE_SECONDS`, default `0`, off). When set, a running delivery renews its Graphile job lock, and a job whose process died is redelivered by another process once its lock has gone unrenewed for that many seconds, instead of after Graphile Worker's fixed 4 hours or the next restart. `30` is a good value when more than one process shares the database. It's off by default because of rolling upgrades: processes on an earlier version don't renew their locks, so turn it on once every process that shares the database runs this version. ### Patch Changes - [#5136](#5136) [`5c9b65d`](5c9b65d) Thanks [@alangenfeld](https://github.com/alangenfeld)! - A `run_started` that races the run ending no longer moves the run back to `running`; it's refused with `RunExpiredError`. - Updated dependencies [[`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/world-local@4.4.5 ## @workflow/astro@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 - @workflow/rollup@4.0.31 - @workflow/vite@4.0.31 ## @workflow/builders@4.1.21 ### Patch Changes - [#5063](#5063) [`e4dbc29`](e4dbc29) Thanks [@joelfickson](https://github.com/joelfickson)! - Export `workflowSerdeComputedPropertyPattern` alongside the other workflow pattern regexes. - Updated dependencies []: - @workflow/core@4.8.16 ## @workflow/cli@4.3.20 ### Patch Changes - [#5182](#5182) [`f3d6142`](f3d6142) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Respect `--project` and `--team` over the inferred `.vercel` link, and prefer the nearest `.vercel/project.json` in monorepos - Updated dependencies [[`e4dbc29`](e4dbc29), [`dbfd1ba`](dbfd1ba), [`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/builders@4.1.21 - @workflow/web@4.1.32 - @workflow/world-local@4.4.5 - @workflow/core@4.8.16 ## @workflow/core@4.8.16 ### Patch Changes - Updated dependencies [[`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/world-local@4.4.5 ## @workflow/nest@4.0.32 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 ## @workflow/next@4.1.20 ### Patch Changes - [#5063](#5063) [`e4dbc29`](e4dbc29) Thanks [@joelfickson](https://github.com/joelfickson)! - Run the Turbopack workflow loader on files that use `static [WORKFLOW_SERIALIZE]` computed keys (such as `@ai-sdk/openai` models), fixing "Class … not found" errors in steps. - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 - @workflow/core@4.8.16 ## @workflow/nitro@4.1.22 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29), [`dbfd1ba`](dbfd1ba)]: - @workflow/builders@4.1.21 - @workflow/web@4.1.32 - @workflow/rollup@4.0.31 - @workflow/vite@4.0.31 - @workflow/core@4.8.16 ## @workflow/nuxt@4.0.32 ### Patch Changes - [#5211](#5211) [`d30f07e`](d30f07e) Thanks [@atinux](https://github.com/atinux)! - Upgrade `@nuxt/kit` to 4.6.3 to pick up the fix for CVE-2026-71314. - Updated dependencies []: - @workflow/nitro@4.1.22 ## @workflow/rollup@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 ## @workflow/sveltekit@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 - @workflow/rollup@4.0.31 - @workflow/vite@4.0.31 ## @workflow/vite@4.0.31 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29)]: - @workflow/builders@4.1.21 ## @workflow/vitest@4.0.32 ### Patch Changes - Updated dependencies [[`e4dbc29`](e4dbc29), [`ceaaa5c`](ceaaa5c), [`b065b0b`](b065b0b)]: - @workflow/builders@4.1.21 - @workflow/world-local@4.4.5 - @workflow/rollup@4.0.31 - @workflow/core@4.8.16 ## @workflow/web@4.1.32 ### Patch Changes - [#5079](#5079) [`dbfd1ba`](dbfd1ba) Thanks [@karthikscale3](https://github.com/karthikscale3)! - Recognize unframed serialized abort packets in the Streams viewer so encrypted packets can be decrypted. ## @workflow/web-shared@4.1.32 ### Patch Changes - Updated dependencies []: - @workflow/core@4.8.16 ## workflow@4.8.16 ### Patch Changes - Updated dependencies [[`f3d6142`](f3d6142), [`d30f07e`](d30f07e), [`e4dbc29`](e4dbc29)]: - @workflow/cli@4.3.20 - @workflow/nuxt@4.0.32 - @workflow/next@4.1.20 - @workflow/astro@4.0.31 - @workflow/nest@4.0.32 - @workflow/nitro@4.1.22 - @workflow/rollup@4.0.31 - @workflow/sveltekit@4.0.31 - @workflow/core@4.8.16 - @workflow/typescript-plugin@4.0.3 ## @workflow/world-local@4.4.5 ### Patch Changes - [#5069](#5069) [`ceaaa5c`](ceaaa5c) Thanks [@AndrewBarba](https://github.com/AndrewBarba)! - Deliver stream chunks in the order they were written. A reader that opened, or polled, while chunks were being written could get a later chunk before an earlier one, and could get a chunk twice. Writes to one stream now land in order, including a `writeToStreamMulti` group and concurrent writes, and the reader holds live chunks while it reads from disk. - [#4909](#4909) [`b065b0b`](b065b0b) Thanks [@fazxes](https://github.com/fazxes)! - Name each stream chunk as it lands, after every chunk already in the stream, so a write that stalled before landing no longer lands ahead of chunks a reader already read. ## @workflow/world-testing@4.1.31 ### Patch Changes - Updated dependencies [[`f3d6142`](f3d6142)]: - @workflow/cli@4.3.20 - workflow@4.8.16 - @workflow/core@4.8.16 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary & Motivation
A
run_startedthat arrives just as the run ends used to set the run back to running. The row was left with its output and completion time still set, and reading the run then failed. Now the laterun_startedleaves the ended run alone and is refused, the same answer it would have gotten had the run already ended when it was checked.RunExpiredErrorand norun_startedis written.This showed up as
ZodErrorfailures fromruns.getin the world-postgres e2e lanes, in thestart({ hook })"start or resume" test: https://github.com/vercel/workflow/actions/runs/40935700151Test Plan
Added a test that reproduces the race every time. A second connection ends the run and holds its row lock while
run_startedwaits on that lock. Against a local Postgres 17 the test fails without the fix (the run comes backrunning) and passes with it. The world-postgres storage and spec tests pass locally.