[world-testing] Wait for run_completed before comparing a run's event reads by alangenfeld · Pull Request #5073 · vercel/workflow

alangenfeld · 2026-10-26T18:29:29Z

Summary & Motivation

The resolveData check used to start comparing a run's event reads as soon as the run reported completed. It now also waits until the run's completion event appears in the log. Without that wait, a World could report the run completed a moment before listing run_completed, so the check's two event reads could see different logs and flake. This happened on #4996's CI.

Test Plan

This changes test code only, and CI runs it.

… reads A World may report the run completed a moment before its run_completed is listed, so the resolveData check could compare two reads of different logs. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>

vercel · 2026-10-26T18:29:32Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
example-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-astro-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-express-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-fastify-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-hono-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-nestjs-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-nitro-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-nuxt-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-python-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workbench-vite-workflow	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workflow-docs	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workflow-swc-playground	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workflow-tarballs	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC
workflow-web	Ready	Preview, v0	Oct 26, 2026 6:32pm UTC

changeset-bot · 2026-10-26T18:29:36Z

🦋 Changeset detected

Latest commit: 3826581

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-testing	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-10-26T18:31:14Z

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 44 total

fence=per-spec

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	6	0ms	ok
✅ `hook-at-step-started`	completed	11	0ms	ok
✅ `hook-at-step-completed`	completed	11	0ms	ok
✅ `hook-at-hook-created`	completed	11	0ms	ok
✅ `claim-hook-at-registration`	completed	11	0ms	ok
✅ `claim-hook-at-step-started`	completed	11	0ms	ok
✅ `deadline-hook-wins`	completed	6	1.0h	ok
✅ `deadline-expires`	completed	6	1.0h	ok
✅ `step-vs-timer-early-settlement`	completed	8	1.0h	ok
✅ `long-sleep`	completed	11	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	10	2.0s	ok
✅ `parallel-steps`	completed	9	0ms	ok
✅ `hook-on-execution-state`	completed	11	0ms	ok
✅ `peek-hook-before-branch`	completed	11	0ms	ok
✅ `peek-hook-after-branch`	completed	11	0ms	ok
✅ `peek-hook-at-registration`	completed	11	0ms	ok
✅ `race-hook-before-probe`	completed	11	0ms	ok
✅ `race-hook-after-probe`	completed	11	0ms	ok
✅ `race-duplicate-delivery`	completed	12	0ms	ok
✅ `attr-hook-before-step`	completed	10	0ms	ok
✅ `attr-hook-after-step`	completed	10	0ms	ok
✅ `attr-from-step-body`	completed	12	0ms	ok
✅ `fork-hook-after-timeout`	completed	13	1.0m	ok
✅ `fork-hook-before-timeout`	completed	13	1.0m	ok
✅ `count-hook-after-timeout`	completed	16	1.0m	ok
✅ `count-hook-before-timeout`	completed	19	1.0m	ok
✅ `stale-read-step-count-fork`	completed	19	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	13	1.0m	ok
✅ `step-vs-step-fork`	completed	12	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	12	0ms	ok
✅ `fence-catches-benign-direction`	completed	12	5ms	ok
✅ `in-flight-before-decision`	completed	16	1.0m	ok
✅ `in-flight-before-decision-counted`	completed	16	1.0m	ok
✅ `in-flight-after-decision`	completed	18	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	19	1.0m	ok
✅ `fork-hook-wins`	completed	12	1.0m	ok
✅ `fork-timeout-wins`	completed	12	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	16	1.0m	ok
✅ `claimed-payload-under-fork`	completed	16	1.0m	ok
✅ `writers-independent-step-bodies`	completed	12	0ms	ok
✅ `writers-scripted-tempo`	completed	11	0ms	ok
✅ `cancel-mid-step`	cancelled	7	0ms	skipped

Full trace: world-sim.txt

github-actions · 2026-10-26T18:31:19Z

📊 Workflow Benchmarks

commit 3826581 · Mon, 26 Oct 2026 18:51:58 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1970 (+379%) 🔻	2089 🔴 (+31%) 🔻	2146 🔴 (+33%) 🔻	2462 🔴 (+48%) 🔻	30
TTFS	stream	1913 (+40%) 🔻	2020 🔴 (+28%) 🔻	2067 🔴 (+26%) 🔻	2141 🔴 (+17%) 🔻	30
TTFS	hook + stream	2178 (+32%) 🔻	2424 🔴 (+39%) 🔻	2466 🔴 (+39%) 🔻	2676 🔴 (+31%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	606 (+12%)	1017 (+21%) 🔻	1035 (+12%)	2439 (+45%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2575 (-27%) 💚	4470 (+3.4%)	4581 (+4.2%)	10958 (+76%) 🔻	10
STSO	1020 steps (inline)	86 (+37%) 🔻	113 (+14%)	131 (+8.3%)	207 (-40%) 💚	1019
WO	1020 steps	112164 (+8.7%)	112164 (+8.7%)	112164 (+8.7%)	112164 (+8.7%)	1

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 102649ms → this run 111824ms (Δ +9175ms, +9%)

 50-100 ms  ██████████┃█████████████  main 765  this 356  -409
100-150 ms  ███████░░░░░░░░░░░░┃      main 210  this 623  +413
150-200 ms  ┃                         main  19  this  28    +9
200-250 ms  ┃                         main   6  this   6    +0
250-300 ms  ┃                         main   6  this   3    -3
300-350 ms  ┃                         main   4  this   2    -2
350-400 ms  ┃                         main   5  this   0    -5
400-450 ms  ┃                         main   1  this   0    -1
450-500 ms  ┃                         main   1  this   0    -1
550-600 ms  ┃                         main   0  this   1    +1
600-650 ms  ┃                         main   1  this   0    -1
700-750 ms  ┃                         main   1  this   0    -1

ℹ️ Metric definitions & methodology

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies. Streaming delivery performance is measured in durabench.}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

github-actions · 2026-10-26T18:33:48Z

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

a run can take over its own earlier hook (example · vercel-ws-transport / vercel / node / ws)
hookCleanupTestWorkflow - hook token reuse after workflow completion (nuxt · local-dev / local / node / stable)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 18:45:22Z · abandoned wrun_01M5WRFNNFVJ8P1DNDRK9GSHEP
cold-start-warmup · suite warmup (tanstack-start) · at 18:45:24Z · abandoned wrun_01M5WRFNNMAN2FVA6CZ88XGRJV
run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 18:52:04Z · abandoned wrun_01M5WRVY9JVTD4AZPYNPC7G0PZ

E2E Test Summary

Summary

	Passed	Skipped	Total
✅ ▲ Vercel Production	3982	878	4860
✅ 💻 Local Development	4669	551	5220
✅ 📦 Local Production	4669	551	5220
✅ 🐘 Local Postgres	4669	551	5220
✅ 🪟 Windows	348	12	360
✅ 🌐 Cross-language Conformance	68	87	155
✅ dynamic-runs	0	0	0
✅ vercel-http-transport	897	183	1080
✅ vercel-multi-region	27	0	27
✅ vercel-ws-transport	607	113	720
Total	19936	2926	22862

Details by Category

✅ ▲ Vercel Production

App	Passed	Skipped
✅ astro-node	145	35
✅ astro-quickjs	145	35
✅ example-node	145	35
✅ example-quickjs	145	35
✅ express-node	145	35
✅ express-quickjs	145	35
✅ fastify-node	145	35
✅ fastify-quickjs	145	35
✅ hono-node	145	35
✅ hono-quickjs	145	35
✅ nest-node	145	35
✅ nest-quickjs	145	35
✅ nextjs-turbopack-node	172	8
✅ nextjs-turbopack-quickjs	172	8
✅ nextjs-webpack-node	172	8
✅ nextjs-webpack-quickjs	172	8
✅ nitro-node	145	35
✅ nitro-quickjs	145	35
✅ nuxt-node	145	35
✅ nuxt-quickjs	145	35
✅ python-node	66	114
✅ sveltekit-node	164	16
✅ sveltekit-quickjs	164	16
✅ tanstack-start-node	145	35
✅ tanstack-start-quickjs	145	35
✅ vite-node	145	35
✅ vite-quickjs	145	35

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	151	29
✅ astro-stable-quickjs	151	29
✅ express-stable-node	151	29
✅ express-stable-quickjs	151	29
✅ fastify-stable-node	151	29
✅ fastify-stable-quickjs	151	29
✅ hono-stable-node	151	29
✅ hono-stable-quickjs	151	29
✅ nest-stable-node	151	29
✅ nest-stable-quickjs	151	29
✅ nextjs-turbopack-canary-node	179	1
✅ nextjs-turbopack-canary-quickjs	179	1
✅ nextjs-turbopack-quickjs-snapshot	179	1
✅ nextjs-turbopack-stable-node	179	1
✅ nextjs-turbopack-stable-quickjs	179	1
✅ nextjs-webpack-canary-node	179	1
✅ nextjs-webpack-canary-quickjs	179	1
✅ nextjs-webpack-stable-node	179	1
✅ nextjs-webpack-stable-quickjs	179	1
✅ nitro-stable-node	151	29
✅ nitro-stable-quickjs	151	29
✅ nuxt-stable-node	151	29
✅ nuxt-stable-quickjs	151	29
✅ sveltekit-stable-node	170	10
✅ sveltekit-stable-quickjs	170	10
✅ tanstack-start-node	151	29
✅ tanstack-start-quickjs	151	29
✅ vite-stable-node	151	29
✅ vite-stable-quickjs	151	29

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	151	29
✅ astro-stable-quickjs	151	29
✅ express-stable-node	151	29
✅ express-stable-quickjs	151	29
✅ fastify-stable-node	151	29
✅ fastify-stable-quickjs	151	29
✅ hono-stable-node	151	29
✅ hono-stable-quickjs	151	29
✅ nest-stable-node	151	29
✅ nest-stable-quickjs	151	29
✅ nextjs-turbopack-canary-node	179	1
✅ nextjs-turbopack-canary-quickjs	179	1
✅ nextjs-turbopack-quickjs-snapshot	179	1
✅ nextjs-turbopack-stable-node	179	1
✅ nextjs-turbopack-stable-quickjs	179	1
✅ nextjs-webpack-canary-node	179	1
✅ nextjs-webpack-canary-quickjs	179	1
✅ nextjs-webpack-stable-node	179	1
✅ nextjs-webpack-stable-quickjs	179	1
✅ nitro-stable-node	151	29
✅ nitro-stable-quickjs	151	29
✅ nuxt-stable-node	151	29
✅ nuxt-stable-quickjs	151	29
✅ sveltekit-stable-node	170	10
✅ sveltekit-stable-quickjs	170	10
✅ tanstack-start-node	151	29
✅ tanstack-start-quickjs	151	29
✅ vite-stable-node	151	29
✅ vite-stable-quickjs	151	29

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	151	29
✅ astro-stable-quickjs	151	29
✅ express-stable-node	151	29
✅ express-stable-quickjs	151	29
✅ fastify-stable-node	151	29
✅ fastify-stable-quickjs	151	29
✅ hono-stable-node	151	29
✅ hono-stable-quickjs	151	29
✅ nest-stable-node	151	29
✅ nest-stable-quickjs	151	29
✅ nextjs-turbopack-canary-node	179	1
✅ nextjs-turbopack-canary-quickjs	179	1
✅ nextjs-turbopack-quickjs-snapshot	179	1
✅ nextjs-turbopack-stable-node	179	1
✅ nextjs-turbopack-stable-quickjs	179	1
✅ nextjs-webpack-canary-node	179	1
✅ nextjs-webpack-canary-quickjs	179	1
✅ nextjs-webpack-stable-node	179	1
✅ nextjs-webpack-stable-quickjs	179	1
✅ nitro-stable-node	151	29
✅ nitro-stable-quickjs	151	29
✅ nuxt-stable-node	151	29
✅ nuxt-stable-quickjs	151	29
✅ sveltekit-stable-node	170	10
✅ sveltekit-stable-quickjs	170	10
✅ tanstack-start-node	151	29
✅ tanstack-start-quickjs	151	29
✅ vite-stable-node	151	29
✅ vite-stable-quickjs	151	29

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	174	0	6
✅ nextjs-turbopack-quickjs	174	0	6

✅ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
✅ python	68	0	87

✅ dynamic-runs

App	Passed	Failed	Skipped
✅ astro-vercel	0	0	0
✅ example-vercel	0	0	0
✅ express-vercel	0	0	0
✅ fastify-vercel	0	0	0
✅ hono-vercel	0	0	0
✅ nest-vercel	0	0	0
✅ nextjs-turbopack-vercel	0	0	0
✅ nextjs-webpack-vercel	0	0	0
✅ nitro-vercel	0	0	0
✅ nuxt-vercel	0	0	0
✅ sveltekit-vercel	0	0	0
✅ tanstack-start-vercel	0	0	0
✅ vite-vercel	0	0	0

✅ vercel-http-transport

App	Passed	Skipped
✅ example	145	35
✅ express	145	35
✅ hono	145	35
✅ nextjs-turbopack	172	8
✅ nitro	145	35
✅ vite	145	35

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Skipped
✅ example	145	35
✅ express	145	35
✅ nextjs-turbopack	172	8
✅ vite	145	35

📋 View full workflow run

github-actions · 2026-10-26T18:45:43Z

Framework	Flow route	Step reg.	Framework output
hono	267.6 KiB (±0)	95.6 KiB (±0)	2.00 MiB (±0)
nextjs-turbopack	275.0 KiB (±0)	426 B (±0)	975.3 KiB (±0)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

3826581 · run

github-actions · 2026-10-26T19:51:19Z

No backport to stable for 5f5299b (AI decision).

This is a flake fix in a test-only file, packages/world-testing/src/resolve-data.mts, which I verified does not exist on stable (git show origin/stable:packages/world-testing/src/resolve-data.mts fails). The check was introduced on main alongside the resolveData: 'skip-step-inputs' feature (#4348), which is not on the maintenance line, so there is no behavior on stable for this fix to cover.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

5f5299b39974a18b1500d2c857c02783fb45f7f7

Brings in main up to #5069. runtime.ts and world-testing's resolve-data.mts keep this branch's side: #5070's runtime.ts change is ported to both orchestrators in the next commit, and #5073's resolve-data wait is already here in an equivalent form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel Bot deployed to Preview – workflow-docs October 26, 2026 18:31

View deployment

alangenfeld mentioned this pull request Oct 26, 2026

Admit start({ hook }) candidates on the single-orchestrator engine #5008

Closed

alangenfeld marked this pull request as ready for review October 26, 2026 18:49

alangenfeld requested a review from a team as a code owner October 26, 2026 18:49

VaguelySerious approved these changes Oct 26, 2026

View reviewed changes

alangenfeld merged commit 5f5299b into main Oct 26, 2026
328 of 331 checks passed

alangenfeld deleted the alangenfeld/resolve-data-terminal-wait branch October 26, 2026 19:50

github-actions Bot mentioned this pull request Oct 26, 2026

Version Packages #4961

Merged

This was referenced Oct 27, 2026

[docs] Catch up docs with recently shipped SDK changes #5094

Open

[docs] Document abort packet framing, World queue/stream contract, and resilient-start public key #5102

Open

Navigation Menu

Repository navigation

[world-testing] Wait for run_completed before comparing a run's event reads - #5073

[world-testing] Wait for run_completed before comparing a run's event reads#5073
alangenfeld merged 1 commit into
mainfrom
alangenfeld/resolve-data-terminal-wait

alangenfeld commented Oct 26, 2026

Uh oh!

vercel Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alangenfeld commented Oct 26, 2026

Summary & Motivation

Test Plan

Uh oh!

vercel Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Oct 26, 2026

🦋 Changeset detected

Uh oh!

github-actions Bot commented Oct 26, 2026

Sim World

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Oct 26, 2026 •

edited

Loading

github-actions Bot commented Oct 26, 2026 •

edited

Loading

github-actions Bot commented Oct 26, 2026 •

edited

Loading