[pranaygp]

Description

Fixes #3935.

When a step's arguments or a run's return value contain a writable forwarded from another run, the reviver called getForwardedWritableEncryptionKey() immediately and stored the promise it returned. The only consumer of that promise is the serialize transform, which awaits it on the first chunk written — so on a writable nobody writes to, nothing ever observes it. When the lookup failed first, Node saw an unhandled rejection and killed the process.

The lookup's slow path is the one that fails: for a descriptor carrying no deploymentId and no encryptionPublicKey it falls back to world.runs.get(ownerRunId), which is GET /v2/runs/:id?remoteRefBehavior=resolve — exactly the request in the report, timing out after ~72s. Only the Vercel World implements getEncryptionKeyForRun, so this is Vercel-only; local and Postgres return early and never make the request.

The cost lands on whatever else shared the invocation. In #3935 that was a GitHub webhook handler with no relationship to the run being resolved, and the symptom presented as an unexplained function failure rather than "reading run metadata was slow."

The fix: wrap the lookup in a memoized thunk. EncryptionKeyParam already accepts a resolver, so the lookup starts on the first write, runs at most once, and a failure rejects that stream — reaching the writer as the cause on writer.closed, cause chain intact:

SerializationError: Failed to serialize stream chunk
  └▶ WorkflowWorldError: GET /v2/runs/…?remoteRefBehavior=resolve timed out after 71779ms
       └▶ TimeoutError: The operation was aborted due to timeout

Both reviver boundaries are covered: getExternalRevivers (the client path, reached from await run.returnValue) and getStepRevivers (forwarded writables in step arguments). This is the same shape the readable side already uses (resolveKey in the ReadableStream reviver, #3871) and that Run#getWritable uses (resolveTarget) — the writable reviver was the one place still resolving eagerly.

Trade-off: forwarded writables without a public key now make their requests on the first write instead of overlapping them with execution, and writables that are never written to make none at all. That prefetch only ever helped descriptors minted by deployments predating public keys on the wire — the case that disappears as deployments roll forward — so paying it lazily is the right side of the trade.

Why this should land before #3678 (lifecycle hooks) — thanks @pranaygp for the pointer

#3678 makes this defect much more reachable, and its own design already argues for this fix.

dispatchRunFailedHooks hydrates the persisted error payload so handlers can inspect it, and it does so deliberately inertly:

getExternalRevivers(globalThis, ops, runId, encryptionKey, {
  lazyStreams: true,
  liveAbortSignals: false,
})

Those two options exist for precisely the reason this PR exists — a reporting handler may never consume what it is handed, so readables must not open I/O and abort signals must not take out live subscriptions. #3678 correctly covers ReadableStream, AbortController and AbortSignal. The WritableStream reviver is the one it leaves eager (it appears in that diff only as context), so a forwarded writable in a failed run's payload still fires runs.get during hydration.

That turns a conditional crash into a systematic one:

• it runs on every run_failed in any app that registers hooks, not only when user code happens to hydrate such a payload;
• a reporting handler writes to nothing by construction, so the promise is guaranteed to stay unobserved — the exact condition for the crash;
• safeWaitUntil around the dispatch does not help. It guards the dispatch promise; the orphaned key promise is created inside the reviver and is never part of that chain.

So an app that adopts registerLifecycleHooks to report failures to Sentry would gain a new way to lose the invocation while reporting a failure. Landing this first means #3678 needs no change: with the thunk in place, lazyStreams/liveAbortSignals and the writable path all agree that terminal reporting does no I/O it wasn't asked for.

Worth considering as a follow-up on #3678 (not required by this PR): an explicit test that hydrating a run_failed payload containing a forwarded writable makes no requests, so the invariant is pinned on that side too.

Relationship to #3955

@withkarann opened #3955 with the same diagnosis and the same thunk approach on 2026-09-03. It has gone stale against main — its diff context still has the two-parameter, non-exported getForwardedWritableEncryptionKey, whereas main now exports it with a third encryptionPublicKey parameter — and it had no companion for stable, where the exposure is larger (see below).

This PR is that fix rebased onto current main, plus regression tests for both reviver boundaries, the memoization guarantee, and the #3678 analysis above. Credit for the original diagnosis is theirs; happy to close whichever of the two is more convenient for the team.

Scope and history

• Introduced by fix(core): resolve forwarded stream keys across deployments #2191 (2026-06-01), first released in @workflow/core@5.0.0-beta.11, backported to the 4.8.x line as Backport #2191: fix forwarded stream encryption across deployments #2202. Not a recent regression.
• Still present on main (5.0.0-beta.52) and stable (4.8.8) before this change.
• stable is more exposed, which is why it gets its own PR rather than waiting on the backport bot: the 4.8.x line has no encryptionPublicKey fast path at all, so every cross-run forwarded writable takes the failing runs.get path. Companion PR: fix(core): start forwarded writable key lookup on first write (backport of #4175) #4176. The auto-backport for this commit can be closed as a duplicate.
• Not addressed here, and not the subject of the issue: the reporter saw 71779ms against a 60s WORKFLOW_REQUEST_TIMEOUT_MS default, even though the deadline is armed per-attempt with elapsed measured from the same per-attempt fetchStart. That ~12s overshoot looks like event-loop starvation delaying the timer inside a busy invocation (consistent with two runs timing out in one invocation), or a raised timeout in their project. Orthogonal to the crash — the reporter is pursuing the underlying slowness through support.

How did you test your changes?

Four regression tests in packages/core/src/serialization.test.ts, alongside the existing forwarded-writable tests:

cd packages/core && WORKFLOW_TARGET_WORLD=local pnpm vitest run src/serialization.test.ts

Test	Guards
client hydrates a return value it never writes to	no unhandledRejection, and no request at all
step hydrates arguments it never writes to	same, on the other reviver boundary
surfaces a failed lookup on the writer that needed the key	the failure is still delivered, as cause on writer.closed with code: 'TIMEOUT'
resolves the owner key at most once across many writes	the thunk memoizes; no per-write request storm

Each mocks runs.get to reject with the exact WorkflowWorldError { code: 'TIMEOUT' } shape world-vercel's makeRequest raises.

Verified the tests fail without the fix. Reverting only the two call sites (keeping the tests) fails both unhandled-rejection tests with AssertionError: expected [ Array(1) ] to deeply equal [], the array holding the timeout message. The other two pass either way by design — they pin behaviour the fix must preserve.

Also reproduced and fixed at the process level. A standalone script installs a fake World on globalThis[Symbol.for('@workflow/world//cache')] whose runs.get rejects like a timed-out read, hydrates a payload with a forwarded writable, and never writes to it — no stubbing inside @workflow/core, the crash comes from built dist:

	before	after
world.runs.get called	true	false
outcome	ERR_UNHANDLED_REJECTION, exit 1	reaches "still alive", exit 0

The error printed before the fix matches the report field for field (status: undefined, code: 'TIMEOUT', the ?remoteRefBehavior=resolve URL, retryAfter: undefined, [cause] TimeoutError). Exit is 1 under bare Node; Vercel's runtime wrapper reports the same condition as 128.

Full suites green on this branch:

• packages/core unit: 2447 passed, 3 expected-fail, 1 skipped (114 files)
• pnpm --filter @workflow/core typecheck: clean
• biome check on both changed files: no new findings (13 pre-existing warnings, identical to main)

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 44733e5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@workflow/core	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
example-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-astro-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-express-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-fastify-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-hono-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-nestjs-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-nitro-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-nuxt-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-python-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workbench-vite-workflow	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workflow-docs	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workflow-swc-playground	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workflow-tarballs	Ready	Preview, v0	Sep 15, 2026 8:42am UTC
workflow-web	Ready	Preview, v0	Sep 15, 2026 8:42am UTC

[github-actions]

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• sleepWinsRaceWorkflow (vite · local-dev / local / node / stable)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

• cold-start-warmup · suite warmup (tanstack-start) · at 09:32:01Z · abandoned wrun_01M2J6GGDFD63TT6SSM0WT3TQZ

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
✅ ▲ Vercel Production	3662	0	685	4347
✅ 💻 Local Development	3998	0	510	4508
✅ 📦 Local Production	3998	0	510	4508
✅ 🐘 Local Postgres	3998	0	510	4508
✅ 🪟 Windows	320	0	2	322
✅ 🌐 Cross-language Conformance	68	0	74	142
✅ vercel-http-transport	823	0	143	966
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	557	0	87	644
Total	17451	0	2521	19972

Details by Category

✅ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	133	0	28
✅ astro-quickjs	133	0	28
✅ example-node	133	0	28
✅ example-quickjs	133	0	28
✅ express-node	133	0	28
✅ express-quickjs	133	0	28
✅ fastify-node	133	0	28
✅ fastify-quickjs	133	0	28
✅ hono-node	133	0	28
✅ hono-quickjs	133	0	28
✅ nest-node	133	0	28
✅ nest-quickjs	133	0	28
✅ nextjs-turbopack-node	158	0	3
✅ nextjs-turbopack-quickjs	158	0	3
✅ nextjs-webpack-node	158	0	3
✅ nextjs-webpack-quickjs	158	0	3
✅ nitro-node	133	0	28
✅ nitro-quickjs	133	0	28
✅ nuxt-node	133	0	28
✅ nuxt-quickjs	133	0	28
✅ python-node	66	0	95
✅ sveltekit-node	152	0	9
✅ sveltekit-quickjs	152	0	9
✅ tanstack-start-node	133	0	28
✅ tanstack-start-quickjs	133	0	28
✅ vite-node	133	0	28
✅ vite-quickjs	133	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	27
✅ astro-stable-quickjs	134	0	27
✅ express-stable-node	134	0	27
✅ express-stable-quickjs	134	0	27
✅ fastify-stable-node	134	0	27
✅ fastify-stable-quickjs	134	0	27
✅ hono-stable-node	134	0	27
✅ hono-stable-quickjs	134	0	27
✅ nest-stable-node	134	0	27
✅ nest-stable-quickjs	134	0	27
✅ nextjs-turbopack-canary-node	160	0	1
✅ nextjs-turbopack-canary-quickjs	160	0	1
✅ nextjs-turbopack-stable-node	160	0	1
✅ nextjs-turbopack-stable-quickjs	160	0	1
✅ nextjs-webpack-canary-node	160	0	1
✅ nextjs-webpack-canary-quickjs	160	0	1
✅ nextjs-webpack-stable-node	160	0	1
✅ nextjs-webpack-stable-quickjs	160	0	1
✅ nitro-stable-node	134	0	27
✅ nitro-stable-quickjs	134	0	27
✅ nuxt-stable-node	134	0	27
✅ nuxt-stable-quickjs	134	0	27
✅ sveltekit-stable-node	153	0	8
✅ sveltekit-stable-quickjs	153	0	8
✅ tanstack-start-node	134	0	27
✅ tanstack-start-quickjs	134	0	27
✅ vite-stable-node	134	0	27
✅ vite-stable-quickjs	134	0	27

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	27
✅ astro-stable-quickjs	134	0	27
✅ express-stable-node	134	0	27
✅ express-stable-quickjs	134	0	27
✅ fastify-stable-node	134	0	27
✅ fastify-stable-quickjs	134	0	27
✅ hono-stable-node	134	0	27
✅ hono-stable-quickjs	134	0	27
✅ nest-stable-node	134	0	27
✅ nest-stable-quickjs	134	0	27
✅ nextjs-turbopack-canary-node	160	0	1
✅ nextjs-turbopack-canary-quickjs	160	0	1
✅ nextjs-turbopack-stable-node	160	0	1
✅ nextjs-turbopack-stable-quickjs	160	0	1
✅ nextjs-webpack-canary-node	160	0	1
✅ nextjs-webpack-canary-quickjs	160	0	1
✅ nextjs-webpack-stable-node	160	0	1
✅ nextjs-webpack-stable-quickjs	160	0	1
✅ nitro-stable-node	134	0	27
✅ nitro-stable-quickjs	134	0	27
✅ nuxt-stable-node	134	0	27
✅ nuxt-stable-quickjs	134	0	27
✅ sveltekit-stable-node	153	0	8
✅ sveltekit-stable-quickjs	153	0	8
✅ tanstack-start-node	134	0	27
✅ tanstack-start-quickjs	134	0	27
✅ vite-stable-node	134	0	27
✅ vite-stable-quickjs	134	0	27

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	27
✅ astro-stable-quickjs	134	0	27
✅ express-stable-node	134	0	27
✅ express-stable-quickjs	134	0	27
✅ fastify-stable-node	134	0	27
✅ fastify-stable-quickjs	134	0	27
✅ hono-stable-node	134	0	27
✅ hono-stable-quickjs	134	0	27
✅ nest-stable-node	134	0	27
✅ nest-stable-quickjs	134	0	27
✅ nextjs-turbopack-canary-node	160	0	1
✅ nextjs-turbopack-canary-quickjs	160	0	1
✅ nextjs-turbopack-stable-node	160	0	1
✅ nextjs-turbopack-stable-quickjs	160	0	1
✅ nextjs-webpack-canary-node	160	0	1
✅ nextjs-webpack-canary-quickjs	160	0	1
✅ nextjs-webpack-stable-node	160	0	1
✅ nextjs-webpack-stable-quickjs	160	0	1
✅ nitro-stable-node	134	0	27
✅ nitro-stable-quickjs	134	0	27
✅ nuxt-stable-node	134	0	27
✅ nuxt-stable-quickjs	134	0	27
✅ sveltekit-stable-node	153	0	8
✅ sveltekit-stable-quickjs	153	0	8
✅ tanstack-start-node	134	0	27
✅ tanstack-start-quickjs	134	0	27
✅ vite-stable-node	134	0	27
✅ vite-stable-quickjs	134	0	27

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	1
✅ nextjs-turbopack-quickjs	160	0	1

✅ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
✅ python	68	0	74

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	133	0	28
✅ express	133	0	28
✅ hono	133	0	28
✅ nextjs-turbopack	158	0	3
✅ nitro	133	0	28
✅ vite	133	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	133	0	28
✅ express	133	0	28
✅ nextjs-turbopack	158	0	3
✅ vite	133	0	28

---

📋 View full workflow run

[github-actions]

📊 Workflow Benchmarks

commit 44733e5 · Tue, 15 Sep 2026 09:54:59 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1657 (+25%) 🔻	1771 🔴 (+23%) 🔻	1804 🔴 (+23%) 🔻	1854 🔴 (+15%)	30
TTFS	stream	162 (-16%) 💚	1683 🔴 (+18%) 🔻	1704 🔴 (+16%) 🔻	1768 🔴 (+17%) 🔻	30
TTFS	hook + stream	615 (+7.1%)	2159 🔴 (+17%) 🔻	2188 🔴 (+12%)	2290 🔴 (-0.6%)	30
Fan-out TTFS	Promise.all(100 steps)	488 (+7.7%)	859 (+27%) 🔻	983 (-47%) 💚	2351 (+27%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2388 (+42%) 🔻	3114 (-4.7%)	3114 (-28%) 💚	10207 (+29%) 🔻	10
STSO	1020 steps (inline)	150 (+19%) 🔻	193 (+21%) 🔻	212 (+18%) 🔻	261 (+23%) 🔻	1019
WO	1020 steps	189878 (+19%) 🔻	189878 (+19%) 🔻	189878 (+19%) 🔻	189878 (+19%) 🔻	1
CRTT	first chunk (pooled)	90 (-1.1%)	124 (-16%) 💚	140 (-32%) 💚	159 (-28%) 💚	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	109 (-25%)	230 (-43%)	331 (-44%)	779 (-5%)	204 (-29%)	10
size sweep (100/s, 160B-12KB)	117 (-12%)	240 (-40%)	406 (-24%)	582 (-38%)	166 (-36%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	110 (-3%)	193 (-28%)	228 (-42%)	353 (-41%)	224 (-45%)	3
replay eve-gpt-5.6-sol-2000t (1x)	99 (-40%)	187 (-27%)	230 (-32%)	748 (-1%)	409 (-40%)	2
replay eve-gpt-5.6-sol-2000t (2x)	104 (-18%)	287 (-38%)	386 (-41%)	710 (-41%)	254 (-41%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 158739ms → this run 189531ms (Δ +30792ms, +19%)

100-150 ms  ┃██████████████           main 512  this   0  -512
150-200 ms  ██████████████░░░░░░░░░┃  main 477  this 836  +359
200-250 ms  █░░░┃                     main  26  this 166  +140
250-300 ms  ┃                         main   2  this  14   +12
300-350 ms  ┃                         main   0  this   2    +2
350-400 ms  ┃                         main   0  this   1    +1
400-450 ms  ┃                         main   2  this   0    -2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant  RTT 1ms→5s+             avg         p50         p90         p99     n
control  ······▁█▅▁···  202.1 (-24%)  178 (-20%)  331 (-44%)   779 (-5%)  3000
sweep    ······▁█▅▁···  196.9 (-31%)  185 (-20%)  406 (-24%)  582 (-38%)  3000
gw 1x    ······▁█▂····  160.3 (-20%)  155 (-12%)  228 (-42%)  353 (-41%)  5295
eve 1x   ······▁█▂▁···  163.5 (-23%)  141 (-21%)  230 (-32%)   748 (-1%)  5186
eve 2x   ······▁██▁···  227.5 (-30%)  200 (-36%)  386 (-41%)  710 (-41%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  ▅█▅▅▄▂▁▄▂▄  176–233ms
sweep    ███▆▇▆▄▃▁▃  166–217ms
gw 1x    █▅▁▂▇▃▄▂▁▅  151–174ms
eve 1x   ▂▁▂▁▃▁█▇▁▁  142–227ms
eve 2x   ▂▁▂▂▁▂▅█▃▃  170–377ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  ▁▆█▇▄▆▆  194–198ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▇▃▇▅▄██▁▆▃  48–74ms
sweep    ▄▆▃▅█▃▃▁▁▂  70–97ms
gw 1x    ▆▆▃▁▄▃█▄▂▄  46–65ms
eve 1x   ▆▄▄▃▆▄█▃▁▄  26–34ms
eve 2x   ▇▆▃▂▄▂▅▂▁█  25–38ms

📜 Previous results (1)

44733e5

Tue, 15 Sep 2026 08:58:08 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1571 (+18%) 🔻	1661 🔴 (+15%) 🔻	1696 🔴 (+16%) 🔻	1761 🔴 (+9.0%)	30
TTFS	stream	158 (-18%) 💚	1617 🔴 (+13%)	1649 🔴 (+13%)	1659 🔴 (+10%)	30
TTFS	hook + stream	606 (+5.6%)	2011 🔴 (+9.3%)	2036 🔴 (+3.9%)	2310 🔴 (±0%)	30
Fan-out TTFS	Promise.all(100 steps)	477 (+5.3%)	1153 (+71%) 🔻	2060 (+11%)	2297 (+24%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	1962 (+17%) 🔻	4037 (+23%) 🔻	5501 (+27%) 🔻	8957 (+13%)	10
STSO	1020 steps (inline)	129 (+2.4%)	153 (-3.8%)	168 (-6.7%)	221 (+4.2%)	1019
WO	1020 steps	152665 (-3.9%)	152665 (-3.9%)	152665 (-3.9%)	152665 (-3.9%)	1
CRTT	first chunk (pooled)	56 (-38%) 💚	84 (-43%) 💚	97 (-53%) 💚	103 (-54%) 💚	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	68 (-53%)	145 (-64%)	241 (-59%)	528 (-36%)	122 (-57%)	10
size sweep (100/s, 160B-12KB)	70.5 (-47%)	147 (-63%)	382 (-28%)	670 (-28%)	150 (-43%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	66 (-42%)	133 (-50%)	169 (-57%)	419 (-30%)	280 (-31%)	3
replay eve-gpt-5.6-sol-2000t (1x)	93.5 (-43%)	134 (-48%)	175 (-48%)	325 (-57%)	234 (-66%)	2
replay eve-gpt-5.6-sol-2000t (2x)	72 (-43%)	207 (-55%)	515 (-21%)	1079 (-11%)	422 (-2%)	3

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	250.7 KiB (±0)	93.0 KiB (±0)	1.89 MiB (+447 B)
nextjs-turbopack	257.2 KiB (±0)	426 B (±0)	897.9 KiB (+8 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

44733e5 · run

[pranaygp]

CI triage — both failures are unrelated to this change

The run finished with exactly two red jobs and no others. Neither is caused by this diff, and neither executed a test that this diff can reach. My token can't rerun --failed on this repo, so flagging for a maintainer to re-run.

1. E2E Vercel WS Transport Test (vite) — Vercel API blip during deploy

Failed at step 6, "Deploy a dedicated WS-transport preview". Steps 7 (Record E2E start time) and 8 (Run E2E Tests) were skipped — no test ran.

Uploading [====================] (2.5MB/2.5MB)
Error: FetchError: invalid json response body at
  https://api.vercel.com/v2/files?teamId=team_nO2mCG4W8IxPIeKoSsqwAxxB
  reason: Unexpected token '<', "<html>\n<h"... is not valid JSON

The API returned an HTML error page instead of JSON during file upload. This lane passed on the base commit (5f723b3d0, the last completed main run) alongside the other three WS-transport apps.

2. Unit Tests (windows-latest) — wall-clock assertions in a file this PR doesn't touch

FAIL src/runtime/run-return-value-real-world.test.ts
  > resolves as soon as the run finishes, not at the next poll tick
    AssertionError: expected 1140 to be less than 800
  > reports a cancellation without waiting out the interval
    AssertionError: expected 1102 to be less than 800

Both assert elapsed < 800ms against an event fired at 300ms — a latency budget, not a correctness check. The Windows runner came in ~340ms over. That file exercises run.returnValue long-poll latency over a local World with no forwarded writables, so the code path this PR changes is never entered.

It passes locally on this branch:

$ cd packages/core && WORKFLOW_TARGET_WORLD=local pnpm vitest run src/runtime/run-return-value-real-world.test.ts
 Test Files  1 passed (1)
      Tests  3 passed (3)

Worth noting this is a known-flaky file: #4119 ("Fix three crash-class defects behind this week's flaky CI") is currently modifying run-return-value-real-world.test.ts for exactly this reason, and its description reports "22 of 30 main pushes red since Sept 8", mostly harness rather than product.

Everything this PR can affect is green, including Biome, DCO, Unit Tests (ubuntu-latest), and every E2E lane that actually ran.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[pranaygp]

Reopening to retrigger CI — the only red is a known-flaky Windows perf assertion in @workflow/world-local (details below).

[pranaygp]

✅ CI is green — 180 passing, 0 failing

Retriggered twice. Both earlier reds were flakes, and each retry produced a different failure, which is itself the tell — a real regression fails the same test every time.

Run	Windows unit	Other	Outcome
34948236190	❌ core/runtime/run-return-value-real-world.test.ts — expected 1140 to be less than 800	❌ WS Transport (vite): api.vercel.com returned HTML mid-upload	red
34950518449	❌ world-local/storage/hook-index.test.ts — expected 3597.44 to be less than 3009.07	✅ all E2E lanes	red
34952563901	✅	✅	green

Three things confirm none of it was this diff:

1. Different test each time, both pure wall-clock/perf assertions. The hook-index one documents itself as jitter-prone: "The generous ratio + constant absorbs fs jitter on slow CI runners (notably Windows)."
2. One of them is in @workflow/world-local, which this PR does not touch — the diff is confined to packages/core/src/serialization.{ts,test.ts} plus a changeset.
3. The stable companion fix(core): start forwarded writable key lookup on first write (backport of #4175) #4176, carrying the same code change, passed Windows on every run. Meanwhile unrelated [world] Add optional invoke method (hook payloads only for now) #4168 was failing Windows on yet another world-local flake, so the runners were broadly noisy.

E2E Required Check was never an independent failure — it declares needs: [..., unit, ...] and gates on needs.unit.result, so it simply inherited the Windows job's status.

Only REVIEW_REQUIRED is left. Ready for a look, @vercel/workflow.

[github-actions]

Backport to stable failed for 56322af due to a workflow error (backport job run).

This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.

Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

56322af66fd1f78a77529353be383bfa93ec3af8