[core] When live stream read waits forever at missing chunk, fail early instead by pranaygp · Pull Request #4334 · vercel/workflow

pranaygp · 2026-09-23T19:23:37Z

Addresses the reader-side part of #4306. It does not close the issue, because the chunk loss itself is not fixed here.

Problem

When a chunk below a stream's known tail is missing from storage, every live-read session opened at that index ends without delivering a frame. createReconnectingFramedStream reconnects from the same index until its consecutive-reconnect budget (50) runs out. Each such session lasts the server's max response duration (~2 min), so a reader sits on the hole for well over an hour before it errors, with a generic "exceeded maximum reconnection attempts" message.

Fix

When a session stays open for at least 10 seconds and ends (clean EOF or errored body) without receiving any bytes, the reader checks streams.getInfo. If the tail is at or past the resume index, the chunk is known to exist but was not served, so this isn't an idle wait for new data. After 3 consecutive stalls like this, the read fails with a StreamError that names the missing index and the tail, and says to resume from startIndex = index + 1.

Sessions that deliver data pay nothing extra: metadata is fetched only for zero-frame sessions, and on EOF it reuses the lookup completion verification already does.
An idle reader at the tail (tail < resume index) keeps waiting as before. So does a reader whose metadata lookup fails.
A session that received partial-frame bytes counts as progress, not a stall. A zero-byte session that ends within 10 seconds (a brief server blip) neither counts nor resets the stall streak, and is retried against the general reconnect budget as before.
The reader does not skip the hole on its own. Silently dropping a chunk (for example a UI message start part) can corrupt consumer state. Failing with the index lets the caller decide.

This does not restore lost chunks. The storage-side loss reported in #4306 is outside this repo.

Tests

packages/core/src/reconnecting-framed-stream.test.ts: new cases for a hole ending in a clean EOF and in an errored body, which fail fast with the index; for two adjacent holes, where resuming at index + 1 fails the same bounded way at the second hole and resuming past it reads the rest; for quick zero-byte sessions, partial-frame cuts and non-consecutive stalls, which don't fail the read; and for an idle reader at the tail, which keeps reconnecting. The existing budget-exhaustion test now uses an empty stream (tail -1), since its old fixture (tail 4 at index 0) is a hole.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

changeset-bot · 2026-09-23T19:23:42Z

🦋 Changeset detected

Latest commit: 4e2cf45

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@workflow/core	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-cloudflare	Patch
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch
@workflow/world-vercel	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-09-23T19:23:57Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
example-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-astro-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-express-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-fastify-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-hono-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-nestjs-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-nitro-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-nuxt-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-python-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workbench-vite-workflow	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workflow-docs	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workflow-swc-playground	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workflow-tarballs	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC
workflow-web	Ready	Preview, v0	Nov 13, 2026 7:58pm UTC

github-actions · 2026-09-23T19:24:02Z

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

a chain of takeovers: each victim ends with HookForceClaimedError, deliveries follow the current owner (nuxt · local-dev / local / node / stable)
awaitChildReturnValueWorkflow - parent waits on the child-run completion hook (fail: false) (nextjs-webpack · local-dev / local / node / canary)
fibonacciWorkflow - recursive workflow composition via start() (nextjs-webpack · local-dev / local / node / canary)
webhookWorkflow (fastify · local-prod / local / quickjs / stable)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 20:00:17Z · abandoned wrun_01M7B7XRW8X0P90R6S6VC3RS15
cold-start-warmup · suite warmup (tanstack-start) · at 20:00:30Z · abandoned wrun_01M7B7XRX1RG1WCBH65QXZYNHC
run-pickup-stall · hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack) · at 20:05:52Z · abandoned wrun_01M7B87ZT8WPSZHD8M645HH0X5
run-pickup-stall · hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack) · at 20:05:54Z · abandoned wrun_01M7B881WZK1XVFZPXV0TFJTA0

E2E Test Summary

Summary

	Passed	Skipped	Total
✅ ▲ Vercel Production	4649	940	5589
✅ 💻 Local Development	5272	731	6003
✅ 📦 Local Production	5272	731	6003
✅ 🐘 Local Postgres	5417	586	6003
✅ 🪟 Windows	401	13	414
✅ dynamic-runs	0	0	0
✅ local-postgres-spec9	0	364	364
✅ local-spec9	409	5	414
✅ vercel-http-transport	1054	188	1242
✅ vercel-multi-region	27	0	27
✅ vercel-queue-hold	4	0	4
✅ vercel-spec9	395	19	414
✅ vercel-ws-transport	712	116	828
Total	23612	3693	27305

Details by Category

✅ ▲ Vercel Production

App	Passed	Skipped
✅ astro-node	171	36
✅ astro-quickjs	170	37
✅ example-node	171	36
✅ example-quickjs	170	37
✅ express-node	171	36
✅ express-quickjs	170	37
✅ fastify-node	171	36
✅ fastify-quickjs	170	37
✅ hono-node	171	36
✅ hono-quickjs	170	37
✅ nest-node	171	36
✅ nest-quickjs	170	37
✅ nextjs-turbopack-node	199	8
✅ nextjs-turbopack-quickjs	198	9
✅ nextjs-webpack-node	199	8
✅ nextjs-webpack-quickjs	198	9
✅ nitro-node	171	36
✅ nitro-quickjs	170	37
✅ nuxt-node	171	36
✅ nuxt-quickjs	170	37
✅ python-node	66	141
✅ sveltekit-node	190	17
✅ sveltekit-quickjs	189	18
✅ tanstack-start-node	171	36
✅ tanstack-start-quickjs	170	37
✅ vite-node	171	36
✅ vite-quickjs	170	37

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	172	35
✅ astro-stable-quickjs	171	36
✅ express-stable-node	172	35
✅ express-stable-quickjs	171	36
✅ fastify-stable-node	172	35
✅ fastify-stable-quickjs	171	36
✅ hono-stable-node	172	35
✅ hono-stable-quickjs	171	36
✅ nest-stable-node	172	35
✅ nest-stable-quickjs	171	36
✅ nextjs-turbopack-canary-node	201	6
✅ nextjs-turbopack-canary-quickjs	200	7
✅ nextjs-turbopack-quickjs-snapshot	200	7
✅ nextjs-turbopack-stable-node	201	6
✅ nextjs-turbopack-stable-quickjs	200	7
✅ nextjs-webpack-canary-node	201	6
✅ nextjs-webpack-canary-quickjs	200	7
✅ nextjs-webpack-stable-node	201	6
✅ nextjs-webpack-stable-quickjs	200	7
✅ nitro-stable-node	172	35
✅ nitro-stable-quickjs	171	36
✅ nuxt-stable-node	172	35
✅ nuxt-stable-quickjs	171	36
✅ sveltekit-stable-node	191	16
✅ sveltekit-stable-quickjs	190	17
✅ tanstack-start-node	172	35
✅ tanstack-start-quickjs	171	36
✅ vite-stable-node	172	35
✅ vite-stable-quickjs	171	36

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	172	35
✅ astro-stable-quickjs	171	36
✅ express-stable-node	172	35
✅ express-stable-quickjs	171	36
✅ fastify-stable-node	172	35
✅ fastify-stable-quickjs	171	36
✅ hono-stable-node	172	35
✅ hono-stable-quickjs	171	36
✅ nest-stable-node	172	35
✅ nest-stable-quickjs	171	36
✅ nextjs-turbopack-canary-node	201	6
✅ nextjs-turbopack-canary-quickjs	200	7
✅ nextjs-turbopack-quickjs-snapshot	200	7
✅ nextjs-turbopack-stable-node	201	6
✅ nextjs-turbopack-stable-quickjs	200	7
✅ nextjs-webpack-canary-node	201	6
✅ nextjs-webpack-canary-quickjs	200	7
✅ nextjs-webpack-stable-node	201	6
✅ nextjs-webpack-stable-quickjs	200	7
✅ nitro-stable-node	172	35
✅ nitro-stable-quickjs	171	36
✅ nuxt-stable-node	172	35
✅ nuxt-stable-quickjs	171	36
✅ sveltekit-stable-node	191	16
✅ sveltekit-stable-quickjs	190	17
✅ tanstack-start-node	172	35
✅ tanstack-start-quickjs	171	36
✅ vite-stable-node	172	35
✅ vite-stable-quickjs	171	36

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	177	30
✅ astro-stable-quickjs	176	31
✅ express-stable-node	177	30
✅ express-stable-quickjs	176	31
✅ fastify-stable-node	177	30
✅ fastify-stable-quickjs	176	31
✅ hono-stable-node	177	30
✅ hono-stable-quickjs	176	31
✅ nest-stable-node	177	30
✅ nest-stable-quickjs	176	31
✅ nextjs-turbopack-canary-node	206	1
✅ nextjs-turbopack-canary-quickjs	205	2
✅ nextjs-turbopack-quickjs-snapshot	205	2
✅ nextjs-turbopack-stable-node	206	1
✅ nextjs-turbopack-stable-quickjs	205	2
✅ nextjs-webpack-canary-node	206	1
✅ nextjs-webpack-canary-quickjs	205	2
✅ nextjs-webpack-stable-node	206	1
✅ nextjs-webpack-stable-quickjs	205	2
✅ nitro-stable-node	177	30
✅ nitro-stable-quickjs	176	31
✅ nuxt-stable-node	177	30
✅ nuxt-stable-quickjs	176	31
✅ sveltekit-stable-node	196	11
✅ sveltekit-stable-quickjs	195	12
✅ tanstack-start-node	177	30
✅ tanstack-start-quickjs	176	31
✅ vite-stable-node	177	30
✅ vite-stable-quickjs	176	31

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	201	0	6
✅ nextjs-turbopack-quickjs	200	0	7

✅ dynamic-runs

App	Passed	Failed	Skipped
✅ astro-vercel	0	0	0
✅ example-vercel	0	0	0
✅ express-vercel	0	0	0
✅ fastify-vercel	0	0	0
✅ hono-vercel	0	0	0
✅ nest-vercel	0	0	0
✅ nextjs-turbopack-vercel	0	0	0
✅ nextjs-webpack-vercel	0	0	0
✅ nitro-vercel	0	0	0
✅ nuxt-vercel	0	0	0
✅ sveltekit-vercel	0	0	0
✅ tanstack-start-vercel	0	0	0
✅ vite-vercel	0	0	0

✅ local-postgres-spec9

App	Passed	Failed	Skipped
✅ nextjs-turbopack-stable-node	0	0	182
✅ nextjs-turbopack-stable-quickjs	0	0	182

✅ local-spec9

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	205	0	2
✅ nextjs-turbopack-quickjs	204	0	3

✅ vercel-http-transport

App	Passed	Skipped
✅ example	171	36
✅ express	171	36
✅ hono	171	36
✅ nextjs-turbopack	199	8
✅ nitro	171	36
✅ vite	171	36

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-queue-hold

App	Passed	Failed	Skipped
✅ nextjs-turbopack	4	0	0

✅ vercel-spec9

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	198	0	9
✅ nextjs-turbopack-quickjs	197	0	10

✅ vercel-ws-transport

App	Passed	Skipped
✅ example	171	36
✅ express	171	36
✅ nextjs-turbopack	199	8
✅ vite	171	36

📋 View full workflow run

github-actions · 2026-09-23T19:24:37Z

📊 Workflow Benchmarks

commit 4e2cf45 · Fri, 13 Nov 2026 20:04:09 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1755 (+23%) 🔻	1888 🔴 (+22%) 🔻	1948 🔴 (+18%) 🔻	2130 🔴 (±0%)	30
TTFS	stream	1748 (+23%) 🔻	1817 🔴 (+21%) 🔻	1841 🔴 (+21%) 🔻	1881 🔴 (+14%)	30
TTFS	hook + stream	1967 (+22%) 🔻	2035 🔴 (+19%) 🔻	2106 🔴 (+20%) 🔻	2327 🔴 (+11%)	30
Fan-out TTFS	Promise.all(100 steps)	336 (+13%)	2105 (+310%) 🔻	2166 (+298%) 🔻	2584 (+47%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2408 (+41%) 🔻	5231 (+33%) 🔻	5450 (+35%) 🔻	7059 (+6.8%)	10
STSO	1020 steps (inline)	81 (+8.0%)	101 (-16%) 💚	116 (-24%) 💚	186 (-43%) 💚	1019
WO	1020 steps	103276 (-9.4%)	103276 (-9.4%)	103276 (-9.4%)	103276 (-9.4%)	1

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 113704ms → this run 101475ms (Δ -12229ms, -11%)

   50-100 ms  ███████████████████░░░░┃  main 574  this 729  +155
  100-150 ms  ████████┃██               main 335  this 266   -69
  150-200 ms  ┃██                       main  84  this  17   -67
  200-250 ms  ┃                         main  11  this   6    -5
  250-300 ms  ┃                         main   4  this   0    -4
  300-350 ms  ┃                         main   2  this   0    -2
  350-400 ms  ┃                         main   1  this   0    -1
  400-450 ms  ┃                         main   1  this   1    +0
  450-500 ms  ┃                         main   1  this   0    -1
  500-550 ms  ┃                         main   2  this   0    -2
  600-650 ms  ┃                         main   1  this   0    -1
  750-800 ms  ┃                         main   1  this   0    -1
  900-950 ms  ┃                         main   1  this   0    -1
1000-1050 ms  ┃                         main   1  this   0    -1

📜 Previous results (4)

`2d04a32`

Fri, 13 Nov 2026 19:42:18 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	247 (-22%) 💚	1859 🔴 (+25%) 🔻	1998 🔴 (+20%) 🔻	2212 🔴 (+2.9%)	30
TTFS	stream	235 (-82%) 💚	1822 🔴 (+22%) 🔻	1883 🔴 (+25%) 🔻	2143 🔴 (+16%) 🔻	30
TTFS	hook + stream	432 (-74%) 💚	2392 🔴 (+38%) 🔻	2429 🔴 (+37%) 🔻	2566 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	336 (+21%) 🔻	490 (-9.3%)	658 (-3.7%)	2178 (+22%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2240 (+33%) 🔻	3631 (-7.8%)	4350 (+2.2%)	7336 (+12%)	10
STSO	1020 steps (inline)	90 (+4.7%)	110 (-17%) 💚	135 (-15%)	195 (-36%) 💚	1019
WO	1020 steps	111029 (-9.9%)	111029 (-9.9%)	111029 (-9.9%)	111029 (-9.9%)	1

`a86b822`

Fri, 06 Nov 2026 01:22:05 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	2132 (+590%) 🔻	2567 🔴 (+51%) 🔻	2700 🔴 (+48%) 🔻	2757 🔴 (+25%) 🔻	30
TTFS	stream	460 (+46%) 🔻	2165 🔴 (+28%) 🔻	2179 🔴 (+13%)	2199 🔴 (+12%)	30
TTFS	hook + stream	2509 (+251%) 🔻	2655 🔴 (+18%) 🔻	2728 🔴 (+18%) 🔻	2751 🔴 (+4.0%)	30
Fan-out TTFS	Promise.all(100 steps)	740 (+23%) 🔻	1105 (+53%) 🔻	2696 (+197%) 🔻	2991 (+34%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	1541 (-2.5%)	4088 (-1.4%)	5897 (+40%) 🔻	9028 (+65%) 🔻	10
STSO	1020 steps (inline)	105 (+2.9%)	132 (-21%) 💚	150 (-29%) 💚	196 (-45%) 💚	1019
WO	1020 steps	129211 (-18%) 💚	129211 (-18%) 💚	129211 (-18%) 💚	129211 (-18%) 💚	1

`e3278e0`

Wed, 23 Sep 2026 20:28:19 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	209 (-71%) 💚	2189 🔴 (+100%) 🔻	2255 🔴 (+102%) 🔻	2731 🔴 (+127%) 🔻	30
TTFS	stream	178 (-19%) 💚	2175 🔴 (+107%) 🔻	2216 🔴 (+102%) 🔻	2294 🔴 (+61%) 🔻	30
TTFS	hook + stream	2233 (+155%) 🔻	2497 🔴 (+88%) 🔻	2530 🔴 (+80%) 🔻	2600 🔴 (+67%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	501 (-42%) 💚	851 (-50%) 💚	873 (-51%) 💚	897 (-52%) 💚	10
Fan-out TTLS	Promise.all(100 steps)	1928 (-16%) 💚	4950 (+36%) 🔻	5554 (+47%) 🔻	8000 (+11%)	10
STSO	1020 steps (inline)	126 (+26%) 🔻	155 (+19%) 🔻	177 (+15%)	300 (+48%) 🔻	1019
WO	1020 steps	157668 (+20%) 🔻	157668 (+20%) 🔻	157668 (+20%) 🔻	157668 (+20%) 🔻	1

`e3278e0`

Wed, 23 Sep 2026 19:57:16 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	2563 (+85%) 🔻	2771 🔴 (+59%) 🔻	2830 🔴 (+59%) 🔻	2911 🔴 (+27%) 🔻	30
TTFS	stream	192 (+18%) 🔻	2466 🔴 (+45%) 🔻	2616 🔴 (+50%) 🔻	2708 🔴 (+38%) 🔻	30
TTFS	hook + stream	2671 (+324%) 🔻	2781 🔴 (+33%) 🔻	2798 🔴 (+23%) 🔻	3047 🔴 (+15%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	688 (+40%) 🔻	939 (+58%) 🔻	1395 (+115%) 🔻	2937 (+29%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2899 (+59%) 🔻	3954 (+43%) 🔻	3963 (+38%) 🔻	12035 (+39%) 🔻	10
STSO	1020 steps (inline)	152 (+33%) 🔻	192 (+39%) 🔻	211 (+39%) 🔻	271 (-76%) 💚	1019
WO	1020 steps	188589 (+19%) 🔻	188589 (+19%) 🔻	188589 (+19%) 🔻	188589 (+19%) 🔻	1

ℹ️ Metric definitions & methodology

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies. Streaming delivery performance is measured in durabench.}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

github-actions · 2026-09-23T19:27:31Z

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 45 total

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	5	0ms	ok
✅ `hook-at-step-started`	completed	9	0ms	ok
✅ `hook-at-step-completed`	completed	9	0ms	ok
✅ `hook-at-hook-created`	completed	9	0ms	ok
✅ `claim-hook-at-registration`	completed	9	0ms	ok
✅ `claim-hook-at-step-started`	completed	9	0ms	ok
✅ `deadline-hook-wins`	completed	6	1.0h	ok
✅ `deadline-expires`	completed	6	1.0h	ok
✅ `step-vs-timer-early-settlement`	completed	6	1.0h	ok
✅ `long-sleep`	completed	9	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	9	2.0s	ok
✅ `parallel-steps`	completed	7	0ms	ok
✅ `hook-on-execution-state`	completed	9	0ms	ok
✅ `peek-hook-before-branch`	completed	9	0ms	ok
✅ `peek-hook-after-branch`	completed	9	0ms	ok
✅ `peek-hook-at-registration`	completed	9	0ms	ok
✅ `race-hook-before-probe`	completed	9	0ms	ok
✅ `race-hook-after-probe`	completed	9	0ms	ok
✅ `race-duplicate-delivery`	completed	10	0ms	ok
✅ `attr-hook-before-step`	completed	9	0ms	ok
✅ `attr-hook-after-step`	completed	9	0ms	ok
✅ `attr-from-step-body`	completed	10	0ms	ok
✅ `fork-hook-after-timeout`	completed	11	1.0m	ok
✅ `fork-hook-before-timeout`	completed	11	1.0m	ok
✅ `count-hook-after-timeout`	completed	13	1.0m	ok
✅ `count-hook-before-timeout`	completed	15	1.0m	ok
✅ `stale-read-step-count-fork`	completed	15	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	11	1.0m	ok
✅ `step-vs-step-fork`	completed	9	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	9	1.0s	ok
✅ `fence-catches-benign-direction`	completed	9	1.0s	ok
✅ `run-ahead-fenced-overlap`	completed	10	1.0s	ok
✅ `in-flight-before-decision`	completed	15	1.0m	ok
✅ `in-flight-before-decision-counted`	completed	13	1.0m	ok
✅ `in-flight-after-decision`	completed	15	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	15	1.0m	ok
✅ `fork-hook-wins`	completed	10	1.0m	ok
✅ `fork-timeout-wins`	completed	10	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	14	1.0m	ok
✅ `claimed-payload-under-fork`	completed	14	1.0m	ok
✅ `writers-independent-step-bodies`	completed	9	0ms	ok
✅ `writers-scripted-tempo`	completed	9	0ms	ok
✅ `cancel-mid-step`	cancelled	6	0ms	skipped

Full trace: world-sim.txt

github-actions · 2026-09-23T19:28:07Z

Framework	Flow route	Step reg.	Framework output
hono	306.2 KiB (±0)	124.0 KiB (±0)	2.39 MiB (+595 B)
nextjs-turbopack	397.8 KiB (±0)	491 B (±0)	1.13 MiB (+197 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

4e2cf45 · run

…tail Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

kvnloo · 2026-09-23T23:24:59Z

yo pranay, long time! funny running into your work here. i've been deep in durable/replay/agent infra lately and ended up reading through #4306 / #4334.

reading through the patch, the bounded reader-side containment makes sense. one edge case from the production evidence: wrun_41M36CF0G60GJN4M233414KTVE has adjacent missing indices 3443, 3444, so startIndex = index + 1 can land directly on another hole and can immediately hit the same stall path again on the next index.

probably fine for this bounded patch, but multi-hole recovery feels worth a follow-up, even if it's just making the error/consumer path cheap to iterate across consecutive gaps.

happy to put together a focused consecutive-hole regression test if useful

…urable-run-streams-lose-ack Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

…ole stall A session at the resume index that is cut partway through a large chunk received bytes but completed no frame, so it was counted as a stall below the tail. Three such cuts in a row failed a healthy read with "the chunk is missing from storage". Bytes arriving show the server is serving the chunk, so treat them as progress. Also cover the consecutive-stall reset. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

VaguelySerious · 2026-11-06T01:07:02Z

I pushed two commits:

c9f20b924 merges current main into the branch. It merged without conflicts. createReconnectingFramedStream on main still has the same reconnect, EOF-verification and budget structure, so the change still fits.
a86b8229c stops counting a session that received partial-frame bytes as a stall below the tail. Before this, a session at the resume index that was cut partway through a large chunk completed no frame, so it counted as a stall. Three such cuts in a row would fail a healthy read with "the chunk is missing from storage". Bytes arriving show the server is serving that chunk, so they now count as progress. This commit adds two tests: one for that case and one for the consecutive-stall reset, which had no test yet.

Left as is: a session that fails right after its headers, with zero bytes, while the reader is behind the tail still counts as a stall. So does a session that ends with a clean EOF the same way. Three of these in quick succession, for example during a brief server blip, would produce the hole error instead of retrying. A minimum session duration would tell these apart from the ~2 min hole sessions, but it adds a tunable. I didn't add it here.

Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

…ent holes A zero-byte session below the tail now counts toward the hole stall cap only if it stayed open for at least 10 seconds. A session at a hole is held open until the server's max response duration (about two minutes), so a session that ends sooner is a brief blip: it is retried against the general reconnect budget as before, instead of three of them failing the read within a second. Drop the WORKFLOW_FRAMED_STREAM_MAX_STALLS_BELOW_TAIL override and its docs entry; the stall cap is a constant. Add a regression test for two adjacent holes: resuming at index + 1 lands on the second hole, fails the same bounded way naming that index, and resuming past it reads the rest of the stream. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

VaguelySerious · 2026-11-13T19:53:14Z

I pushed two commits:

631d66fbd merges current main. It merged without conflicts. createReconnectingFramedStream and the live-read path are unchanged on main, and the two hole tests still fail against main's reader, so the change is still needed.
2d04a32af addresses the open feedback.

Quick zero-byte sessions (the leftover risk from my last comment). A zero-byte session below the tail now counts as a stall only if it stayed open for at least 10 seconds (FRAMED_STREAM_MIN_STALL_SESSION_MS). A session at a hole stays open until the server's max response duration (about 2 minutes), so this doesn't delay hole detection. A session that ends sooner is a blip: it neither counts toward the streak nor resets it, and it is retried against the general reconnect budget (50) as before. The check runs before the metadata lookup, so on an errored body a blip also skips the getInfo call. New test: nine quick zero-byte sessions at an index below the tail, then the chunk arrives, and the read completes. Without the rule, it fails after three.

Adjacent holes. A resume at index + 1 that lands on a second hole already fails the same bounded way: a new read has its own stall count, so it errors after 3 held sessions and names the new index (read from startIndex index + 2). It doesn't wait out the 50-session budget. The new test covers chunks 2 and 3 both missing. A consumer loop that resumes from the startIndex in each error gets errors at 2 and then 3, with 3 sessions at each, and then reads 4..5 to the end. Each hole costs at most 3 server sessions (about 6 minutes). Going lower would need a different probe, because the reader can only tell a hole from an idle wait once a session ends. I kept the error's shape: the message names the missing index, the tail and the startIndex to resume from. A typed field on StreamError would be a public @workflow/errors API addition, which I'd leave for a separate decision.

Configuration. I removed WORKFLOW_FRAMED_STREAM_MAX_STALLS_BELOW_TAIL and its runtime-tuning entry, so the PR no longer changes docs. The stall cap is a constant. Recent main (#5227) has been removing knobs that only guard against a default being wrong. With the duration rule, a false positive needs three consecutive zero-byte sessions of 10 seconds or more while the metadata already reports the chunk. The new duration threshold is a constant too, not an env var.

Smaller changes: the isVerifiedComplete doc comment had ended up on getStreamInfo and now sits on the right function. The hole tests drive a fake clock: scripted streams pull lazily, so a session's duration elapses after it opens. The fail-fast tests now assert exactly 3 sessions at the hole. I updated the PR description where it named the removed env var.

pnpm test and pnpm typecheck in packages/core pass, and so does biome check on the changed files.

CI on 2d04a32af: 192 checks pass, including E2E Required Check and Unit Tests (windows-latest). The one failure is E2E Python Conformance, which hit its 20-minute job timeout. That job has hit the same timeout on every main run since c11fc40 (#5235), the last 8 in a row (for example run 43489805104 on e7259b1), so it isn't caused by this PR.

…-ack

[core] Add failing test: live read retries a hole below the known tail

34d4673

Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

vercel Bot deployed to Preview – workbench-python-workflow September 23, 2026 19:24

View deployment

vercel Bot deployed to Preview – workflow-web September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-astro-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-express-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – example-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workflow-tarballs September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-hono-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-vite-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow September 23, 2026 19:25

View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow September 23, 2026 19:26

View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack September 23, 2026 19:26

View deployment

vercel Bot deployed to Preview – workflow-docs September 23, 2026 19:26

View deployment

vercel Bot deployed to Preview – workflow-swc-playground September 23, 2026 19:28

View deployment

[core] Fail a live stream read that stalls at a hole below the known …

e3278e0

…tail Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

vercel Bot deployed to Preview – workbench-python-workflow September 23, 2026 19:29

View deployment

vercel Bot deployed to Preview – workflow-web September 23, 2026 19:30

View deployment

vercel Bot deployed to Preview – workbench-hono-workflow September 23, 2026 19:30

View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow September 23, 2026 19:30

View deployment

vercel Bot deployed to Preview – example-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workbench-astro-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workflow-tarballs September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workbench-vite-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack September 23, 2026 19:31

View deployment

vercel Bot deployed to Preview – workflow-docs September 23, 2026 19:32

View deployment

vercel Bot deployed to Preview – workflow-swc-playground September 23, 2026 19:33

View deployment

pranaygp closed this Sep 23, 2026

pranaygp reopened this Sep 23, 2026

pranaygp mentioned this pull request Sep 23, 2026

Durable run streams lose acknowledged chunks, and the live read blocks forever at the hole #4306

Open

VaguelySerious changed the title ~~[core] Fix live stream read waiting forever at a missing chunk below the tail~~ [core] When live stream read waits forever at missing chunk, fail early instead Sep 24, 2026

Merge remote-tracking branch 'origin/main' into triage-workflow4306-d…

c9f20b9

…urable-run-streams-lose-ack Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

vercel Bot deployed to Preview – workflow-docs November 6, 2026 01:05

View deployment

vercel Bot deployed to Preview – workflow-docs November 6, 2026 01:09

View deployment

vercel Bot and others added 2 commits November 13, 2026 19:25

Merge remote-tracking branch 'origin/main' into pr-4334

631d66f

Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>

vercel Bot deployed to Preview – workflow-docs November 13, 2026 19:28

View deployment

Merge branch 'main' into triage-workflow4306-durable-run-streams-lose…

4e2cf45

…-ack

VaguelySerious marked this pull request as ready for review November 13, 2026 19:54

VaguelySerious requested a review from a team as a code owner November 13, 2026 19:54

vercel Bot deployed to Preview – workflow-docs November 13, 2026 19:57

View deployment

Navigation Menu

Repository navigation

[core] When live stream read waits forever at missing chunk, fail early instead - #4334

[core] When live stream read waits forever at missing chunk, fail early instead#4334
pranaygp wants to merge 7 commits into
mainfrom
triage-workflow4306-durable-run-streams-lose-ack

pranaygp commented Sep 23, 2026 •

edited by VaguelySerious

Loading

Uh oh!

changeset-bot Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 23, 2026 •

edited

Loading

`2d04a32`

`a86b822`

`e3278e0`

`e3278e0`

Uh oh!

github-actions Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

kvnloo commented Sep 23, 2026

Uh oh!

VaguelySerious commented Nov 6, 2026

Uh oh!

VaguelySerious commented Nov 13, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pranaygp commented Sep 23, 2026 • edited by VaguelySerious Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Tests

Uh oh!

changeset-bot Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

2d04a32

a86b822

e3278e0

e3278e0

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sim World

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kvnloo commented Sep 23, 2026

Uh oh!

VaguelySerious commented Nov 6, 2026

Uh oh!

VaguelySerious commented Nov 13, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pranaygp commented Sep 23, 2026 •

edited by VaguelySerious

Loading

changeset-bot Bot commented Sep 23, 2026 •

edited

Loading

vercel Bot commented Sep 23, 2026 •

edited

Loading

github-actions Bot commented Sep 23, 2026 •

edited

Loading

github-actions Bot commented Sep 23, 2026 •

edited

Loading

`2d04a32`

`a86b822`

`e3278e0`

`e3278e0`

github-actions Bot commented Sep 23, 2026 •

edited

Loading

github-actions Bot commented Sep 23, 2026 •

edited

Loading