Repository navigation
[world-vercel] Retry throttled stream WebSocket writes - #4381
Conversation
🦋 Changeset detectedLatest commit: a50ad75 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 131526ms → this run 207020ms (Δ +75494ms, +57%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (2)6c02f0eTue, 29 Sep 2026 22:21:52 GMT · run logs
Streams
8249f2aThu, 24 Sep 2026 20:24:32 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3904 | 0 | 875 | 4779 |
| ✅ 💻 Local Development | 4406 | 0 | 550 | 4956 |
| ✅ 📦 Local Production | 4406 | 0 | 550 | 4956 |
| ✅ 🐘 Local Postgres | 4406 | 0 | 550 | 4956 |
| ✅ 🪟 Windows | 342 | 0 | 12 | 354 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 84 | 152 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ✅ vercel-http-transport | 879 | 0 | 183 | 1062 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 595 | 0 | 113 | 708 |
| Total | 19033 | 0 | 2917 | 21950 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 142 | 0 | 35 |
| ✅ astro-quickjs | 142 | 0 | 35 |
| ✅ example-node | 142 | 0 | 35 |
| ✅ example-quickjs | 142 | 0 | 35 |
| ✅ express-node | 142 | 0 | 35 |
| ✅ express-quickjs | 142 | 0 | 35 |
| ✅ fastify-node | 142 | 0 | 35 |
| ✅ fastify-quickjs | 142 | 0 | 35 |
| ✅ hono-node | 142 | 0 | 35 |
| ✅ hono-quickjs | 142 | 0 | 35 |
| ✅ nest-node | 142 | 0 | 35 |
| ✅ nest-quickjs | 142 | 0 | 35 |
| ✅ nextjs-turbopack-node | 169 | 0 | 8 |
| ✅ nextjs-turbopack-quickjs | 169 | 0 | 8 |
| ✅ nextjs-webpack-node | 169 | 0 | 8 |
| ✅ nextjs-webpack-quickjs | 169 | 0 | 8 |
| ✅ nitro-node | 142 | 0 | 35 |
| ✅ nitro-quickjs | 142 | 0 | 35 |
| ✅ nuxt-node | 142 | 0 | 35 |
| ✅ nuxt-quickjs | 142 | 0 | 35 |
| ✅ python-node | 66 | 0 | 111 |
| ✅ sveltekit-node | 161 | 0 | 16 |
| ✅ sveltekit-quickjs | 161 | 0 | 16 |
| ✅ tanstack-start-node | 142 | 0 | 35 |
| ✅ tanstack-start-quickjs | 142 | 0 | 35 |
| ✅ vite-node | 142 | 0 | 35 |
| ✅ vite-quickjs | 142 | 0 | 35 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 148 | 0 | 29 |
| ✅ astro-stable-quickjs | 148 | 0 | 29 |
| ✅ express-stable-node | 148 | 0 | 29 |
| ✅ express-stable-quickjs | 148 | 0 | 29 |
| ✅ fastify-stable-node | 148 | 0 | 29 |
| ✅ fastify-stable-quickjs | 148 | 0 | 29 |
| ✅ hono-stable-node | 148 | 0 | 29 |
| ✅ hono-stable-quickjs | 148 | 0 | 29 |
| ✅ nest-stable-node | 148 | 0 | 29 |
| ✅ nest-stable-quickjs | 148 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 176 | 0 | 1 |
| ✅ nitro-stable-node | 148 | 0 | 29 |
| ✅ nitro-stable-quickjs | 148 | 0 | 29 |
| ✅ nuxt-stable-node | 148 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 148 | 0 | 29 |
| ✅ sveltekit-stable-node | 167 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 167 | 0 | 10 |
| ✅ tanstack-start-node | 148 | 0 | 29 |
| ✅ tanstack-start-quickjs | 148 | 0 | 29 |
| ✅ vite-stable-node | 148 | 0 | 29 |
| ✅ vite-stable-quickjs | 148 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 148 | 0 | 29 |
| ✅ astro-stable-quickjs | 148 | 0 | 29 |
| ✅ express-stable-node | 148 | 0 | 29 |
| ✅ express-stable-quickjs | 148 | 0 | 29 |
| ✅ fastify-stable-node | 148 | 0 | 29 |
| ✅ fastify-stable-quickjs | 148 | 0 | 29 |
| ✅ hono-stable-node | 148 | 0 | 29 |
| ✅ hono-stable-quickjs | 148 | 0 | 29 |
| ✅ nest-stable-node | 148 | 0 | 29 |
| ✅ nest-stable-quickjs | 148 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 176 | 0 | 1 |
| ✅ nitro-stable-node | 148 | 0 | 29 |
| ✅ nitro-stable-quickjs | 148 | 0 | 29 |
| ✅ nuxt-stable-node | 148 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 148 | 0 | 29 |
| ✅ sveltekit-stable-node | 167 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 167 | 0 | 10 |
| ✅ tanstack-start-node | 148 | 0 | 29 |
| ✅ tanstack-start-quickjs | 148 | 0 | 29 |
| ✅ vite-stable-node | 148 | 0 | 29 |
| ✅ vite-stable-quickjs | 148 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 148 | 0 | 29 |
| ✅ astro-stable-quickjs | 148 | 0 | 29 |
| ✅ express-stable-node | 148 | 0 | 29 |
| ✅ express-stable-quickjs | 148 | 0 | 29 |
| ✅ fastify-stable-node | 148 | 0 | 29 |
| ✅ fastify-stable-quickjs | 148 | 0 | 29 |
| ✅ hono-stable-node | 148 | 0 | 29 |
| ✅ hono-stable-quickjs | 148 | 0 | 29 |
| ✅ nest-stable-node | 148 | 0 | 29 |
| ✅ nest-stable-quickjs | 148 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 176 | 0 | 1 |
| ✅ nitro-stable-node | 148 | 0 | 29 |
| ✅ nitro-stable-quickjs | 148 | 0 | 29 |
| ✅ nuxt-stable-node | 148 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 148 | 0 | 29 |
| ✅ sveltekit-stable-node | 167 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 167 | 0 | 10 |
| ✅ tanstack-start-node | 148 | 0 | 29 |
| ✅ tanstack-start-quickjs | 148 | 0 | 29 |
| ✅ vite-stable-node | 148 | 0 | 29 |
| ✅ vite-stable-quickjs | 148 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 171 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 171 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 84 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 142 | 0 | 35 |
| ✅ express | 142 | 0 | 35 |
| ✅ hono | 142 | 0 | 35 |
| ✅ nextjs-turbopack | 169 | 0 | 8 |
| ✅ nitro | 142 | 0 | 35 |
| ✅ vite | 142 | 0 | 35 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 142 | 0 | 35 |
| ✅ express | 142 | 0 | 35 |
| ✅ nextjs-turbopack | 169 | 0 | 8 |
| ✅ vite | 142 | 0 | 35 |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 42 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
shalabhc
left a comment
shalabhc
left a comment
There was a problem hiding this comment.
In general do we have a policy on when we fall back to HTTP? Do we see long term always maintaining the HTTP fallback path?
Behavior changes flagged by agent below, with my comments below.
-
Declined writers now stop after one HTTP failure: this changes existing behavior (ws-stream-session.ts:303-305, :354-363).
Repro: upgrade declined, first writeHttp rejects with a 503.
On main: write fails, next write succeeds, close() succeeds (closeHttp called once).
On this PR: write fails, next write fails, close() fails, and closeHttp is never called.Is this intentional? Can we add a test?
-
A drain that arrives during a throttle wait drops the writer to HTTP for good (:500/:740-745, :846).
Repro: 429 with retryAfter: '1', then drain{max_duration}.
The resend throws NotSent because the mode is draining. fallbackToHttpBeforeSend then closes the socket from the client side with 1000, and every later write goes over HTTP.
Result: 1 socket, both writes over HTTP, closed: [[1000,'HTTP fallback before send']].
It's safe, but this is the normal drain-and-reconnect case, now losing the socket for good.
Suggested fix: when mode === 'draining' after the wait, await transportDecision and try again, instead of treating the resend as not sent.Seems like we shouldn't fallback to http here?
-
A TCP reset after a 429 fails the writer instead of falling back (:662-668).
Repro: 429, then error(ECONNRESET) and close 1006. The write rejects with ECONNRESET, and writeHttp is never called.
At that moment nothing is outstanding and the throttled request did not apply, so HTTP would be safe.
The 'error' handler after open calls failUnknown without checking throttledSocket. handleSocketClose is the only place that checks it.
The old-server contract is a clean close after the error. But an abrupt reset (for example, a proxy tearing down under load, which is exactly what 429s coincide with) takes the fatal path.
Suggested fix: in the post-open 'error' handler, when !this.pending && socket === this.throttledSocket, switch to HTTP. -
When the budget runs out after the socket has closed, the writer fails rather than trying HTTP (:1027-1040).
Repro: 429 with retryAfter: '60', then close. The write fails in about 1 ms without trying HTTP.
Maybe ok, but unsure if http has a separate retry allowance in which case an http retry may be fine?
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…l throttle waits on dispose Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…ins; hand them to HTTP on reset or budget exhaustion Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
6c02f0e to
a50ad75
Compare
shalabhc
left a comment
shalabhc
left a comment
There was a problem hiding this comment.
Approving but curious about your thoughts on this. Could be documented in the protocol spec.
In general do we have a policy on when we fall back to HTTP? Do we see long term always maintaining the HTTP fallback path?
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: no blocking issues
| * whose socket closed meanwhile, or one past the budget, throws | ||
| * StreamWsRequestNotSentError, which callers turn into an HTTP fallback. | ||
| */ | ||
| private async requestRetryingThrottle( |
There was a problem hiding this comment.
AI Review: Note
This conflicts with #4510, which adds a maxMessageBytes argument to request() and passes it from the write() loop. Git flags the import and protocol-comment hunks, but there is also a conflict it won't flag: after resolving those, write() passes a 5th argument to requestRetryingThrottle, which doesn't accept it. Whichever PR lands second has to add the parameter here and forward it to request().
I merged the two locally, forwarded the arg, and ran a few combined cases (not committed). All passed:
- a 429 on the second byte-split request resends only that request, with
chunkSeqadvanced past the first (acked) request and the same body - a
retryAfterover the budget on the second request sends only the unapplied tail to HTTP (not the acked first request), and later writes stay on HTTP without a reconnect - a throttle resend followed by a single chunk too large for one message sends that chunk over HTTP
So the two features compose once the argument is threaded through.
| * likewise never replayed. The whole WebSocket-message ceiling is deliberately | ||
| * closes from executing and poisons the writer. The exception is a correlated | ||
| * 429, which did not apply: a client with no other request outstanding may | ||
| * resend it after `retryAfter`. An unknown client outcome is never replayed. The whole WebSocket-message ceiling is deliberately |
There was a problem hiding this comment.
AI Review: Nit
This comment line wasn't re-wrapped after the insertion (about 130 columns).
| "@workflow/world-vercel": patch | ||
| --- | ||
|
|
||
| Retry throttled (429) stream WebSocket writes and closes after `Retry-After`, moving to HTTP when the socket closes or the wait budget runs out, and retry a close 5xx over HTTP instead of failing the writer. |
There was a problem hiding this comment.
AI Review: Nit
This ends up in the end-user changelog. Something like "Retry throttled stream WebSocket writes and closes" is enough. The HTTP handoff and close-5xx details belong in the docs, which already cover them.
|
No backport to The change builds entirely on the experimental To override, re-run the Backport to stable workflow manually via |
Summary & Motivation
A correlated 429 on
workflow-stream-ws/v1means the write never applied, so the session now waits out the server'sRetry-Afterand resends on the same socket (or its post-drain replacement), matching what the HTTP writer already does. Past the shared 30s throttle budget, or if the connection ends during the wait, the write moves to HTTP rather than failing the writer; a close that fails with a 5xx likewise retries over HTTP, since close is idempotent.Two behavior changes ride along: an HTTP write that fails after the session has settled on HTTP now fails the writer (its outcome is unknown, so no queued write may apply ahead of it), and a throttle wait is cancelled on dispose or poison instead of stalling until its timer fires.
Test Plan
Tests added.