Repository navigation
feat(streams): add WebSocket writer lifecycle - #3833
Conversation
🦋 Changeset detectedLatest commit: 89a1fe6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3662 | 0 | 685 | 4347 |
| ✅ 💻 Local Development | 3922 | 0 | 586 | 4508 |
| ✅ 📦 Local Production | 3922 | 0 | 586 | 4508 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 586 | 4508 |
| ✅ 🪟 Windows | 320 | 0 | 2 | 322 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 74 | 142 |
| ✅ vercel-http-transport | 823 | 0 | 143 | 966 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 557 | 0 | 87 | 644 |
| Total | 17223 | 0 | 2749 | 19972 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 133 | 0 | 28 |
| ✅ astro-quickjs | 133 | 0 | 28 |
| ✅ example-node | 133 | 0 | 28 |
| ✅ example-quickjs | 133 | 0 | 28 |
| ✅ express-node | 133 | 0 | 28 |
| ✅ express-quickjs | 133 | 0 | 28 |
| ✅ fastify-node | 133 | 0 | 28 |
| ✅ fastify-quickjs | 133 | 0 | 28 |
| ✅ hono-node | 133 | 0 | 28 |
| ✅ hono-quickjs | 133 | 0 | 28 |
| ✅ nest-node | 133 | 0 | 28 |
| ✅ nest-quickjs | 133 | 0 | 28 |
| ✅ nextjs-turbopack-node | 158 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 158 | 0 | 3 |
| ✅ nextjs-webpack-node | 158 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 158 | 0 | 3 |
| ✅ nitro-node | 133 | 0 | 28 |
| ✅ nitro-quickjs | 133 | 0 | 28 |
| ✅ nuxt-node | 133 | 0 | 28 |
| ✅ nuxt-quickjs | 133 | 0 | 28 |
| ✅ python-node | 66 | 0 | 95 |
| ✅ sveltekit-node | 152 | 0 | 9 |
| ✅ sveltekit-quickjs | 152 | 0 | 9 |
| ✅ tanstack-start-node | 133 | 0 | 28 |
| ✅ tanstack-start-quickjs | 133 | 0 | 28 |
| ✅ vite-node | 133 | 0 | 28 |
| ✅ vite-quickjs | 133 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 27 |
| ✅ astro-stable-quickjs | 134 | 0 | 27 |
| ✅ express-stable-node | 134 | 0 | 27 |
| ✅ express-stable-quickjs | 134 | 0 | 27 |
| ✅ fastify-stable-node | 134 | 0 | 27 |
| ✅ fastify-stable-quickjs | 134 | 0 | 27 |
| ✅ hono-stable-node | 134 | 0 | 27 |
| ✅ hono-stable-quickjs | 134 | 0 | 27 |
| ✅ nest-stable-node | 134 | 0 | 27 |
| ✅ nest-stable-quickjs | 134 | 0 | 27 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 1 |
| ✅ nitro-stable-node | 134 | 0 | 27 |
| ✅ nitro-stable-quickjs | 134 | 0 | 27 |
| ✅ nuxt-stable-node | 134 | 0 | 27 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 27 |
| ✅ sveltekit-stable-node | 153 | 0 | 8 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 8 |
| ✅ tanstack-start-node | 134 | 0 | 27 |
| ✅ tanstack-start-quickjs | 134 | 0 | 27 |
| ✅ vite-stable-node | 134 | 0 | 27 |
| ✅ vite-stable-quickjs | 134 | 0 | 27 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 27 |
| ✅ astro-stable-quickjs | 134 | 0 | 27 |
| ✅ express-stable-node | 134 | 0 | 27 |
| ✅ express-stable-quickjs | 134 | 0 | 27 |
| ✅ fastify-stable-node | 134 | 0 | 27 |
| ✅ fastify-stable-quickjs | 134 | 0 | 27 |
| ✅ hono-stable-node | 134 | 0 | 27 |
| ✅ hono-stable-quickjs | 134 | 0 | 27 |
| ✅ nest-stable-node | 134 | 0 | 27 |
| ✅ nest-stable-quickjs | 134 | 0 | 27 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 1 |
| ✅ nitro-stable-node | 134 | 0 | 27 |
| ✅ nitro-stable-quickjs | 134 | 0 | 27 |
| ✅ nuxt-stable-node | 134 | 0 | 27 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 27 |
| ✅ sveltekit-stable-node | 153 | 0 | 8 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 8 |
| ✅ tanstack-start-node | 134 | 0 | 27 |
| ✅ tanstack-start-quickjs | 134 | 0 | 27 |
| ✅ vite-stable-node | 134 | 0 | 27 |
| ✅ vite-stable-quickjs | 134 | 0 | 27 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 27 |
| ✅ astro-stable-quickjs | 134 | 0 | 27 |
| ✅ express-stable-node | 134 | 0 | 27 |
| ✅ express-stable-quickjs | 134 | 0 | 27 |
| ✅ fastify-stable-node | 134 | 0 | 27 |
| ✅ fastify-stable-quickjs | 134 | 0 | 27 |
| ✅ hono-stable-node | 134 | 0 | 27 |
| ✅ hono-stable-quickjs | 134 | 0 | 27 |
| ✅ nest-stable-node | 134 | 0 | 27 |
| ✅ nest-stable-quickjs | 134 | 0 | 27 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 1 |
| ✅ nitro-stable-node | 134 | 0 | 27 |
| ✅ nitro-stable-quickjs | 134 | 0 | 27 |
| ✅ nuxt-stable-node | 134 | 0 | 27 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 27 |
| ✅ sveltekit-stable-node | 153 | 0 | 8 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 8 |
| ✅ tanstack-start-node | 134 | 0 | 27 |
| ✅ tanstack-start-quickjs | 134 | 0 | 27 |
| ✅ vite-stable-node | 134 | 0 | 27 |
| ✅ vite-stable-quickjs | 134 | 0 | 27 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 1 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 74 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 133 | 0 | 28 |
| ✅ express | 133 | 0 | 28 |
| ✅ hono | 133 | 0 | 28 |
| ✅ nextjs-turbopack | 158 | 0 | 3 |
| ✅ nitro | 133 | 0 | 28 |
| ✅ vite | 133 | 0 | 28 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 133 | 0 | 28 |
| ✅ express | 133 | 0 | 28 |
| ✅ nextjs-turbopack | 158 | 0 | 3 |
| ✅ vite | 133 | 0 | 28 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 129960ms → this run 156942ms (Δ +26982ms, +21%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
ab87b64 to
9853758
Compare
9853758 to
064bde0
Compare
38f930a to
538cea5
Compare
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM, with some specific issues the agent found, noted below
| this.pending = undefined; | ||
| clearTimeout(pending.timer); | ||
| if (reply.type === 'close_ack') this.closeAcknowledged = true; | ||
| if (reply.type === 'error') { |
There was a problem hiding this comment.
AI Review: Note
Every correlated server error poisons the session permanently, with no distinction between a retryable rejection and a fatal one. A throttle response, an ambiguous-write 503, and a conflict all arrive as the same frame shape and all end the writer's life. parseStreamWsReply gives you reply.status right here and it is discarded except for the message string.
Over HTTP each write is independent, so a throttled write fails once and the writer stays usable. Over this transport it is terminal, which means an opted-in deployment behaves worse than the default one precisely under load. stream-ws-protocol-v1.ts already names the missing piece ("A future retry contract must add a dedicated machine-readable code"), so this is a known gap rather than an oversight, but it is the thing that decides when the gate can be turned on for anything but low-volume traffic. Even ahead of a machine-readable code, treating a 4xx-with-reqId as "this request was rejected, the connection is fine" and only poisoning on genuinely unknown outcomes would recover most of the resilience.
| private async handleMessage(raw: Uint8Array): Promise<void> { | ||
| try { | ||
| const frame = await decodeOne(raw); | ||
| const reply = parseStreamWsReply(frame.meta, frame.body); | ||
| const pending = this.pending; | ||
| if ( |
There was a problem hiding this comment.
AI Review: Note
This is where an error frame carrying no reqId becomes fatal, which the protocol schema does define ("An absent request id makes the error connection-fatal"). The consequence is worth spelling out, because it fires on an expected lifecycle event rather than a failure: a server that ends a connection because the captured token is about to expire has no way to say so except an uncorrelatable error frame, and this treats that as an unknown outcome.
I drove those exact bytes into a session with the writer idle:
later write rejected with: stream WebSocket reply cannot be correlated
writeHttp fallback calls: 0
close() rejected with: stream WebSocket reply cannot be correlated
closeHttp fallback calls: 0
So: the writer dies while doing nothing; close() also rejects and never falls back, meaning the stream is never marked done over either transport and a reader awaiting the terminal record gets nothing; and the server's own message, which is the one string that would tell an operator what happened, is thrown away in favour of "cannot be correlated".
At minimum, an uncorrelatable error frame should surface reply.status and reply.message in the poison error instead of the correlation failure. Better would be a drain-shaped frame in the response union so a scheduled teardown can be answered by reconnecting with a fresh token, which the docs in this PR already flag as a rollout prerequisite.
| this.assertUsable(); | ||
| const ws = this.socket; | ||
| if (this.mode !== 'ws' || !ws || ws.readyState !== 1) { |
There was a problem hiding this comment.
AI Review: Note
Both pre-send failures poison, though in both the outcome is unambiguous: nothing reached the wire, so a duplicate append cannot have happened and HTTP fallback is provably safe. The socket-not-open branch here, and the buildFrame throw just below it, are the two cases where the session's own justification for refusing to fall back does not apply.
The second one is reachable through ordinary input. A single chunk above the v1 per-chunk cap fails inside encodeStreamWsWriteRequest:
rejected with: stream WebSocket chunk is 10485761 bytes; maximum is 10485760
bytes actually sent: 0
writeHttp fallback calls: 0
and every later write on that writer, including small ones, is dead. The existing tests assert this poisoning deliberately so I take it as a decision rather than a miss, but it is the wrong side of the trade when the frame provably never left: the same oversized chunk over HTTP is one failed request with the writer intact.
| }, | ||
| async () => | ||
| new Promise<Record<string, unknown>>((resolve, reject) => { | ||
| const timer = setTimeout(() => { |
There was a problem hiding this comment.
AI Review: Note
getRequestTimeoutMs() defaults to 60s, and expiry here goes to failUnknown, so one slow write ends the writer for the remainder of its life. A single stalled backend call is a plausible way to reach that, and over HTTP it would be one slow request rather than a dead writer.
The reasoning is sound as far as it goes: after 60s of silence the outcome genuinely is unknown. But the blast radius is the whole writer rather than the one frame, and there is no separate, shorter liveness signal that would let the session distinguish "this request is slow" from "this socket is gone". A ping/pong or a server-side in-progress signal would let the timeout be about the connection rather than the request.
| // Clean idle infrastructure close: reconnect with the same writer identity | ||
| // and next writer-local sequence, but cap eager attempts so a draining | ||
| // server cannot create an open/close hot loop for the invocation lifetime. | ||
| if (this.idleReconnects >= MAX_IDLE_RECONNECTS) { |
There was a problem hiding this comment.
AI Review: Note
The reconnect path only covers the clean-idle case, and the session has no lifetime bound of its own, so it depends on the counterparty always closing cleanly and always doing so while nothing is in flight. Neither is guaranteed: the invocation serving a socket is recycled on its own schedule, and a writer that is actively streaming is far more likely to have a frame outstanding at that moment than to be idle. That path lands in failUnknown above, not here.
A client-side recycle, closing and reopening while idle at some interval comfortably inside the server's connection ceiling, would convert most of those unclean deaths into the clean-idle case this already handles well. As written, the longer a stream runs the more likely it is to end poisoned.
The three-attempt cap and the reuse of the same writer id are both right, and the comment explaining the hot-loop risk is a good one.
Separately, and not about this diff: #2731 is still open and implements this same feature with a different architecture (writer markers so the read side dedupes overlap, ack-evicted buffer, recycle inside the server ceiling). That design answered the retry problem by making a resend safe rather than forbidding it. Worth closing one of the two paths deliberately rather than by attrition.
| await this.writeHttp(chunks); | ||
| return; | ||
| } | ||
| for ( |
There was a problem hiding this comment.
AI Review: Nit
This split cannot trigger at default settings: STREAM_WS_V1_MAX_CHUNKS_PER_WRITE is MAX_CHUNKS_PER_STREAM_WRITE (1000) and core caps a group at MAX_CHUNKS_PER_BATCH, also 1000, so chunks.length never exceeds one batch unless WORKFLOW_STREAM_MAX_CHUNKS_PER_BATCH is raised above the wire cap. The test covers it by doing exactly that, which is the right way to test it.
Worth a comment saying so, because the loop reads as a live path and it is really a guard against the two caps drifting apart. Note also that the split is by count only, with no byte budget, so the total frame size a batch produces is unbounded by anything in this function.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
|
No backport to This is feature work: it implements the client half of the new To override, re-run the Backport to stable workflow manually via |
Summary & Motivation
Implements the client half of
workflow-stream-ws/v1behind the existing default-offWORKFLOW_STREAMS_TRANSPORT=wsgate, populating thecreateWriteSessionseam only when opted in.workflow.stream.ws.connectspan and each frame a synthesizedhttp POSTspan, so per-event tracing survives the non-fetchtransport.Test Plan
New unit tests cover the lifecycle, fallback, and poisoning paths; 598
@workflow/world-verceltests plus package build/typecheck and workspace lint/format pass. Root build/typecheck couldn't run locally (missing Rust toolchain for the unrelated@workflow/swc-plugin).