Repository navigation
[core] Don't download recorded step inputs on replay (resolveData: 'skip-step-inputs') - #4348
Merged
Merged
Conversation
Replay recomputes every step's arguments by re-running workflow code; a step's input is only ever read by step execution, from the step entity. Yet every replay read (the event list, the run_started / hook_received preloads, and inline sinceCursor deltas) transfers each step_created / step_started input. For a workflow passing growing state into its steps, those inputs are the quadratic part of the replay log. Add an advisory `omitStepInputs` to ListEventsParams and CreateEventParams. The runtime sets it on every replay read and on every write whose returned replay events feed the log; world-vercel sends it as a list query param and POST frame meta flag. A World (or server) that doesn't know it keeps returning the inputs, so this is compatible in both directions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
🦋 Changeset detectedLatest commit: 6d38f95 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (62 failed)python-node (62 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3816 | 62 | 739 | 4617 |
| ✅ 💻 Local Development | 4238 | 0 | 550 | 4788 |
| ✅ 📦 Local Production | 4238 | 0 | 550 | 4788 |
| ✅ 🐘 Local Postgres | 4238 | 0 | 550 | 4788 |
| ✅ 🪟 Windows | 340 | 0 | 2 | 342 |
| ✅ vercel-http-transport | 873 | 0 | 153 | 1026 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 591 | 0 | 93 | 684 |
| Total | 18361 | 62 | 2637 | 21060 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 141 | 0 | 30 |
| ✅ astro-quickjs | 141 | 0 | 30 |
| ✅ example-node | 141 | 0 | 30 |
| ✅ example-quickjs | 141 | 0 | 30 |
| ✅ express-node | 141 | 0 | 30 |
| ✅ express-quickjs | 141 | 0 | 30 |
| ✅ fastify-node | 141 | 0 | 30 |
| ✅ fastify-quickjs | 141 | 0 | 30 |
| ✅ hono-node | 141 | 0 | 30 |
| ✅ hono-quickjs | 141 | 0 | 30 |
| ✅ nest-node | 141 | 0 | 30 |
| ✅ nest-quickjs | 141 | 0 | 30 |
| ✅ nextjs-turbopack-node | 168 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 168 | 0 | 3 |
| ✅ nextjs-webpack-node | 168 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 168 | 0 | 3 |
| ✅ nitro-node | 141 | 0 | 30 |
| ✅ nitro-quickjs | 141 | 0 | 30 |
| ✅ nuxt-node | 141 | 0 | 30 |
| ✅ nuxt-quickjs | 141 | 0 | 30 |
| ❌ python-node | 4 | 62 | 105 |
| ✅ sveltekit-node | 160 | 0 | 11 |
| ✅ sveltekit-quickjs | 160 | 0 | 11 |
| ✅ tanstack-start-node | 141 | 0 | 30 |
| ✅ tanstack-start-quickjs | 141 | 0 | 30 |
| ✅ vite-node | 141 | 0 | 30 |
| ✅ vite-quickjs | 141 | 0 | 30 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 142 | 0 | 29 |
| ✅ astro-stable-quickjs | 142 | 0 | 29 |
| ✅ express-stable-node | 142 | 0 | 29 |
| ✅ express-stable-quickjs | 142 | 0 | 29 |
| ✅ fastify-stable-node | 142 | 0 | 29 |
| ✅ fastify-stable-quickjs | 142 | 0 | 29 |
| ✅ hono-stable-node | 142 | 0 | 29 |
| ✅ hono-stable-quickjs | 142 | 0 | 29 |
| ✅ nest-stable-node | 142 | 0 | 29 |
| ✅ nest-stable-quickjs | 142 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 170 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 170 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 170 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 170 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 170 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 170 | 0 | 1 |
| ✅ nitro-stable-node | 142 | 0 | 29 |
| ✅ nitro-stable-quickjs | 142 | 0 | 29 |
| ✅ nuxt-stable-node | 142 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 142 | 0 | 29 |
| ✅ sveltekit-stable-node | 161 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 161 | 0 | 10 |
| ✅ tanstack-start-node | 142 | 0 | 29 |
| ✅ tanstack-start-quickjs | 142 | 0 | 29 |
| ✅ vite-stable-node | 142 | 0 | 29 |
| ✅ vite-stable-quickjs | 142 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 142 | 0 | 29 |
| ✅ astro-stable-quickjs | 142 | 0 | 29 |
| ✅ express-stable-node | 142 | 0 | 29 |
| ✅ express-stable-quickjs | 142 | 0 | 29 |
| ✅ fastify-stable-node | 142 | 0 | 29 |
| ✅ fastify-stable-quickjs | 142 | 0 | 29 |
| ✅ hono-stable-node | 142 | 0 | 29 |
| ✅ hono-stable-quickjs | 142 | 0 | 29 |
| ✅ nest-stable-node | 142 | 0 | 29 |
| ✅ nest-stable-quickjs | 142 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 170 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 170 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 170 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 170 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 170 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 170 | 0 | 1 |
| ✅ nitro-stable-node | 142 | 0 | 29 |
| ✅ nitro-stable-quickjs | 142 | 0 | 29 |
| ✅ nuxt-stable-node | 142 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 142 | 0 | 29 |
| ✅ sveltekit-stable-node | 161 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 161 | 0 | 10 |
| ✅ tanstack-start-node | 142 | 0 | 29 |
| ✅ tanstack-start-quickjs | 142 | 0 | 29 |
| ✅ vite-stable-node | 142 | 0 | 29 |
| ✅ vite-stable-quickjs | 142 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 142 | 0 | 29 |
| ✅ astro-stable-quickjs | 142 | 0 | 29 |
| ✅ express-stable-node | 142 | 0 | 29 |
| ✅ express-stable-quickjs | 142 | 0 | 29 |
| ✅ fastify-stable-node | 142 | 0 | 29 |
| ✅ fastify-stable-quickjs | 142 | 0 | 29 |
| ✅ hono-stable-node | 142 | 0 | 29 |
| ✅ hono-stable-quickjs | 142 | 0 | 29 |
| ✅ nest-stable-node | 142 | 0 | 29 |
| ✅ nest-stable-quickjs | 142 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 170 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 170 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 170 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 170 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 170 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 170 | 0 | 1 |
| ✅ nitro-stable-node | 142 | 0 | 29 |
| ✅ nitro-stable-quickjs | 142 | 0 | 29 |
| ✅ nuxt-stable-node | 142 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 142 | 0 | 29 |
| ✅ sveltekit-stable-node | 161 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 161 | 0 | 10 |
| ✅ tanstack-start-node | 142 | 0 | 29 |
| ✅ tanstack-start-quickjs | 142 | 0 | 29 |
| ✅ vite-stable-node | 142 | 0 | 29 |
| ✅ vite-stable-quickjs | 142 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 170 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs | 170 | 0 | 1 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 141 | 0 | 30 |
| ✅ express | 141 | 0 | 30 |
| ✅ hono | 141 | 0 | 30 |
| ✅ nextjs-turbopack | 168 | 0 | 3 |
| ✅ nitro | 141 | 0 | 30 |
| ✅ vite | 141 | 0 | 30 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 141 | 0 | 30 |
| ✅ express | 141 | 0 | 30 |
| ✅ nextjs-turbopack | 168 | 0 | 3 |
| ✅ vite | 141 | 0 | 30 |
Contributor
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 147518ms → this run 185732ms (Δ +38214ms, +26%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (2)6c7dd46Thu, 24 Sep 2026 03:29:47 GMT · run logs
Streams
ead616eThu, 24 Sep 2026 01:39:58 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 42 total
Full trace: |
Contributor
About these numbersSizes are gzip; parentheses show the change against
|
Review feedback: omitting step inputs is a data-resolution choice, so make it one instead of a separate flag. - @workflow/world: `EventsResolveData` = `ResolveData | 'skip-step-inputs'` for `ListEventsParams.resolveData` and `CreateEventParams.resolveData` (on a create it applies to the returned event-log page only), plus `entityResolveData()` for code that passes create params on to entity reads. `omitStepInputs` is gone. - @workflow/world-vercel: lists send `remoteRefBehavior=skip-step-inputs`; creates send `eventsRemoteRefBehavior: 'skip-step-inputs'` in the POST meta, since a POST's `remoteRefBehavior` already selects how its created event and entities come back (lazy for most event types). A backend that predates the value rejects the list with a 400; the first rejection moves the process to `resolve` for that backend (kept on globalThis). - @workflow/core: the replay reads and delta-requesting writes ask for `resolveData: 'skip-step-inputs'`. - world-local / world-postgres / world-sim: accept the wider type; they return inputs as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
| '@workflow/world-postgres': patch | ||
| --- | ||
|
|
||
| Replay no longer downloads recorded step inputs. The runtime reads the event log with `resolveData: 'skip-step-inputs'`, a new event-read resolution mode in which the World may leave `input` out of `step_created` and `step_started` events. It uses the mode for the event list, the `run_started` and hook preloads, and inline deltas. Replay recomputes step arguments by re-running workflow code, and steps read their input from the step entity. For workflows that pass growing state into their steps, this removes the part of the replay transfer that grows quadratically. Worlds that don't support the mode keep returning the inputs, and world-vercel falls back to full resolution against a server that predates it. |
Member
There was a problem hiding this comment.
Suggested change
| Replay no longer downloads recorded step inputs. The runtime reads the event log with `resolveData: 'skip-step-inputs'`, a new event-read resolution mode in which the World may leave `input` out of `step_created` and `step_started` events. It uses the mode for the event list, the `run_started` and hook preloads, and inline deltas. Replay recomputes step arguments by re-running workflow code, and steps read their input from the step entity. For workflows that pass growing state into their steps, this removes the part of the replay transfer that grows quadratically. Worlds that don't support the mode keep returning the inputs, and world-vercel falls back to full resolution against a server that predates it. | |
| Added `resolveData: 'skip-step-inputs'` option, which directs the World to leave out `input` from `step_created` and `step_started` events. Replay recomputes step arguments by re-running workflow code, so it does not need to fetch step inputs. |
Contributor
Author
There was a problem hiding this comment.
Applied in 6d38f95, merged with the precision nit in the other thread on this line. It now says steps take their input from the step_started response or from memory. I also added that a World that doesn't implement the option must treat it as 'all'.
VaguelySerious
approved these changes
Sep 24, 2026
VaguelySerious
left a comment
VaguelySerious
left a comment
Member
There was a problem hiding this comment.
LGTM. I checked out 6c7dd46f and ran world-vercel events.test.ts (61 passed) and core retained-vm-loop, runtime/helpers and quickjs-entrypoint.preload (126 passed). I also read workflow-server#1039.
I independently confirmed the key assumption, that nothing in the runtime reads a step input from the replay log:
- no
eventData.inputreads on step events in core; ReplayPayloadCachenever prepares step inputs;- the resilient-dispatch re-ensure takes the input from the queue message;
- the runtime never calls
world.steps.get/list.
The step executor gets its input from the step_started response (still remoteRefBehavior: 'resolve' via eventsNeedingResolve) or from in-memory dehydrated bytes (lazy and pre-claimed inline starts). None of the notes below block merging.
| * that passes growing state into its steps those inputs are the part of the | ||
| * log that grows quadratically. A World that ignores it returns them. | ||
| */ | ||
| export type EventsResolveData = ResolveData | 'skip-step-inputs'; |
Member
There was a problem hiding this comment.
Contract risk for Worlds outside this repo. "A World that ignores it returns them" holds only if the World tests resolveData !== 'none'. A third-party World that tests === 'all' would treat 'skip-step-inputs' as 'none' and strip every payload, step results included, which breaks replay. One that validates against a ['none','all'] enum would reject the read. All the in-repo Worlds are fine. Could the doc here state the requirement explicitly (treat unknown values as 'all')? A world-testing conformance case for it would also help.
Contributor
Author
There was a problem hiding this comment.
Done in 6d38f95.
- Doc:
EventsResolveDatanow says a World must treat any value other than'none'as'all'. It spells out both failure modes (an=== 'all'test strips step results; a['none','all']enum rejects the read) and points toentityResolveData(). - Conformance: new world-testing case
resolveData 'skip-step-inputs' returns every payload but step inputs. It runs a three-step workflow and lists the log withalland withskip-step-inputs. It asserts the same events, and every payload other than step inputs byte-identical. Step inputs may be either unchanged or absent. - Proof it bites: I checked it against world-local patched to read anything but
'all'as'none'. The case fails, and already the run itself fails to replay.
| * The {@link ResolveData} an {@link EventsResolveData} asks for on everything | ||
| * other than an event-log page's step inputs: `'skip-step-inputs'` is `'all'`. | ||
| */ | ||
| export function entityResolveData<T extends EventsResolveData | undefined>( |
Member
There was a problem hiding this comment.
Nit: widening CreateEventParams.resolveData means any code that forwards create params to an entity read now has to remember entityResolveData(). You covered the legacy paths in world-local, world-postgres and world-sim; worth a one-line note on CreateEventParams.resolveData so future forwarding sites don't miss it.
Contributor
Author
There was a problem hiding this comment.
Added in 6d38f95: CreateEventParams.resolveData now says code that forwards create params to an entity read (runs, steps, hooks) must map them with entityResolveData() first.
| * On `globalThis` (see `globalSingleton`) so that every bundled copy of this | ||
| * module learns from one rejection instead of paying it once per copy. | ||
| */ | ||
| const skipStepInputsSupport = globalSingleton( |
Member
There was a problem hiding this comment.
Once a backend is remembered as unsupported, that lasts for the life of the process. A long-lived process that hits an old server instance during a rolling deploy keeps doing full reads after the server upgrades. It only costs performance and is never a correctness problem, but a TTL (or re-probing after N minutes) would be cheap.
Contributor
Author
There was a problem hiding this comment.
Done in 6d38f95. A rejection is now remembered for 10 minutes (SKIP_STEP_INPUTS_REPROBE_MS), and the next read after that asks for skip-step-inputs again. A new test pins the sequence: reject → resolve; within the window, resolve without asking; after it, skip-step-inputs again. I checked that the test fails with the TTL removed. With #1039 merged this path should now only matter for self-hosted or older backends.
| '@workflow/world-postgres': patch | ||
| --- | ||
|
|
||
| Replay no longer downloads recorded step inputs. The runtime reads the event log with `resolveData: 'skip-step-inputs'`, a new event-read resolution mode in which the World may leave `input` out of `step_created` and `step_started` events. It uses the mode for the event list, the `run_started` and hook preloads, and inline deltas. Replay recomputes step arguments by re-running workflow code, and steps read their input from the step entity. For workflows that pass growing state into their steps, this removes the part of the replay transfer that grows quadratically. Worlds that don't support the mode keep returning the inputs, and world-vercel falls back to full resolution against a server that predates it. |
Member
There was a problem hiding this comment.
Small precision nit: "steps read their input from the step entity" is true for queued execution, where the input comes back on the step_started POST response. Lazy and pre-claimed inline starts use the in-memory dehydrated bytes instead (lazyStepInput / preclaimedStart.step.input). Maybe "steps take their input from the step_started response or from memory, never from the replay log"?
Contributor
Author
There was a problem hiding this comment.
Fixed in 6d38f95: "steps take their input from the step_started response or from memory, never from the replay log". The EventsResolveData doc uses the same wording.
- Document that a World MUST treat any resolveData other than 'none' as 'all' (a `=== 'all'` test would strip step results on every replay), and add a world-testing conformance case: `skip-step-inputs` returns the same events as `all`, every payload except step inputs unchanged. A World with that bug fails it (the run itself fails to replay). - Note on CreateEventParams.resolveData that forwarding it to an entity read needs entityResolveData(). - Remember a backend's rejection of `skip-step-inputs` for 10 minutes rather than for the life of the process, so a process that met an old instance mid-deploy goes back to the cheaper read. - Changeset wording: steps take their input from the step_started response or from memory, never from the replay log. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Contributor
|
No backport to This is a replay-transfer performance optimization that introduces new public API surface: a new To override, re-run the Backport to stable workflow manually via |
This was referenced Sep 24, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replay never reads recorded step inputs. Workflow code recomputes every step's arguments when it re-runs, and a step's input is only read by step execution, which takes it from the step entity (
startResult.stepinstep-executor.ts, the pre-claimed batch'sstep, or in-memory bytes on the lazy inline path). Yet every replay read transfers eachstep_created/step_startedinput:run_startedandhook_receivedpreloadssinceCursordeltasFor a workflow that passes growing state into its steps (#4238), those inputs are the part of the replay log that grows quadratically. #3026 reports the resulting cost.
Following review, the omission is a data-resolution mode rather than a separate flag:
@workflow/worldEventsResolveData = ResolveData | 'skip-step-inputs'forListEventsParams.resolveDataandCreateEventParams.resolveData.'skip-step-inputs'is'all', except that the World may leaveinputout ofstep_created/step_startedevents.stepentity.entityResolveData()maps it to'all'for code that passes create params on to entity reads.'all'; world-local, world-postgres and world-sim do, and keep returning inputs. A new@workflow/world-testingconformance case checks this.@workflow/world-vercelremoteRefBehavior=skip-step-inputs.eventsRemoteRefBehavior: 'skip-step-inputs'in the v4 POST meta. That is the same enum, applied to the event-log page the POST returns: therun_startedpreload, the hook replay stream, the inline delta, and the skipped-slot page.remoteRefBehavior. That field already chooses how the created event and entities come back, and it is deliberatelylazyfor every event type exceptrun_created,run_startedandstep_started. Forcing it to resolve would echo everystep_completedresult back.@workflow/coreREPLAY_RESOLVE_DATA('skip-step-inputs'):loadWorkflowRunEventswithReplayDelta(); itsstep_startedwrite, whosestep.inputit reads, is untouchedWhat replay reads from these events:
hasCreatedEventonstep_created(step.ts) and readsownerMessageIdfromstep_started.markCreated.The web UI and CLI don't ask for the mode, so they keep reading inputs.
Compatibility
remoteRefBehavior=skip-step-inputson the list with400 {"error":"validation-error","details":[{"path":["remoteRefBehavior"],…}]}. world-vercel then retries that read once withresolveand remembers the backend for 10 minutes, onglobalThis, so a process that met an old instance mid-deploy goes back to the cheaper read afterwards. The replay costs what it did before instead of failing.eventsRemoteRefBehaviorPOST meta key (verified against workflow-servermain).Server PR
https://github.com/vercel/workflow-server/pull/1039 (merged)
CI notes
step-executor.tson cleanmainfail them too. Dev-mode step frames get their function names from source-map lookups that depend on that file's line offsets, and their positions are already wrong onmain. Details are in Local-dev step stack traces: wrong lines, and function names that depend on step-executor.ts line offsets #4352.step-executor.ts's layout unchanged by sharing awithReplayDelta()helper. The step-error tests pass locally for sveltekit, node and quickjs, withdev.test.tsrun first as in CI.createHook({ experimental_force: true })takes a held token over #4193. That PR raisedSPEC_VERSION_CURRENTto 8, and the Python SDK (vercel/vercel-py) accepts at most 7 (spec_version … less than or equal to 7). PRs based before feat(hooks):createHook({ experimental_force: true })takes a held token over #4193 pass, andmain's own Python jobs have been cancelled since then. It isn't related to this change.How did you test your changes?
@workflow/world-testing(world-local): 17 passed, including the newskip-step-inputscase. I checked that case fails against a World that reads anything but'all'as'none'.@workflow/corepnpm test: 2,670 passed.@workflow/world-vercel: 757 passed.@workflow/world-local: 598 passed.@workflow/world-sim: 71 passed.turbo typecheckacross all non-workbench packages: passed.retained-vm-loop.test.ts): runs a growing-state loop against a World that serves every replay event without step inputs when asked (list pages, therun_startedpreload, inline deltas) and keeps them on thestepentity. It covers inline deltas, list-only reads, and a fresh replay after a 412. Each case asserts the same result, identical step inputs, that stripping actually happened, and that every replay read and delta-requesting write asked forskip-step-inputs.quickjs-runtime.test.tscases already replaystep_createdevents with noinput.quickjs-entrypoint.preload.test.tsnow asserts that its fetches ask for the mode.events.test.ts):step_createdframe;mainreturns, and remembering the backend;🤖 Generated with Claude Code