Repository navigation
[archived] Performance Megathread for v5 beta #1930
Description
This issue gathers all known performance issues that are NOT due to edge-cases.
If your workflow performs well in most cases, but hits performance problems in some cases, please open a separate issue, and optionally link it here.
Current bottlenecks
- Every ~2 minutes we create a new invocation to stay ahead of serverless function timeouts. This could be tightened (Inline step execution optimization on Vercel doesn't make use of full function timeout #1927), will be fixed in Use max function duration for inline execution limit #2964.
- Using
Promise.allwith over 3 steps will still cause a full event log reload for all steps, delaying step processing by ~200ms-2s. - Passing a step output as the input for a subsequent step still duplicates the payloads. We should de-duplicate this intelligently.
- Fluid cold starts still affect workflows due to large bundlesize. We should reduce this where possible.
- Streams are still slower to flush than we'd like it to be. Could be sped up by using websockets ([core] Ack'd write-channel stream writes with framed-v2 writer markers #2731)
- Creating the VM every replay is expensive for long runs (fixed in perf(core): retain workflow VM across inline steps (primitives-gated) #3046)
- Time to first step is ~200ms, but could be reduced to ~100ms for the median case by eliminating the first network requests
Solved bottlenecks (as of 5.0.0-beta.36)
- Cross-region and multi-region deployments on Vercel were slow, which we fixed by introducing region-pinned runs and multi-region support in
>=5.0.0-beta.33. - Between every step, the workflow enqueues an event replay. This means invocation overhead, queue overhead, and initial state fetching overhead on each step call.
>=5.0.0-beta.5removed 90%+ of this overhead ([core] Combine flow+step bundle and process steps eagerly #1338) - Large network overhead for event log payload fetching was solved in
>=5.0.0-beta.16([world-vercel] Switch event endpoints to v4 wire format #2055) - DurableAgent/WorkflowAgent excessive step size overhead was fixed in DurableAgent/WorkflowAgent duplicates doStreamStep return size #1929
- Consider turning off
collectUIChunksif your steps are still too large
- Consider turning off
- Step input/output size was reduced by 80% in RFC: compress serialized payload refs — zstd (gzip fallback), specVersion 5 #2394
- Large bundlesize due to sourcemaps (Allow disabling step sourcemap with new
sourcemapoption in builders #1842) was fixed by defaulting sourcemaps to off - Stream performance was improved in
>=5.0.0-beta.36
A note on pre-releases
Performance improvements go through a QA phase before being released as 5.x..x-beta releases currently. If you want the latest improvements ahead of the beta release, considering using pre-releases (nightly build) by installing tarballs from https://workflow-tarballs.labs.vercel.dev/
Activity
schnitzel112 commentedon May 5, 2026
Quick question on the nightly tarballs from workflow-tarballs.labs.vercel.dev: are these considered safe enough to run against the hosted Vercel World, or are they currently only meant for self-hosted / other World backends?
Asking because the perf improvements would be relevant for us, but we're running on Vercel and don't want to hit incompatibilities between a nightly client and the prod World backend.
shtefcs commentedon May 5, 2026
I am wondering the same.
VaguelySerious commentedon May 9, 2026
@schnitzel112 @shtefcs Any given commit on
main(nightly tarballs) are extensively e2e tested, but we can't guarantee that any particular commit won't negatively affect your setup when you upgrade. We've seen e.g. some compile-time issues or framework-specific issues in the past. Skew protection on Vercel should limit the impact of these sorts of bugs.That said, if you try out a tarball for a specific commit and it works for you, it's very likely to keep working for you. Especially now for the v2 flow performance improvements, if you try running
https://workflow-tarballs-hdqkpb1dh.labs.vercel.dev/workflow.tgzand it works for you, you can keep staying on that without any upgrade/downgrade issues later.Otherwise, we do recommend using the
5.x.x-beta.xversion, or4.x.xif you need an SLA. We're prepping the above tarball to be5.0.0-beta.5in a few days.VaguelySerious commentedon May 22, 2026
Many stability improvements for the beta release just went out in
5.0.0-beta.7VaguelySerious commentedon Jun 24, 2026
We're now at
5.0.0-beta.24.Since
5.0.0-beta.7, we've shipped "turbo mode" (fast-path first execution), optimistic step execution payload compression, network multiplexing, and many smaller optimizations, which significantly speed up time to first step and time between steps1napz commentedon Jun 26, 2026
❤️❤️❤️
shtefcs commentedon Jul 5, 2026
Hi @VaguelySerious, checking in as a production user who has been running the durable v5 beta for several months. Our workload is multi-step coding agents, roughly 40 to 50 steps per run, with large conversation payloads.
Credit where it is due: the fixes so far have helped. Setting WORKFLOW_V2_TIMEOUT_MS closer to our function budget, plus turbo mode and the v4 event wire, measurably cut our per-invocation cycling. Thank you for that.
We are still feeling the remaining bottlenecks on real workloads, and we have been waiting a while, so it would genuinely help us plan if you could share rough status or timelines (even a loose "weeks versus next quarter" is fine) on these:
Snapshot Runtime (Snapshot Runtime: QuickJS WASM VM with snapshot/restore for workflow execution #1300). This is the big one for us. The per-boundary event fetch and re-hydration is the dominant steady cost on our long agent runs. Any sense of when the snapshot path might land, even behind a flag?
Payload compression in the World (RFC: compress serialized payload refs — zstd (gzip fallback), specVersion 5 #2394, WIP) and the payload de-dup / diff-only approach mentioned above (currently not in progress). Our step inputs and outputs carry the full conversation, so payload size is one of our biggest costs. Is compression close to landing, and is de-dup something you expect to pick up this cycle?
Event fetching and pagination (Event pagination is inefficient for long runs #1928), which compounds the above on longer runs.
Stream delivery lag (Stream event delivery to clients lags 2-3 minutes behind durable writes (steps finish in seconds) #2767). We still see the client fall a couple of minutes behind durable writes on longer runs.
We are not asking you to rush anything unsafe, and we are committed to this SDK for the long run. We just need enough visibility to decide what to build around versus what to wait for. Any concrete signal on ordering and rough timing would mean a lot. Thanks again for the continued work here.
VaguelySerious commentedon Jul 5, 2026
@shtefcs
shtefcs commentedon Jul 6, 2026
Follow-up with hard numbers, and an offer: we built and verified two fixes on our fork that we are happy to upstream as PRs if you want them.
Context: durable coding agent on workflow@5.0.0-beta.24 + @workflow/ai@5.0.0-beta.11 (DurableAgent), ~50-70 LLM rounds per build, large payloads. We decrypted full event logs (run-key API) for before/after runs of the same app build.
Every UIMessageChunk written to getWritable() gates on a durable-store round-trip of roughly 65-70ms. Models emitting 1-token deltas cap at ~14 chunks/s, so effective throughput was 12-14 tok/s wall against 78 tok/s of real provider inference. Measured on a ~70-round build:
The patch is a small TransformStream before .pipeTo(writable) in doStreamStep that merges consecutive same-id text/reasoning/tool-input deltas, with an 8KB size cap and a 200ms age cap so live viewers still get ~5 updates/s. Chunks carrying providerMetadata are never merged. blobStorageBytes stayed flat across the change, which isolates the entire effect to the stream-delta layer.
We measured provider-side dead-air stalls (zero chunks mid-stream) of 340s and, earlier, 17.7 minutes on a wedged gateway stream. Byte-level timeouts (undici bodyTimeout) cannot catch these because gateway keepalive bytes reset byte-idle timers while decoding produces no chunks. The same TransformStream now errors the pipe after 120s with zero decoded chunks (300s grace before the first chunk for long prefills), so the step retries in ~2min instead of riding to a coarse platform timeout.
All three currently live in a pnpm patch on @workflow/ai@5.0.0-beta.11 in production preview for us. If any of this is useful, tell us which target branch/package layout you want and we will open PRs (coalescing may overlap with your acked write-channel work in #2731, happy to align with that instead).
1napz commentedon Jul 7, 2026
feat
shtefcs commentedon Jul 8, 2026
Followed up after reading the beta.25 to beta.27 source, validating against our workload (durable coding agents, 50 to 70 LLM rounds per build, large payloads). Three things that might be useful. Glad to PR any of them.
1. Why the 10ms writeMulti batching does not help token streams. We traced this since it kept surprising us. The flushable pump awaits each
writer.writebefore reading the next chunk (flushable-stream.jsaround line 206), so when the flush timer fires the buffer holds exactly one chunk andwriteMulti(which needs length > 1) never runs. Throughput ends up around one write per (flush interval + storage round trip), roughly 14 writes/s in our traces, regardless of token rate. RaisingWORKFLOW_STREAM_FLUSH_INTERVAL_MSdoes not help because it just delays each single-chunk write. What worked for us is a small TransformStream beforepipeTothat merges consecutive same-id UI deltas into one write. That took one of our builds from 57.7 min to 16.2 min. If a native same-id coalescer is interesting, happy to send it.2. #1929 looks like it is still there in beta.13, just moved. The full prompt and tools JSON is still attached at
stream-text-iterator.jsaround line 556 (request: { body: JSON.stringify(...) }), which is about 350KB per step for us. We drop it with a patch, can PR the guard.3. Small one next to the #2699 invalid-input fix (which is great). An unknown tool name still throws before the recoverable path (
durable-agent.js, theif (!tool) throw). In a multi-agent setup a misrouted tool call kills the run after the retries. Returning an error-text tool-result there, same shape as #2699, makes misroutes self-correct instead. Tiny diff.Also, the compression and step-payload reduction you shipped are landing well for us. Appreciate the fast iteration.
VaguelySerious commentedon Jul 14, 2026
🚀 Multi-region support has been released with
5.0.0-beta.33and above, and runs will now start and run in the region of the function that created them, which should greatly networking overhead for anyone running workflows outside ofiad1/us-eastsaasjesus commentedon Jul 15, 2026
Durable stream write-per-chunk still dominates agent turns (post multi-region)
We're on
workflow@5.0.0-beta.34/@ai-sdk/workflow@1.0.0, app + runs infra1(multi-region default). Same agent on Trigger.dev is still meaningfully faster, so we instrumentedWorkflowAgentturns end-to-end.What we measured
For a triage turn that calls one tool (
list_tickets) then answers:doStreamStep#1 (reason + tool call)writeToolResultsWithStepBoundarydoStreamStep#2 (short text reply)Package-level pattern matches what we reported earlier in #2708: provider stream starts quickly, but awaiting
writer.write(part)for every model delta dominates step wall time. Multi-region (#1981 / beta.33+) reduced per-write RTT vsiad1, but the architecture is still one durable write per stream part.On a slow turn the durable stream contained ~230 parts, mostly:
reasoning-deltatool-input-deltaAt ~65–70ms per write that alone is ~15s of step time — independent of model latency.
Workaround that helped a lot
We patched
@ai-sdk/workflow'sdoStreamStepto coalesce consecutive same-id deltas beforewriter.write():text-delta/reasoning-delta(mergetext)tool-input-delta(mergedelta) — this was the big one; tool-arg streaming is extremely chattytype,id, and serializedproviderMetadatamatch (Azure attaches a constant per-blockitemId; refusing anyproviderMetadatamade reasoning coalescing a no-op)After deploy, the same shape of turn wrote 1 coalesced
tool-input-deltainstead of 133, and total delta parts for the whole turn dropped to ~7. Step 1 went 24s → ~6s.Downstream consumers already concatenate delta text, so stream semantics stayed intact; UI still updates progressively within the flush window.
What's still left (can't fix from app side)
getWritable()(or coalescing upstream in@ai-sdk/workflow/ world). Until then, chatty providers (tool-input-delta, reasoning summaries) will keep punishing WorkflowAgent vs non-durable runtimes.doStreamStep→executeWorkflowToolStep→writeToolResultsWithStepBoundary→ nextdoStreamStep, each with ~0.3–0.6s queue/replay overhead. A single tool round-trip still costs ~1–3s of pure workflow machinery even when the tool itself is ~1s.writeToolResultsWithStepBoundarycan be fat — ~2s in our latest run just to persist one small tool result (serialization / replay payload size? see DurableAgent notes in this megathread about large step I/O).doStreamSteptime is much closer to genuine model TTFT/generation (we still see ~8s for a short text step on a large-toolset prompt). Without writer diagnostics it's easy to mis-attribute that entirely to the model.Happy to share the patch / diag markers (
cx-dostream-writer-diag:rawParts,writtenParts,writeMs,maxWriteMs) if useful. Bottom line: multi-region helped; forWorkflowAgentstreaming, chunk-level durable writes + per-tool step hops are still the main gap vs Trigger.dev for the same agent/model.joeyhotz commentedon Jul 18, 2026
Hey guys, was running big burst tests and was finding some CPU hog, turns out this change improves CPU requirements by alot!
#2996
2 remaining items
shtefcs commentedon Jul 21, 2026
LFG
VaguelySerious commentedon Jul 22, 2026
@shtefcs @saasjesus @Siva271195 Stream improvements are live as
5.0.0-beta.36shtefcs commentedon Jul 22, 2026
Thanks for shipping this. We upgraded core to 5.0.0-beta.36 (with @workflow/world-postgres 5.0.0-beta.28) and measured it against our own app-side delta coalescer, the one I described earlier in this thread.
Result on our stack: the per-write cost is still there when app-side coalescing is off. We added a kill switch to our coalescer and ran the same prompt on the same build. With our coalescer on, a warm step wrote 6 batches for 16 chunks and the stream phase took about 290ms. With it off, 17 writes for 18 chunks and the stream phase took 1118ms. That is roughly 65 to 75ms per write, the same figure we measured before beta.36.
Reading the code, my guess is the new writeMulti batching only fills a batch when writes queue up while one is already in flight. Our stream is consumed by a sequential awaited pipe, so each write completes before the next chunk arrives and the batch size stays at 1. If that is right, token streams with small deltas never benefit unless the producer coalesces first.
Question: is the awaited pipe path expected to be covered by these improvements, or is app-side coalescing still the recommended approach for one-token deltas? Happy to share the full numbers. This was measured on the postgres world locally, so the hosted backend may behave differently.
Separate but related, we filed #3026 about per-dispatch overhead growing with run history. Would appreciate eyes on that one too.
karthikscale3 commentedon Jul 23, 2026
@shtefcs - Thanks! Your diagnosis is right. In beta.36, batching is currently path-dependent. The
getWritable()path uses our internal read-ahead pipe and can combine chunks that arrive while a write is in flight. A raw ReadableStream piped directly to the server-backed writable can still wait for every write to finish, leaving batches at one chunk. We’re going to fix this by moving the buffering into the underlying server writable so both paths behave consistently.In the meantime, could you run one comparison with your app-side coalescer disabled but route the token stream through getWritable()?
There’s no need to import flushablePipe() directly - getWritable() uses it internally. If possible, please share the resulting write count, chunks per write, and stream duration. That will confirm whether this path improves batching on your workload. If you are already using getWritable() this way, please share the relevant pipeline code, because that would indicate the internal batching path is not engaging as expected.
saasjesus commentedon Jul 27, 2026
Thanks @karthikscale3 for the
getWritable()batching ask — we re-measured onworkflow@5.0.0-beta.36, already writing throughgetWritable().We instrumented the stream writer path to count model parts vs durable
write()calls and accumulate write wall time (app-side delta coalescing can be disabled for A/B).Coalesce off (raw path → one
write()per part)≈ 0.4–1 ms per write on average. Step wall is dominated by model time, not durable write RTT. This is a big change vs the earlier ~65–70 ms/write behavior we saw before the
getWritable()/ beta.36 path.Coalesce on (same writer, merge consecutive text/reasoning/tool-input deltas)
App-side coalescing still cuts durable writes by roughly ~10×, but absolute write cost is already small either way.
Takeaway
On beta.36 +
getWritable(), the old per-chunk durable RTT tax looks largely fixed for our workload. Coalescing remains useful as write-count amortization, not as a necessity to make streaming usable. Happy to run any further experiment shape you want (different models, longer tool loops, etc.).shtefcs commentedon Jul 28, 2026
@karthikscale3 We are already on the getWritable() path, so this is the second case you mentioned, the one where the internal batching does not engage as expected. Details of our pipeline:
With our coalescer disabled that produced 17 writes for 18 chunks at roughly 65 to 75 ms per write (world postgres against a remote Neon database, so each unbatched insert pays the network round trip, which is why it is so visible for us).
Our working theory: because the writable crosses the use step boundary as a step argument, the object the step ends up writing to may not be the wrapper that carries your read ahead pipe, so writes stay strictly sequential and the batch never fills. saasjesus measured 0.4 to 1 ms per write on the same beta with getWritable, and the visible difference on our side is the step boundary crossing plus the postgres world.
Happy to test a build where the buffering lives in the underlying server writable, since that would cover this path too. Full numbers and the exact pipe code available if useful.
karthikscale3 commentedon Jul 28, 2026
@shtefcs @saasjesus
Thanks! this explains the discrepancy, and your are correct. I was treating “created through getWritable()” as sufficient, but the important detail is where the pipe actually executes. Because the writable crosses the "use step" serialization boundary, the write loop inside the step receives the underlying server-backed writable and uses native pipeTo(). In beta.36, each write() on that path still waits for the durable world operation, so your remote Postgres RTT is paid serially.
We’ve now merged #3078, which addresses this exact case. Buffering and group commit now live inside WorkflowServerWritableStream, below the workflow/step serialization boundary. A write() resolves when the chunk enters a bounded buffer, native pipeTo() can continue pulling, and accumulated chunks are sent through writeMulti(). Step completion and close still wait on a separate durability barrier, so this does not weaken durability. This is merged but not included in a published release yet. Once it is available, I will let you know and we’d appreciate the same A/B test. One expectation to clarify: the number of application-level write() calls may remain 18 for 18 parts; what should decrease is the number of network/database batches and the time each producer-facing write() remains blocked.
There is also a Postgres-specific follow-up: world-postgres.writeMulti() currently performs one batched INSERT but still sends one awaited Postgres notification per chunk. That means your app-side coalescer may continue to reduce total Neon work even after #3078, although it should no longer be required to avoid blocking the producer on every chunk. We should optimize that notification loop separately.
If possible, after the release please share:
That will distinguish the core path fix from the remaining Postgres notification cost. Also, #3026 is separate: #3078 addresses stream-write serialization and batching, not dispatch overhead growing with run history. We’ll track that independently.
shtefcs commentedon Jul 30, 2026
Following up with the measurements you asked for after #3078, plus one release question.
Numbers from a beta.37 run (so #3078 group commit active) with our app-side coalescer also on, kimi-k2.7 coding build, 22 sampled stream steps:
So with both layers active the producer is effectively never blocked on persistence. We kept our app-side coalescer for now since it still cuts row count on top of the group commit (your Postgres-notification point) and the same patch carries our mid-stream idle watchdog, but the native path is clearly doing its job.
One question: we are currently pinned to beta.35 because of the parallel-batch dormancy (#3183). TooTallNate's fix (#3198) merged but the beta dist-tag still points at 5.0.0-beta.37. Is a beta.38 with #3198 expected soon? We are holding an engine upgrade on our side for it and will re-run the #3183 repro plus this same A/B the day it ships.
saasjesus commentedon Jul 30, 2026
btw, while instrumenting TTFT on
workflow@5.0.0-beta.36+@ai-sdk/workflowwe hit three smaller things that might be worth a look:StepResult.performanceis never populated. Every step across every turn reports{ stepTimeMs: 0, responseTimeMs: 0, timeToFirstOutputMs: null, toolExecutionMs: {} }.timeToFirstOutputMsis the one field that separates model latency from orchestration without reconstructing it as a remainder, so it'd be a nice win ifstreamTextpopulates these and they're just getting dropped in the workflow wrapper.No aggregate
usage/finishReasonon the stream result. Plain AI SDK gives youtotalUsageandfinishReasonon the result;WorkflowAgentexposesstepsbut no aggregate, so callers sum per-step usage or reach foronEnd. Not hard once you know, but it isn't discoverable from the types — either exposing the aggregates or a line in the type docs would match the AI SDK surface.process.uptimediffers between the handler and'use step'. It works in the HTTP handler and throwsTypeError: process.uptime is not a functioninside a step. Same module, two sandboxes — so code shared between a route and a step can fail only on one path. Documenting whatprocessthe step sandbox provides would be enough.One positive while I'm here:
stream()→ first model step measured 0ms on every turn. No orchestration tax there at all.karthikscale3 commentedon Jul 31, 2026
@shtefcs - beta.38 is released now. Please try it out and let us know if you face any issues.
@saasjesus - Thanks for providing those details. Let me take a pass and get back to you on these.
shtefcs commentedon Aug 1, 2026
beta.38 validation results, two independent rounds on our production-shaped workload (durable coding agent, parking hooks, kimi builds, world "vercel"):
Dormancy (#3183): confirmed fixed, twice. Round 1: 0 of 17 parallel batches stalled across 5 runs, gaps 0.2 to 5.5s. Round 2: 0 of 24 batches (widest 7-wide), worst gap 7.8s. Versus beta.37's 5 of 6 batches stalled at 1446 to 1479s. Also measured beta.38 trending faster per stream step than beta.35 (5.25s vs 6.32s median on matched pairs).
Hook path (the #3198 determinism concern): we ran an interleaved A/B, 8 runs per arm, beta.38 vs its immediate parent commit on beta.35, matched prompts, park delays and resume styles, 11 to 13 hook lifecycles per arm. Zero corrupted event logs in both arms, Fisher p = 1.0. The only replay churn we saw (a duplicate step_started, run still completed) was on the beta.35 control, which runs against a beta.38-caused hypothesis. One caveat for precision: our control predates both #3139 and #3198, so this A/B bounds the whole beta.36 to beta.38 window, not #3198 alone.
We did observe exactly one CORRUPTED_EVENT_LOG (ReplayDivergenceError on step_created) on a hook-holding beta.38 run in round 1, at low concurrency (4 to 5 runs). It did not reproduce across 16 further runs and has no distinguishing replay signature, so we are treating it as unattributed noise for now, adopting beta.38, and adding monitoring that surfaces that errorCode loudly. If production shows it again we will come back with data.
VaguelySerious commentedon Oct 6, 2026
Workflow v5 is out of beta, and most performance issues have been resolved. We are continuing to work on performance in upcoming versions, and we're closing this thread to take new/separate github issues for any new performance issues
[-]Performance Megathread - Is your workflow run slow? Look here first[/-][+][archived] Performance Megathread for v5 beta[/+]