Repository navigation
Step-dispatch latency grows linearly with accumulated run history (~0.3s per round per transition) — 65% of a 311-step run's wall time #3026
Description
We run @workflow/ai@5.0.0-beta.11 + workflow@5.0.0-beta.24 on Vercel (IAD1). On long agent runs, the time between step completions grows linearly with the number of accumulated steps, even for trivial bookkeeping steps.
Measurement (a 311-step, 92-minute agent run on a preview deployment, all numbers from step createdAt/startedAt/completedAt timestamps via the workflow API):
- Every step's own execution is fast: non-LLM steps complete in 0.1–0.5s once started; queue delay (startedAt − createdAt) ≈ 0 for all 311 steps.
- But the per-TRANSITION gap (previous step end → next step created) grows from ~1.9s avg (round 2) to ~16.6s avg (round 50). Linear fit across rounds: gap = 1.45s + 0.297s × round, R² = 0.94. With ~6 transitions per agent round that's ~96s of pure engine overhead per round by round 50.
- Budget: of 70.6 min of active run time, only 24.7 min (35%) is inside LLM steps; 45.9 min (65%) is inter-step time, 97.9% of which is not attributable to any step's execution or queue delay.
- The growth is uniform across step types — a 0.1s credit-deduction step pays the same ~15s transition cost as an LLM step by round 50 — which points at per-dispatch run-history replay/rehydration cost rather than anything payload-specific.
- Context: our conversation payloads are large (268K→589K tokens across the run), so O(history) work that re-reads/deserializes accumulated step results would scale badly here. We already strip the
request.bodyecho from doStreamStep returns via a local patch (the core of [ai] Reduce doStreamStep step boundary payload #1932 / DurableAgent/WorkflowAgent duplicates doStreamStep return size #1929) — this measurement is with that strip active. - History: we reported the same shape in April at much smaller magnitude (961ms→2,135ms over 77 steps) and [core] Combine flow+step bundle and process steps eagerly #1338 flattened it to 0.27–0.47s. It has returned at ~10× on longer runs.
Our own mitigation (collapsing per-round bookkeeping steps) reduced dispatches per round from 6–7 to 3, which helps proportionally — but the per-dispatch growth itself looks engine-side.
Question: is per-dispatch cost expected to scale with accumulated run history in the current scheduler, and is there a knob or planned fix? Happy to share the full 311-step timing table and run IDs privately.
Activity
NathanColosimo commentedon Jul 21, 2026
Step to step overhead shouldn't be 1.9s on the same run especially at the start (for example your 1.9s round 2)
Can you double check your otel?
Additionally, upgrade to the latest v5 beta version, this should help with large payloads, workflow@5.0.0-beta.24 is not the latest.
TooTallNate commentedon Jul 23, 2026
You'd probably be interested in RFC: VM Snapshot Runtime which intends to solve long-running / forever workflow runs.
kaijiabo commentedon Aug 4, 2026
Adding a second dataset on this. Same shape, different scale, and measured on newer versions than the original report — so the two open questions above (otel accuracy, and whether upgrading fixes it) both get an answer.
Setup:
workflow@5.0.0-beta.36,@workflow/core@5.0.0-beta.36,@ai-sdk/workflow@1.0.28, Vercel managed world, production. Two completed agent runs, 767 and 756 steps.Method
No otel. Every number below comes from the workflow API's own step records (
/v1/workflow/v2/runs/{runId}/steps, fully paginated), using the timestamps the engine writes:clientCreatedAt[i+1] − completedAt[i]— engine time between one step finishing and the next being createdstartedAt − clientCreatedAt— queue/dispatch timecompletedAt − startedAt— the step's own workSteps sorted by
clientCreatedAt. Gaps where the next step was created before the previous finished (overlapping parallel steps) are negative and excluded: 565 of 766 usable in run A, 578 of 755 in run B.On statistics: raw per-step gaps are heavy-tailed, so an OLS fit on raw points gives a misleadingly low R² (0.008 for i ≥ 100; 0.001 across all points) even though the trend is strong and monotonic. I'm therefore reporting Spearman rank correlation on the raw data — which assumes neither linearity nor normality — plus decile medians. Both are in the table so the result doesn't rest on a modelling choice.
Result
blobStorageBytesRun A gap medians across 7 equal index buckets (seconds): 0.51 · 0.78 · 1.05 · 1.51 · 2.13 · 2.39 · 3.07
Only the gap grows. Dispatch and duration are flat or very slightly decreasing across the same runs (|ρ| ≤ 0.10). So this isn't queue pressure and it isn't steps getting slower — the growth is entirely in engine time between steps. In run A that's 33.0 of 47.7 minutes of wall clock.
On the two questions raised above
"Step-to-step overhead shouldn't be 1.9s at round 2 — check your otel." That looks right, and this dataset supports the pushback. Measured from engine timestamps rather than otel, the early-run median gap is 0.5–0.6s, not 1.9s. The elevated baseline in the original report appears to be a measurement artifact. The growth, however, reproduces cleanly at ρ ≈ 0.85–0.90.
"Upgrade to the latest v5 beta, this should help with large payloads." We did — beta.24 → beta.36, five days before these runs. The growth persists at this scale. Worth noting the merged payload work (#1932/#1929) reduced the
doStreamStepreturn; the step input is unchanged.Possible mechanism
Between these two runs the fitted slope differs 1.94× and
blobStorageBytesdiffers 1.75×, in the same direction. That's two runs, so it's suggestive rather than conclusive — but it is consistent with per-dispatch cost scaling with accumulated journal size.A concrete contributor:
doStreamStep(conversationPrompt, …)takes the conversation as its first argument, so the full and growing prompt is journaled as step input on every model call — 78 and 80 times respectively in these runs. That signature is unchanged in@ai-sdk/workflow@1.0.50, and I couldn't find an existing issue covering the input side (as opposed to the return). Happy to open one separately if that's the right place for it.Questions
Run IDs, the full per-step timing tables, and the analysis script are available privately if useful.
shtefcs commentedon Sep 1, 2026
Some production data that might help quantify this. We run a coding agent on @ai-sdk/workflow 1.0.30 with workflow core 5.0.0-beta.42 (later beta.46), Vercel managed world, region iad1.
One real run: the actual conversation was 463KB. The journal for that run was 54.41MB, about 117x larger. We decrypted the step payloads to see where the bytes go:
Replay cost is what makes the size matter. We measured journal replay at roughly 0.5 to 0.7 seconds per MB on invocation re-entry. On long runs that compounds into minutes of pure re-reading.
Happy to share methodology. Everything above came from workflow inspect steps --decrypt on real runs plus timing the re-entry gaps from the event log.
rpinckne commentedon Sep 23, 2026
We hit the same shape: a linear per-transition gap on long
WorkflowAgentruns. It went from about 0.36 s to 3.7 s between the model step and the next tool step over an 837-step run, and the prepare-to-model hop grew too. We traced the largest part of it to retained-VM demotion, which we filed as #4316.WorkflowAgentpasses the provider model instance into its model step. AI SDK provider models implement[WORKFLOW_SERIALIZE], so dehydrating that argument records guest-code execution.getRetentionDecisionthen returnsserialization_executed_workflow_code, and the next inline pass is a full replay of the event log, once per model call. That gives O(steps) work per round, which matches the linear fit above.How to confirm on your run:
DEBUG=workflow:runtime:debug.serialization_executed_workflow_codeafter each model step.What worked for us: reducing the model argument to a plain
{ classId, data }envelope in the workflow body, and reviving it inside the step. Retention then held across model calls, and the per-call replay disappeared. The repro and details are in #4316.