Repository navigation
perf(core): don't re-publish step messages this invocation already published - #4099
Conversation
…blished On a fan-out that runs some steps inline, the orchestrator invocation publishes the queued siblings' step-execution messages, runs its inline steps, falls back into the replay loop, reloads the log, and its pending-step dispatch pass publishes every one of those messages again (measured 19-28 re-sends per pass on a 32-branch fan-out, ~400 ms after the originals). The queue dedupes them by idempotency key, but the sends still cost round-trips on the shared connection pool and hold the invocation open past its useful work. Track, per delivery, the correlation ids this invocation has already published a step message for (the suspension handler's resilient publishes plus the dispatch pass's own immediate enqueues) and skip the immediate re-enqueue for those on later passes, unless a step_retrying has been observed since (a new schedule). The set is invocation-scoped and never derived from the log: a step_created does not prove the message was ever sent, so a different delivery still re-enqueues unconditionally. Reported as a debug log and the `workflow.dispatch.republish_skipped` span attribute. The QuickJS engine already keeps the equivalent invocation-scoped `queuedStepIds` set, so it is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: a1d0ea6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 136104ms → this run 136383ms (Δ +279ms, +0%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3662 | 0 | 685 | 4347 |
| ✅ 💻 Local Development | 3922 | 0 | 586 | 4508 |
| ✅ 📦 Local Production | 3922 | 0 | 586 | 4508 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 586 | 4508 |
| ✅ 🪟 Windows | 320 | 0 | 2 | 322 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 74 | 142 |
| ✅ vercel-http-transport | 823 | 0 | 143 | 966 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 557 | 0 | 87 | 644 |
| Total | 17223 | 0 | 2749 | 19972 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 133 | 0 | 28 |
| ✅ astro-quickjs | 133 | 0 | 28 |
| ✅ example-node | 133 | 0 | 28 |
| ✅ example-quickjs | 133 | 0 | 28 |
| ✅ express-node | 133 | 0 | 28 |
| ✅ express-quickjs | 133 | 0 | 28 |
| ✅ fastify-node | 133 | 0 | 28 |
| ✅ fastify-quickjs | 133 | 0 | 28 |
| ✅ hono-node | 133 | 0 | 28 |
| ✅ hono-quickjs | 133 | 0 | 28 |
| ✅ nest-node | 133 | 0 | 28 |
| ✅ nest-quickjs | 133 | 0 | 28 |
| ✅ nextjs-turbopack-node | 158 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 158 | 0 | 3 |
| ✅ nextjs-webpack-node | 158 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 158 | 0 | 3 |
| ✅ nitro-node | 133 | 0 | 28 |
| ✅ nitro-quickjs | 133 | 0 | 28 |
| ✅ nuxt-node | 133 | 0 | 28 |
| ✅ nuxt-quickjs | 133 | 0 | 28 |
| ✅ python-node | 66 | 0 | 95 |
| ✅ sveltekit-node | 152 | 0 | 9 |
| ✅ sveltekit-quickjs | 152 | 0 | 9 |
| ✅ tanstack-start-node | 133 | 0 | 28 |
| ✅ tanstack-start-quickjs | 133 | 0 | 28 |
| ✅ vite-node | 133 | 0 | 28 |
| ✅ vite-quickjs | 133 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 27 |
| ✅ astro-stable-quickjs | 134 | 0 | 27 |
| ✅ express-stable-node | 134 | 0 | 27 |
| ✅ express-stable-quickjs | 134 | 0 | 27 |
| ✅ fastify-stable-node | 134 | 0 | 27 |
| ✅ fastify-stable-quickjs | 134 | 0 | 27 |
| ✅ hono-stable-node | 134 | 0 | 27 |
| ✅ hono-stable-quickjs | 134 | 0 | 27 |
| ✅ nest-stable-node | 134 | 0 | 27 |
| ✅ nest-stable-quickjs | 134 | 0 | 27 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 1 |
| ✅ nitro-stable-node | 134 | 0 | 27 |
| ✅ nitro-stable-quickjs | 134 | 0 | 27 |
| ✅ nuxt-stable-node | 134 | 0 | 27 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 27 |
| ✅ sveltekit-stable-node | 153 | 0 | 8 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 8 |
| ✅ tanstack-start-node | 134 | 0 | 27 |
| ✅ tanstack-start-quickjs | 134 | 0 | 27 |
| ✅ vite-stable-node | 134 | 0 | 27 |
| ✅ vite-stable-quickjs | 134 | 0 | 27 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 27 |
| ✅ astro-stable-quickjs | 134 | 0 | 27 |
| ✅ express-stable-node | 134 | 0 | 27 |
| ✅ express-stable-quickjs | 134 | 0 | 27 |
| ✅ fastify-stable-node | 134 | 0 | 27 |
| ✅ fastify-stable-quickjs | 134 | 0 | 27 |
| ✅ hono-stable-node | 134 | 0 | 27 |
| ✅ hono-stable-quickjs | 134 | 0 | 27 |
| ✅ nest-stable-node | 134 | 0 | 27 |
| ✅ nest-stable-quickjs | 134 | 0 | 27 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 1 |
| ✅ nitro-stable-node | 134 | 0 | 27 |
| ✅ nitro-stable-quickjs | 134 | 0 | 27 |
| ✅ nuxt-stable-node | 134 | 0 | 27 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 27 |
| ✅ sveltekit-stable-node | 153 | 0 | 8 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 8 |
| ✅ tanstack-start-node | 134 | 0 | 27 |
| ✅ tanstack-start-quickjs | 134 | 0 | 27 |
| ✅ vite-stable-node | 134 | 0 | 27 |
| ✅ vite-stable-quickjs | 134 | 0 | 27 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 27 |
| ✅ astro-stable-quickjs | 134 | 0 | 27 |
| ✅ express-stable-node | 134 | 0 | 27 |
| ✅ express-stable-quickjs | 134 | 0 | 27 |
| ✅ fastify-stable-node | 134 | 0 | 27 |
| ✅ fastify-stable-quickjs | 134 | 0 | 27 |
| ✅ hono-stable-node | 134 | 0 | 27 |
| ✅ hono-stable-quickjs | 134 | 0 | 27 |
| ✅ nest-stable-node | 134 | 0 | 27 |
| ✅ nest-stable-quickjs | 134 | 0 | 27 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 20 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 20 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 1 |
| ✅ nitro-stable-node | 134 | 0 | 27 |
| ✅ nitro-stable-quickjs | 134 | 0 | 27 |
| ✅ nuxt-stable-node | 134 | 0 | 27 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 27 |
| ✅ sveltekit-stable-node | 153 | 0 | 8 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 8 |
| ✅ tanstack-start-node | 134 | 0 | 27 |
| ✅ tanstack-start-quickjs | 134 | 0 | 27 |
| ✅ vite-stable-node | 134 | 0 | 27 |
| ✅ vite-stable-quickjs | 134 | 0 | 27 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 1 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 74 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 133 | 0 | 28 |
| ✅ express | 133 | 0 | 28 |
| ✅ hono | 133 | 0 | 28 |
| ✅ nextjs-turbopack | 158 | 0 | 3 |
| ✅ nitro | 133 | 0 | 28 |
| ✅ vite | 133 | 0 | 28 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 133 | 0 | 28 |
| ✅ express | 133 | 0 | 28 |
| ✅ nextjs-turbopack | 158 | 0 | 3 |
| ✅ vite | 133 | 0 | 28 |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
There was a problem hiding this comment.
🟡 Changes recommended
Resilient step publishes can be duplicated on early-exit paths because their IDs are seeded too late.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Improves workflow performance by preventing duplicate step-message publishes during replay passes.
Changes:
- Tracks invocation-scoped step publishes and skips redundant dispatches.
- Adds telemetry and regression tests.
- Adds a core patch changeset.
File summaries
| File | Summary |
|---|---|
packages/core/src/telemetry/semantic-conventions.ts |
Adds republish-suppression telemetry. |
packages/core/src/runtime.ts |
Implements publish tracking and deduplication. |
packages/core/src/runtime.test.ts |
Tests replay, retry, and resilient dispatch behavior. |
.changeset/dispatch-skip-republished.md |
Declares the core patch release. |
Review details
Suppressed comments (2)
packages/core/src/runtime.ts:4006
- This set is keyed only by
correlationId, but step-dispatch identity intentionally includesstepNamebecause a stale-snapshot restart can re-derive the same correlation ID for a different step (packages/core/src/runtime/helpers.ts:1158-1173).publishedStepCorrelationIdssurvivesrestartReplayInProcess, so if an earlier pass published step A and a corrected replay maps that ID to step B, this condition skips B's legitimate enqueue; the old A message can then be rejected by the step-name fence and B never runs. Track the same correlation-id/step-name identity when seeding and checking this set.
publishedStepCorrelationIds.has(
step.correlationId
) &&
step.sawRetrying !== true
packages/core/src/runtime.ts:4006
step.sawRetryingis a sticky flag:step.tssets it on the firststep_retryingand never clears it. After this invocation performs the first re-enqueue allowed by the carve-out, every later replay pass still seessawRetrying === true, so this condition keeps publishing the same step message and defeats the invocation-local deduplication for multi-pass retry flows. Track/consume the retry handoff separately from the permanent ownership flag so only a new retry schedule bypasses the skip.
if (
publishedStepCorrelationIds.has(
step.correlationId
) &&
step.sawRetrying !== true
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| for (const correlationId of suspensionResult.queuedStepCorrelationIds) { | ||
| publishedStepCorrelationIds.add(correlationId); | ||
| } |
There was a problem hiding this comment.
Good catch, this is real: the set was only seeded at the dispatch pass, so a pass that exited early via the hook-conflict, attribute-event or serialization-failure branch forgot the handler's resilient publishes and the next pass re-sent them. Fixed in #4128 by seeding immediately after handleSuspension returns, before any early exit, with a test that drives a hook_conflict pass and asserts one send (it sends twice without the fix).
|
No backport to This is a performance optimization ( To override, re-run the Backport to stable workflow manually via |
…eturns Follow-up to #4099. The invocation-scoped set of step correlation ids this delivery already published was seeded from the suspension handler's `queuedStepCorrelationIds` only at the dispatch pass, which runs after the hook-conflict, attribute-event and serialization-failure branches `continue` the replay loop in-process. The handler's resilient publishes on such a pass were therefore forgotten: on the next pass the step already existed, the handler no longer reported it, and the dispatch pass published its message a second time. Move the seeding to immediately after `handleSuspension` returns, before any early exit, and drop the dispatch-pass-only placement. Adds a test that drives a hook_conflict pass with resilient dispatch on and asserts the queued step's message is sent exactly once across the invocation (it sends twice without the fix). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Motivation (measured)
On a 32-branch fan-out (durabench, 2026-09-11, Datadog traces of 3 orchestrator invocations), the orchestrator invocation publishes the 29 queued steps' messages ~450-750 ms into the invocation, runs its 3 inline steps, then falls back into the replay loop, reloads the log, and its pending-step dispatch pass re-publishes 28 / 20 / 19 of those same steps ~400 ms after it first published them. VQS dedupes them by idempotency key, but the sends take 150-550 ms on the 8-connection HTTP/1.1 pool and hold the invocation open (one run acked at 1455 ms instead of ~1050 ms). Every redundant send came from the same invocation that had just published the step.
What changed
packages/core/src/runtime.ts: the queue-message handler now keeps an invocation-scoped set,publishedStepCorrelationIds, of the steps this delivery has already published a step-execution message for. It lives next tocontinuedHookIds(the closure that owns the replay loop), so it persists across replay passes within one delivery and dies with it. It is populated from:suspensionResult.queuedStepCorrelationIdson every pass (the suspension handler's resilient publishes), andIt is not populated by delayed backstop wakes (they target the run, not a step) or owned-recovery steps (executed in-process). On a later pass the dispatch pass skips the immediate re-enqueue of a pending step whose id is in the set, unless
step.sawRetryingis true (a retry handoff is a new schedule). The check sits before the TTR hand-off so a skipped step never consumes the resume-timing measurement.Skips are reported as a debug log (
republishesSkipped,loopIteration) and a new span attributeworkflow.dispatch.republish_skipped, set alongside the existingbackstop_wakes_armed/owned_recovery_stepsattributes.Safety argument
Only invocation-local knowledge is used. A
step_createdin the log proves the step was created, not that its message was ever sent (the publisher may have crashed between the two), so the set is never derived from the log. A different invocation (crash-recovery redelivery, a concurrent wake) still re-enqueues unconditionally, exactly as before. Within one invocation, every publish that seeds the set is joined before the delivery can ack (the dispatch join anddeferredBatchWork), so a publish that fails fails the delivery and the redelivery starts with an empty set. This is stated in a code comment on the set.QuickJS engine
packages/core/src/runtime/quickjs-entrypoint.tsalready has the equivalent: an invocation-scopedqueuedStepIdsset, seeded fromdispatchPendingOps'squeuedStepCidsand its own overflow publishes, which the later pending-step requeue pass consults (if (queuedStepIds.has(step.correlationId)) continue;). It is unchanged. One small asymmetry worth knowing: that engine skips unconditionally, without thesawRetryingcarve-out added here.Tests
Three tests added to the
workflowEntrypoint step-dispatch ack orderingsuite inpackages/core/src/runtime.test.ts, reusing its harness (one inline step, one queued step, a sleep, a stateful event log so the loop runs a second pass):stepIdsend for the whole invocation; assertsevents.listran at least twice). Fails onmainwith 2 sends.step_retryingobserved since is still re-enqueued (2 sends, same correlation id).WORKFLOW_RESILIENT_STEP_DISPATCH=1the suspension handler's own publish (carryingstepInput) counts as already published (1 send). Fails onmainwith 2 sends.pnpm vitest run src/runtime.test.ts(ack-ordering suite)cd packages/core && FORCE_COLOR=0 pnpm testtsc --noEmitinpackages/core🤖 Generated with Claude Code