Repository navigation
A hook_received while a step attempt is in flight re-dispatches the step; both attempts run concurrently in-process #2780
Description
Summary
While a workflow is suspended awaiting a step, receiving a payload on an unrelated hook wakes the run, and the wake's replay re-dispatches the in-flight step under the same correlation id. The original attempt is still running in the same process and is not cancelled: both attempts execute the step body to completion, duplicating every side effect. Only one step_completed is recorded, so the duplication is invisible to the event log's completion accounting.
For steps whose side effects are externally visible — stream writes, resumeHook calls to other runs, third-party API calls — the duplicate is user-observable. In our case a duplicated resumeHook from a re-executed step also produced a resume landing on an already-disposed hook, corrupting the receiving run's replay (filed separately, see below).
At-least-once execution is expected for crash recovery. This report is about the live-process case: the runtime re-dispatches a step whose original attempt is demonstrably still running on the same event loop, and the trigger is any unrelated hook_received — routine traffic for hook-driven workflows.
Versions observed
@workflow/core@5.0.0-beta.26@workflow/world-local@5.0.0-beta.22(viacreateLocalWorld; in-process handler,WORKFLOW_TURBO=0)- Node v24
Reproduced 4/4 attempts with the minimal repro below.
Event log
One run; the step's side-effect log shows its body ran twice (starts=2):
step_started corr=step_…CN0S spikeSlowStep ← attempt 1 (with input)
step_created corr=step_…CN0S spikeSlowStep
hook_received corr=hook_… (unrelated wake token)
step_started corr=step_…CN0S spikeSlowStep ← attempt 2, same correlation id
step_completed corr=step_…CN0S spikeSlowStep ← one completion
Repro
import { appendFile } from "node:fs/promises";
import { createHook } from "workflow";
export async function spikeSlowStep(input: { logPath: string }) {
"use step";
await appendFile(input.logPath, "slow-step\n"); // side-effect marker
await new Promise((resolve) => setTimeout(resolve, 1500));
return "slow-done";
}
export async function wakeMidStepWorkflow(input: { token: string; logPath: string }) {
"use workflow";
const hook = createHook<{ n: number }>({ token: input.token });
const iterator = hook[Symbol.asyncIterator]();
const pending = iterator.next(); // unrelated wake source
await spikeSlowStep({ logPath: input.logPath }); // in flight when the wake lands
const first = await pending;
hook.dispose();
return { received: first.done ? -1 : first.value.n };
}Client:
const run = await start(wakeMidStepWorkflow, [{ token, logPath }]);
await waitForHookRegistered(token);
await sleep(400); // let the slow step start
await resumeHook(token, { n: 7 }); // unrelated wake, mid-step
await run.returnValue; // resolves fine: { received: 7 }
// logPath now contains "slow-step" twiceExpected
A step in created/started but not completed state whose attempt is still executing in the current process is recognized as pending during a wake's replay and not re-dispatched (or the superseded attempt is torn down before the new one starts, so at most one body's side effects land).
Actual
The step body executes twice concurrently in one process; all side effects flush from both attempts; one completion is recorded.
Impact / workaround
Any workflow that holds a hook open while running steps (the natural shape for cancellation, inbox, or control-channel patterns) gets duplicate step side effects under routine hook traffic. We had to (a) make cancellation-related step results pure markers, (b) move stream-emitting settle logic into a different run whose wake sources are quieter, and (c) add an in-process single-flight around the remaining emission step.
Related
- Recreating a hook token after dispose() in the same run self-conflicts: suspension processing validates the new hook_created before the pending hook_disposed persists #2777, Hook token claim is not released in order with hook_disposed: a fast next-run claimant gets HookConflictError against an already-disposed hook #2778 — hook-disposal ordering; same investigation.
- A duplicated
resumeHookfrom a re-executed step can land after the receiving run disposed the target hook, which is journaled and then corrupts that run's replay — filed separately.
Activity
robertyates commentedon Jul 8, 2026
Corroborating this from a production app (Next.js on Vercel, hosted world — not
world-local),@workflow/core@5.0.0-beta.26. Same mechanism, and I want to add the streaming-step angle because it makes the duplication user-visible and also defeats the very operation that triggers it.Our shape. One durable workflow run per chat turn. At turn start we construct a hook-backed
AbortControllerand use its token purely as a Stop channel: a/cancelAPI route callsresumeHook(abortToken, { aborted: true }), and the workflow readssignal.abortedat each agent step boundary (stopWhen) to end the loop. We deliberately do not thread the signal into the step args (that path hitsmergeAbortSignals'x instanceof AbortSignal, and in the workflow VMAbortSignalis a non-callable statics object → throws), so this is the "read.abortedat the boundary" variant rather than the "signal-in-step" variant from #2795.The catch: the run is stopped while a step is in flight by definition — Stop only fires against a
runningturn, and our in-flight step is a streaming LLM call that runs for 5–30s. So theresumeHookwake lands squarely mid-step, and per this issue the step is re-dispatched under the same correlation id with the original attempt still running.What that does to a streaming step specifically:
text-startfor an id that's already open,text-deltas arriving aftertext-end, two answers woven mid-sentence. Onestep_completedwins the journal, so the persisted transcript is clean, but the client renders the live stream mid-turn and shows the garble (and, in anask_uservariant, a "ghost" question card from the losing attempt's chunks that has no backing hook and can never be answered).finishReason: "stop", no "stopped" marker. The user clicked Stop; nothing stopped.Natural experiment (same run, back to back): a
/cancelthat returnedcancelled:false(didn't resume the hook — our Option-A gate) produced a clean single generation; a/cancel~2s in that returnedcancelled:true(resumed the hook mid-step) produced the interleaved duplicate. The wake is the trigger, reproducibly.Worth noting the wake source here is the run's own intended control hook, not unrelated traffic — so this hits the canonical "hook as cancellation signal" pattern head-on, which I think reinforces the reporter's point that this isn't just crash-recovery at-least-once.
Our mitigation matches the workarounds above: we're moving Stop off the hook channel entirely — the
/cancelroute writes a resettable flag to our own store, and the run reads it at its step boundary during normal execution, so nothing wakes the run and no second invocation is ever spawned. Happy to share more repro detail if it's useful.VaguelySerious commentedon Jul 9, 2026
Will have a solution for this soon
Backstop as delayed run continuation, off the step message's idempote…
Rewrite ownership changelog as a defense-of-design document
fix(e2e): shorten inline-ownership lease in mid-step restart test; do…
Remove unnecessary runtime and World env vars; update configuration d…