Repository navigation
resumeHook racing hook disposal is journaled after hook_disposed; the owner's replay then diverges into CorruptedEventLogError #2781
Description
Summary
A resumeHook that races the target hook's disposal can be accepted and journaled after hook_disposed in the owning run's event log. Once that sequence is in the log, every subsequent replay of the owning run diverges at the post-disposal hook_received event (REPLAY_DIVERGENCE), and after the 3 recovery replays the run fails with CorruptedEventLogError — a healthy, correctly-written workflow is permanently corrupted by a late duplicate resume it never asked for.
Resumes that arrive clearly after disposal has persisted are properly rejected. The defect is the window where the resume is accepted but ordered after the disposal in the journal: acceptance and replay disagree about whether a disposed hook can receive.
This is a third surface of the disposal-ordering inconsistency reported in #2777 (same-run recreate conflicts against a pending disposal) and #2778 (cross-run claim not released with disposal): claim validation, cross-run claim release, and resume acceptance/replay each observe hook_disposed at different points.
Versions observed
@workflow/core@5.0.0-beta.26@workflow/world-local@5.0.0-beta.22(viacreateLocalWorld; in-process handler,WORKFLOW_TURBO=0)- Node v24
Event log smoking gun
Owning run (a long-lived driver workflow). It created a per-turn control hook, consumed one payload from it, and disposed it; a duplicate resume from the sender landed in the disposal window:
evnt_…3TWA hook_received hook_…EAVT wrun_…:turn-control:0 ← consumed
evnt_…S56H hook_disposed hook_…EAVT wrun_…:turn-control:0
evnt_…S57W hook_received hook_…EAVT wrun_…:turn-control:0 ← accepted AFTER disposal
Every wake of this run afterwards logs:
[workflow-sdk] Workflow replay diverged; queueing a recovery replay before declaring the event log corrupted
code REPLAY_DIVERGENCE
divergenceEventId evnt_…S57W ← the post-disposal hook_received
divergenceCount 1 → 2 → 3
[workflow-sdk] Error while running workflow
CorruptedEventLogError
code CORRUPTED_EVENT_LOG
How the duplicate resume is produced
The duplicate came from an at-least-once re-executed step in the sending run: a wake landed while its resumeHook-calling step was in flight, the step was re-dispatched under the same correlation id, and both attempts resumed the token (see #2780). The receiver consumed the first payload, disposed the hook, and the second resume landed milliseconds later — inside the acceptance window.
Before we mitigated (by deferring the receiver's hook disposal until the sender's run has provably completed), this reproduced in roughly 40–50% of runs of our integration suite.
Standalone repro status
The duplicated-resume mechanism reproduces deterministically with #2780's shape (resumeStarts=2 every run). Landing the duplicate inside the acceptance window is timing-dependent; in a simplified two-workflow repro the duplicate consistently arrived after disposal had persisted and was correctly rejected, so we could not synthesize the corruption on demand. The invariant violation is visible in the journal regardless of how it is produced: once hook_received is ordered after hook_disposed for one hook, replay divergence is deterministic.
Expected
One consistent enforcement point: either the resume is rejected whenever it would be ordered after hook_disposed (acceptance validates against the same order replay will see), or replay treats a post-disposal hook_received as inert instead of diverging.
Actual
The resume is accepted and journaled after hook_disposed; replay then diverges at that event on every wake and the run escalates to CorruptedEventLogError.
Impact / workaround
Any pattern where the resuming side can retry/duplicate (which #2780 makes true for any step-issued resume) risks corrupting the receiving run around hook disposal. We work around it by deferring hook disposal until the producer of potential duplicates has completed (disposing turn-scoped hooks one turn late).
Related
- A hook_received while a step attempt is in flight re-dispatches the step; both attempts run concurrently in-process #2780 — the at-least-once step re-execution that produced the duplicate resume.
- Recreating a hook token after dispose() in the same run self-conflicts: suspension processing validates the new hook_created before the pending hook_disposed persists #2777, Hook token claim is not released in order with hook_disposed: a fast next-run claimant gets HookConflictError against an already-disposed hook #2778 — sibling disposal-ordering inconsistencies.
Activity
AndrewBarba commentedon Jul 7, 2026
Note on PR #2779 (fixes #2777/#2778): it narrows but does not close this issue.
The new dispose-lock check in
findLiveHookCreatedEventdoes eliminate one entry path — a late resume can no longer be accepted via an event-log rebuild resurrecting a hook that is mid-teardown. However:hook_disposedappend, and a resume that passed its liveness check before the lock was written can still journal itshook_receivedafterhook_disposed(acceptance is not re-validated at event-write time), andhook_received-after-hook_disposedis in the log, every replay still diverges at that event and escalates toCorruptedEventLogError.Either serializing resume acceptance against the dispose lock, or making replay treat post-disposal receipts as inert, would close it. #2780 (the at-least-once step re-execution that produces such duplicate resumes) is likewise out of #2779's scope.
Fix resumeHook racing hook disposal being journaled after hook_disposed
VaguelySerious commentedon Jul 7, 2026
(AI) Thanks for the detailed report — and for the precise follow-up on #2779's remaining window, which was accurate.
Reproduction. Like you, we couldn't land the duplicate inside the acceptance window on demand via timing alone. The window itself reproduces deterministically at the storage layer though: committing a hook's disposal mid-teardown (dispose lock written, entity not yet deleted — i.e. the disposer stalled or crashed between its first and second durable op) and then sending a resume showed the
hook_receivedbeing accepted and journaled, because acceptance only checked hook-entity existence and was not serialized against the disposer. That is exactly the sequence in your journal: a resume that passes its liveness check before the disposal begins can append afterhook_disposed.Related issues. #2777 / #2778 (sibling disposal-ordering bugs) were fixed by #2779. The duplicate-resume producer you identified — #2780's re-dispatch of an in-flight step — is a separate defect and remains open; this fix makes the receiving run immune to duplicate resumes around disposal regardless of how they are produced (queue redeliveries can duplicate resumes even after #2780 is fixed).
Fix. #2808 (CI green):
hook_receivedandhook_disposednow share the per-hook lock, and thehook_receivedwrite re-validates the durable dispose lock (at acceptance and again immediately before the event append). A resume that would order after the disposal is rejected withHookNotFoundError— the same behavior as one arriving after teardown finished. Acceptance now observes the same order replay will see.(runId, hookId).One honest caveat: this closes the acceptance side; a
hook_received-after-hook_disposedorder already journaled before this fix will still diverge on replay. Making replay treat post-disposal receipts as inert (which would also heal existing logs) is a reasonable follow-up we're considering alongside #2780.To see if this fix solves your issue, could you try to install the pre-release in your package.json? Just replace the
workflowdep like so:{ "dependencies": { "workflow": "https://workflow-tarballs-h1z1pblmn.labs.vercel.dev/workflow.tgz", "@workflow/world-local": "https://workflow-tarballs-h1z1pblmn.labs.vercel.dev/workflow-world-local.tgz" } }[world-local] Hold the victim's hook lock during a forced hook claim (#…