Repository navigation
Idempotency follow-ups: atomic keyed start(), post-completion dedupe, and attribute-based run lookup #2376
Description
Follow-up to the hook-based run idempotency work in #2015, #2373, and #2011. Those PRs ship the in-flight story: deterministic hook tokens as run idempotency keys, hook.getConflict() resolving with the conflicting Run, and code-driven conflict-handling strategies (reject, adopt result, inspect, signal via resumeHook(), supersede via cancel()). This issue tracks the structural gaps that remain before the idempotency story is rock-solid.
Where we stand vs. comparable frameworks
Idempotency is a lifecycle; each framework answers four questions:
| Phase | Temporal | Inngest | DBOS | Workflow (today) |
|---|---|---|---|---|
| Admission (dedupe atomic with start?) | Yes — workflow ID enforced server-side at start | Yes — idempotency key dedupes scheduling |
Yes — workflow ID is effectively a DB primary key | No — start() always creates a run; the claim happens inside the run body |
| In-flight conflict policy | Enum: Fail / UseExisting / TerminateExisting | None (first wins) | Implicit UseExisting | Code — getConflict() hands the duplicate the owner's Run |
| Post-completion memory | RejectDuplicate, bounded by namespace retention | Fixed 24h TTL per key | Durable record | None — hook released at terminal state |
| Result reuse | No (rejects; caller queries the closed run) | Yes, within window | Yes — same ID returns stored result | Only while the owner is running (conflict.returnValue) |
Where we're ahead: in-flight expressiveness. Policy-as-code beats a static enum ("inspect the owner's status, then decide" or "forward this request's payload to the owner" are inexpressible as configuration). And because the duplicate run is itself durable, the conflict-handling logic gets retries, replay, and observability — in admission-time systems that logic lives in a crashable client.
Where we're behind: admission atomicity and post-completion memory. Notably, no framework offers unbounded memory — Temporal's reject-duplicate is retention-bounded, Inngest is 24h — so the target is a retention window, not forever.
Gaps
- Admission isn't atomic (root cause of the rest). The duplicate run is created, billed, queued, and executed before discovering it's a duplicate, and routes need the resume-with-retry dance to bridge the
start()→ hook-registration window. The docs already note a native atomic start-and-hook-registration API is planned. - No post-completion dedupe. The hook is a lease and the lease dies with the run. A retried request arriving seconds after the owner completes starts fresh duplicate-sensitive work. (Temporal: retention-bounded reject; DBOS: returns the stored result.)
- Attributes are writable but not queryable. Runs can set (
experimental_setAttributes) and even seed (CreateWorkflowRunParams.attributes) attributes, butListWorkflowRunsParamsonly filters byworkflowName/status— nothing can find a run by attribute, so attributes can't yet serve as post-completion memory. - Optimistic-strategy races (acceptable, but inherent). Supersede (cancel-and-reclaim) can lose the reclaim to a third arrival (ABA-shaped; the documented retry loop handles it); signal-the-owner can hit
HookNotFoundErrorif the owner completes mid-forward. - Flat token namespace.
order:123collides across unrelated workflows sharing a key scheme. Same property as Temporal; a documented prefix convention is probably sufficient.
Recommendations (in order)
- P0 — atomic keyed
start()with{ run, created }return semantics and a retention-bounded uniqueness window. Closes gaps 1 and 2 at once and yields DBOS-style result reuse (created === false+ completed →run.returnValue) while keeping policies in code: no policy enum — the caller inspects the existing run and decides, which subsumes both of Temporal's knobs (conflict policy + reuse policy). - P1 —
runs.listattribute filtering in the World contract, then document the attribute pattern as the post-completion bridge: hook claim = in-flight mutex; attribute (idempotency: <key>) = retention-bounded memory; a duplicate that wins the token after the owner finished queries completed runs by attribute in a step and adopts the prior result viagetRun(prior.runId).returnValue. Must be documented as advisory (the query and subsequent work aren't atomic — residual race in the just-completed window) and retention-bounded. Stays useful after keyed start lands, for richer queries. - Document the two patterns that work today so users aren't stranded: the entity pattern (a long-lived run per key looping
for awaiton its hook — strict serialization at the cost of one perpetual run per key and deployment pinning) and the app-record pattern (storerunIdunder the domain key in your own DB inside a step; replays resolve viagetRun). - Non-goal: a standalone lease/TTL primitive or a
dispose: falsehook option. A claim that outlives its run is a leak generator; keyed start with a retention window does the same job with better semantics.
🤖 Generated with Claude Code
Activity
pranaygp commentedon Jun 12, 2026
Cross-referencing this against the patterns registry in #1858, which ships six coordination components (
semaphore,rate-limiter,circuit-breaker,debounce,batch-aggregator,singleton-run) pluskill-switch,child-workflows, andrecurring-cron— all built on hook channels and deterministic tokens. They turn out to be a useful stress test: every gap in this issue shows up as a concrete contortion or race in pattern code, and the patterns surface two gaps this issue doesn't cover yet (bottom).How the gaps manifest in the patterns
Gap 1 (admission isn't atomic) is the
sendWithEnsuredance, hand-rolled five times:resume(token)→ catch →start(coordinator)→setTimeout(500)→ retry ×3. The 500ms wall-clock sleep exists purely to bridge the start → hook-registration window. It also forces a design contortion: coordinators must take no initial payload viastart()args (the naturalstart(debounce, [key, firstPayload])is unsafe because the conflict-loser run would drop the payload), so first payloads travel via resume like everything else.Gap 1's cost side (duplicate runs are created/billed/queued before discovering they're duplicates) is the documented caveat in
singleton-run: "a failed duplicate withHookConflictErrorin the run list is the mechanism working, not a bug."getConflict()(#2373) converts that into a cleanreturn { dedupedTo: conflict.runId }— and awaiting it commits registration early, shrinking the same window the 500ms sleep papers over.Gap 2 (no post-completion dedupe) is live in the
stripedunning pattern: apayment_failedwebhook redelivery arriving after the run completed starts a fresh dunning timeline. The hook claim can't help — it died with the run.At-least-once
start()/resumeHook()from steps (adjacent to gap 1, worth naming explicitly):child-workflows' spawn step can start a child twice if the step crashes between thestart()call and its recording. The parent consumes one completion; the duplicate's side effects still ran. Keyedstart()withkey = completionTokenmakes spawns exactly-once.recurring-cron's continue-as-new step has the same duplication — which would fork the schedule into two parallel loops forever. It's only saved because the successor's stop-hook token is generation-keyed (cron:<name>:<iteration>), so the duplicate dies onHookConflictError. That a self-perpetuating duplication is prevented accidentally, by a token chosen for a different purpose, is a good argument for keyed start being P0.Per-pattern severity
start().signalnever firesgetConflict()now; keyed start /resumeOrStartproperlydispose()self-healing is sound, just adds latencyresumeOrStartremoves the danceTwo gaps to add
Channel close + drain. A coordinator that decides to terminate (e.g. the aggregator flushing at
MAX_ITEMS) has no way to atomically stop accepting resumes and then consume what's already buffered. Today an item resumed between the coordinator's lastawait hookand run termination is acknowledged to the sender and then silently dropped — the buffered payload queue dies with the run. The window is the full duration of the flush step.dispose()is close but inverted: it stops future events without exposing what's already buffered, and there's no non-blocking way to poll the buffer. Something likehook.close()→ subsequentresumeHookrejects (senders re-ensure) →hook.drain()/iterator-until-empty yields the already-accepted backlog.Idempotency keys on
resumeHook(). A send from inside a step is at-least-once: if the step crashes after the resume lands, the retry delivers the payload again, and channel-consuming patterns (aggregator counts, semaphore acquires) see duplicates. Userland mitigation is carrying a dedupe ID in the payload and deduping in coordinator state — which works but every consumer must remember to do it. AresumeHook(token, payload, { idempotencyKey })deduped server-side (retention-bounded, same window as keyed start) is the hook-delivery analog of the post-completion run dedupe this issue already tracks. The natural key exists for free in steps:stepId(+ a counter for multi-send steps), mirroring the documented step-idempotency guidance.On the proposed atomic
resumeOrStartEmphatic +1 — it's the single missing primitive behind five of the six coordinators; every
send*helper in #1858 is a racy userland approximation of it. One spec nuance from building those: it must deliver the payload exactly once whichever branch wins — a freshly started run receives the triggering payload as its first channel message, atomically with creation. That's strictly stronger than composing keyed-start + retry-resume in userland (which still has the registration window), and it's what lets coordinators accept initial payloads naturally again. Framing: keyedstart()is the admission face,resumeOrStartis the delivery face of the same primitive.🤖 Generated with Claude Code
docs(patterns): idempotency hardening from the #2376 analysis
pranaygp commentedon Jun 12, 2026
Empirical reproduction: CI hit the gap-1 × gap-2 interaction on every slow runtime
#2387 (e2e coverage for the documented conflict-handling strategies) accidentally produced a measured reproduction of gaps 1 and 2 interacting, in its first CI run (
27444182929): the adopt-the-owner's-result test timed out at exactly 90s on every slow matrix entry — nextjs-turbopack/webpack (all variants), sveltekit, nuxt, astro local-dev, and both postgres jobs — while passing locally every time.The trace:
start(owner)→ owner registers the hook (verified) →start(duplicate)→ test resumes the owner once.start()time (gap 1) — finds the token free, and becomes a legitimate fresh owner.Two takeaways worth recording here:
resumeOrStartmust deliver the triggering payload exactly once whichever branch wins. The CI hang is precisely the failure of the userland composition — the payload was delivered to the admission-time owner, while a different run won execution-time ownership. The same comment'ssendWithEnsure500ms-sleep contortion is the workaround for this exact window; CI just measured the window being wider than any constant sleep.The test was fixed in 52370fb by gating the owner's completion on the duplicate's recorded
hook_conflictevent — correct for a test (it verifies the conflict-handling branch, so it must arrange the interleaving that reaches it), but a production application cannot arrange interleavings. Until keyedstart()/resumeOrStartexist, the docs' guidance stands: pair workflow-side claims with route-side resume-with-retry (self-healing for the strand), or pass inputs viastart()args and use the hook claim-only.🤖 Generated with Claude Code
fix(hooks): dedup cost-tracker's metrics append on a repeated Stop (GH …
Add start({ hook }) atomic admission for idempotent starts (#4996)
pranaygp commentedon Nov 2, 2026
Status update: start-time Hooks and retention have landed
#4996 added
start({ hook })and #5156 enabled it on the single-orchestrator (spec 9) engine. Here's where each gap stands.Done
start(workflow, args, { hook: { token } })admits the run and its start Hook in one step. A duplicate throwsHookConflictError(kind: 'start',conflictingRunId) and never becomes a run, so it isn't created, queued or billed. The start Hook acceptsresumeHook()as soon asstart()returns, so the resume-with-retry /setTimeout(500)workaround isn't needed. Whenstart()can't confirm the outcome, it throwsWorkflowStartError, and retrying with the same token is safe. Docs: Run idempotency,start()→ Prevent duplicate starts.experimental_minRetentiononstart({ hook })(orcreateHook()) keeps refusing duplicate starts after the run ends, until the retention passes. To reuse the result, callgetRun(error.conflictingRunId).returnValue. There is no default retention. Docs: Keep the token after the run ends,createHook()minimum retention.start()args, and an existing owner gets it through its start Hook. See Start or route to the owner.The shape differs from the P0 proposal in one way: there is no
{ run, created }return. A duplicate is signalled byHookConflictErrorcarrying the owner's run ID.What's left
WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK=1and the Vercel World needsWORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK=1(World support). After that, drop theexperimental_prefix fromminRetention.hookisn't supported for dynamic workflows yet.runs.listattribute filtering.ListWorkflowRunsParamsstill filters only byworkflowNameandstatus. Retention now covers a bounded post-completion window, so this matters less, but it's still needed to remember runs beyond the retention window and for richer queries.resumeHook(). A retriedresumeHook()creates a newresumeIdand can append a secondhook_received(resumeHook()). Consumers still have to carry and dedupe their own request key.await hookcan still drop a payload it already acknowledged.resumeOrStart. This is now possible in userland with the pattern above, which takes at most two calls while the owner's Hook is open. A single built-in call would be a convenience rather than a correctness fix, so this may no longer need to be its own item.VaguelySerious commentedon Nov 13, 2026
(AI) Closing. Atomic keyed
start()(gap 1) and post-completion dedupe (gap 2) shipped asstart({ experimental_hook })andexperimental_minRetention(#4996, #5156, #5238), and the resume-or-start pattern now works without a registration window. The remaining open items (stable start Hooks plus dynamic-workflow support,runs.listattribute filtering,resumeHook()idempotency keys, hook close/drain) are tracked in #5424.