Skip to content

Add start({ hook }) atomic admission for idempotent starts - #4996

Merged
alangenfeld merged 3 commits into
mainfrom
alangenfeld/2026-10-21-run-idempotency-sdk-prototype
Oct 30, 2026
Merged

alangenfeld merged 3 commits into
mainfrom
alangenfeld/2026-10-21-run-idempotency-sdk-prototype

Conversation

@alangenfeld

@alangenfeld alangenfeld commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary & Motivation

start() used to create a new run on every call, so a workflow could only spot a duplicate after the duplicate run already existed. Now a start can carry a key: the World admits the run and creates its hook in the same atomic step. A second start with the same key is refused, gets the owner's run ID back, and never becomes a run.

Workflows need to start at most once per key, and their hook needs to accept payloads from the moment the run is admitted.

  • Idempotent start. start(workflow, args, { hook: { token, metadata?, experimental_minRetention? } }) admits the run together with its start Hook (startHookId(runId)). A duplicate throws HookConflictError with kind: 'start' and conflictingRunId, and none of its code runs.
  • Resumable from admission. Before, resumeHook() failed until the workflow body called createHook(), so callers had to retry. Now the start Hook accepts resumes as soon as start() returns. In the workflow, createHook({ token }) binds to the start Hook without a write, and any earlier payloads are delivered in order.
  • Token lifetime. The run holds the token until it ends, then for any experimental_minRetention. There is no default retention. Disposing the start Hook releases the token at once.
  • Unconfirmed outcome. When start() can't confirm that its run was queued or admitted, it asks the World again with the same candidate. If it still can't confirm, it throws the new WorkflowStartError with the candidate's runId and stage, and retrying with the same token is safe. cancel() of a candidate that never became a run stops it from ever being created.
  • Postgres token claims. Before, an admission and a concurrent createHook() could both win the same token. Now every Hook row claims its token through a unique index (migration 0028_add_start_hook_admissions), so only one can. The migration is required even with the flag off.
  • Single-orchestrator runs (spec 9). start({ hook }) throws a contract error before it publishes anything, because that engine isn't wired for start Hooks yet.
  • Observability. The dashboard and CLI show the start Hook token and revive WorkflowStartError. CLI re-runs don't forward the source run's start Hook. Admission outcomes and rejection reasons are traced.

This addresses #2376 and #85 and carries forward #3425, #3426, #3449 and #3440. #5156 wires start Hooks into the single-orchestrator engine and lifts the spec 9 refusal.

Rollout: always on for the Local World. On the Postgres World it's behind WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK=1, which the CI Postgres lane sets. On the Vercel World it's behind WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK=1, which needs a backend that supports start Hooks.

Test Plan

Added unit tests, a shared World conformance suite and a 19-test start({ hook }) e2e block. Locally, typecheck and the unit suites pass for the affected packages. The only failure is a Postgres listen-reconnect test that needs Docker-managed Postgres. The e2e block passes on nextjs-turbopack against the Local and Postgres Worlds, on both the Node and QuickJS VMs. A run that forced the Vercel capability on passed it on the prod, WS and HTTP lanes against the production backend.

@vercel

vercel Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
example-nextjs-workflow-webpack Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
example-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-astro-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-express-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-fastify-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-hono-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-nestjs-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-nitro-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-nuxt-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-python-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-sveltekit-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-tanstack-start-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workbench-vite-workflow Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workflow-docs Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workflow-swc-playground Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workflow-tarballs Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC
workflow-web Ready Ready Preview, v0 Oct 30, 2026 6:27pm UTC

@changeset-bot

changeset-bot Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9181a8f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 21 packages
Name Type
@workflow/world Minor
@workflow/errors Minor
@workflow/core Minor
workflow Minor
@workflow/web-shared Patch
@workflow/cli Patch
@workflow/world-local Minor
@workflow/world-postgres Minor
@workflow/world-vercel Minor
@workflow/vitest Patch
@workflow/web Patch
@workflow/world-testing Patch
@workflow/builders Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 45 total
scenario outcome events virt replay violations
✅ smoke-no-steps completed 3 0ms ok 0
✅ smoke-one-step completed 5 0ms ok 0
✅ hook-at-step-started completed 9 0ms ok 0
✅ hook-at-step-completed completed 9 0ms ok 0
✅ hook-at-hook-created completed 9 0ms ok 0
✅ claim-hook-at-registration completed 9 0ms ok 0
✅ claim-hook-at-step-started completed 9 0ms ok 0
✅ deadline-hook-wins completed 6 1.0h ok 0
✅ deadline-expires completed 6 1.0h ok 0
✅ step-vs-timer-early-settlement completed 6 1.0h ok 0
✅ long-sleep completed 9 30.0d ok 0
✅ hook-never-arrives stalled 3 0ms skipped 0
✅ step-retries-twice completed 9 2.0s ok 0
✅ parallel-steps completed 7 0ms ok 0
✅ hook-on-execution-state completed 9 0ms ok 0
✅ peek-hook-before-branch completed 9 0ms ok 0
✅ peek-hook-after-branch completed 9 0ms ok 0
✅ peek-hook-at-registration completed 9 0ms ok 0
✅ race-hook-before-probe completed 9 0ms ok 0
✅ race-hook-after-probe completed 9 0ms ok 0
✅ race-duplicate-delivery completed 10 0ms ok 0
✅ attr-hook-before-step completed 9 0ms ok 0
✅ attr-hook-after-step completed 9 0ms ok 0
✅ attr-from-step-body completed 10 0ms ok 0
✅ fork-hook-after-timeout completed 11 1.0m ok 0
✅ fork-hook-before-timeout completed 11 1.0m ok 0
✅ count-hook-after-timeout completed 13 1.0m ok 0
✅ count-hook-before-timeout completed 15 1.0m ok 0
✅ stale-read-step-count-fork completed 15 1.0m ok 0
✅ stale-read-equal-step-counts completed 11 1.0m ok 0
✅ step-vs-step-fork completed 9 0ms ok 0
✅ step-vs-step-fork-fenced completed 9 1.0s ok 0
✅ fence-catches-benign-direction completed 9 1.0s ok 0
✅ run-ahead-fenced-overlap completed 10 1.0s ok 0
✅ in-flight-before-decision completed 15 1.0m ok 0
✅ in-flight-before-decision-counted completed 13 1.0m ok 0
✅ in-flight-after-decision completed 15 2.0m ok 0
✅ stale-read-step-count-fork-fenced completed 15 1.0m ok 0
✅ fork-hook-wins completed 10 1.0m ok 0
✅ fork-timeout-wins completed 10 1.0m ok 0
✅ unclaimed-payload-under-fork completed 14 1.0m ok 0
✅ claimed-payload-under-fork completed 14 1.0m ok 0
✅ writers-independent-step-bodies completed 9 0ms ok 0
✅ writers-scripted-tempo completed 9 0ms ok 0
✅ cancel-mid-step cancelled 6 0ms skipped 0

Full trace: world-sim.txt

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • resumes in flight during the takeover are never lost: each lands in exactly one log (nuxt · local-prod / local / quickjs / stable)
  • RetryableError respects custom retryAfter delay (example · vercel-ws-transport / vercel / node / ws)
  • sleepingWorkflow (python · vercel-prod / vercel / node / production)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 18:29:49Z · abandoned wrun_01M67162B8AFZVPZS5VH07Z0JB
  • cold-start-warmup · suite warmup (tanstack-start) · at 18:30:02Z · abandoned wrun_01M67162BBENRV6QA50BYET18V
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 18:39:07Z · abandoned wrun_01M671Q2VC4YA6DKAF9ZNR03KV

E2E Test Summary

Summary
Passed Failed Skipped Total
✅ ▲ Vercel Production 4008 0 1392 5400
✅ 💻 Local Development 5249 0 551 5800
✅ 📦 Local Production 5249 0 551 5800
✅ 🐘 Local Postgres 5249 0 551 5800
✅ 🪟 Windows 388 0 12 400
✅ 🌐 Cross-language Conformance 68 0 107 175
✅ dynamic-runs 0 0 0 0
✅ vercel-http-transport 903 0 297 1200
✅ vercel-multi-region 27 0 0 27
✅ vercel-spec9 344 0 56 400
✅ vercel-ws-transport 611 0 189 800
Total 22096 0 3706 25802
Details by Category

✅ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 146 0 54
✅ astro-quickjs 146 0 54
✅ example-node 146 0 54
✅ example-quickjs 146 0 54
✅ express-node 146 0 54
✅ express-quickjs 146 0 54
✅ fastify-node 146 0 54
✅ fastify-quickjs 146 0 54
✅ hono-node 146 0 54
✅ hono-quickjs 146 0 54
✅ nest-node 146 0 54
✅ nest-quickjs 146 0 54
✅ nextjs-turbopack-node 173 0 27
✅ nextjs-turbopack-quickjs 173 0 27
✅ nextjs-webpack-node 173 0 27
✅ nextjs-webpack-quickjs 173 0 27
✅ nitro-node 146 0 54
✅ nitro-quickjs 146 0 54
✅ nuxt-node 146 0 54
✅ nuxt-quickjs 146 0 54
✅ python-node 66 0 134
✅ sveltekit-node 165 0 35
✅ sveltekit-quickjs 165 0 35
✅ tanstack-start-node 146 0 54
✅ tanstack-start-quickjs 146 0 54
✅ vite-node 146 0 54
✅ vite-quickjs 146 0 54

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 171 0 29
✅ astro-stable-quickjs 171 0 29
✅ express-stable-node 171 0 29
✅ express-stable-quickjs 171 0 29
✅ fastify-stable-node 171 0 29
✅ fastify-stable-quickjs 171 0 29
✅ hono-stable-node 171 0 29
✅ hono-stable-quickjs 171 0 29
✅ nest-stable-node 171 0 29
✅ nest-stable-quickjs 171 0 29
✅ nextjs-turbopack-canary-node 199 0 1
✅ nextjs-turbopack-canary-quickjs 199 0 1
✅ nextjs-turbopack-quickjs-snapshot 199 0 1
✅ nextjs-turbopack-stable-node 199 0 1
✅ nextjs-turbopack-stable-quickjs 199 0 1
✅ nextjs-webpack-canary-node 199 0 1
✅ nextjs-webpack-canary-quickjs 199 0 1
✅ nextjs-webpack-stable-node 199 0 1
✅ nextjs-webpack-stable-quickjs 199 0 1
✅ nitro-stable-node 171 0 29
✅ nitro-stable-quickjs 171 0 29
✅ nuxt-stable-node 171 0 29
✅ nuxt-stable-quickjs 171 0 29
✅ sveltekit-stable-node 190 0 10
✅ sveltekit-stable-quickjs 190 0 10
✅ tanstack-start-node 171 0 29
✅ tanstack-start-quickjs 171 0 29
✅ vite-stable-node 171 0 29
✅ vite-stable-quickjs 171 0 29

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 171 0 29
✅ astro-stable-quickjs 171 0 29
✅ express-stable-node 171 0 29
✅ express-stable-quickjs 171 0 29
✅ fastify-stable-node 171 0 29
✅ fastify-stable-quickjs 171 0 29
✅ hono-stable-node 171 0 29
✅ hono-stable-quickjs 171 0 29
✅ nest-stable-node 171 0 29
✅ nest-stable-quickjs 171 0 29
✅ nextjs-turbopack-canary-node 199 0 1
✅ nextjs-turbopack-canary-quickjs 199 0 1
✅ nextjs-turbopack-quickjs-snapshot 199 0 1
✅ nextjs-turbopack-stable-node 199 0 1
✅ nextjs-turbopack-stable-quickjs 199 0 1
✅ nextjs-webpack-canary-node 199 0 1
✅ nextjs-webpack-canary-quickjs 199 0 1
✅ nextjs-webpack-stable-node 199 0 1
✅ nextjs-webpack-stable-quickjs 199 0 1
✅ nitro-stable-node 171 0 29
✅ nitro-stable-quickjs 171 0 29
✅ nuxt-stable-node 171 0 29
✅ nuxt-stable-quickjs 171 0 29
✅ sveltekit-stable-node 190 0 10
✅ sveltekit-stable-quickjs 190 0 10
✅ tanstack-start-node 171 0 29
✅ tanstack-start-quickjs 171 0 29
✅ vite-stable-node 171 0 29
✅ vite-stable-quickjs 171 0 29

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 171 0 29
✅ astro-stable-quickjs 171 0 29
✅ express-stable-node 171 0 29
✅ express-stable-quickjs 171 0 29
✅ fastify-stable-node 171 0 29
✅ fastify-stable-quickjs 171 0 29
✅ hono-stable-node 171 0 29
✅ hono-stable-quickjs 171 0 29
✅ nest-stable-node 171 0 29
✅ nest-stable-quickjs 171 0 29
✅ nextjs-turbopack-canary-node 199 0 1
✅ nextjs-turbopack-canary-quickjs 199 0 1
✅ nextjs-turbopack-quickjs-snapshot 199 0 1
✅ nextjs-turbopack-stable-node 199 0 1
✅ nextjs-turbopack-stable-quickjs 199 0 1
✅ nextjs-webpack-canary-node 199 0 1
✅ nextjs-webpack-canary-quickjs 199 0 1
✅ nextjs-webpack-stable-node 199 0 1
✅ nextjs-webpack-stable-quickjs 199 0 1
✅ nitro-stable-node 171 0 29
✅ nitro-stable-quickjs 171 0 29
✅ nuxt-stable-node 171 0 29
✅ nuxt-stable-quickjs 171 0 29
✅ sveltekit-stable-node 190 0 10
✅ sveltekit-stable-quickjs 190 0 10
✅ tanstack-start-node 171 0 29
✅ tanstack-start-quickjs 171 0 29
✅ vite-stable-node 171 0 29
✅ vite-stable-quickjs 171 0 29

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 194 0 6
✅ nextjs-turbopack-quickjs 194 0 6

✅ 🌐 Cross-language Conformance

App Passed Failed Skipped
✅ python 68 0 107

✅ dynamic-runs

App Passed Failed Skipped
✅ astro-vercel 0 0 0
✅ example-vercel 0 0 0
✅ express-vercel 0 0 0
✅ fastify-vercel 0 0 0
✅ hono-vercel 0 0 0
✅ nest-vercel 0 0 0
✅ nextjs-turbopack-vercel 0 0 0
✅ nextjs-webpack-vercel 0 0 0
✅ nitro-vercel 0 0 0
✅ nuxt-vercel 0 0 0
✅ sveltekit-vercel 0 0 0
✅ tanstack-start-vercel 0 0 0
✅ vite-vercel 0 0 0

✅ vercel-http-transport

App Passed Failed Skipped
✅ example 146 0 54
✅ express 146 0 54
✅ hono 146 0 54
✅ nextjs-turbopack 173 0 27
✅ nitro 146 0 54
✅ vite 146 0 54

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

✅ vercel-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-node 172 0 28
✅ nextjs-turbopack-quickjs 172 0 28

✅ vercel-ws-transport

App Passed Failed Skipped
✅ example 146 0 54
✅ express 146 0 54
✅ nextjs-turbopack 173 0 27
✅ vite 146 0 54

📋 View full workflow run

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 9181a8f · Fri, 30 Oct 2026 18:36:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1893 (+484%) 🔻 2074 🔴 (+30%) 🔻 2117 🔴 (+17%) 🔻 2314 🔴 (+22%) 🔻 30
TTFS stream 1966 (+536%) 🔻 2071 🔴 (+12%) 2114 🔴 (+11%) 2155 🔴 (+9.7%) 30
TTFS hook + stream 572 (-2.9%) 2336 🔴 (+27%) 🔻 2380 🔴 (+20%) 🔻 2402 🔴 (+11%) 30
Fan-out TTFS Promise.all(100 steps) 568 (-2.7%) 981 (+14%) 2495 (+139%) 🔻 2515 (+35%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 1441 (-61%) 💚 4313 (-2.3%) 4380 (-2.8%) 8018 (+68%) 🔻 10
STSO 1020 steps (inline) 84 (-28%) 💚 101 (-50%) 💚 122 (-52%) 💚 171 (-64%) 💚 1019
WO 1020 steps 103458 (-44%) 💚 103458 (-44%) 💚 103458 (-44%) 💚 103458 (-44%) 💚 1
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 184058ms → this run 103183ms (Δ -80875ms, -44%)

 50-100 ms  ░░░░░░░░░░░░░░░░░░░░░░░┃  main   0  this 703  +703
100-150 ms  █████████┃█████           main 440  this 297  -143
150-200 ms  ┃██████████               main 313  this  14  -299
200-250 ms  ┃████                     main 154  this   4  -150
250-300 ms  ┃█                        main  49  this   0   -49
300-350 ms  ┃                         main  28  this   1   -27
350-400 ms  ┃                         main  18  this   0   -18
400-450 ms  ┃                         main   6  this   0    -6
450-500 ms  ┃                         main   6  this   0    -6
500-550 ms  ┃                         main   1  this   0    -1
550-600 ms  ┃                         main   2  this   0    -2
750-800 ms  ┃                         main   2  this   0    -2
📜 Previous results (9)

e22ecb2

Fri, 30 Oct 2026 17:40:25 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 2014 (+78%) 🔻 2342 🔴 (+37%) 🔻 2365 🔴 (+37%) 🔻 2412 🔴 (+31%) 🔻 30
TTFS stream 1962 (+40%) 🔻 2307 🔴 (+32%) 🔻 2329 🔴 (+31%) 🔻 2349 🔴 (+18%) 🔻 30
TTFS hook + stream 2245 (+227%) 🔻 2656 🔴 (+26%) 🔻 2765 🔴 (+28%) 🔻 4962 🔴 (+89%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 495 (-12%) 664 (-26%) 💚 704 (-27%) 💚 2347 (+16%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 2647 (+5.7%) 4134 (-7.9%) 4200 (-8.7%) 7915 (+67%) 🔻 10
STSO 1020 steps (inline) 80 (+33%) 🔻 100 (-14%) 118 (-16%) 💚 152 (-32%) 💚 1019
WO 1020 steps 101459 (-10%) 101459 (-10%) 101459 (-10%) 101459 (-10%) 1

b99a1b6

Thu, 29 Oct 2026 19:13:02 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 375 (-69%) 💚 2152 🔴 (+32%) 🔻 2218 🔴 (+31%) 🔻 2249 🔴 (-5.1%) 30
TTFS stream 372 (+4.8%) 2107 🔴 (+37%) 🔻 2151 🔴 (+38%) 🔻 2328 🔴 (+42%) 🔻 30
TTFS hook + stream 2269 (+58%) 🔻 2405 🔴 (+29%) 🔻 2542 🔴 (+32%) 🔻 2799 🔴 (+38%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 620 (±0%) 787 (-3.8%) 977 (-50%) 💚 2202 (-4.5%) 10
Fan-out TTLS Promise.all(100 steps) 1859 (+17%) 🔻 4156 (-3.5%) 4310 (±0%) 5092 (-22%) 💚 10
STSO 1020 steps (inline) 73 (-17%) 💚 101 (-19%) 💚 120 (-19%) 💚 171 (-28%) 💚 1019
WO 1020 steps 101852 (-16%) 💚 101852 (-16%) 💚 101852 (-16%) 💚 101852 (-16%) 💚 1

80ddf9b

Thu, 29 Oct 2026 18:01:40 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 439 (-64%) 💚 2024 🔴 (+24%) 🔻 2051 🔴 (+21%) 🔻 2094 🔴 (-12%) 30
TTFS stream 1897 (+434%) 🔻 1982 🔴 (+29%) 🔻 2015 🔴 (+30%) 🔻 2072 🔴 (+26%) 🔻 30
TTFS hook + stream 533 (-63%) 💚 2313 🔴 (+24%) 🔻 2374 🔴 (+23%) 🔻 2478 🔴 (+22%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 428 (-31%) 💚 797 (-2.6%) 926 (-52%) 💚 5337 (+132%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 1242 (-22%) 💚 4140 (-3.8%) 4323 (±0%) 10767 (+64%) 🔻 10
STSO 1020 steps (inline) 83 (-5.7%) 104 (-17%) 💚 123 (-17%) 💚 177 (-25%) 💚 1019
WO 1020 steps 105318 (-13%) 105318 (-13%) 105318 (-13%) 105318 (-13%) 1

6457a0b

Thu, 29 Oct 2026 17:02:40 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1941 (+60%) 🔻 2156 🔴 (+33%) 🔻 2202 🔴 (+30%) 🔻 2324 🔴 (-1.9%) 30
TTFS stream 1957 (+451%) 🔻 2146 🔴 (+40%) 🔻 2186 🔴 (+41%) 🔻 2242 🔴 (+36%) 🔻 30
TTFS hook + stream 2422 (+69%) 🔻 2549 🔴 (+36%) 🔻 2692 🔴 (+40%) 🔻 5934 🔴 (+193%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 519 (-16%) 💚 1006 (+23%) 🔻 2630 (+36%) 🔻 2779 (+21%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 1337 (-16%) 💚 4131 (-4.0%) 4152 (-3.6%) 8126 (+24%) 🔻 10
STSO 1020 steps (inline) 78 (-11%) 112 (-10%) 131 (-11%) 201 (-15%) 💚 1019
WO 1020 steps 112460 (-7.5%) 112460 (-7.5%) 112460 (-7.5%) 112460 (-7.5%) 1

cba1d09

Thu, 29 Oct 2026 16:50:45 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1969 (+62%) 🔻 2107 🔴 (+30%) 🔻 2158 🔴 (+28%) 🔻 2211 🔴 (-6.7%) 30
TTFS stream 1966 (+454%) 🔻 2103 🔴 (+37%) 🔻 2127 🔴 (+37%) 🔻 2189 🔴 (+33%) 🔻 30
TTFS hook + stream 2332 (+63%) 🔻 2474 🔴 (+32%) 🔻 2494 🔴 (+30%) 🔻 2596 🔴 (+28%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 583 (-6.1%) 951 (+16%) 🔻 2857 (+48%) 🔻 3030 (+31%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 2669 (+69%) 🔻 4293 (±0%) 4573 (+6.2%) 8708 (+33%) 🔻 10
STSO 1020 steps (inline) 114 (+30%) 🔻 139 (+11%) 160 (+8.1%) 210 (-11%) 1019
WO 1020 steps 138254 (+14%) 138254 (+14%) 138254 (+14%) 138254 (+14%) 1

89875b6

Thu, 29 Oct 2026 16:11:58 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1967 (+62%) 🔻 2075 🔴 (+28%) 🔻 2098 🔴 (+24%) 🔻 2174 🔴 (-8.2%) 30
TTFS stream 1958 (+452%) 🔻 2054 🔴 (+34%) 🔻 2149 🔴 (+38%) 🔻 2183 🔴 (+33%) 🔻 30
TTFS hook + stream 2337 (+63%) 🔻 2439 🔴 (+30%) 🔻 2473 🔴 (+29%) 🔻 2625 🔴 (+30%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 703 (+13%) 846 (+3.4%) 994 (-49%) 💚 2847 (+24%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 2627 (+66%) 🔻 4425 (+2.8%) 4436 (+3.0%) 8661 (+32%) 🔻 10
STSO 1020 steps (inline) 106 (+20%) 🔻 131 (+4.8%) 149 (+0.7%) 189 (-20%) 💚 1019
WO 1020 steps 130731 (+7.5%) 130731 (+7.5%) 130731 (+7.5%) 130731 (+7.5%) 1

f7d93e9

Wed, 28 Oct 2026 21:59:27 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 322 (-70%) 💚 2096 🔴 (+37%) 🔻 2254 🔴 (+42%) 🔻 2525 🔴 (+9.9%) 30
TTFS stream 349 (-76%) 💚 2021 🔴 (+34%) 🔻 2226 🔴 (+46%) 🔻 2394 🔴 (+51%) 🔻 30
TTFS hook + stream 2229 (+33%) 🔻 2351 🔴 (+32%) 🔻 2414 🔴 (+28%) 🔻 2565 🔴 (+13%) 30
Fan-out TTFS Promise.all(100 steps) 664 (+20%) 🔻 873 (+27%) 🔻 2406 (+199%) 🔻 2409 (+30%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 3531 (+160%) 🔻 4325 (+2.5%) 6012 (+42%) 🔻 7278 (+18%) 🔻 10
STSO 1020 steps (inline) 102 (+36%) 🔻 139 (+32%) 🔻 165 (+33%) 🔻 302 (+42%) 🔻 1019
WO 1020 steps 140272 (+32%) 🔻 140272 (+32%) 🔻 140272 (+32%) 🔻 140272 (+32%) 🔻 1

1ead5fd

Tue, 27 Oct 2026 19:11:40 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1906 (+62%) 🔻 2020 🔴 (+45%) 🔻 2089 🔴 (+44%) 🔻 2249 🔴 (+25%) 🔻 30
TTFS stream 1644 (+55%) 🔻 1971 🔴 (+36%) 🔻 1985 🔴 (+35%) 🔻 2001 🔴 (+31%) 🔻 30
TTFS hook + stream 2157 (+35%) 🔻 2293 🔴 (+30%) 🔻 2522 🔴 (+38%) 🔻 3120 🔴 (+46%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 553 (+7.6%) 701 (-10%) 709 (-12%) 2377 (+40%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 3878 (+163%) 🔻 4283 (+4.0%) 4394 (+6.6%) 5098 (-15%) 10
STSO 1020 steps (inline) 77 (-6.1%) 95 (-13%) 111 (-15%) 146 (-49%) 💚 1019
WO 1020 steps 95972 (-15%) 95972 (-15%) 95972 (-15%) 95972 (-15%) 1

b37d7fd

Tue, 27 Oct 2026 19:01:02 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1692 (+44%) 🔻 2163 🔴 (+55%) 🔻 2192 🔴 (+51%) 🔻 2222 🔴 (+23%) 🔻 30
TTFS stream 1982 (+87%) 🔻 2091 🔴 (+44%) 🔻 2119 🔴 (+44%) 🔻 2166 🔴 (+42%) 🔻 30
TTFS hook + stream 2270 (+43%) 🔻 2515 🔴 (+42%) 🔻 2592 🔴 (+41%) 🔻 5957 🔴 (+179%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 641 (+25%) 🔻 887 (+14%) 1087 (+35%) 🔻 2634 (+55%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 1617 (+9.6%) 4418 (+7.3%) 4929 (+20%) 🔻 9254 (+54%) 🔻 10
STSO 1020 steps (inline) 80 (-2.4%) 110 (+0.9%) 128 (-1.5%) 185 (-35%) 💚 1019
WO 1020 steps 108563 (-3.6%) 108563 (-3.6%) 108563 (-3.6%) 108563 (-3.6%) 1
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies. Streaming delivery performance is measured in durabench.

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor
Framework Flow route Step reg. Framework output
hono 271.7 KiB (+2.2 KiB) 100.1 KiB (+3.4 KiB) 2.15 MiB (+28.3 KiB)
nextjs-turbopack 279.1 KiB (+2.2 KiB) 426 B (±0) 998.4 KiB (+9.7 KiB)
About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9181a8f · run

Comment thread packages/core/e2e/e2e.test.ts Resolved
@vercel-internal-security-reviewer

Copy link
Copy Markdown

Security review details

@vercel
vercel Bot temporarily deployed to Preview – workflow-docs October 22, 2026 17:50 Inactive
@alangenfeld
alangenfeld force-pushed the alangenfeld/2026-10-21-run-idempotency-sdk-prototype branch from c59fc83 to cc849a0 Compare October 22, 2026 22:22
alangenfeld added a commit that referenced this pull request Oct 22, 2026
…rator branch

Squash of #4996 at 8ea0c2b (its own commits only),
resolved against this branch: the spec-8 engine's admission handling
moves to runtime/spec8/delivery.ts with a shared start-hook-delivery.ts,
capability and queue-schema edits sit beside the single-orchestrator
ones, and the start() unit tests declare the in-band fence spec 9 needs.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
alangenfeld added a commit that referenced this pull request Oct 22, 2026
…wStartError a WorkflowError

Brings #4996's recut onto the single-orchestrator engine: setup on both
engines treats exactly a run_started response that failed its schema at
`run` as a redelivery (warned, span attribute
workflow.run_started.runless_response) instead of a fatal contract error;
WorkflowStartError extends WorkflowError and drops its inert `retryable`
field; and the conformance suite covers an adoption without its own
retention keeping the start's.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@alangenfeld
alangenfeld force-pushed the alangenfeld/2026-10-21-run-idempotency-sdk-prototype branch from cc849a0 to e133836 Compare October 23, 2026 15:15
alangenfeld added a commit that referenced this pull request Oct 23, 2026
…rator branch

Squash of #4996 at 8ea0c2b (its own commits only),
resolved against this branch: the spec-8 engine's admission handling
moves to runtime/spec8/delivery.ts with a shared start-hook-delivery.ts,
capability and queue-schema edits sit beside the single-orchestrator
ones, and the start() unit tests declare the in-band fence spec 9 needs.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
alangenfeld added a commit that referenced this pull request Oct 23, 2026
…wStartError a WorkflowError

Brings #4996's recut onto the single-orchestrator engine: setup on both
engines treats exactly a run_started response that failed its schema at
`run` as a redelivery (warned, span attribute
workflow.run_started.runless_response) instead of a fatal contract error;
WorkflowStartError extends WorkflowError and drops its inert `retryable`
field; and the conformance suite covers an adoption without its own
retention keeping the start's.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@alangenfeld

Copy link
Copy Markdown
Collaborator Author

Addressed as of 1ead5fdc5.

Where it differs from your suggestion:

  • F1: inside a step, resumeHook() and getHookByToken() wait up to 30s on HookReservedError, honoring retryAfter. That's instead of an executor retry budget, which would have had to bypass the timeout guard.
  • F2: Postgres start Hooks are opt-in, behind WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK.

F7, F8, F9, F10, F12 and F13 are in a separate commit, "[core][world] Trace start Hook admission outcomes".

F6 is deferred to its own PR, because it changes serialization for every World error.


Local agent review (`anthropic/claude-opus-5.5`)

@alangenfeld
alangenfeld requested a review from pranaygp October 27, 2026 19:24
alangenfeld added a commit that referenced this pull request Oct 27, 2026
…rator branch

Squash of #4996 at 8ea0c2b (its own commits only),
resolved against this branch: the spec-8 engine's admission handling
moves to runtime/spec8/delivery.ts with a shared start-hook-delivery.ts,
capability and queue-schema edits sit beside the single-orchestrator
ones, and the start() unit tests declare the in-band fence spec 9 needs.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
alangenfeld added a commit that referenced this pull request Oct 27, 2026
…wStartError a WorkflowError

Brings #4996's recut onto the single-orchestrator engine: setup on both
engines treats exactly a run_started response that failed its schema at
`run` as a redelivery (warned, span attribute
workflow.run_started.runless_response) instead of a fatal contract error;
WorkflowStartError extends WorkflowError and drops its inert `retryable`
field; and the conformance suite covers an adoption without its own
retention keeping the start's.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>

@pranaygp pranaygp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 1ead5fd. The first round is largely fixed:

  • the kill-switch refusal is no longer final;
  • world-local uses the 24h bound;
  • the holder wake runs in the background;
  • the confirm loop covers non-streamed starts;
  • forced claims get a refusal reason;
  • admission rows are cleaned up;
  • the tracing commit covers F7–F10, F12 and F13;
  • error messages are reworded;
  • world-postgres start Hooks are opt-in.

start() without hook still makes the same World calls, in the same order, with the same awaits as main.

Two fixes, though, introduced regressions for apps that never use start({ hook }), on world-postgres and on the Local World.

What I ran:

  • core start-hook, resume-hook and serialization tests: 585 pass;
  • world: 280 pass;
  • world-vercel: 1019 pass;
  • world-local: 747 pass. The 3 failures in vercel-deployment-warning.test.ts are outside this PR; they come from the VERCEL* env on my machine.

Status of the first round

  • Fixed: C1, C3–C9, F1, F3–F5, F7–F10, F12, F13. F2 is fixed by making start Hooks opt-in.
  • F6 is deferred to its own PR, which is fine. But document now that code, status and reason on WorkflowWorldError are undefined inside workflow code (workflow-world-error.mdx).
  • C2: the flag-off path is close to main, but see N1 and the migration note below.

Correctness

N1. world-postgres, callers who don't use the feature: a concurrent replay of the same createHook can write a hook_conflict naming the run's own hook.

  • Where: packages/world-postgres/src/storage.ts:2767.
  • The fresh-hook insert now uses onConflictDoNothing({ target: Schema.hooks.hookId }) and sets claims_token = true.
  • When two writes of the same (runId, hookId, token) both pass the ON CONFLICT pre-check, the second raises 23505 on workflow_hooks_token_claim_unique. That isn't the arbiter, so it's caught as "another writer claimed the token" (:2679), and the re-read finds the run's own hook as the owner. The run then gets both a hook_created and a hook_conflict for one correlationId.
  • On main, the untargeted ON CONFLICT DO NOTHING made the second write a no-op, and the event insert deduplicated it.
  • Reproduced with the flag unset: 7 of 400 concurrent rounds.
  • Suggested fix: after a 23505 on the claim index, re-read the token. If the owner is this (runId, hookId), treat it as the idempotent duplicate (EntityConflictError). Add a looped regression test.

N2. Local World: cancelling a missing run now blocks any later creation of that run ID, ordinary runs included.

  • Where: packages/world-local/src/storage/events-storage.ts:1737, packages/core/src/runtime/runs.ts:194.
  • cancelRun now sends run_cancelled for a missing run whenever the World advertises atomicStartHook, and the Local World always does.
  • The Local World then writes a token-less fence at the run path for any ID younger than 24h. Ordinary creation doesn't recognise the fence: run_created fails with "already exists", and run_started with not-found, while runs.get still says not found.
  • Example: an ordinary resilient start that is cancelled before its run_started lands can never be created.
  • Suggested fix: make only start Hook admission honour the fence. For instance, ordinary creation could treat a token-less fence as absent, or create the run already cancelled. Alternatively, fence only when a reservation claim names the ID. Add a test that cancels an ordinary resilient run before its run_started.

Risks

N3. world-postgres, mixed versions: placeholder reservation rows are returned as real Hooks by the previous release's lookup.

  • Where: packages/world-postgres/src/drizzle/schema.ts:362, storage.ts:3820-3844.
  • With the flag on, a reservation is an ordinary workflow_hooks row (shr_<runId>), and it stays in place after adoption.
  • The previous release's getByToken is where token = ? and available limit 1, with no ordering and no filter, so it can return the placeholder. resumeHook then appends hook_received under a correlationId nothing awaits, and resolves successfully: the payload is silently lost. hooks.list shows the placeholder too.
  • postgres.mdx only warns about older queue workers. Extend it to every process that reads the database (resume and webhook handlers, the CLI, the dashboard), or keep the placeholder out of the rows an older query matches.

N4. world-postgres: the two copies of one candidate can fail with a raw workflow_hooks_pkey violation.

  • Where: packages/world-postgres/src/start-hook.ts:198.
  • The placeholder insert arbitrates on the token's partial index only, but both copies share shr_<runId>.
  • Correctness holds: the direct copy retries and replays, and the queued copy is redelivered. The cost is a confirm sleep, a burned queue retry and an error log, in a race every start() creates on purpose (3 + 3 of 300 rounds).
  • Suggested fix: use an untargeted onConflictDoNothing(), or translate the pkey 23505 into the race path.

N5. world-postgres: deleteExpiredDecisions makes concurrent admissions queue behind each other.

  • Where: packages/world-postgres/src/start-hook.ts:317.
  • Every admission deletes the same oldest expired rows inside its own transaction, so admissions for unrelated tokens wait on each other's commit.
  • Suggested fix: add FOR UPDATE SKIP LOCKED, or move the cleanup off the admission transaction.

N6. The in-step reserved-token wait also runs when the token is reserved by the step's own run.

  • Where: packages/core/src/runtime/resume-hook.ts:337.
  • The owner can't register its hook while its own step is running, so the 30s wait can't succeed. It costs about 30s per attempt, roughly 2 minutes with the default retries.
  • Suggested fix: skip the wait when error.runId is the current workflowRunId.

N7. The raw token is in world-postgres's new 503 messages.

  • Where: packages/world-postgres/src/start-hook.ts:212, storage.ts:3091.
  • These messages flow into WorkflowStartError's message, the workflow.start span status and the new runtimeLogger.warn. Everything else in the PR keeps the token out.
  • Name the run instead.

N8. The reworded HookConflictError doesn't survive the step boundary.

  • Where: packages/core/src/serialization/reducers/common.ts:559, common-vm.ts:472.
  • The revivers rebuild the error through its constructor and discard the message. Inside a workflow, a refused start({ hook }) therefore shows the createHook wording, raw token included, and if it's uncaught that becomes the run's failure message.
  • Rewriting message after construction also leaves the stack stale and drops the docs link (start.ts:597).
  • Suggested fix: carry a kind: 'start' field through the reducer and build the message from it.

Compatibility notes

  • Migration 0028. It's now required even with the flag off, because every hook insert writes claims_token, and it takes table locks with no lock_timeout. Say so in postgres.mdx and the changeset, and set a lock_timeout.
  • Older viewers. Dashboards and CLIs released before this one can't read the new HookReservedError / WorkflowStartError tags ("Unknown type …"), and these now routinely reach step_failed / run_failed. Either encode them through a tag older revivers understand (the named-Error encoding), or say in the changesets that viewers must be upgraded first.

Nits

  • world-local answers WorkflowRunNotFoundError after fencing a cancelled candidate (events-storage.ts:1771), unlike START_HOOK_CANDIDATE_CANCELLED elsewhere. The WorkflowStartError docs (workflow-start-error.mdx:50) describe the Local World behaviour wrongly as a result.
  • hook-reserved-error.mdx:17 doesn't mention the automatic in-step wait, and its example retry loop stacks on top of it.
  • An unsupported or switched-off World gets the same error code as a World that broke the contract (start.ts:898). The refusal doesn't name the env var.
  • A stale JSDoc block now sits on the wrong declaration (start.ts:433).
  • world-vercel sends x-workflow-hook-conflict-reasons on every request (utils.ts:282), including callers who don't use the feature.
  • Still open from the first round, all small:
    • world-local parses every run read as a fence-or-run union (helpers.ts:517);
    • a lookup of a disposed adopted reservation does the full hooks/ scan (hooks-storage.ts:309-365);
    • health checks always parse capabilities (helpers.ts:244, :410).
  • The holder-wake result is now only in logs. The span attribute was removed with no replacement (start-hook-holder.ts).

Requesting changes for N1 and N2. Both are regressions for apps that don't use start({ hook }). N3–N8 can follow, but N3 should be fixed or documented before anyone turns on the Postgres flag.

Copy link
Copy Markdown
Contributor

Pushed f7d93e9 with e2e cases for how a run uses its reserved token after admission. They run in the start({ hook }) block, and all 10 tests in it pass on the Local World and on world-postgres (with WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK=1):

  • Never claimed. The token was only an idempotency key: the run completes, and a duplicate start is refused naming it. A resume gets HookReservedError and nothing is written to the run's log. After the run ends, the lookup gives HookNotFoundError while the token stays held for its retention.
  • Never claimed, then cancelled, with no retention. The token is free, and a new start owns it.
  • Claimed late. A resume before the claim is refused and not written. The run's createHook() (with no getConflict()) then claims the token with exactly one hook_created and no hook_conflict.
  • Claimed but never read. Payloads resumed to the Hook stay in the log, and the run still completes.
  • Disposed, then claimed again. Lookups are not found in between while the token stays the run's, and a second createHook() claims it again.
  • Resumed from a step with maxRetries = 0. resumeHook waits out HookReservedError until the owner claims the token.

One World gap the investigation turned up (world-postgres, raw World API only):

  • hook_received and hook_disposed addressed to the reservation placeholder's hook id are accepted. The existence checks around storage.ts:1840, the locked re-check near :3240, and the hook_disposed DELETE near :3163 don't filter on is_start_reservation. The id is deterministic (shr_<runId>).
  • With a raw world.events.create(runId, { eventType: 'hook_received', correlationId: 'shr_' + runId, … }), nothing ever consumes the event. It strands, and the run fails with ReplayDivergenceError when the workflow returns.
  • A raw hook_disposed with that id deletes the placeholder and silently releases the reservation.
  • The SDK can't reach this, because getByToken, get and list never return the placeholder.
  • Suggested fix: add is_start_reservation = false to those three queries, and a world-postgres unit test.

Copy link
Copy Markdown
Contributor

New requirement: a start Hook can be resumed from the moment its run is admitted. This removes HookReservedError and the 24-hour default retention.

resumeHook(token) has to work as soon as start({ hook }) admits a run. A window where the token is reserved but can't be resumed breaks "start a run, or resume the owner's Hook", which a planned higher-level helper guarantees. So:

  • The admission creates the start Hook. run_created implies its hook_created, and its id is derived from the run's: startHookId(runId), wrun_<ulid> → hook_<ulid>. Resumes can land as soon as the run is admitted, while it's still pending.
  • createHook({ token }) in the run with the start token just returns a handle bound to that Hook. It writes no hook_created, needs no suspension, and getConflict() resolves null.
  • Payloads resumed before the workflow opens the Hook are kept and delivered in order once it does.
  • Never opening it is supported. The token was then only an idempotency key: the run completes, and leftover hook_received events don't fail it.
  • No default retention. The start Hook follows the ordinary Hook lifecycle, like a createHook() in the run. experimental_minRetention stays an optional argument on start({ hook }).

I prototyped this on top of this PR so it can be validated:

Merge them into this PR, or rework them however you like. The PR description lists the open decisions and one known gap: world-postgres's existing createHook() race.

🤖 Generated with Claude Code

@alangenfeld

Copy link
Copy Markdown
Collaborator Author

Replying to #4996 (comment)

Thanks. We're adopting the resumable-from-admission design. #4996 will absorb #5119, #5120 and f7d93e9 as one commit, and #1292 gets recut the same way against your change list. We changed two things and kept your choices everywhere else. Change either if you see it differently:

  • Disposing the start Hook keeps the token until the run ends. As prototyped, disposal (including a using scope exit) frees the token even with experimental_minRetention, so a duplicate start() becomes a second live run for the same key. Disposal now detaches the handle; the claim lasts until the run is terminal, then retention applies. A force-claim still transfers the token.
  • The world-postgres admission vs createHook race is closed. Ordinary hook rows claim the token's unique index too (no extra statements). A 23505 for the same (runId, hookId) maps to EntityConflictError, which keeps main's dedup of a concurrent replay.

Kept as you have them:

  • Uniform force-claim. It answers a retryable 429 while the victim has no run row. An in-run createHook({ token }) after a force-claim surfaces as claimed rather than getConflict() === null.
  • The other defaults. No default retention; throwing on start-only options in the in-run createHook(); the ~1.5s backend and ~30s world-vercel budgets.

Also folded in: a direct copy reads the constraint before its transaction, which saves the cancelled 5-item write on duplicates. Accepted decisions expire at window + 1h.

Open:

I'll re-request review once both are green.


Local agent review (anthropic/claude-opus-5.5)

@pranaygp pranaygp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final pass at 6457a0b. I confirmed this absorbed #5119, #5120 and f7d93e9, and closed those two:

  • all 19 start({ hook }) e2e tests are here;
  • the start Hook binding is in core on both engines, with eager parking;
  • HookReservedError and the 24-hour default are gone;
  • the Idempotency rewrite is in.

The two deviations are good calls:

  • Disposal keeps the token until the run ends, so a key never has two live runs.
  • world-postgres's createHook race is closed by the unique index.

Of the earlier findings (C1–C9, F1–F13, N1–N8), 29 are fixed or obsolete. CF6 (WorkflowWorldError.code at the step boundary) stays deferred, as agreed.

What I ran locally

  • Unit tests:

    Package Result
    core 3438 passed
    world 280 passed
    world-local 763 passed
    world-vercel 1027 passed
    world-postgres 472 passed
    errors, web-shared, cli all pass

    world-local's 3 failures are in vercel-deployment-warning.test.ts, which this PR doesn't touch; they come from the VERCEL* env on my machine.

  • e2e:

    • the start({ hook }) block passes 19/19 on the Local World and on Postgres (with WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK=1), each under Node and QuickJS;
    • the full suite passes on the Local World (173 passed, 1 skipped).
  • CI: the only red check is Windows unit, from world-local/src/storage/fence-lock.test.ts, which this PR doesn't touch; main's test runs are failing intermittently too.

Still blocking

1. world-postgres: every hook_disposed now runs an extra SELECT on workflow_runs (packages/world-postgres/src/storage.ts:3115).

  • It runs whatever the flag says, so every Postgres app pays it on every dispose, using exit and webhook, even if it never uses start({ hook }).
  • The same event already reads the run in getRunForValidation.
  • Suggested fix: add start_hook_cbor (or only its token) to that read and reuse it, or decide from the hook row itself (hookId === startHookId(runId), or a column set when a Hook takes the start token). Either way, apps that don't use start Hooks are back to main's statements.

Non-blocking

  • Migration 0028's lock ordering (0028_add_start_hook_admissions.sql:44).
    • It is now required for every Postgres user, and drizzle runs it in one transaction.
    • ALTER TABLE workflow_runs ADD COLUMN takes ACCESS EXCLUSIVE, and then the workflow_hooks unique index builds while that lock is held.
    • Build the index before the workflow_runs ALTER, or split it out. At least say in postgres.mdx that the build scans workflow_hooks while workflow_runs is locked.
  • Unread start Hook payloads are re-offered before every event (packages/core/src/events-consumer.ts:472).
    • In the idempotency-key-only use, eager-parked start Hook events stay in parked, and drainParked() offers each one to every callback before every event. That costs O(K × callbacks) per event on every replay, and the list grows for the run's whole life.
    • QuickJS also hydrates each payload into the heap and its snapshots.
    • Offer them only once the start Hook is bound, and in QuickJS keep raw bytes until a payload is taken.
  • Test gap: no concurrent test covers two different hook ids of one run on one token (world-postgres/test/atomic-start-hook.test.ts:626). That case must give a hook_conflict naming the run, never EntityConflictError.
  • Docs and small fixes:
    • The start-or-route example throws when the owner never got a run (an orphan candidate) (idempotency.mdx:117).
    • upgrading-to-v5.mdx:154 still says hook_disposed always releases the token.
    • The World support table doesn't mention the Postgres migration prerequisite.
    • workflow-world-error.mdx:49: retryAfter doesn't mention the run-pending 503.
    • start-hook-binding.ts:98 puts raw start tokens in a WorkflowRuntimeError message.
    • world-local's hook_disposed re-reads the run record it already loaded.

Not approving yet, because of item 1. It's a small change. The cross-region resume on the Vercel World also depends on a backend fix being tracked separately, so the Vercel opt-in should stay off until that ships.

🤖 Generated with Claude Code

@pranaygp pranaygp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at b99a1b6. The blocker from my last pass is fixed:

  • world-postgres hook_disposed is back to main's single DELETE … RETURNING (identical to main), so apps that don't use start Hooks pay nothing extra. The only added read is the admission's, inside if (startHook).

Also addressed:

  • Disposal releases the token, as any Hook does (4750f87), consistently with the backend.
  • Unread start Hook payloads are no longer offered to every callback until the Hook is opened (events-consumer.ts).
  • Migration 0028 builds the workflow_hooks index before it locks workflow_runs, with a 10s lock_timeout.
  • Docs and error text: the retryAfter doc, the postgres prerequisite, and no raw tokens in the binding's error.
  • A concurrent test for two Hooks of one run on one token (one Hook, plus a conflict naming the run).

What I ran locally at this head:

  • unit tests: core 3419, world 280, world-vercel 1027, world-postgres 470, world-local 762. world-local's 3 vercel-deployment-warning failures come from the VERCEL* env on my machine.
  • e2e: the start({ hook }) block passes 19/19 on the Local World and on Postgres (with WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK=1), each under Node and QuickJS.
  • CI: green.

Follow-ups:

  • WorkflowWorldError.code/status/reason are dropped at the step boundary (CF6). These are deferred to their own PR, as agreed. Until then, a workflow can't tell START_HOOK_ADMISSION_REJECTED from WORLD_CONTRACT_ERROR.
  • The Vercel lanes skip the start({ hook }) e2e block until WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK is set for them. Turn it on there once the backend is deployed, so cross-region resumes get e2e coverage before any caller opts in.

🤖 Generated with Claude Code

alangenfeld and others added 2 commits October 30, 2026 10:08
start(workflow, args, { hook: { token, metadata?, experimental_minRetention? } })
admits the run and creates its start Hook in one atomic World step. A duplicate
start throws HookConflictError (kind 'start') naming the owner and never
becomes a run. The start Hook (startHookId(runId)) can be resumed from
admission, before the run starts; the workflow's createHook({ token }) binds to
it without a write, and payloads resumed earlier are delivered in order.

The token is the run's idempotency key: the run holds it until it ends, even
after disposing the start Hook, then any experimental_minRetention applies.

Implemented for world-local, world-postgres (behind
WORKFLOW_POSTGRES_EXPERIMENTAL_START_HOOK, migration 0028; every Hook row now
claims its token through a unique index) and world-vercel (behind
WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK), with a shared conformance suite,
tracing of admission outcomes, docs and e2e coverage.

Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…-inert scan

The terminal-inert check in TERMINAL_STATE_JS reads a drawn hook id with no
__hooks entry as a closed Hook. The start Hook's id is derived from the run's
id rather than drawn, so it sorts before every id the VM draws: once the VM
had drawn one, a payload resumed into the start Hook before the body opened it
was dropped as a closed Hook's straggler. reflected() now answers false for
the start Hook until it is bound.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
… it is wired

The spec 9 consumer does not yet skip turbo for a start Hook candidate, carry
the request on run_started, check the echo, or acknowledge a refusal. A
refused candidate's queued copy could therefore create its run unkeyed and run
it next to the token's owner, and Node would not bind createHook({ token }) to
the start Hook.

start() now throws a contract error, before it publishes or admits anything,
when the effective spec version is 9 or later (an explicit specVersion and a
probed target included). The spec 9 consumer acknowledges a keyed first
delivery, from another SDK version, before any write or execution. The
start({ hook }) e2e block skips on the spec 9 lane.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@github-actions github-actions Bot mentioned this pull request Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for eb07a8d (AI decision).

This is new feature work: it adds the start(workflow, args, { hook }) atomic admission API, the new WorkflowStartError, the atomicStartHook World capability, new env flags, and a new Postgres migration (0028_add_start_hook_admissions), all with minor changesets and v5-only docs. The Postgres change where every Hook row claims its token through a unique index closes a race between admission and a concurrent createHook(), but that race only matters alongside the new start-Hook admission path. It is part of the feature and is not a separate fix for stable.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

eb07a8d261de014499ca0eb7f259e178d610082f

This branch was successfully deployed

1 active deployment
Preview – workflow-docs — 9181a8f9 Deployed Oct 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants