Skip to content

[e2e] Add an opt-in start({ hook }) chaos drill - #5010

Draft
alangenfeld wants to merge 21 commits into
mainfrom
alangenfeld/2026-10-22-run-idempotency-chaos-e2e
Draft

alangenfeld wants to merge 21 commits into
mainfrom
alangenfeld/2026-10-22-run-idempotency-chaos-e2e

Conversation

@alangenfeld

@alangenfeld alangenfeld commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary & Motivation

This adds end-to-end coverage for start({ hook }) (#4996, the atomic keyed start asked for in #2376). The drill sends real start Hook traffic, mostly cross region, to a workflow-server that has its start Hook faults turned on (workflow-server#1300). For each token it checks that there is exactly one owner and that every caller resolves to that owner. It also checks that refused candidates leave no trace and that the owner's log replays.

It only runs when WORKFLOW_E2E_START_HOOK_DRILL=1 is set. Setting it to dry instead runs a fault-free smoke check against a deployment before a full drill.

Test Plan

I ran the full drill in the multi-region lane against the workflow-server#1300 preview, and all five cases passed:

  • concurrent callers for each fault
  • a start inside a step with a forced step retry
  • token retention after the owner ends
  • an orphaned start that a later contender fences and releases
  • queue redelivery

Adds the start({ hook }) wire contract: StartHookSchema on run_created,
resilient run_started and the queued runInput; the atomicStartHook
capability; the run.startHook echo; WorkflowStartError and
START_HOOK_ADMISSION_REJECTED; and world-vercel's startHook frame meta,
409 start-hook-conflict / 400 start-hook-* error mapping, and the
WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK opt-in.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…es TBD

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Adds a shared atomic-start-hook conformance suite and implements it in
world-local (reservation on the token claim file, run created under its
lock, refusals recorded per candidate) and world-postgres (decision and
run in one transaction, serialized per token with an advisory lock shared
with createHook; migration 0028). Both let the owner's createHook adopt
the reservation and move it with a force-claimed Hook. The run detail
panel shows the start Hook token.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
start() publishes the candidate first and then decides it with
run_created; the queued copy's run_started carries the same startHook,
skips turbo, and acks a refusal (HookConflictError or
START_HOOK_ADMISSION_REJECTED) without running code. Unconfirmed queue or
admission throws WorkflowStartError. Stream arguments pipe only once the
candidate is admitted. Cross-deployment starts check the target's probed
capabilities (health checks now carry them, parsed tolerantly), keeping
the no-probe-channel guard. run_started now forwards encryptionPublicKey.
WorkflowStartError round-trips through the host, VM and QuickJS serdes;
re-runs never forward a source run's start Hook.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Replaces the idempotency callout promising native atomic start with the
start({ hook }) API, a start-or-route-to-owner pattern, WorkflowStartError,
the World contract, and WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…ntion

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…nsitions and re-adoption

An adopted Hook's retention now extends the reservation itself, so
disposing it and adopting again with a shorter one cannot shorten the
promise. world-local also carried run.startHook through none of its
field-by-field run rewrites, so a run_created retry after run_started
found the run 'without this start Hook'. Both are now conformance cases.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
… reviver fixture

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
A by-token lookup that lands on a start reservation its run has not
adopted answers HookReservedError (425 hook-reserved on the wire),
retryable and never a HookNotFoundError. startHook carries optional
serialized metadata the adopting Hook inherits when it brings none.
WorkflowStartError is documented and marked retryable.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…n and carry start Hook metadata

Lookups of an unadopted reservation of a live run throw HookReservedError,
and not found once the run ends without adopting it. Adoption inherits
the reservation's metadata unless the Hook brings its own. Postgres tracks
adoption on the admission row (migration 0028), and its timezone test
counts the new table.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…retry rules

hook.metadata is serialized like createHook metadata. Without an explicit
experimental_minRetention a start Hook keeps its token for 24 hours from
start() where the World supports Hook retention (0 opts out). Inside a
workflow a refusal, permanent rejection or contract error is final for
the start() step, while WorkflowStartError stays retryable. HookReservedError
round-trips through every serde.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…t retention

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…ugh an interrupted adoption

A lookup between the adopting createHook's claim write and its
hook_created publish answered not found, so callers retrying on
HookReservedError gave up; it now stays HookReservedError while the
owner runs. A retried adoption whose first attempt died before
publishing now still inherits the reservation's metadata. The e2e
routing test now actually passes metadata to start().

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…s lost

A lookup that finds the claim naming the adopting Hook but no entity now
rebuilds the Hook from its published hook_created before answering
HookReservedError, so a crash between the publish and the entity write
does not leave the token reserved forever.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Points world-vercel at the server branch alias and advertises
atomicStartHook unconditionally, so the Vercel e2e lanes run the
start({ hook }) tests against the server's admission. Reverted after
pairing.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…created lands

Holds the direct run_created back so the queued copy's run_started
admits the candidate through the resilient start against the real
backend, then checks a duplicate is refused naming it, the late direct
write replays the decision, and the log has one run_created.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Pairing passed: the start({ hook }) e2e tests ran and passed on the
node and quickjs Vercel lanes against the server preview.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot

changeset-bot Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: eb12df6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
example-nextjs-workflow-webpack Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
example-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-astro-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-express-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-fastify-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-hono-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-nestjs-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-nitro-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-nuxt-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-python-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-sveltekit-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-tanstack-start-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workbench-vite-workflow Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workflow-docs Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workflow-swc-playground Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workflow-tarballs Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC
workflow-web Ready Ready Preview, v0 Oct 22, 2026 7:37pm UTC

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • admits exactly one of several concurrent starts; the rest name the owner (nextjs-turbopack · local-prod / local / node / canary)
  • step-argument serialization failure is catchable in workflow code (fastify · vercel-prod / vercel / quickjs / production)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 19:43:25Z · abandoned wrun_01M5JJ72TRE5CDFN2HFGS8BVCQ
  • cold-start-warmup · suite warmup (tanstack-start) · at 19:43:27Z · abandoned wrun_01M5JJ72V5KYSNPHGWR6HR647Z

E2E Test Summary

Summary
Passed Failed Skipped Total
✅ ▲ Vercel Production 3982 0 986 4968
✅ 💻 Local Development 4785 0 551 5336
✅ 📦 Local Production 4785 0 551 5336
✅ 🐘 Local Postgres 4785 0 551 5336
✅ 🪟 Windows 356 0 12 368
✅ 🌐 Cross-language Conformance 68 0 91 159
✅ dynamic-runs 0 0 0 0
✅ vercel-http-transport 897 0 207 1104
✅ vercel-multi-region 27 0 0 27
✅ vercel-ws-transport 607 0 129 736
Total 20292 0 3078 23370
Details by Category

✅ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 145 0 39
✅ astro-quickjs 145 0 39
✅ example-node 145 0 39
✅ example-quickjs 145 0 39
✅ express-node 145 0 39
✅ express-quickjs 145 0 39
✅ fastify-node 145 0 39
✅ fastify-quickjs 145 0 39
✅ hono-node 145 0 39
✅ hono-quickjs 145 0 39
✅ nest-node 145 0 39
✅ nest-quickjs 145 0 39
✅ nextjs-turbopack-node 172 0 12
✅ nextjs-turbopack-quickjs 172 0 12
✅ nextjs-webpack-node 172 0 12
✅ nextjs-webpack-quickjs 172 0 12
✅ nitro-node 145 0 39
✅ nitro-quickjs 145 0 39
✅ nuxt-node 145 0 39
✅ nuxt-quickjs 145 0 39
✅ python-node 66 0 118
✅ sveltekit-node 164 0 20
✅ sveltekit-quickjs 164 0 20
✅ tanstack-start-node 145 0 39
✅ tanstack-start-quickjs 145 0 39
✅ vite-node 145 0 39
✅ vite-quickjs 145 0 39

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 155 0 29
✅ astro-stable-quickjs 155 0 29
✅ express-stable-node 155 0 29
✅ express-stable-quickjs 155 0 29
✅ fastify-stable-node 155 0 29
✅ fastify-stable-quickjs 155 0 29
✅ hono-stable-node 155 0 29
✅ hono-stable-quickjs 155 0 29
✅ nest-stable-node 155 0 29
✅ nest-stable-quickjs 155 0 29
✅ nextjs-turbopack-canary-node 183 0 1
✅ nextjs-turbopack-canary-quickjs 183 0 1
✅ nextjs-turbopack-quickjs-snapshot 183 0 1
✅ nextjs-turbopack-stable-node 183 0 1
✅ nextjs-turbopack-stable-quickjs 183 0 1
✅ nextjs-webpack-canary-node 183 0 1
✅ nextjs-webpack-canary-quickjs 183 0 1
✅ nextjs-webpack-stable-node 183 0 1
✅ nextjs-webpack-stable-quickjs 183 0 1
✅ nitro-stable-node 155 0 29
✅ nitro-stable-quickjs 155 0 29
✅ nuxt-stable-node 155 0 29
✅ nuxt-stable-quickjs 155 0 29
✅ sveltekit-stable-node 174 0 10
✅ sveltekit-stable-quickjs 174 0 10
✅ tanstack-start-node 155 0 29
✅ tanstack-start-quickjs 155 0 29
✅ vite-stable-node 155 0 29
✅ vite-stable-quickjs 155 0 29

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 155 0 29
✅ astro-stable-quickjs 155 0 29
✅ express-stable-node 155 0 29
✅ express-stable-quickjs 155 0 29
✅ fastify-stable-node 155 0 29
✅ fastify-stable-quickjs 155 0 29
✅ hono-stable-node 155 0 29
✅ hono-stable-quickjs 155 0 29
✅ nest-stable-node 155 0 29
✅ nest-stable-quickjs 155 0 29
✅ nextjs-turbopack-canary-node 183 0 1
✅ nextjs-turbopack-canary-quickjs 183 0 1
✅ nextjs-turbopack-quickjs-snapshot 183 0 1
✅ nextjs-turbopack-stable-node 183 0 1
✅ nextjs-turbopack-stable-quickjs 183 0 1
✅ nextjs-webpack-canary-node 183 0 1
✅ nextjs-webpack-canary-quickjs 183 0 1
✅ nextjs-webpack-stable-node 183 0 1
✅ nextjs-webpack-stable-quickjs 183 0 1
✅ nitro-stable-node 155 0 29
✅ nitro-stable-quickjs 155 0 29
✅ nuxt-stable-node 155 0 29
✅ nuxt-stable-quickjs 155 0 29
✅ sveltekit-stable-node 174 0 10
✅ sveltekit-stable-quickjs 174 0 10
✅ tanstack-start-node 155 0 29
✅ tanstack-start-quickjs 155 0 29
✅ vite-stable-node 155 0 29
✅ vite-stable-quickjs 155 0 29

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 155 0 29
✅ astro-stable-quickjs 155 0 29
✅ express-stable-node 155 0 29
✅ express-stable-quickjs 155 0 29
✅ fastify-stable-node 155 0 29
✅ fastify-stable-quickjs 155 0 29
✅ hono-stable-node 155 0 29
✅ hono-stable-quickjs 155 0 29
✅ nest-stable-node 155 0 29
✅ nest-stable-quickjs 155 0 29
✅ nextjs-turbopack-canary-node 183 0 1
✅ nextjs-turbopack-canary-quickjs 183 0 1
✅ nextjs-turbopack-quickjs-snapshot 183 0 1
✅ nextjs-turbopack-stable-node 183 0 1
✅ nextjs-turbopack-stable-quickjs 183 0 1
✅ nextjs-webpack-canary-node 183 0 1
✅ nextjs-webpack-canary-quickjs 183 0 1
✅ nextjs-webpack-stable-node 183 0 1
✅ nextjs-webpack-stable-quickjs 183 0 1
✅ nitro-stable-node 155 0 29
✅ nitro-stable-quickjs 155 0 29
✅ nuxt-stable-node 155 0 29
✅ nuxt-stable-quickjs 155 0 29
✅ sveltekit-stable-node 174 0 10
✅ sveltekit-stable-quickjs 174 0 10
✅ tanstack-start-node 155 0 29
✅ tanstack-start-quickjs 155 0 29
✅ vite-stable-node 155 0 29
✅ vite-stable-quickjs 155 0 29

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 178 0 6
✅ nextjs-turbopack-quickjs 178 0 6

✅ 🌐 Cross-language Conformance

App Passed Failed Skipped
✅ python 68 0 91

✅ dynamic-runs

App Passed Failed Skipped
✅ astro-vercel 0 0 0
✅ example-vercel 0 0 0
✅ express-vercel 0 0 0
✅ fastify-vercel 0 0 0
✅ hono-vercel 0 0 0
✅ nest-vercel 0 0 0
✅ nextjs-turbopack-vercel 0 0 0
✅ nextjs-webpack-vercel 0 0 0
✅ nitro-vercel 0 0 0
✅ nuxt-vercel 0 0 0
✅ sveltekit-vercel 0 0 0
✅ tanstack-start-vercel 0 0 0
✅ vite-vercel 0 0 0

✅ vercel-http-transport

App Passed Failed Skipped
✅ example 145 0 39
✅ express 145 0 39
✅ hono 145 0 39
✅ nextjs-turbopack 172 0 12
✅ nitro 145 0 39
✅ vite 145 0 39

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

✅ vercel-ws-transport

App Passed Failed Skipped
✅ example 145 0 39
✅ express 145 0 39
✅ nextjs-turbopack 172 0 12
✅ vite 145 0 39

📋 View full workflow run

Comment on lines +4411 to +4414
const run = await start(startHookDrillOwnerWorkflow, [token], {
hook: { token },
region,
});

@vercel vercel Bot Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The in-step child start omits drillToken attributes, so the final listing cannot recognize these owners and reports them missing even when they complete successfully.

Fix on Vercel

Comment on lines +463 to +468
afterAll(async () => {
if (records.some((r) => r.owner)) {
await checkListing(owner.workflowId, since).catch((error) => {
console.error('[drill] listing check failed', error);
});
}

@vercel vercel Bot Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the final listing finds multiple runs for a token, it only appends failures after all assertions have run, so the drill still passes despite violating its single-owner invariant.

Fix on Vercel

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 42 total

fence=per-spec

scenario outcome events virt replay violations
✅ smoke-no-steps completed 3 0ms ok 0
✅ smoke-one-step completed 6 0ms ok 0
✅ hook-at-step-started completed 12 0ms ok 0
✅ hook-at-step-completed completed 12 0ms ok 0
✅ hook-at-hook-created completed 12 0ms ok 0
✅ deadline-hook-wins completed 7 1.0h ok 0
✅ deadline-expires completed 7 1.0h ok 0
✅ step-vs-timer-early-settlement completed 8 1.0h ok 0
✅ long-sleep completed 11 30.0d ok 0
✅ hook-never-arrives stalled 3 0ms skipped 0
✅ step-retries-twice completed 10 2.0s ok 0
✅ parallel-steps completed 9 0ms ok 0
✅ hook-on-execution-state completed 12 0ms ok 0
✅ peek-hook-before-branch completed 12 0ms ok 0
✅ peek-hook-after-branch completed 12 0ms ok 0
✅ peek-hook-at-registration completed 12 0ms ok 0
✅ race-hook-before-probe completed 12 0ms ok 0
✅ race-hook-after-probe completed 12 0ms ok 0
✅ race-duplicate-delivery completed 13 0ms ok 0
✅ attr-hook-before-step completed 11 0ms ok 0
✅ attr-hook-after-step completed 11 0ms ok 0
✅ attr-from-step-body completed 13 0ms ok 0
✅ fork-hook-after-timeout completed 14 1.0m ok 0
✅ fork-hook-before-timeout completed 14 1.0m ok 0
✅ count-hook-after-timeout completed 17 1.0m ok 0
✅ count-hook-before-timeout completed 20 1.0m ok 0
✅ stale-read-step-count-fork completed 20 1.0m ok 0
✅ stale-read-equal-step-counts completed 14 1.0m ok 0
✅ step-vs-step-fork completed 12 0ms ok 0
✅ step-vs-step-fork-fenced completed 12 0ms ok 0
✅ fence-catches-benign-direction completed 12 5ms ok 0
✅ in-flight-before-decision completed 17 1.0m ok 0
✅ in-flight-before-decision-counted completed 17 1.0m ok 0
✅ in-flight-after-decision completed 19 2.0m ok 0
✅ stale-read-step-count-fork-fenced completed 20 1.0m ok 0
✅ fork-hook-wins completed 13 1.0m ok 0
✅ fork-timeout-wins completed 13 1.0m ok 0
✅ unclaimed-payload-under-fork completed 17 1.0m ok 0
✅ claimed-payload-under-fork completed 17 1.0m ok 0
✅ writers-independent-step-bodies completed 12 0ms ok 0
✅ writers-scripted-tempo completed 12 0ms ok 0
✅ cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim.txt

Runs concurrent start-or-route callers, an in-step start with a forced
step retry, a retention case, an orphan case and a forced queue
redelivery against a workflow-server whose start Hook faults are on, and
checks one owner per token, no trace of refused candidates, and that each
owner's log replays. Skipped unless WORKFLOW_E2E_START_HOOK_DRILL=1.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…ver drill preview

Points world-vercel at the workflow-server#1300 branch alias, advertises
atomicStartHook unconditionally, and runs the drill in the multi-region
lane. Reverted after the drill.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…flow-server drill preview"

This reverts commit fabfd4d.

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@alangenfeld
alangenfeld force-pushed the alangenfeld/2026-10-21-run-idempotency-sdk-prototype branch from c59fc83 to cc849a0 Compare October 22, 2026 22:22
@alangenfeld
alangenfeld force-pushed the alangenfeld/2026-10-21-run-idempotency-sdk-prototype branch 13 times, most recently from f7d93e9 to 89875b6 Compare October 29, 2026 15:55
@alangenfeld
alangenfeld force-pushed the alangenfeld/2026-10-21-run-idempotency-sdk-prototype branch from b99a1b6 to e22ecb2 Compare October 30, 2026 17:27
Base automatically changed from alangenfeld/2026-10-21-run-idempotency-sdk-prototype to main October 30, 2026 19:02

This branch was successfully deployed

1 active deployment
Preview – workflow-docs — eb12df62 Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant