Repository navigation
[e2e] Add an opt-in start({ hook }) chaos drill - #5010
alangenfeld wants to merge 21 commits into
Conversation
Adds the start({ hook }) wire contract: StartHookSchema on run_created,
resilient run_started and the queued runInput; the atomicStartHook
capability; the run.startHook echo; WorkflowStartError and
START_HOOK_ADMISSION_REJECTED; and world-vercel's startHook frame meta,
409 start-hook-conflict / 400 start-hook-* error mapping, and the
WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK opt-in.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…es TBD Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Adds a shared atomic-start-hook conformance suite and implements it in world-local (reservation on the token claim file, run created under its lock, refusals recorded per candidate) and world-postgres (decision and run in one transaction, serialized per token with an advisory lock shared with createHook; migration 0028). Both let the owner's createHook adopt the reservation and move it with a force-claimed Hook. The run detail panel shows the start Hook token. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
start() publishes the candidate first and then decides it with run_created; the queued copy's run_started carries the same startHook, skips turbo, and acks a refusal (HookConflictError or START_HOOK_ADMISSION_REJECTED) without running code. Unconfirmed queue or admission throws WorkflowStartError. Stream arguments pipe only once the candidate is admitted. Cross-deployment starts check the target's probed capabilities (health checks now carry them, parsed tolerantly), keeping the no-probe-channel guard. run_started now forwards encryptionPublicKey. WorkflowStartError round-trips through the host, VM and QuickJS serdes; re-runs never forward a source run's start Hook. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Replaces the idempotency callout promising native atomic start with the
start({ hook }) API, a start-or-route-to-owner pattern, WorkflowStartError,
the World contract, and WORKFLOW_VERCEL_EXPERIMENTAL_START_HOOK.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…ntion Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…nsitions and re-adoption An adopted Hook's retention now extends the reservation itself, so disposing it and adopting again with a shorter one cannot shorten the promise. world-local also carried run.startHook through none of its field-by-field run rewrites, so a run_created retry after run_started found the run 'without this start Hook'. Both are now conformance cases. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
… reviver fixture Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
A by-token lookup that lands on a start reservation its run has not adopted answers HookReservedError (425 hook-reserved on the wire), retryable and never a HookNotFoundError. startHook carries optional serialized metadata the adopting Hook inherits when it brings none. WorkflowStartError is documented and marked retryable. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…n and carry start Hook metadata Lookups of an unadopted reservation of a live run throw HookReservedError, and not found once the run ends without adopting it. Adoption inherits the reservation's metadata unless the Hook brings its own. Postgres tracks adoption on the admission row (migration 0028), and its timezone test counts the new table. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…retry rules hook.metadata is serialized like createHook metadata. Without an explicit experimental_minRetention a start Hook keeps its token for 24 hours from start() where the World supports Hook retention (0 opts out). Inside a workflow a refusal, permanent rejection or contract error is final for the start() step, while WorkflowStartError stays retryable. HookReservedError round-trips through every serde. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…t retention Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…ugh an interrupted adoption A lookup between the adopting createHook's claim write and its hook_created publish answered not found, so callers retrying on HookReservedError gave up; it now stays HookReservedError while the owner runs. A retried adoption whose first attempt died before publishing now still inherits the reservation's metadata. The e2e routing test now actually passes metadata to start(). Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…s lost A lookup that finds the claim naming the adopting Hook but no entity now rebuilds the Hook from its published hook_created before answering HookReservedError, so a crash between the publish and the entity write does not leave the token reserved forever. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Points world-vercel at the server branch alias and advertises
atomicStartHook unconditionally, so the Vercel e2e lanes run the
start({ hook }) tests against the server's admission. Reverted after
pairing.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
…created lands Holds the direct run_created back so the queued copy's run_started admits the candidate through the resilient start against the real backend, then checks a duplicate is refused naming it, the late direct write replays the decision, and the log has one run_created. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Pairing passed: the start({ hook }) e2e tests ran and passed on the
node and quickjs Vercel lanes against the server preview.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
|
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3982 | 0 | 986 | 4968 |
| ✅ 💻 Local Development | 4785 | 0 | 551 | 5336 |
| ✅ 📦 Local Production | 4785 | 0 | 551 | 5336 |
| ✅ 🐘 Local Postgres | 4785 | 0 | 551 | 5336 |
| ✅ 🪟 Windows | 356 | 0 | 12 | 368 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 91 | 159 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ✅ vercel-http-transport | 897 | 0 | 207 | 1104 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 607 | 0 | 129 | 736 |
| Total | 20292 | 0 | 3078 | 23370 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 145 | 0 | 39 |
| ✅ astro-quickjs | 145 | 0 | 39 |
| ✅ example-node | 145 | 0 | 39 |
| ✅ example-quickjs | 145 | 0 | 39 |
| ✅ express-node | 145 | 0 | 39 |
| ✅ express-quickjs | 145 | 0 | 39 |
| ✅ fastify-node | 145 | 0 | 39 |
| ✅ fastify-quickjs | 145 | 0 | 39 |
| ✅ hono-node | 145 | 0 | 39 |
| ✅ hono-quickjs | 145 | 0 | 39 |
| ✅ nest-node | 145 | 0 | 39 |
| ✅ nest-quickjs | 145 | 0 | 39 |
| ✅ nextjs-turbopack-node | 172 | 0 | 12 |
| ✅ nextjs-turbopack-quickjs | 172 | 0 | 12 |
| ✅ nextjs-webpack-node | 172 | 0 | 12 |
| ✅ nextjs-webpack-quickjs | 172 | 0 | 12 |
| ✅ nitro-node | 145 | 0 | 39 |
| ✅ nitro-quickjs | 145 | 0 | 39 |
| ✅ nuxt-node | 145 | 0 | 39 |
| ✅ nuxt-quickjs | 145 | 0 | 39 |
| ✅ python-node | 66 | 0 | 118 |
| ✅ sveltekit-node | 164 | 0 | 20 |
| ✅ sveltekit-quickjs | 164 | 0 | 20 |
| ✅ tanstack-start-node | 145 | 0 | 39 |
| ✅ tanstack-start-quickjs | 145 | 0 | 39 |
| ✅ vite-node | 145 | 0 | 39 |
| ✅ vite-quickjs | 145 | 0 | 39 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 155 | 0 | 29 |
| ✅ astro-stable-quickjs | 155 | 0 | 29 |
| ✅ express-stable-node | 155 | 0 | 29 |
| ✅ express-stable-quickjs | 155 | 0 | 29 |
| ✅ fastify-stable-node | 155 | 0 | 29 |
| ✅ fastify-stable-quickjs | 155 | 0 | 29 |
| ✅ hono-stable-node | 155 | 0 | 29 |
| ✅ hono-stable-quickjs | 155 | 0 | 29 |
| ✅ nest-stable-node | 155 | 0 | 29 |
| ✅ nest-stable-quickjs | 155 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 183 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 183 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 183 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 183 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 183 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 183 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 183 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 183 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 183 | 0 | 1 |
| ✅ nitro-stable-node | 155 | 0 | 29 |
| ✅ nitro-stable-quickjs | 155 | 0 | 29 |
| ✅ nuxt-stable-node | 155 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 155 | 0 | 29 |
| ✅ sveltekit-stable-node | 174 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 174 | 0 | 10 |
| ✅ tanstack-start-node | 155 | 0 | 29 |
| ✅ tanstack-start-quickjs | 155 | 0 | 29 |
| ✅ vite-stable-node | 155 | 0 | 29 |
| ✅ vite-stable-quickjs | 155 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 155 | 0 | 29 |
| ✅ astro-stable-quickjs | 155 | 0 | 29 |
| ✅ express-stable-node | 155 | 0 | 29 |
| ✅ express-stable-quickjs | 155 | 0 | 29 |
| ✅ fastify-stable-node | 155 | 0 | 29 |
| ✅ fastify-stable-quickjs | 155 | 0 | 29 |
| ✅ hono-stable-node | 155 | 0 | 29 |
| ✅ hono-stable-quickjs | 155 | 0 | 29 |
| ✅ nest-stable-node | 155 | 0 | 29 |
| ✅ nest-stable-quickjs | 155 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 183 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 183 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 183 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 183 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 183 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 183 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 183 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 183 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 183 | 0 | 1 |
| ✅ nitro-stable-node | 155 | 0 | 29 |
| ✅ nitro-stable-quickjs | 155 | 0 | 29 |
| ✅ nuxt-stable-node | 155 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 155 | 0 | 29 |
| ✅ sveltekit-stable-node | 174 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 174 | 0 | 10 |
| ✅ tanstack-start-node | 155 | 0 | 29 |
| ✅ tanstack-start-quickjs | 155 | 0 | 29 |
| ✅ vite-stable-node | 155 | 0 | 29 |
| ✅ vite-stable-quickjs | 155 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 155 | 0 | 29 |
| ✅ astro-stable-quickjs | 155 | 0 | 29 |
| ✅ express-stable-node | 155 | 0 | 29 |
| ✅ express-stable-quickjs | 155 | 0 | 29 |
| ✅ fastify-stable-node | 155 | 0 | 29 |
| ✅ fastify-stable-quickjs | 155 | 0 | 29 |
| ✅ hono-stable-node | 155 | 0 | 29 |
| ✅ hono-stable-quickjs | 155 | 0 | 29 |
| ✅ nest-stable-node | 155 | 0 | 29 |
| ✅ nest-stable-quickjs | 155 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 183 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 183 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 183 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 183 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 183 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 183 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 183 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 183 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 183 | 0 | 1 |
| ✅ nitro-stable-node | 155 | 0 | 29 |
| ✅ nitro-stable-quickjs | 155 | 0 | 29 |
| ✅ nuxt-stable-node | 155 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 155 | 0 | 29 |
| ✅ sveltekit-stable-node | 174 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 174 | 0 | 10 |
| ✅ tanstack-start-node | 155 | 0 | 29 |
| ✅ tanstack-start-quickjs | 155 | 0 | 29 |
| ✅ vite-stable-node | 155 | 0 | 29 |
| ✅ vite-stable-quickjs | 155 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 178 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 178 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 91 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 145 | 0 | 39 |
| ✅ express | 145 | 0 | 39 |
| ✅ hono | 145 | 0 | 39 |
| ✅ nextjs-turbopack | 172 | 0 | 12 |
| ✅ nitro | 145 | 0 | 39 |
| ✅ vite | 145 | 0 | 39 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 145 | 0 | 39 |
| ✅ express | 145 | 0 | 39 |
| ✅ nextjs-turbopack | 172 | 0 | 12 |
| ✅ vite | 145 | 0 | 39 |
| const run = await start(startHookDrillOwnerWorkflow, [token], { | ||
| hook: { token }, | ||
| region, | ||
| }); |
| afterAll(async () => { | ||
| if (records.some((r) => r.owner)) { | ||
| await checkListing(owner.workflowId, since).catch((error) => { | ||
| console.error('[drill] listing check failed', error); | ||
| }); | ||
| } |
Sim WorldSimulated world deterministic testing for races. Traces 🟢 world-sim scenario book — 0 fail of 42 total
Full trace: |
12a71df to
755baa9
Compare
755baa9 to
3fc669d
Compare
Runs concurrent start-or-route callers, an in-step start with a forced step retry, a retention case, an orphan case and a forced queue redelivery against a workflow-server whose start Hook faults are on, and checks one owner per token, no trace of refused candidates, and that each owner's log replays. Skipped unless WORKFLOW_E2E_START_HOOK_DRILL=1. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
3fc669d to
de98bb9
Compare
…ver drill preview Points world-vercel at the workflow-server#1300 branch alias, advertises atomicStartHook unconditionally, and runs the drill in the multi-region lane. Reverted after the drill. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
de98bb9 to
fabfd4d
Compare
…flow-server drill preview" This reverts commit fabfd4d. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
c59fc83 to
cc849a0
Compare
f7d93e9 to
89875b6
Compare
b99a1b6 to
e22ecb2
Compare
Summary & Motivation
This adds end-to-end coverage for
start({ hook })(#4996, the atomic keyed start asked for in #2376). The drill sends real start Hook traffic, mostly cross region, to a workflow-server that has its start Hook faults turned on (workflow-server#1300). For each token it checks that there is exactly one owner and that every caller resolves to that owner. It also checks that refused candidates leave no trace and that the owner's log replays.It only runs when
WORKFLOW_E2E_START_HOOK_DRILL=1is set. Setting it todryinstead runs a fault-free smoke check against a deployment before a full drill.Test Plan
I ran the full drill in the multi-region lane against the workflow-server#1300 preview, and all five cases passed: