Repository navigation
[core] Start the World at server boot so self-hosted Worlds recover in-flight runs - #4906
VaguelySerious wants to merge 1 commit into
Conversation
…t runs Add ensureWorldStarted() to workflow/runtime, register it as a Nitro server plugin for production builds, and stop world-local's start() from failing on the 'bundled' version sentinel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 525cfd4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed 🛠 Infra Events (absorbed by the harness)Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 🌐 Cross-language Conformance
✅ dynamic-runs
✅ vercel-http-transport
✅ vercel-multi-region
✅ vercel-ws-transport
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 127649ms → this run 105981ms (Δ -21668ms, -17%) ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟢 world-sim scenario book — 0 fail of 42 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
(AI) Preliminary review: AI review: blocking issues found
| } | ||
|
|
||
| if ( | ||
| !isVercelDeploy && |
There was a problem hiding this comment.
AI Review: Blocking
The start plugin also runs inside Nitro's build-time prerender server, so nitro build starts the World and runs workflow code.
The prerenderer creates a second Nitro instance with preset: 'nitro-prerender' and dev: false. Modules run again on it, so isVercelDeploy is false (the preset isn't 'vercel') and !nitro.options.dev is true. The plugin gets registered, and its ensureWorldStarted() runs in the build process as soon as there is a route to prerender. That holds even with NITRO_PRESET=vercel.
Repro on this branch: workbench/nitro-v3 with prerender: { routes: ['/prerendered'] } plus a trivial route. I left one run stranded in .workflow-data (started it, then SIGKILLed the server). Then DEBUG='workflow:*' pnpm exec nitro build:
[nitro] ℹ Prerendering 1 routes
[world-local] Upgrading from version 1.0.0 to 5.1.1
[world-local] Re-enqueued 1 active run(s) on startup
[workflow:runtime:debug] Starting workflow execution { workflowRunId: 'wrun_…', executionMode: 'replay' }
[workflow:runtime:debug] Workflow suspended { … waits: 1 }
[nitro] prerender:generate …
The build process replayed the stranded run through the prerender server's flow route. A run with a pending step had its delivery picked up and then cut off when the prerenderer closed. I got the same Re-enqueued 1 active run(s) line with NITRO_PRESET=vercel. With world-postgres and a database URL in the build environment, which is common because the same env serves build and runtime, every build would start Graphile Worker and re-enqueue the whole active set into the production database. Every build would also rewrite the data directory's version.txt.
Suggested fix: skip the plugin when nitro.options.preset === 'nitro-prerender', which is the check Nitro itself uses for dev-or-prerender, or return early on import.meta.prerender inside the virtual plugin. Add a unit test for the prerender preset next to the Vercel one.
| // A server bundle that inlines this package has no package.json to read, so | ||
| // the version is the 'bundled' sentinel (see getPackageInfo). There is no | ||
| // version to compare or record, so the version file is left as it is. | ||
| if (packageInfo.version === BUNDLED_VERSION) { |
There was a problem hiding this comment.
AI Review: Note
In a Nitro (rolldown) production build, getPackageInfo() never returns 'bundled'. It reads Nitro's generated .output/server/package.json.
world-local is inlined into .output/server/_chunks/*.mjs, so path.join(moduleDir, '../package.json') at line 64 resolves to .output/server/package.json. That file is {"name":"traced-node-modules","version":"1.0.0"}. The sentinel check here never fires for this layout, and initDataDir() goes on to record the wrong package. Observed on workbench/nitro-v3 with this branch:
# version.txt was written by the CLI/API process
@workflow/world-local@5.1.1
# boot the production server (the new start plugin runs initDataDir)
[world-local] Upgrading from version 5.1.1 to 1.0.0
# version.txt afterwards
traced-node-modules@1.0.0
upgradeVersion() is a no-op today, so nothing breaks yet. But every production boot now rewrites the data directory's version with a foreign package name, and it flips back and forth with the CLI or dashboard. Any future migration keyed on that version would misfire. On main this path was not hit at boot, because nothing called start(). The same misread also produces the dpl_local@1.0.0 vs dpl_local@5.1.1 deployment-pinning warning I saw on runs started from outside the server; that part already exists on main.
Suggest trusting the file only when name === PACKAGE_NAME, and falling back to BUNDLED_VERSION otherwise. The PR description says the bundled fix covers Vite-based Nitro. It may not cover Nitro v3's own bundler output.
| * | ||
| * @default !nitro.options.dev | ||
| */ | ||
| startWorld?: boolean; |
There was a problem hiding this comment.
AI Review: Note
Default-on for every non-Vercel preset, including serverless ones.
The gate is !isVercelDeploy && !dev, so the plugin is also registered for netlify, aws-lambda, cloudflare-*, deno-deploy, and similar presets. With world-postgres there, each cold start would start a Graphile Worker inside a short-lived function and re-enqueue the full active set. #5237 already reports that a full re-enqueue per boot can OOM a long-lived process. With node-cluster, every worker process repeats the scan.
Consider defaulting to on only for long-lived server presets (node-server, node-cluster, node-middleware, bun, deno-server) and off elsewhere. At minimum, the docs should say which presets get it.
| - **Not deduplicated against direct calls.** Code that calls `world.start()` itself runs a second start. Replace such calls with `ensureWorldStarted()`. | ||
|
|
||
| <Callout type="warn"> | ||
| Every process that calls `ensureWorldStarted()` re-enqueues every active run in storage, including runs another instance is executing. With several instances sharing one database, each instance's boot repeats that scan and re-enqueues the same runs, so boot-time work grows with the number of instances and the number of active runs. To recover from one instance only, call `ensureWorldStarted()` there and nowhere else (for Nitro, set [`startWorld: false`](/docs/configuration/framework-options#workflowstartworld) on the others). |
There was a problem hiding this comment.
AI Review: Note
"set startWorld: false on the others" has a side effect for world-postgres that the callout doesn't mention.
For world-postgres, start() does two jobs. It starts this instance's Graphile runner, so the instance consumes jobs, and it runs recovery. An instance with startWorld: false doesn't consume anything until its own first queue() call, which lazily runs queue.start(). A pure web replica that never enqueues never becomes a consumer. That may be what someone wants (see #3968 / #5129), but the docs should say it. Otherwise "recover from one instance only" quietly becomes "execute on one instance only".
This is the underlying problem: start() bundles "attach a consumer" with "sweep and re-enqueue the active set", and they want different policies. The consumer should start per instance. The sweep should run once, or be bounded or leader-elected (#3993, #5237).
| ## Behavior | ||
|
|
||
| - **Once per World.** Repeated and concurrent calls share one `start()`, including calls from the separate module copies a bundler creates per layer. A World injected later with [`setWorld()`](/docs/api-reference/workflow-runtime/set-world) is started by the next call. | ||
| - **Never rejects.** A failed start, such as a database that is unreachable at boot, is logged and the next call tries again. Runs stay durable in storage until a start succeeds. Awaiting it is optional: leave it unawaited if the server should accept requests while recovery scans storage. |
There was a problem hiding this comment.
AI Review: Note
"the next call tries again": in the Nitro plugin, and in the documented Next.js / SvelteKit snippets, there is no next call.
ensureWorldStarted() runs once at boot. If the database is unreachable at that moment, the failure is logged and recovery doesn't happen again until the process restarts. With world-postgres, the runner still starts lazily on the next enqueue, but the active-set re-enqueue never runs, so stranded runs stay stranded. Either retry with backoff inside the plugin, or reword this to say a failed start is retried only if your code calls it again.
| * forgotten, and the next call tries again. A queue-backed World also starts | ||
| * its workers on the next enqueue regardless. | ||
| */ | ||
| export const ensureWorldStarted = async (): Promise<void> => { |
There was a problem hiding this comment.
AI Review: Note
@workflow/nest still calls world.start?.() directly in #startWorld() (packages/nest/src/workflow.module.ts), so it isn't deduplicated against ensureWorldStarted().
A Nest app with manageWorldLifecycle: true that also follows the new docs starts the World twice, and runs recovery twice. Nest isn't in this diff, but this PR introduces the helper and says to replace direct calls with it, so switching Nest over here, or in a follow-up, would keep the integrations consistent.
Also on observability: a successful start logs nothing, and the re-enqueue count is debug-only. So an operator can't tell from the logs whether boot recovery ran at all. One info line from the plugin path, such as "World started; recovered N runs", would help when debugging "my run never resumed".
| expect(nitro.options.plugins).toEqual([]); | ||
| }); | ||
|
|
||
| it('never registers the plugin for Vercel deploys', async () => { |
There was a problem hiding this comment.
AI Review: Note
Missing tests:
- The prerender preset (see the blocking comment on
index.ts). - An e2e or integration test of the actual behavior change: production server, SIGKILL mid-sleep, restart with no further workflow operation, run completes. The PR description verifies this by hand. I reproduced it on
workbench/nitro-v3(8s sleep, SIGKILL at 2s,completedwithin 15s of the restart). Nothing in CI guards it, and feat(recovery): start the World at server boot to recover in-flight runs #2544 had such a test. nuxtpasses options throughnitro.workflow, butstartWorldisn't surfaced or documented for Nuxt users. They need to know aboutnitro: { workflow: { startWorld: false } }.
Closes #679. Supersedes #2544. Related: #1531 (closed, same root cause).
Problem
Self-hosted Worlds recover interrupted runs in
World.start(): world-local and world-postgres re-enqueue everypending/runningrun from storage. Onmain, nothing callsstart()at server boot, except@workflow/nest's opt-inmanageWorldLifecycle. A run that was sleeping, waiting on a hook, or between steps when the server stopped staysrunninguntil the user wiresstart()by hand. For world-local there is no other trigger. For world-postgres, Graphile Worker only starts on the next enqueue.Reproduced on
mainwithworkbench/nitro-v3(production build): startsleepingWorkflow(8000), SIGKILL the server 2s in, restart it, and issue no further workflow operations. The run is stillrunning25s after the restart. Same result on world-postgres with the workbench's own start plugin removed.Changes
workflow/runtime:ensureWorldStarted(). CallsgetWorld().start?.()once per World instance. Calls are deduplicated through aWeakMaponglobalThis, so the module copies a bundler creates per layer share one start, and a World injected later withsetWorld()gets started by the next call. It never rejects: a failed start is logged and retried on the next call. It does nothing on the Vercel World, which has nostart().workflow/nitro: start plugin, on by default for production builds. It covers Nitro, Nuxt, and Express/Hono/Fastify/Vite/TanStack Start on Nitro. It is a virtual server plugin with a bare dynamicimport('workflow/runtime'), so the bundler resolves it to the same runtime copy the flow handler uses. feat(recovery): start the World at server boot to recover in-flight runs #2544 found that an externalfile://import breaks withERR_INTERNAL_ASSERTIONfrom a CJS/ESM dual load. The plugin is never registered for Vercel deploys. A newstartWorldoption controls it (default!nitro.options.dev).@workflow/world-local: bundled-version fix (ported from feat(recovery): start the World at server boot to recover in-flight runs #2544). When world-local is inlined into a server bundle (Vite-based Nitro, Next standalone),getPackageInfo()returns'bundled', andinitDataDir()threwInvalid version string: "bundled". That madestart()fail before recovery. It now skips the version-file check for that sentinel. @rovo89 asked in feat(recovery): start the World at server boot to recover in-flight runs #2544 for this part to land separately.ensureWorldStarted()instead ofworld.start(). In production that dedupes them against the Nitro plugin. TheirWORKFLOW_TARGET_WORLDgates stay, so the local-world lanes behave as before.ensureWorldStartedAPI page; Postgres "Server startup" now uses the helper and says Nitro production builds need no setup; a new Local World section, "Recovering runs after a restart";startWorldis documented in the Nitro and framework-options references.Verification
workbench/nitro-v3andworkbench/viteproduction builds, with the server SIGKILLed mid-sleep and restarted with no further workflow operation:mainrunning(25s after restart)completed; slept 8.8s / 20.8s, so the remaining sleep was honored, not woken at rebootcompleted; slept 8.2s / 20.3sstartWorld: falserunning(36s), the negative controlworkbench/vite, world-local inlined)completed. Before the world-local fix,start()loggedInvalid version string: "bundled"and the run stayed stuckUnit tests:
ensureWorldStarted(dedupe under concurrent calls, waiting forstart(),setWorldre-start, fail-open with retry, synchronous throw, nostart()), Nitro plugin registration (prod on, dev off, explicit opt-in/opt-out, never on Vercel v2/v3, idempotent setup), and the world-local bundled sentinel.@workflow/core(3132),@workflow/nitro(39), and@workflow/world-local(676) pass, typecheck passes, and the docs link linter is clean.Reproduction script (not committed)
Run from
workbench/nitro-v3afterpnpm build. For Postgres, exportWORKFLOW_TARGET_WORLD=@workflow/world-postgresandWORKFLOW_POSTGRES_URLfor both the build and the script.Docs Preview
ensureWorldStarted()(new)workflow.startWorldworkflow/nitrooptions(Preview sits behind deployment protection and requires Vercel team access.)
What this PR deliberately does not carry over from #2544
#2544 grew to 41 files over four review rounds. This PR keeps the parts that were settled and leaves out the parts that drew blocking review comments or are already tracked elsewhere:
onRestart: 'cancel' | 'recover' | 'ignore'and terminally cancelled in-flight runs on every dev restart. The review threads found several hazards with it: aNODE_ENVpolarity that cancels production runs, a cancel that is not namespace-scoped on a shared Postgres DB (no namespace column onruns), custom Worlds that ignore the option, and a behavior change in the changelog. This PR leaves dev as it is onmain. The Nitro plugin is off undernitro devunlessstartWorld: true, and nothing cancels runs.WORKFLOW_SKIP_BOOT_RECOVERY,WORKFLOW_RECOVER_IN_DEV). The opt-out isstartWorld: falsefor Nitro, or not calling the helper elsewhere. world-local still honorsWORKFLOW_LOCAL_RECOVER_ACTIVE_RUNS.World.start()contract changes, and nostart()added to world-vercel.startstays optional, andensureWorldStarted()handles its absence.restart-recovery.test.ts(711 lines, a new CI job, de-gated to soak) is not ported. Coverage here is unit-level plus the manual repro above. It could follow up as a report-only lane.Open questions and alternatives
These are the places where the right answer was not clear-cut. Each lists what this PR picked and the alternatives.
1. Where the start is triggered
instrumentation.ts/ SvelteKitinitfor the rest.getWorld()or on the first flow-route delivery: this needs no user wiring, but it does not fix the bug. A stuck run produces no request, so nothing would trigger the start. It would also make the CLI and observability UI, which callgetWorld(), start workers and re-enqueue runs.@workflow/nextinjects instrumentation:instrumentation.tsis user-owned, and feat(recovery): start the World at server boot to recover in-flight runs #2544 found that re-exporting aregisterhelper throughworkflow/next/instrumentationbroke Turbopack through CJS interop. Next.js stays manual and documented.2. Dev-mode policy for runs interrupted by a dev restart
ReplayDivergenceError.deploymentIdmatches the current build, and leave the others. This is the most precise option, but in local dev the deployment id comes from the SDK version and not the code, so it would need a code-hash deployment id first.3. What happens when the start fails
manageWorldLifecycle, which this PR does not change and which still awaitsworld.start()directly. A strict mode could be an option on the helper if anyone needs it.4. Multi-instance and shared-DB deployments
startWorld: falseon all but one instance.recoverActiveRunsfor world-postgres), fix(world-postgres): skip parked runs and dedupe recovery jobs on startup #3162 (skip parked runs, dedupe recovery jobs), Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments #3138 (advisory-lock single-flight or owner-aware recovery), and [world-postgres] Renew running deliveries' job locks and redeliver abandoned jobs after a configurable stale window (opt-in) #4759 (job-lock renewal, so a dead instance's jobs are redelivered without boot recovery). These sit in world-postgres and compose with this PR.5.
World.start()optional or required🤖 Generated with Claude Code