Bug: workerd segfaults intermittently on shutdown under vitest-pool-workers (linux-x64 and macOS-arm64) #6763

Activity

1. trafgals commented on Jul 21, 2026

   

   trafgals

   on Jul 21, 2026

   More actions

   Confirming this on WSL2 (linux-x64, Ubuntu 22.04 inside Windows) with a slightly older toolchain, and offering one piece of additional root-cause / mitigation insight that may help triage:

   Versions that reproduce locally:

   • @cloudflare/vitest-pool-workers 0.12.21
   • wrangler 4.71.x
   • vitest 2.0.x
   • node 20.x

   Symptom shape (matches yours, but with a useful wrinkle): every test reports passing in stdout (e.g. Tests 794 passed (794)), but the vitest process exits 1 and prints [vpw:debug] Disposing remote proxy sessions... followed by [vitest-pool]: Worker cloudflare-pool emitted error … Worker exited unexpectedly. The "Worker exited unexpectedly" comes from miniflare's WebSocket close handler reacting to the workerd subprocess dying — not from a test failure.

   Insight that may be the actual blocker for upstream: in pool.close() inside node_modules/@cloudflare/vitest-pool-workers/dist/pool/index.mjs (lines ~8050–8072 at the 0.12.21 bundled version, the same shape survives into 0.16.x), both dispose promises are pushed onto an array and awaited via Promise.all(promises) with no .catch() wrapping:

   promises.push(
     forEachMiniflare(project.mf, async (mf) => {
       await waitForStorageReset(mf);
       await mf.dispose();          // <- rejects when workerd segfaults on shutdown
     })
   );
   // ...
   promises.push(sessionData.session.dispose()); // <- same here
   await Promise.all(promises);

   So the workerd segfault is the upstream trigger, but the test-side non-zero exit is the pool refusing to tolerate dispose rejection. Any single teardown rejection aborts the whole shutdown and bubbles out as exit 1, even though no test logic failed.

   Local mitigation (so the test suite can finish deterministically on linux-x64 / WSL2 / macOS-arm64 dev machines while this is open): wrap both dispose promises with .catch() that logs but doesn't propagate. We've shipped this in our repo via patch-package, which generates the following diff (manually trimmed of file-mode noise):

   --- a/node_modules/@cloudflare/vitest-pool-workers/dist/pool/index.mjs
   +++ b/node_modules/@cloudflare/vitest-pool-workers/dist/pool/index.mjs
   @@ -8056,6 +8056,8 @@ function pool_default(ctx) {
                forEachMiniflare(project.mf, async (mf) => {
                  await waitForStorageReset(mf);
                  await mf.dispose();
   +            }).catch((err) => {
   +              console.error("[vitest-pool-workers] miniflare dispose rejected (likely workerd shutdown segfault — see cloudflare/workerd#6763):", err?.message ?? err);
                })
              );
            }
   @@ -8064,7 +8066,9 @@ function pool_default(ctx) {
          log2.debug("Disposing remote proxy sessions...");
          for (const sessionData of remoteProxySessionsDataMap.values()) {
            if (sessionData?.session?.dispose) {
   -          promises.push(sessionData.session.dispose());
   +          promises.push(sessionData.session.dispose().catch((err) => {
   +            console.error("[vitest-pool-workers] remote proxy session dispose rejected (likely workerd shutdown segfault — see cloudflare/workerd#6763):", err?.message ?? err);
   +          }));
            }
          }
          remoteProxySessionsDataMap.clear();

   This makes our pre-commit hook (which runs frontend + workers vitest in parallel and treats exit 1 as a gate failure) behave the same as our Linux ARM64 self-hosted CI runners, which evidently use a different workerd binary path and don't trip this — that's likely why the failure surfaces only on certain hosts and not in cloudflare/workerd's own CI.

   Things this patch does not fix and which still point at workerd:

   • The Received signal #11 itself — needs the actual workerd-side fix.
   • Test files that get silently dropped from the pass counter when their worker dies mid-teardown. With the patch above they still report as passed because the in-file test bodies completed, but anyone depending on teardown assertions will need to wait for the upstream fix.

   Happy to gather coredumps or run with MINIFLARE_WORKERD_PATH pointed at a debug build if that would help narrow the call site. Mentioning in case it's useful for triage.
2. trafgals commented on Jul 21, 2026

   

   trafgals

   on Jul 21, 2026

   More actions

   Adding an update with the complete 3-layer chain I traced from the workerd segfault to the observable vitest exit-1, since the dispose-time .catch() wrapping from my previous comment turned out to only fix one of the amplifiers.

   The full chain (verified empirically with a local patch-package patch where 794/794 tests now pass deterministically while workerd still segfaults underneath):

   1. Upstream trigger: workerd SIGSEGV (this issue). The crash is the actual bug; everything downstream is how the surrounding tooling reacts to it.

   2. Amplifier Rename BUILD -> BUILD.bazel #1 in @cloudflare/vitest-pool-workers. Inside the bundled dist/pool/index.mjs runTests() (~line 7850 in 0.12.x, similar in 0.16.x), the WebSocket close handler does:

      if (event.code !== 1e3) {
        throw startupError ?? new Error("Failed to run tests");
      }

      When workerd dies post-task, the socket closes with a non-normal code, startupError is undefined, and this throw kills the fork — after all tests in the file have already been reported via the birpc listener. We patched this to log + debuglog2('DONE', files); return; when there's no startupError, and the patch is enough locally for the test results to stand. A real startupError still propagates unchanged.

   3. Amplifier Add tips for developing with vscode #2 in tinypool. When the vitest-side fork dies before terminate() was called, ProcessWorker.onUnexpectedExit emits Error("Worker exited unexpectedly") on the worker regardless of whether any task was in flight. There's no distinction between "worker crashed mid-task" and "worker exited cleanly after its task was done." I opened a small PR upstream proposing options.allowWorkerIdleExit (default false; when true, swallows the post-task case) — pending review.

   4. Closing indirection in vitest's CLI layer. The error from Fix clang 11 and make icudata embed portable #3 is then surfaced as [vitest-pool]: Worker cloudflare-pool emitted error at vitest/dist/chunks/cli-api.*.js ~3012, which is where vitest's exit-1 decision happens. So even if Add tips for developing with vscode #2 and Fix clang 11 and make icudata embed portable #3 are both mitigated and the fork death is silent, you still need Add tips for developing with vscode #2 fixed (the throw) for the kill-the-fork pathway.

   So three independent retry layers all assume "if my downstream died, that's a real problem" — none of which is true for teardown-after-task-completion noise. The workerd segfault needs fixing on its own merits; the pool-side layers are independently worth fixing because they make every flavor of upstream crash silent noise instead of test-side failures.

   For anyone reproducing locally: the patch-package mitigation I shipped that gets us green is the union of the dispose-.catch() from my prior comment + the throw-suppression in runTests(). With both applied, the segfault is still printed to stderr (visible via debuglog) but no longer affects exit code.

   Willing to bisect against a debug workerd build if that would help isolate the crashing frame — happy to coordinate on next steps.
3. trafgals commented on Jul 22, 2026

   

   trafgals

   on Jul 22, 2026

   More actions

   Adding the missing third layer that completes the chain — a never-resolving dispose promise that neither of my earlier fixes covered, plus the upstream PRs that came out of this triage.

   What my prior fix missed

   The two earlier patches (dispose .catch() wrapping + the runTests() throw-suppression) cover rejection paths. But when miniflare's I/O is waiting on a workerd process that no longer exists, the dispose promise doesn't reject — it just never resolves. Promise.all(promises) then waits indefinitely. Adding .catch() doesn't help a promise that never throws.

   The actual cap is a Promise.race against a timer:

   const DISPOSE_TIMEOUT_MS = 30_000;
   const timeout = new Promise((resolve) => setTimeout(() => {
     console.error("[vitest-pool-workers] dispose timeout after 30s (likely workerd shutdown segfault — see cloudflare/workerd#6763)");
     resolve();
   }, DISPOSE_TIMEOUT_MS));
   promises.push(timeout);
   await Promise.all(promises);

   30s is enough for healthy teardown while bounding the worst case. The timeout promise resolves, so Promise.all settles and the runner exits cleanly.

   Verification in mynextrow (60 test files / 794 tests)

   With all three patches applied (dispose .catch() + runTests() throw suppression + dispose-timeout race), the suite now finishes deterministically:

   Test Files  60 passed (60)
   Tests       794 passed (794)
   Start at    17:50:07
   Duration    363.88s (transform 39.40s, setup 0ms, collect 3341.98s, tests 638.17s)
   [vpw:debug] Shutting down runtimes...
   [vpw:debug] Disposing remote proxy sessions...
   [vitest-pool-workers] dispose timeout after 30s (likely workerd shutdown segfault — see cloudflare/workerd#6763)
   [All pre-commit unit tests passed]
   

   The 30s timeout log fires on every run — confirming workerd is still segfaulting underneath, but the dispose path no longer blocks the runner. Total wall-clock is unchanged from before; we just no longer wedge on the last dispose.

   Adjacent finding: WSL2-only forks-pool overload (separate issue)

   While validating, I hit a different failure on WSL2 (linux-x64 inside Windows) that initially looked like the same bug but isn't: when frontend vitest (forks pool) and workers vitest (cloudflare-pool) run in parallel via a pre-commit hook, the 30s vitest forks-pool handshakeTimeout is exceeded for ~10% of worker spawns. Manifests as Failed to start forks worker / Timeout waiting for worker to respond. Single-suite runs (frontend alone or workers alone) never reproduce.

   This is upstream vitest fork-spawn timing under WSL2 with combined fork + workerd subprocess pressure — not workerd itself. Sequential (frontend) && (workers) in the hook makes it disappear. Mentioning in case anyone else is debugging similar WSL-only flakes and lands here.

   Upstream PRs opened out of this triage

   • vitest-pool-workers — socket-error tolerance during teardown (the runTests() throw-suppression). PR open against cloudflare/vitest-pool-workers, review pending.
   • tinypool — options.allowWorkerIdleExit (default false; when true, swallows onUnexpectedExit if no task was in flight, distinguishing post-task crash from mid-task crash). PR open against tinypool/tinypool, review pending.
   • patch-package — workers/patches/@cloudflare+vitest-pool-workers+0.12.21.patch in mynextrow bundles all three local mitigations so the test suite ships green while upstream lands.

   Happy to coordinate on bisecting the actual workerd crash frame with a debug build — the long-running dispose suggests there's still an I/O handle keeping workerd's loop alive after the task channel closes.