UsageService test "upgrades a v4 cache" flakes with ENOTEMPTY: restarted service's cache write races temp-dir teardown #17546

What happened

apps/server/src/usage/UsageService.test.ts › "upgrades a v4 cache: reprices live Codex tiers, keeps deleted rollouts, leaves v4 intact" fails intermittently (about 1 run in 5 on my machine) with:

PlatformError: Unknown: FileSystem.makeTempDirectoryScoped (/var/folders/…/usage-service-v4-upgrade-testXXXXXX)
Caused by: Error: ENOTEMPTY: directory not empty, rmdir '/var/folders/…/usage-service-v4-upgrade-testXXXXXX/userdata'

Diagnosis

The second UsageService.make migrates the v4 cache and marks it dirty, so readSummary schedules a background persistCaches (Effect.forkDetach). The test never awaits that write. The service's own awaitPersisted finalizer is registered in the outer Effect.scoped scope, but the temp state directory belongs to the Effect.provide(layerService(...)) scope, which closes first. The cache write can therefore land while makeTempDirectoryScoped is removing userdata, and the rmdir fails with ENOTEMPTY.

I traced the order in every run (persist start/end and test end). In every run, passing or failing, the second persist starts before the inner test block ends and finishes after it. The run fails only when the write lands inside the rm -r window. The first service in the same test already calls awaitPersisted; the restarted one does not.

Steps to reproduce

From apps/server, run the single test repeatedly:

for i in $(seq 30); do pnpm exec vp test run src/usage/UsageService.test.ts -t "upgrades a v4 cache" >/dev/null 2>&1 || echo fail; done | wc -l

On main 31b04e2 this failed 6 of 30 runs.

Version

main at 31b04e2. The test arrived with #15101.

Environment

macOS (Apple silicon), Node v26.10.0, pnpm 11.10.0.

Evidence

With the fix below: 0 of 30 failures, against 6 of 30 without it, same machine and loop.

Related issues

None found (searched for the test name and ENOTEMPTY).

Fix applied or workaround

Await the restarted service's write before the layer tears down the state directory:

-          const summary = yield* (yield* UsageService.make).readSummary(WINDOW);
+          const restarted = yield* UsageService.make;
+          const summary = yield* restarted.readSummary(WINDOW);
@@
             legacy,
           );
+          // The migrated cache is written in the background; let it land before
+          // the layer removes the state directory under it.
+          yield* restarted.awaitPersisted;
         }).pipe(

I'm happy to open a PR with this if it's useful.

Filed by

@astarktc, while absorbing main into a downstream build.