Coverage job: "Worker exited unexpectedly" kills one fork in the last ~40s of the run (4 of 17 recent CI failures) #1824

Activity

1. thymikee commented on Aug 18, 2026

   

   thymikee

   on Aug 18, 2026

   MemberAuthor

   More actions

   One more instance while wave-2 PRs ran: #1825 first commit, Coverage job — same signature, rerun passed. (Also #1828 run 32140685486 Coverage failed on android-lifecycle.test.ts 'IME recovery records could not be persisted for Pixel 8' — different symptom, first-seen, noting only in case it's the same fork dying mid-write.)
2. thymikee

   added

   needs-triageNew or unreviewed issueNew or unreviewed issue

   on Aug 18, 2026
3. thymikee commented on Aug 18, 2026

   

   thymikee

   on Aug 18, 2026

   MemberAuthor

   More actions

   Diagnosis (wave 3, PR #1854). Not OOM, and not the tail of the run: a unit test sends real signals to a fabricated pid, and on the CI runner that pid periodically belongs to a sibling vitest fork.

   What the six deaths have in common (the four runs above plus 32151259761 and 31783654175, found by scanning the last 40 red CI runs for the same signature):

   run	file whose results vanished	tests collected before death
   32041468051	src/daemon/handlers/__tests__/snapshot-handler.test.ts	0 (died during import)
   32044136488	src/platforms/apple/core/__tests__/runner-session.test.ts	73 (died before the first result)
   32047734673	snapshot-handler.test.ts	0
   32126085337 (attempt 2)	src/utils/__tests__/daemon-client.test.ts	not even queued (died at fork start)
   32151259761	runner-session.test.ts	73
   31783654175	snapshot-handler.test.ts	0

   Those are the three largest unit-core files by bytes after remote-connection.test.ts. The sequencer sorts projects by name and files by size, so they are the first unit-core files, all alive in the same ~10 s window right after provider-integration finishes (~50–60 s into the run). The "last ~40 s" is the coverage report being written: the pool's AggregateError is only surfaced once every task settles, so the print time says nothing about the death time.

   The sender. runner-session.test.ts fabricates a runner child (makeBackgroundRunner(4242)) and mocks the liveness reads in host-process.ts (isProcessAlive → true) but not the signal writes, so killRunnerProcessTree sends real SIGINT/SIGTERM/SIGKILL to 4242 and to process group -4242 — 146 signals per run of the file (measured with a process.kill tracer in every fork; runner-request-cancellation.test.ts does the same to 4141/4242/4343/4444, ~70 s later). It runs at exactly the moment above, and its two siblings in that window are snapshot-handler and daemon-client.

   CI confirmation (https://github.com/callstack/agent-device/actions/runs/32165517507, Coverage job 95804064013): sentinel processes were parked on pids 4141/4242/4343/4444 before pnpm test:coverage:ci. All four were dead afterwards; 4242 logged received SIGTERM at 17:27:01.909 — 220 ms after the first unit-core file started its tests. In that run the forks alive at that moment had pids ~6100 after ~2 000 throwaway spawns from planting, i.e. ~4100–4300 in a normal run: 4242 sits inside the band. dmesg shows no OOM kill; runner has 16 GB / 4 cores.

   Fix (same PR): a shared setup file refuses any process.kill from a vitest worker to a pid it did not spawn (signal 0 stays free) and fails the sending test by name — planted red against the current tests: 2 files / 28 tests fail naming pid 4242 and killRunnerProcessTree. The group signal in runner-disposal.ts moves behind signalProcessGroupBestEffort in host-process.ts so the three runner tests mock signal writes where they already mock liveness reads.

   Open question I could not close: two of the six victims were runner-session.test.ts itself, and its own signalPidsBestEffort excludes process.pid; nothing else in the trace (macOS or Linux CI) signals that band in that window. The guard covers every process.kill path from any test regardless, so the 20-green-runs kill criterion is what settles it.
4. thymikee

   added 2 commits that reference this issue on Aug 19, 2026

   ci: w3-1824 experiment — trace fork signals and plant pid sentinels i…

   ...

   a8a0200

   test: refuse foreign-pid signals from unit-test workers

   ...

   84ffcc7
5. thymikee commented on Aug 19, 2026

   

   thymikee

   on Aug 19, 2026

   MemberAuthor

   More actions

   Confirmed, both directions, on the Coverage job itself.

   Before the fix — run 32165517507 (instrumentation-only head), sentinels parked on the pids the runner tests fabricate:

   pid 4141: DEAD   received SIGTERM
   pid 4242: DEAD   received SIGTERM   (17:27:01.909 — 220 ms after the first unit-core file began)
   pid 4343: DEAD   received SIGTERM
   pid 4444: DEAD   received SIGTERM
   kernel OOM / kill lines: (none)
   runner-session.test.ts: 26x kill -4242 SIGTERM | 27x kill -4242 SIGKILL | 20x kill -4242 SIGINT (+ the same to 4242)
   

   After the fix — run 32222448997, same probe, same job:

   pid 4141: ALIVE   pid 4242: ALIVE   pid 4343: ALIVE   pid 4444: ALIVE
   signals aimed at a pid that was also a vitest fork in this run: 0
   Test Files  983 passed (983)   Tests  7238 passed | 3 skipped
   

   Neither runner test appears in the kill trace any more; the only remaining signals in the whole run are tests killing daemon/Metro children they spawned themselves (daemon-client.test.ts, client-metro.test.ts) — legitimate, and allowed by the guard.

   PR #1854 carries the fix: the group-signal write moves behind signalProcessGroupBestEffort in src/utils/host-process.ts (next to signalPidsBestEffort), so signal writes and liveness reads share one seam, and src/__tests__/hermetic-signal-setup.ts refuses any process.kill from a worker to a pid it did not spawn — answering ESRCH so best-effort kill paths proceed, and failing the sending test by name. The instrumentation above is reverted in the PR's last commit.

   Worth naming: the negative-pid half (process.kill(-4242, …)) is a process-group signal, so this was never only a CI hazard — a local pnpm test could deliver SIGKILL to an unrelated group leader's children on a developer machine.

   Kill criterion for this issue

   Leaving this open until the fix has held. Done when 20 consecutive Coverage runs show no fork death:

   • 1 — run 32222448997 (fix head, instrumented, 983/983)
   • 2–20 — count post-merge Coverage runs on main; close when 20 consecutive are free of Worker exited unexpectedly

   Any recurrence before 20 means the guard missed a sender (it covers process.kill from a test worker; a signal sent by a subprocess a test spawned would not be refused) — reopen with the run URL and the vanished file rather than restarting the count.
6. thymikee commented on Aug 19, 2026

   

   thymikee

   on Aug 19, 2026

   MemberAuthor

   More actions

   There are two causes behind this signature, not one. Checking #1866's failure (run 32223038279, corpus-replay.test.ts made heavier, then fixed by shrinking its instrumented module graph) against my theory: it is not my mechanism with the timing shifted.

   • The vanished file is scripts/fuzz/corpus-replay.test.ts itself — a subprocess-stub member. That project runs fileParallelism: false, maxWorkers: 1 in a group that starts only after unit-core drains: last unit-core ✓ 06:27:32, stub files 06:27:38 → 06:27:49, death reported 06:28:43. Nothing else was executing.
   • The pid-signalling files finish ~3.5 min earlier — in my instrumented run, runner-session.test.ts ✓ 17:27:02 vs corpus-replay.test.ts 17:30:33. Their forks are already gone, and forks that late carry pids ~14 000, nowhere near 4141–4444.
   • My kill tracer recorded every foreign-pid signal across a full 983-file run: only those two files, only in the early window. It would have logged nothing at corpus-replay's death.
   • Their log has [vitest-pool]: Timeout terminating forks worker for … corpus-replay.test.ts and no JavaScript heap out of memory. A V8 heap-limit abort prints that line; a kernel OOM-kill prints only to dmesg. So: silent SIGKILL of a bloated fork — the original OOM hypothesis, correct for this shape.

   So:

   • Shape (A) — an early unit-core file dies while other files are running and the fabricated-pid signals are in flight. Sentinels dead → alive proves the mechanism and the fix. 5 of the 6 deaths I catalogued look like this. PR test: refuse foreign-pid signals from unit-test workers (Coverage fork death, #1824) #1854 closes it.
   • Shape (B) — one heavy V8-instrumented file dies alone in the tail, under memory pressure. test(fuzz): structured CLI/Maestro generators that reach command validation and assert error codes (#1781 B2) #1866's failure. PR test: refuse foreign-pid signals from unit-test workers (Coverage fork death, #1824) #1854 does not fix this. The remedy there is what test(fuzz): structured CLI/Maestro generators that reach command validation and assert error codes (#1781 B2) #1866 did (keep the instrumented graph small), or, if it recurs, a fork heap ceiling (poolOptions.forks.execArgv: ['--max-old-space-size=…']) or splitting the coverage run.

   I've switched #1854 from Closes to Refs so this issue stays open on its criterion rather than being auto-closed by a partial fix. Distinguishing the shapes at triage time is cheap: if the vanished file is a subprocess-stub member, or the run's only file in flight, it is (B); if it is an early unit-core file with siblings running, it is (A) and means the guard missed a sender.

   Checklist (unchanged, counting shape-(A)-free Coverage runs):

   • 1 — run 32222448997 (fix head, instrumented, 983/983, sentinels alive)
   • 2–20 — post-merge Coverage runs on main
7. thymikee commented on Aug 19, 2026

   

   thymikee

   on Aug 19, 2026

   MemberAuthor

   More actions

   Correction to my earlier comment, from review of #1854.

   I wrote that the guard's residual risk was "a signal sent by a subprocess a test spawned would not be refused — no such sender exists in the trace today". The trace I cited contradicts that. src/daemon/__tests__/request-router-open.test.ts spawns a real pkill -f 'xcodebuild.*test-without-building.*AgentDeviceRunner\.env\.session-…' twice per suite run — visible in both my before and after logs — because it mocks host-process.ts but not the exec seam. runner-disposal.ts reaches the process table two ways, process.kill and a spawned pkill (-P for a child tree, -f for stale xcodebuild), and my first pass guarded only the first. On a developer machine with a live Apple runner, pnpm test could have killed it.

   Fixed rather than merely documented: the setup file now refuses spawned kill/pkill/killall as well, answering ENOENT (which those best-effort callers already tolerate) and failing the sending test by name; request-router-open.test.ts stubs the Apple tool seam. A full 983-file run under the extended guard is clean, so that was the only such sender in the repo.

   Two other corrections while I'm here:

   • The signals aimed at a pid that was also a vitest fork: 0 line proves nothing and I should not have quoted it only under "after". It printed 0 in both runs, and could not have printed anything else: planting the sentinels burns ~1977 throwaway pids, which pushes every fork in an instrumented run above 4444. The sentinel deaths are the real result.
   • The fork-was-the-victim step is an inference, not an observation. Observed: the tests deliver real signals to 4141/4242/4343/4444; those signals kill whatever holds them (4242 at 17:27:01.909, 223 ms after the first unit-core file starts — that file signals only 4242; the other three together 69 s later, where runner-request-cancellation.test.ts runs); and the fix stops both. Inferred: that a sibling fork held one of those pids during the six catalogued deaths — arithmetic, since without the planting spawns the first src/ worker (6128) lands near 4151, inside the band. No run captures a fork's death certificate.

   On the checklist: at the observed base rate (6 shape-(A) deaths in ~100 runs), a broken fix still shows 20 consecutive clean Coverage runs about 29% of the time. So the count is a guard against regression, not the proof — the mechanism evidence above is what carries this, and shape-(B) tail deaths should not be counted against it either way.
8. thymikee

   added 3 commits that reference this issue on Aug 19, 2026

   test: refuse spawned kill/pkill writes too, and share the guard with …

   ...

   090cfb9

   test: end signal authority at child exit, and guard the promisified e…

   ...

   10c4db8

   test: refuse foreign-pid signals from unit-test workers (Coverage for…

   ...

   e4c3b42
9. thymikee

   closed this as completedon Aug 20, 2026
10. thymikee commented on Aug 20, 2026

    

    thymikee

    on Aug 20, 2026

    MemberAuthor

    More actions

    Closing: fixed at the source by #1854 (e4c3b420a).

    The OOM hypothesis in this issue was wrong, and the instrumentation run in that PR proved the real cause. The Apple runner tests fabricate runner child pids (4141/4242/4343/4444) and mocked the liveness reads in host-process.ts but not the signal writes, so killRunnerProcessTree delivered real SIGINT/SIGTERM/SIGKILL to those pids and their process groups — 146 signals per run of runner-session.test.ts. On the CI runner the sibling vitest forks live in that pid band, so the Coverage job periodically lost one fork mid-file.

    That explains every fact recorded here that OOM only loosely fit: exactly one file’s results vanishing, the SIGKILL-shaped death with no test emitting an error, and the silent gap at the end of a long run. It also explains why it was Coverage-only rather than a property of memory pressure.

    The fix is a permanent guard rather than a mitigation: src/__tests__/hermetic-signal-setup.ts records any process.kill to a process a worker neither is nor spawned, answers it with ESRCH so best-effort kill paths proceed as if the pid were dead, and fails the sending test by name in afterEach. A future test that fabricates a pid cannot reintroduce this silently.

    Verified since the merge: 3 CI failures in the window, all Swift Runner Host XCTests or Layering Guard. Zero fork deaths. With the contention retry removed (#1781 A4) there is no rerun layer masking it either, so the quiet window is a real signal.
11. thymikee

    added 8 commits that reference this issue on Aug 24, 2026

    fix(ci): run the nested-Worker fuzz replay outside the coverage lane

    ...

    0ac0fcc

    fix(ci): run the fuzz corpus replay outside the coverage lane

    ...

    12ab7fa

    fix(ci): run the fuzz corpus replay outside the coverage lane

    ...

    8a496ea

    fix(ci): run the fuzz corpus replay outside the coverage lane

    ...

    76ae7d2

    fix(ci): run the fuzz corpus replay outside the coverage lane

    ...

    1dd134f

    fix(ci): run the fuzz corpus replay outside the coverage lane

    ...

    eb169d7

    fix(ci): run the fuzz corpus replay outside the coverage lane

    ...

    ca3dd64

    fix(ci): run the fuzz corpus replay outside the coverage lane (#1994)

    ...

    104fe75