Skip to content

fix(queen): a beat thread is ended only once it has loaded, so a node stopped at its start no longer crashes Bun 1.3.x (Closes #1813) - #1837

Merged
gHashTag merged 2 commits into
queenfrom
fix-1813-pglive-bun-panic
Oct 11, 2026
Merged

gHashTag merged 2 commits into
queenfrom
fix-1813-pglive-bun-panic

Conversation

@gHashTag

Copy link
Copy Markdown
Collaborator

Closes #1813

Cause

Not a leaked pool or socket: a Bun 1.3.x crash that our own stop path triggered.

  • Both CI crashes (runs 38099896355 and 38103211817, attempt 1) happened during the last test of tests/pglive/queen-actors-pg-edges-live.test.ts, "a node whose UNLISTEN fails at stop". Their bun.report link is identical. It decodes to SavedSourceMap.putValue <- js_printer.printAst <- RuntimeTranspilerStore.TranspilerJob.runFromWorkerThread, on Bun's transpiler thread pool, with workers_spawned(23) workers_terminated(13).
  • That test starts a PgLink and stops it at once. createPgLink spawns the beat thread (queen-actors-pg-beat.ts), and halt() called terminate() while the thread's module graph was still being transpiled. The transpile job then wrote into the source map table of the worker that was gone.
  • A minimal repro with no trios code reproduces it: a worker importing 300 small TS modules, terminated 2-20 ms after spawn, 40 times per run. It panics on almost every run on Bun 1.3.6 and 1.3.11 (macOS arm64), with the same stack as CI. The panic does not appear on Bun 1.4.3.
  • Our handles are not the cause. A link that is stopped and whose pool is ended leaves 0 connections in pg_stat_activity, because terminate() closes the beat thread's socket.

Fix

  • queen-actors-pg-beat.ts posts { loaded: true } once its module body runs, which means every one of its modules is loaded. It does not post this on the main thread, where the beat live test imports nodeBeat.
  • In createPgLink, halt() now ends the beat thread through endBeat(). That function waits for the loaded word, or for the error or close event that ends a failed load, and only then calls terminate(). Every halt shares this one end, so fence() followed by stop() terminates the thread once.
  • There is no timeout. Every way a load can end fires one of those three events (checked on 1.3.6 and 1.4.3: a missing import, a syntax error, a top-level throw and a missing entry each fire error).
  • Production benefits too: a node fenced or stopped right after its start no longer crashes the server process on Bun 1.3.x.
  • No new decision rule and no threshold: this is thread lifecycle glue only.

Test

New edges test: "a node stopped as soon as it starts ends its beat thread only once the thread has loaded". It wraps Worker, records whether each terminate() came after the thread said it loaded, and runs 20 start/stop cycles.

  • Negative control: with this test on the old runtime code, it failed 6 of 6 runs (all 20 terminates came before the load), on both Bun 1.3.6 and 1.3.11.
  • With the fix it passes, in about 1 s.

Results (local, Postgres 16, this Mac)

what Bun runs panics test failures
minimal repro, terminate 2-20 ms after spawn (old pattern) 1.3.6 4 3 (plus 1 SIGSEGV exit 139) n/a
minimal repro, terminate 2-20 ms after spawn (old pattern) 1.3.11 4 4 n/a
minimal repro + real beat worker, many delays 1.4.3 20 x 40 cycles 0 n/a
terminate only after the worker said it loaded 1.3.6 / 1.3.11 10 x 40 cycles each 0 / 0 n/a
whole pglive suite with the fix (probe) 1.3.6 1 0 0 (156 pass)

The 20-run results for the edges file and the whole pglive suite follow in a comment.

Not done

  • Not filed upstream: Bun 1.4.3 no longer reproduces it, so a report would describe a bug the current release does not have.
  • Bun was not bumped. The CI pin stays at 1.3.6, the same as packageManager and production.

🤖 Generated with Claude Code

… stopped at its start no longer crashes Bun 1.3.x (Closes #1813)

Bun 1.3.6 and 1.3.11 segfault (0xFFFFFFFFFFFFFFF8, SavedSourceMap.putValue
on a transpiler pool thread) when a Worker is terminated while its module
graph is still being transpiled. The pglive edges test stops a PgLink
within milliseconds of its start, so createPgLink's halt terminated the
beat thread mid-load and the whole pglive suite died after its tests had
passed.

The beat thread now says it has loaded once its module body runs, and
halt ends it only after that word, or after the error or exit that ends
a failed load. A new edges test watches every beat thread's terminate and
fails on the old code 6 of 6 times. Bun 1.4.3 survives the same churn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tri-claw

tri-claw Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Summary by t27-bees

The change fixes a Bun 1.3.x crash caused by terminating a worker thread while its modules were still being transpiled. The beat thread now signals { loaded: true } once fully initialized, and halt() waits for that signal (or an error/close) before calling terminate().

Worth a look

  • trios/agent-server/apps/server/src/api/services/queen-actors-pg.ts:312-313 — beatLoaded starts as Promise.resolve() but is reassigned only when url is provided and a Worker is created. If url is falsy, endBeat() will immediately resolve and call beatThread?.terminate() on undefined (safe via optional chaining), but the logic implicitly assumes a beat thread exists whenever halt() is called.
  • trios/agent-server/apps/server/src/api/services/queen-actors-pg.ts:324-330 — endBeat() memoizes beatEnded. If halt() is called concurrently (e.g., fence() then stop()), both await the same promise. The first caller triggers termination; the second waits on the already-settled promise. This is correct but relies on beatThread not being reassigned between calls (it isn't).
  • trios/agent-server/apps/server/src/api/services/queen-actors-pg.ts:333-334 — halt() calls endBeat() before releasing the listener, then awaits beatGone after. If UNLISTEN hangs, the beat thread termination is already in flight (waiting on beatLoaded). This order is fine, but the beat thread could outlive the listener cleanup by an unbounded time if beatLoaded never resolves (though the PR asserts error/close always fire).
  • trios/agent-server/apps/server/src/api/services/queen-actors-pg.ts:350-356 — The beatLoaded promise resolves on message (with loaded: true), error, or close. If the worker crashes after sending loaded: true but before terminate() is called, the close handler would fire on an already-resolved promise (no-op). If it crashes before, error/close resolves the promise and terminate() is called on a dead worker (safe). No explicit handling for a worker that never fires any of the three events — the PR claims this cannot happen on tested Bun versions.
  • trios/agent-server/apps/server/tests/pglive/queen-actors-pg-edges-live.test.ts:473-474 — The test replaces globalThis.Worker with a wrapper that tracks loaded state. It assumes the message event for { loaded: true } fires before any terminate() call in the fixed code. If the wrapper's message listener is added after the worker has already posted the message (race at construction), this.loaded would stay false and the test would fail spuriously. The wrapper adds the listener in its constructor before super(...args) returns, so the race window is minimal but non-zero.

Read: 3 files, +70 -3, the whole diff. Head ba4a685. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

✅ Tests passed — 3312/3381

Suite Passed Failed Skipped
✅ agent 87/87 0 0
✅ build 9/9 0 0
✅ cdp-protocol 5/5 0 0
✅ eval 93/93 0 0
✅ server-agent 280/280 0 0
✅ server-api 2018/2086 0 68
✅ server-browser 6/6 0 0
✅ server-integration 10/11 0 1
✅ server-lib 279/279 0 0
✅ server-pglive 164/164 0 0
✅ server-root 68/68 0 0
✅ server-sim 4/4 0 0
✅ server-skills 31/31 0 0
✅ server-tools 244/244 0 0
✅ shared 14/14 0 0

View workflow run

@gHashTag

Copy link
Copy Markdown
Collaborator Author

20-run results (local: Postgres 16, macOS arm64)

Every run used the command run-test-group.ts builds for a group (bun --env-file=.env.development test --preload=./tests/__helpers__/test-env.ts <target>), minus its cleanup step, which kill -9s whatever holds ports 9005/9105/9305 on a shared machine.

what tree Bun runs clean panics test failures
queen-actors-pg-edges-live.test.ts this PR (a2ec6bb) 1.3.6 20 20 (12 pass each) 0 0
queen-actors-pg-edges-live.test.ts this PR (a2ec6bb) 1.3.11 20 20 (12 pass each) 0 0
queen-actors-pg-edges-live.test.ts queen 55d575b, unmodified 1.3.6 20 19 1 (run 20) 0
queen-actors-pg-edges-live.test.ts queen 55d575b, unmodified 1.3.11 20 19 1 (run 7) 0
whole pglive suite (./tests/pglive) this PR (a2ec6bb) 1.3.6 20 consecutive 20 (156 pass each, about 507 s per run) 0 0

At about 1 in 20 runs, the file loop alone cannot separate the two trees. The same input with a higher dose does. Each run below is 200 cycles of a real createPgLink start, a random 0-30 ms wait, then stop():

tree Bun runs crashed
queen unmodified 1.3.6 10 7
queen unmodified 1.3.11 10 3
this PR 1.3.6 10 0
this PR 1.3.11 10 0

Every crash had the same stack as both CI failures: SavedSourceMap.putValue on a RuntimeTranspilerStore job, segfault at 0xFFFFFFFFFFFFFFF8. The CI run of this head (38105164273) passed, server-pglive 156/156.

On the t27-bees review points:

  • No endBeat before a thread exists. endBeat cannot run before the beat thread exists. The thread is created in the same synchronous continuation as the end of LISTEN, and nothing before it can fence.
  • The listeners go with the thread. They live on the terminated Worker and are collected with it.
  • The UNLISTEN window holds no renewal. The beat thread has its own pool, and its interval is armed only after it loads, with its first renewal 5 s later. So nothing renews in the window before terminate().
  • loaded is posted only from a thread on purpose. Only threads are ever awaited.
  • The Worker swap is always undone. The restore runs in afterEach, which runs on a failure too.

🤖 Generated with Claude Code

@gHashTag
gHashTag merged commit f1b59b2 into queen Oct 11, 2026
17 of 19 checks passed
dmitrii-f-t27 pushed a commit to dmitrii-f-t27/trios that referenced this pull request Oct 11, 2026
…so a stop at its start no longer crashes Bun 1.3.x (Closes Trinity-S-AI-DNA#1865)

threadWork terminated its worker at once on stop, even while the thread
was still transpiling its entry or its work module. Bun 1.3.6 (the CI and
production pin) then crashes the whole process (SavedSourceMap.putValue,
Segmentation fault at 0xFFFFFFFFFFFFFFF8), the crash Trinity-S-AI-DNA#1813 hit on the
PgLink beat thread.

The thread entry now posts { loaded: true } once its work module's import
has settled. Every terminate goes through one end() that waits for that
word, or for the error or close event that ends a failed load, and then
terminates once. Same pattern as Trinity-S-AI-DNA#1837. No rule changed: when to stop
stays with kill_effect and escalation.

New test: 200 cycles of start, a random 0-30 ms wait, then stop, on the
work's own stop and through the runtime with turn stop on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dmitrii-f-t27 added a commit to dmitrii-f-t27/trios that referenced this pull request Oct 15, 2026
…loses Trinity-S-AI-DNA#1810)

The flaky pglive case "mail to a node that died is expired, not kept for
ever" saw 0 rows where it expected 5. The racing pair: the poll's read of
node 2's mailbox that was on its way when b.link.stop() was called, and
the sends to node 2 right after the stop. stop() did not wait for that
drain, so the read could reach the store after the sends, hand the mail
to the stopped node's actor and acknowledge (delete) it.

- stop() now waits for the drain that was running.
- A read that answers after the stop hands nothing on and acknowledges
  nothing; a drain checks the stop before each read.

New live case holds that read until the sends are in, so the race runs
on every run. It fails on the old code (all 5 handled on the stopped
node) and with only the post-read check removed; passes with the fix.

No .t27 change: no netlink card decision changes; stop() ordering is the
I/O shell, as in Trinity-S-AI-DNA#1837.

Refs Trinity-S-AI-DNA#1862
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant