[Bug]: headless t3 serve runs at the default V8 heap and dies with "Ineffective mark-compacts near heap limit" after ~3 days with 40+ threads; a server restart drops every running turn #21370

Before submitting

• I searched existing issues and did not find a duplicate. [Bug]: headless t3 serve crashes on an unhandled 'read ECONNRESET' from one provider child's pipe; all 44 sessions stopped #16794 (ours) is a different crash cause (unhandled ECONNRESET) with the same consequence; Server exits on unhandled 'socket idle timeout' (UND_ERR_INFO) from HTTP/2 fetch since 0.0.46-nightly.20261010.2948, cancelling all running threads #18141 (closed) another. This one is the heap, plus the cost of any restart.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Run t3 serve headlessly with 40+ concurrent threads (Claude Code, Codex, ACP) and a multi-GB state database.
2. Leave it up for about 3 days.
3. Watch the process RSS and the V8 heap.

Expected behavior

• The launcher sets a heap ceiling scaled to the machine (or exposes a documented flag / setting), so a 20-core, 128 GB box does not run the orchestrator at the ~4 GB Node default.
• The server unloads idle thread state under memory pressure instead of growing until it dies.
• A crash restart resumes in-flight turns, or at least re-wakes the threads it stopped, so one process crash is not N lost turns.

Actual behavior

• 2026-09-24 23:57Z, after 3 days up: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory (Mark-Compact 3893 MB), core dump, cgroup peak 38.1 GB. systemd restarted the server; every running lane lost its turn.
• With a 12 GB heap flag added by us the process now peaks at ~14 GB RSS with 40–45 live threads (reported in [Bug]: headless t3 serve crashes on an unhandled 'read ECONNRESET' from one provider child's pipe; all 44 sessions stopped #16794), so the default ceiling is simply too small for this load.
• Every restart (crash or deliberate) stops every session. The restart reconciliation logged stoppedSessions: 44 in [Bug]: headless t3 serve crashes on an unhandled 'read ECONNRESET' from one provider child's pipe; all 44 sessions stopped #16794; after another restart 34 threads were left in error, their provider sessions dropped, and re-woken turns had no memory of their context. Our workaround has to re-wake each thread by hand.
• A power-cut reboot on 2026-10-17 had the same cost: 0 threads resumed on their own.

Impact

Major degradation or frequent failure

Version or commit

Heap crash: 0.0.43-nightly.20260924.2200. Restart cost: every nightly since, last seen 0.0.46-nightly.20261008.2819. Not re-tested without the heap flag on 2819.

Environment

Linux (Ubuntu, kernel 7.0.0-38-generic), compiled @t3code/t3-linux-x64 binary (Node.js v26.8.2 in crash traces), t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3 as a systemd --user service, headless; Tailscale serve in front; clients: macOS desktop app, iOS app, web. 40–50 live threads at a time over 8 Claude Code instances (claudeAgent), 3 Codex instances and ACP providers (Grok CLI, OpenCode with GLM). ~16,900 threads in statev2.sqlite (40.4 GB). Threads are created, woken and stopped by our own tooling over the /ws RPC (orchestration.dispatchCommand), not by a human in the UI.

Running build: 0.0.46-nightly.20261008.2819 with two local server patches (GitHub PR-sync cadence, Grok multi-task wait). Each draft says whether the patch touches the reported area. 128 GB RAM, 20 cores.

Logs or stack traces

# 2026-09-24 23:57Z (0.0.43-nightly.20260924.2200)
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 ... Mark-Compact 3893.0 (4134.9) -> 3887.1 (4136.4) MB ...
systemd[...]: t3-server.service: Main process exited, code=dumped, status=6/ABRT

# 2026-10-07 (0.0.46-nightly.20261005.2702), see #16794
restart reconciliation ... stoppedSessions: 44

Workaround

A systemd drop-in sets NODE_OPTIONS=--max-old-space-size=12288 for the server process (with a patch so the server removes it from the environment it passes to provider children), plus a timer that alarms when the server's heap climbs. After each restart our tooling re-wakes every thread that had work in flight.