Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load #12584

Activity

1. juliusmarminge commented on Sep 19, 2026

   

   juliusmarminge

   on Sep 19, 2026

   Member

   More actions

   Triage

   Confirmed as a real background-service V8 heap abort, not a one-off OOM from a single oversized payload and not a misconfigured unit. Reproduced twice on v0.0.42 (719a76ca1d) with the same main-thread stack. Current main @ dfbb11bdd still has the long-lived retention paths that fit this shape. I did not find a duplicate.

   Related, not the same ticket:

   • [Bug]: Desktop server private memory climbs ~25-30 MB/h while idle (immortal Effect parent spans + always-on tracing) #5410 / open fix(server): detach immortal parent spans from idle background work #9824 — idle desktop Private Bytes from immortal Effect parent spans (~25–30 MB/h with no provider work). That leak is still unfixed on main (forkParked still inherits the caller span). It is a contributor, not this crash: idle growth alone does not reach a V8 ceiling in 30–48h.
   • fix(server): bound provider event log records before serialization #12305 (on main, after 0.0.42) — bound provider event-log serialization.
   • fix(server): release oversized pull request diff cache entries #12523 (on main, after 0.0.42) — drop oversized PR diff cache entries.
   • fix(server): keep tool payloads out of completion queues #12345 — keep tool payloads out of completion queues (merged to t3code/codex-turn-mapping, not on main).
   • fix(server): bound getThreadProjection history with explicit paging #11514 — bound getThreadProjection with paging (open).
   • fix(server): forget a deleted thread's provider binding #8796 — delete provider_session_runtime rows when a thread is deleted (open).

   None of those is a multi-day busy-service OOM under hundreds of live sessions.

   What the report shows

   Two SIGABRTs, both v8::internal::V8::FatalProcessOutOfMemory on the server thread during Heap::CollectGarbage / AllocateRawWithLightRetrySlowPath:

   crash 1   2026-09-18 05:48:00Z   uptime 48.0h   cgroup memory peak 18.4 G
   crash 2   2026-09-19 11:35:36Z   uptime 29.8h   cgroup memory peak 15.4 G
   

   RSS 230 MB after 90 minutes, then fatal after a day-plus. That is retention, not a single huge allocation. The 15–18 GiB cgroup peaks include children (claude, git, extra node). The abort itself is the server process V8 heap.

   Workload matches the expensive path: many long-lived provider sessions, frequent turn churn, lots of git/worktree work, ~16.7k orchestration events, ~430 live session bindings. 13h57m CPU / 29.8h wall on the second window.

   The generated user unit still ships with no heap cap:

   t3code/apps/server/src/cloud/bootService.ts

   Lines 95 to 125 in dfbb11b

   	export function renderBootServiceUnit(plan: BootServicePlan): string {
   	// The user manager has no reliable network-online target; server networking retries itself.
   	return [
   	"[Unit]",
   	"Description=T3 Code server",
   	"StartLimitIntervalSec=300",
   	"StartLimitBurst=5",
   	"",
   	"[Service]",
   	"Type=simple",
   	"WorkingDirectory=%h",
   	`Environment=T3CODE_HOME=${quoteSystemdValue(plan.baseDir)}`,
   	`Environment=${BOOT_SERVICE_UNIT_ENV}=${BOOT_SERVICE_UNIT_FILE}`,
   	`ExecStart=${plan.program.map(quoteSystemdValue).join(" ")}`,
   	// Let the launcher mark an explicit stop before it signals the server.
   	// systemd still SIGKILLs the whole cgroup if graceful shutdown times out.
   	"KillMode=mixed",
   	// Agent tool calls run as children of the server, so they share this cgroup.
   	// With the systemd default of OOMPolicy=stop, the kernel killing one greedy
   	// child stops the whole unit: the server, every live agent, and the user's
   	// connection. Keep running and let Restart=always cover the main process.
   	"OOMPolicy=continue",
   	"Restart=always",
   	"RestartSec=5",
   	`StandardOutput=append:${escapeSystemdSpecifiers(plan.logPath)}`,
   	`StandardError=append:${escapeSystemdSpecifiers(plan.logPath)}`,
   	"",
   	"[Install]",
   	"WantedBy=default.target",
   	"",
   	].join("\n");

   Environment= is only T3CODE_HOME and the unit name. No NODE_OPTIONS, no --max-old-space-size. KillMode=mixed + Restart=always is why recovery is ~25s and why the cgroup teardown SIGKILLs in-flight provider/git children. OOMPolicy=continue only covers kernel OOM of a child, not a V8 abort of the main process.

   What the process still retains

   Several independent sites can grow for the life of t3 serve. Together they fit “small at 90 minutes, dead after 1–2 days” on a busy orchestration host.

   1. Adapter session maps keep every completed turn.

   Claude’s in-memory session is a process-lifetime Map, and each finished turn is appended — not replaced:

       context.turns.push({
         id: turnState.turnId,
         items: [...turnState.items],
       });

   items is the turn’s tool/content payload. The array is only rewritten on rollback. Codex / Cursor / OpenCode / Grok / Antigravity keep the same kind of Map<ThreadId, …SessionContext>. stopSession deletes the entry; if the session is still considered live, the history stays.

   2. The inactivity reaper will not touch a “busy” or background-live session.

   t3code/apps/server/src/provider/Layers/ProviderSessionReaper.ts

   Lines 42 to 95 in dfbb11b

   	for (const binding of bindings) {
   	if (binding.status === "stopped") {
   	continue;
   	}
   	const lastSeenMs = Date.parse(binding.lastSeenAt);
   	if (Number.isNaN(lastSeenMs)) {
   	yield* Effect.logWarning("provider.session.reaper.invalid-last-seen", {
   	threadId: binding.threadId,
   	provider: binding.provider,
   	lastSeenAt: binding.lastSeenAt,
   	});
   	continue;
   	}
   	if (now - lastSeenMs < inactivityThresholdMs) {
   	continue;
   	}
   	const thread = yield* projectionSnapshotQuery
   	.getThreadShellById(binding.threadId)
   	.pipe(Effect.map(Option.getOrUndefined));
   	// Ingestion updates this timestamp alongside activeTurnId when a turn
   	// settles. Long turns must get a full idle window after that transition,
   	// even though the binding was last touched when the turn was sent.
   	const lastActivityMs = Math.max(
   	lastSeenMs,
   	Date.parse(thread?.session?.updatedAt ?? binding.lastSeenAt),
   	);
   	const idleDurationMs = now - lastActivityMs;
   	if (idleDurationMs < inactivityThresholdMs) {
   	continue;
   	}
   	if (thread?.session?.activeTurnId != null) {
   	yield* Effect.logDebug("provider.session.reaper.skipped-active-turn", {
   	threadId: binding.threadId,
   	activeTurnId: thread.session.activeTurnId,
   	idleDurationMs,
   	});
   	continue;
   	}
   	// The turn can settle while background work runs on (subagent
   	// fleets, workflow runs, Monitor watch loops). Those live inside the
   	// provider process, so stopping the session would kill them silently,
   	// and nothing bumps lastSeenAt between turns.
   	if (thread?.backgroundLiveness != null) {
   	yield* Effect.logDebug("provider.session.reaper.skipped-background-work", {
   	threadId: binding.threadId,
   	backgroundLiveness: thread.backgroundLiveness,
   	idleDurationMs,
   	});
   	continue;
   	}

   Default idle window is 30 minutes. It skips stopped, anything with activeTurnId, and anything with backgroundLiveness (working / monitoring — subagent fleets, workflow runs, Monitor loops). On this host that is the intended workload, so ~430 live bindings can be real in-process sessions, not leftover SQLite rows.

   ThreadBackgroundLiveness is in-memory and only clears on task.completed (terminal status) or session.exited. A monitor/subagent that never emits a terminal event keeps the session off-limits to the reaper.

   Open #8796 is the other half: deleted threads stop the session but leave the provider_session_runtime row. That inflates the binding count; it does not by itself keep adapter maps warm.

   3. Live event / worker queues are unbounded.

   Every adapter publishes on Queue.unbounded<ProviderRuntimeEvent>(). Ingestion, checkpoint, PR, and command reactors use makeDrainableWorker (TxQueue.unbounded). A slow git/diff/ingestion worker holds every queued event, including diffs and tool payloads, until it drains. #12345 is the same class of bug and is not on main.

   4. Immortal parent spans (#5410) are still on main.

   forkParked still does Effect.forkScoped(effect) with the caller’s span:

   t3code/apps/server/src/serverActivation.ts

   Lines 12 to 26 in dfbb11b

   	export const forkParked = <A, E, R>(
   	effect: Effect.Effect<A, E, R>,
   	): Effect.Effect<void, never, Scope.Scope | R> =>
   	Effect.gen(function* () {
   	const activation = yield* ServerActivation;
   	if (activation === undefined) {
   	yield* Effect.forkScoped(effect);
   	return;
   	}
   	const parked = yield* Deferred.make<void>();
   	yield* Effect.forkScoped(
   	Deferred.succeed(parked, undefined).pipe(Effect.andThen(activation), Effect.andThen(effect)),
   	);
   	yield* Deferred.await(parked);
   	});

   Local file tracing is always on (server.trace.ndjson, Info + timing). Under idle load this was 25–30 MB/h; under this host’s turn/git churn it will be worse. #9824 is the proposed detach. Not merged.

   5. Full thread projections are still unbounded (#11514).

   orchestration.getThreadProjection still materializes every turn/payload/checkpoint for a long thread. A reconnect or snapshot refresh of many busy threads is a heap spike on top of the leak.

   16.7k orchestration events in SQLite is not large by itself. It only hurts if those rows (or their payloads) are decoded and held — replay, projection, live catch-up, or the completion-queue path above.

   What this is not

   • Not kernel OOM of a child. The stack is V8 FatalProcessOutOfMemory on the server thread.
   • Not “raise the heap and we are done.” --max-old-space-size only delays the abort. The generated unit also has no place to put it without a systemd drop-in.
   • Not already fixed in a release newer than 0.0.42. fix(server): bound provider event log records before serialization #12305 / fix(server): release oversized pull request diff cache entries #12523 help log/cache spikes; they do not bound session maps, reaper skips, unbounded workers, or parent spans.

   Suggested direction

   Do not start with a larger heap. Bound the process-lifetime sets:

   1. Drop completed-turn payloads from adapter session maps (or keep a small rolling window). Resume/rollback should read history from the provider store, not from an ever-growing turns[].
   2. Land fix(server): detach immortal parent spans from idle background work #9824 so parked reactors do not pin startup spans for the life of the service.
   3. Bound the runtime-event / DrainableWorker queues (drop or coalesce when the consumer is behind; bring fix(server): keep tool payloads out of completion queues #12345 onto main).
   4. Land fix(server): bound getThreadProjection history with explicit paging #11514 so projection reads cannot materialize a whole thread.
   5. Land fix(server): forget a deleted thread's provider binding #8796 so deleted threads do not accumulate bindings. Separately: sample provider.session.reaper.skipped-background-work / skipped-active-turn on this class of host — stuck liveness is how 430 “live” sessions stay in the Map.

   Graceful restart is a follow-up, not the fix. KillMode=mixed will still SIGKILL the cgroup if the main process aborts.

   What would help on the next occurrence

   The offer of --heapsnapshot-near-heap-limit=1 is the right next artifact. Also useful, if cheap:

   • process.memoryUsage() / v8.getHeapStatistics() on a 15–30 min timer (heapUsed vs external vs rss).
   • Counts: provider_session_runtime by status, adapter.listSessions() vs that table, and reaper skip reasons from the server log.
   • Whether server.trace.ndjson is rotating (traceMaxBytes is 10 MiB) or the file is huge.

   Raising the heap via a systemd drop-in is a valid delay, not a diagnosis:

   # ~/.config/systemd/user/t3code.service.d/heap.conf
   [Service]
   Environment=NODE_OPTIONS=--max-old-space-size=8192 --heapsnapshot-near-heap-limit=1
   

   Then systemctl --user daemon-reload && systemctl --user restart <unit>. Snapshots land in the service cwd (%h). Redact before attaching.

   No workaround is required for the reporter — Restart=always is already holding. Filing this is correct; a long-running t3 serve should not hit the V8 ceiling under normal sustained use.
2. juliusmarminge

   added

   acceptedfeature request acceptedfeature request accepted

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 19, 2026
3. OscarsOrbit commented on Oct 15, 2026

   

   OscarsOrbit

   on Oct 15, 2026

   More actions

   Data from another long-running host, from T3's own trace spans and a source read. No heap contents are shared.

   Runtime

   • 0.0.46-nightly.20261008.2833 (a6ec88f), SEA linux-x64, Node 26.8.2, V8 14.6.202.34, in a container. Our build carries a small local patch (MCP wiring and provider instructions only); it touches none of the files named below.
   • Since a graceful restart on 2026-10-15 at 08:00 UTC: NODE_OPTIONS=--max-old-space-size=12288 --heapsnapshot-near-heap-limit=1 --diagnostic-dir=<dir>. RSS was about 1.0 GiB shortly after.
   • At our next rebuild we move to a current nightly, which includes perf(shared): cached SQLite statements no longer hold large parameters #19305, and raise the cap to about 20 GiB.

   What we saw

   • Snapshot residue (an earlier process). One snapshot from the in-app SIGUSR2 handler at RSS 2926 MiB. The server.heapSnapshot span took 8.3 s and succeeded, writing a 267 MB file. The next server.eventLoop.stall span, 23 s later, had rssMb 4337. RSS settled near 5.5 GiB (high-water mark about 5.9 GiB) and stayed there. Stalls of 3-5 s recurred from then on.
   • Slope (a clean process, 6 GiB cap, no snapshot). A sawtooth of about ±500 MB of anonymous memory. The troughs rose about 0.3 GiB/h; trough-to-peak was up to about 0.9 GiB/h. RSS alone cannot separate retained objects from heap headroom.

   Source note

   v8.writeHeapSnapshot leaves V8's HeapObjectsMap allocated and object-move tracking on for the life of the process. HeapProfiler::TakeSnapshot sets is_tracking_object_moves_ = true (deps/v8/src/profiler/heap-profiler.cc:167 at nodejs/node v26.8.2, f2f2c2f2). Only ClearHeapObjectMap (:344-353) turns it off, and Node's writeHeapSnapshot path never calls it. Every later scavenge and compaction then reports each moved object to the profiler. So the SIGUSR2 handler (apps/server/src/observability/HeapSnapshot.ts:24, :46-49 at 26fffe5) can permanently raise RSS and per-GC cost on a long-running server. This is from source; we have not measured how much of our rise is the map and how much is the allocator keeping memory.

   Suggestions

   1. Take the in-app snapshot through an in-process node:inspector Session: HeapProfiler.takeHeapSnapshot, then HeapProfiler.disable, which calls ClearObjectIds (deps/v8/src/inspector/v8-heap-profiler-agent-impl.cc:333-341, same node commit). Otherwise, document that the residue lasts until restart.
   2. Put heapUsed, heapTotal, external, arrayBuffers and the heap limit on the server.eventLoop.stall span next to rssMb (apps/server/src/observability/EventLoopMonitor.ts:146, :151 at 26fffe5). That is the triage ask's data, taken at the moment that matters, and it separates GC-near-limit stalls from sync-I/O stalls without a snapshot.

   Disk

   Disk, not heap. Our statev2.sqlite is 10.24 GB; orchestration_events is 6.48 GB of it (63%) over about 1.70M rows, and orchestration_v2_events is empty. Since migration 055 orchestration_events also holds every v2 event, so we cannot yet say how much is v1 history. compactEventStore (apps/server/src/orchestration-v2/ProjectionMaintenance.ts:242 at 5a4e998) has no caller outside tests, which matches #18559.

   Offer

   We will take no further manual snapshot. If the near-limit snapshot fires, we will post a summary here: retainer classes and their sizes, never heap contents.