Oversized Codex diff event exhausts desktop backend heap

[alexfertel]

What happened

The T3 Code desktop backend stops without warning. The UI then marks active work as failed and shows:

Provider session did not survive a server restart. Send a new message to continue.

The fault happened three times over two days.

Diagnosis

The desktop backend runs out of V8 heap while it handles a large Codex turn/diff/updated notification.

CodexAdapter places the same diff in two fields of the canonical event:

• raw.payload.diff
• payload.unifiedDiff

Source:

• apps/server/src/provider/Layers/CodexAdapter.ts:985
• apps/server/src/provider/Layers/CodexAdapter.ts:1646

ProviderService sends this canonical event to the provider event logger before it publishes the event:

• apps/server/src/provider/Layers/ProviderService.ts:916

EventNdjsonLogger serializes the complete event and holds the resulting line in memory:

• apps/server/src/provider/Layers/EventNdjsonLogger.ts:497
• apps/server/src/provider/Layers/EventNdjsonLogger.ts:620

In the observed crash, one redacted canonical log record exceeded 250 MiB. The duplicate diff and temporary serialization copies pushed the backend past its heap limit.

After the backend restarts, reconcileProviderSessions marks sessions whose provider process is gone as failed. This creates the user-facing restart message:

• apps/server/src/serverRuntimeStartup.ts:338

Expected behavior: T3 Code should bound, omit, or summarize large raw diff data before diagnostic serialization. One provider event should not stop the backend or unrelated work.

Steps to reproduce

Proposed source-level reproduction:

1. Start the T3 Code desktop backend with the Codex provider.
2. Send a valid turn/diff/updated notification with a large generated string in params.diff.
3. Let CodexAdapter convert it to a canonical event.
4. Let the canonical provider event logger serialize the event.
5. Observe that the serialized record contains the diff twice.
6. With a large enough diff or existing heap use, V8 stops the backend with an out-of-memory error.

A standalone synthetic test has not yet confirmed the minimum payload size.

Version

0.0.41-nightly.20260909.1439, commit 6c583620ff7a

This was the newest nightly, and its commit matched main when checked.

Environment

• T3 Code desktop app with local backend
• Darwin 25.6.0, arm64
• Node.js 25.6.0
• Codex CLI 0.153.4

Evidence

<--- Last few GCs --->

Scavenge 1856.5 (...) -> 1856.7 (...) MB
Scavenge 1856.7 (...) -> 1848.5 (...) MB

FATAL ERROR: Zone Allocation failed - process out of memory

Sanitized provider log measurement:

stream: CANON
method: turn/diff/updated
record size: greater than 250 MiB
payload content: redacted

No secrets, source paths, thread identifiers, thread titles, commands, query data, or payload content are included.

Related issues

• [Bug]: app-server stdin reader is quadratic in line length — OOM crash on large tool payloads #5389: A large Codex protocol line caused quadratic copying in the input reader. That issue is closed. This report concerns later canonical event duplication and logging.
• [Bug]: Desktop backend still SIGABRTs with V8 OOM after 9–18h on 0.0.35 (hydration bounds already present) #8648: The desktop backend can still stop with a V8 out-of-memory error. This report provides a specific active-event path.
• Codex thread resume can exhaust server heap on large histories #7075: Large Codex history can exhaust heap during resume. This report does not use the resume path.

Fix applied or workaround

Restarting the desktop app restores the backend for a time. Avoiding very large single-event diffs reduces the risk. No local data was changed.

Filed by

Codex, GPT-5, via t3 triage

[juliusmarminge]

Triage

Confirmed on current main (6c583620ff7a, same commit as the nightly in the report). This is a real backend heap bug, not a duplicate of #8648 or #7075.

What the code does

CodexAdapter builds every canonical event with the native payload attached, then maps turn/diff/updated onto payload.unifiedDiff as well:

// apps/server/src/provider/Layers/CodexAdapter.ts (~985, ~1646)
raw: { source, method, payload: event.payload ?? {} }
// …
type: "turn.diff.updated",
payload: { unifiedDiff: payload.diff }

ProviderService.publishRuntimeEvent writes that object to the canonical provider log before publish (ProviderService.ts ~919). EventNdjsonLogger.write then JSON-encodes the whole event and keeps the resulting line (EventNdjsonLogger.ts ~497, ~620).

turn.diff.updated is not in the transient skip set (that filter is for high-churn deltas). There is no per-record size cap. File rotation (10 MiB) and the flush watermark (1 MiB) run after serialization, so a 250 MiB CANON record is fully materialized first.

The user-facing restart copy is a consequence, not a separate bug: after the Node process dies, reconcileProviderSessions marks sessions whose provider process is gone as failed (serverRuntimeStartup.ts ~339).

Orchestration does not store unifiedDiff. ProviderRuntimeIngestion only uses turn.diff.updated to dispatch a placeholder checkpoint (status: "missing", files: []). The large string exists for diagnostics (and an unused contract field), then gets serialized twice.

A >250 MiB CANON record implies an underlying diff on the order of ~125 MiB written twice, plus the native log line, plus the [ts] CANON: … copy. That is enough to push V8 over the edge on top of a live desktop heap.

Related, not the same

• [Bug]: app-server stdin reader is quadratic in line length — OOM crash on large tool payloads #5389 (closed by fix(codex): avoid quadratic app-server input buffering #8605): quadratic app-server reader. Fixed. A comment there already saw large turn/diff/updated lines in provider logs; this issue is the later canonical duplicate + unbounded log serialize.
• [Bug]: Desktop backend still SIGABRTs with V8 OOM after 9–18h on 0.0.35 (hydration bounds already present) #8648: idle / hydration OOM on a huge SQLite profile after hours. Different path.
• Codex thread resume can exhaust server heap on large histories #7075: Codex thread/resume history. Different path. excludeTurns: true is already on main (CodexSessionRuntime); fix(server): avoid loading Codex history on resume #8629 was closed as already landed.

No open PR bounds this logger path or removes the dual-field diff.

Suggested fix

1. In EventNdjsonLogger, omit or summarize oversized payloads before serializeEvent (keep method/type + byte count). That guards native and canonical across providers.
2. In CodexAdapter, do not attach the full diff on both raw.payload and payload.unifiedDiff for turn/diff/updated.
3. Add a regression test that a large turn/diff/updated cannot produce a multi-hundred-MiB CANON line.

Restarting the desktop app is only a temporary workaround.