Skip to content

Bound Codex proxy mirroring and debug log pressure during Responses streams #372

Closed

Description

@Juliusolsson05

Status (quality-loop audit, 2026-09-25)

Landed: #729 made retention a single scan.

Remaining: codex-headless responsesProxy.ts:408-432 still mirrors every Responses event synchronously (appendFileSync + base64, no cap), and it is on for every Codex session.

Acceptance: Chunks are no longer mirrored synchronously by default; proxy, feed-debug and perf logs have per-run byte caps and dropped counters. Evidence: a real 100–280 MiB proxy-events.jsonl.

Original report

Problem

The heap-crash investigation found a separate pressure amplifier from #369: packages/codex-headless/src/proxy/responsesProxy.ts mirrors every proxy event synchronously, and response-chunk events serialize raw Buffers through JSON/base64 before normal listeners run. This creates large transient strings, blocks the event loop, and writes unbounded active proxy-events.jsonl files.

Local evidence from the investigation found proxy storage near 965 MiB, with individual proxy-events.jsonl files over 100-280 MiB. Feed-debug and performance logging have similar per-file cap gaps.

Findings

  • ResponsesProxy.emit mirrors before super.emit, so slow/string-heavy mirror writes delay the stream adapter.
  • response-chunk mirroring can hold raw bytes, base64 text, JSON text, and append buffers at once.
  • Request capture buffers and parses full bodies for shape extraction and may base64 encode request bodies.
  • Retention eventually prunes disk, but active writers have no per-run byte cap or drop mode.
  • Feed-debug renderer memory is count-capped, but disk files are not per-file byte-capped.
  • Retention sweeps are cleanup, not pressure control, and can themselves be expensive during pressure.

Proposed Fix

  • Stop raw/base64 mirroring of every response-chunk by default. Store metadata plus bounded samples/tails; gate full raw capture behind an explicit forensic flag.
  • Add per-run/per-file byte caps and dropped counters for proxy mirrors, feed-debug, and performance logs.
  • Move mirror writes off the hot synchronous path into a bounded async queue/drop policy.
  • Add lightweight counters: chunk bytes, serialized bytes, mirror write ms, cap drops, queue depth, per-run bytes.
  • Make retention cheaper and visible: one scan per prune, protected bytes, active-grace skips, removals, and scan duration.

Acceptance

  • Large Responses streams no longer synchronously base64/JSON mirror every chunk by default.
  • Active proxy/feed/perf log files have hard caps and record dropped counters.
  • A stream stress test demonstrates bounded proxy mirror memory/disk behavior.
  • Debug/retention metrics make it clear when diagnostics are being dropped because of caps.

Related: #327, #365, #369, #370

Activity

  1. Juliusolsson05 commented on Sep 25, 2026

    @Juliusolsson05
    OwnerAuthor

    Status after #1317 (merged via batch #1318, 2026-09-25).

    Shipped: codex-headless#53. The proxy mirror now writes response chunks as base64 ({ _buffer_b64 }), as documented, instead of decimal byte arrays: JSON.stringify ran Buffer#toJSON before the replacer. That was 3.65× the payload, 2,266 MiB of chunk lines for 621 MiB in one recorded 3.19 GB session. The same file's chunk lines drop to about 830 MiB, with identical bytes.

    Still open: per-file byte caps with dropped counters, request-body mirroring (775 MiB in that file), and the synchronous write.

  2. Juliusolsson05 commented on Sep 27, 2026

    @Juliusolsson05
    OwnerAuthor

    Merged in #1332. Its three defaults (rotate, 64 MiB / 16 MiB, no /v1/models dedup) are pending the owner's confirmation; see #1332's comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    class:C6-unboundedUnbounded growthprovider:codexAffects Codex integrationsev:P2Real bug with a workaroundtype:perfPerformance or resource use

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions