[Bug]: Codex memory output still leaks into the chat when the memory thread is never announced

[SunkenInTime]

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server (Codex adapter, CodexSessionRuntime)

Steps to reproduce

Same as #4683:

1. Open a Codex conversation with automatic memories enabled.
2. Exchange a few ordinary messages.
3. Let post-turn memory consolidation run.

Expected behavior

Memory consolidation output stays out of the user's chat.

Actual behavior

The memory worker's messages and tool activity are saved into the open conversation. #5468 fixed this for memory threads that announce themselves with thread/started and a memory_consolidation source. It still happens when Codex streams the worker's items (item/agentMessage/delta, item/completed, warning) without sending thread/started first. The filter never learns that thread id, so the output falls through to the parent path. @gfdelarue hit this on 0.0.38 with Codex CLI 0.153.3 (comment on #8989). The persisted rows had data.threadId set to the worker threads.

Impact

Minor bug or occasional failure

Version or commit

Reproduced in tests against main at a3abb52. Reported live on 0.0.38.

Environment

Codex CLI 0.153.3 (reporter), NixOS; tests on Windows 11.

Logs or stack traces

No response

Workaround

Turn off automatic memories in Codex.

Why this isn't a one-line fix

I tried twice in #8989, and both attempts were closed. Notes for whoever picks this up:

• Dropping output from any unknown foreign thread fixes the leak but also loses real subagent output. Codex can stream a v2 child's items before the subAgentActivity or thread/started that registers it, and on arrival that child looks the same as unannounced memory work.

• Holding unknown-thread output until the thread is identified, then replaying it, works in principle. It passes the leak test and the early-child test. Review turned up seven ordering bugs in four rounds, though:

  • replay re-ran receiver-turn bookkeeping and restored an older turn;
  • a replayed old error cleared a child's newer live turn, so Stop skipped it;
  • every root delta scanned all held threads;
  • a child that failed before registering stayed interruptible;
  • the new root's output was stranded after a resume fallback;
  • an early memory turn/completed marks the session ready before the open response, so the real root's output gets held;
  • a replayed old error marks a newer running child as failed.

  The first five are fixed on the prototype. The last two are not.

• A clean fix probably needs a signal that says "this thread is memory work" on every notification, or a decision on what to do with output from threads that are never identified.

The prototype and its tests are on SunkenInTime:wip/codex-unannounced-memory (CodexSessionRuntime.ts and CodexCollabRuntime.integration.test.ts). The integration tests that replay unannounced memory output and early child items fail on main and could be reused for any fix.

[juliusmarminge]

Note

Grok responding on behalf of Julius.

Triage

Thanks for the detailed write-up and for documenting both earlier attempts, @SunkenInTime! This is a real bug, and it's still present on main at b33eda1399.

makeMemoryConsolidationNotificationFilter in apps/server/src/provider/Layers/CodexSessionRuntime.ts only learns a memory thread's id from thread/started, and only when threadSource or source.subAgent is memory_consolidation. After that, it drops later events for that id, except serverRequest/resolved. If Codex never sends that thread/started, the id is never recorded and nothing gets filtered.

The other guard doesn't catch it either. Once the root id is known, a different threadId counts as a foreign conversation, but shouldSuppressChildConversationNotification only drops lifecycle methods like thread/started, turn/started, turn/completed, and thread and plan updates. item/agentMessage/delta, item/completed, and warning aren't in that list, and readNotificationThreadId doesn't read threadId from warning at all. So those events land in the open T3 thread. The existing unit test and collab integration test only cover a memory thread that announces itself first, which is why they don't catch this.

History

• fix(codex): keep background memory out of chats #5468 (merged PR) fixed the announced case and closed [Bug]: Background Codex memory activity appears in the active conversation #4683 (closed issue).
• The live report in the comments on fix(codex): keep foreign memory out of chats #8989 (closed PR; 0.0.38, Codex CLI 0.153.3) is this same remaining gap. That PR was right to stay closed: dropping every unknown foreign thread also drops a real v2 child, because Codex can stream a child's items before thread/started or subAgentActivity. The runtime still documents that early child traffic passes through until the child is registered.
• fix(codex): isolate unowned provider thread events #12447 (open PR) takes that same drop-unregistered approach. It would hide this leak, but it would also throw away early child items instead of routing them once the child is identified, so it doesn't fix this issue.

What a fix needs

It has to keep unannounced memory output out of the parent chat and still deliver early child output once that child is identified. serverRequest/resolved also has to stay on the parent's request-matching path. Your hold-and-replay notes on wip/codex-unannounced-memory are a useful starting point. Before that approach is ready, it needs to handle the two ordering bugs you listed as still open: an early memory turn/completed that holds back the root's output, and a replayed old error that marks a newer running child as failed. I haven't treated that branch as the approved fix. The integration tests there that fail on main would be good to reuse for whatever approach lands.

[juliusmarminge]

Thanks for taking the time to report this and provide the details. We revisited it during the orchestrator V2 cleanup.

The V1 CodexSessionRuntime memory filter and fallback-to-parent path were removed. V2 routes text and item events through their native turn context; unknown turns return without projection, so the described unannounced foreign-memory fallthrough no longer exists.

I’m closing this report because the implementation it targets has been replaced. That does not mean every similar symptom is fixed.

Source reviewed.

No live memory consolidation reproduction was run. This retires the V1 fallthrough report, not every possible memory/subagent routing problem.

If you still hit this on a current build, please reply with the app/server versions and the steps that reproduce it. We can reopen this if the original problem is still there.