Fix feed clearing and missing user rows during MCP/tool-call turns #159

Activity

1. Juliusolsson05 commented on May 16, 2026

   

   Juliusolsson05

   on May 16, 2026

   OwnerAuthor

   More actions

   Investigation update from the current debugging pass.

   Findings so far:

   • The MCP call path is not the first failure. orchestration_list_agents shows up as a normal Responses function_call, then turn_completed, then stream_phase: awaiting-tool, then the follow-up Responses turn.
   • User input reception is also not the first failure. The trace has optimistic_user_add for the user messages, so the agent reacting while the user row is not visible means the row is lost or hidden after runtime insertion.
   • The main render ownership gap is that completed semantic turns can leave currentTurn before JSONL has durable rows. The old semantic history shape was summary-only and Feed did not render it, so MCP/tool-call gaps could make the feed visually shrink/clear.
   • Codex must not suppress an entire archived semantic turn by shared codexTurnId; Codex commits one response_item at a time, so duplicate suppression must be per block/item.

   Local changes made while investigating:

   • Keep bounded full semantic history instead of text-only summaries.
   • Render archived semantic history in Feed until committed transcript rows supersede it.
   • Add per-block duplicate suppression for no-block assistant text fallback.
   • Archive ended Codex tool turns on tool_completed when no pending tools remain.
   • Add debug counts to optimistic_user_add and JSONL optimistic reconciliation.
   • Make visible_rows render logging include entry-count/visible-count changes, not only row-key changes.
   • Move render logging to all mounted feeds rather than debug-panel/focus gated.

   Current caveat:

   The running app still has not emitted RENDER visible_rows after the latest TileLeaf logging change, which strongly suggests the active renderer did not pick up that source change via HMR. A restart from this working tree should be the next validation step so the trace includes the final render breakpoint.

   Validation run locally:

   • npm run test:session-ownership passed.
   • npm run test:ghost-fallback passed.
   • npm --prefix packages/codex-headless run build passed.
   • git diff --check passed.
2. Juliusolsson05 commented on May 16, 2026

   

   Juliusolsson05

   on May 16, 2026

   OwnerAuthor

   More actions

   Findings — root cause traced end to end

   Investigated against the live feed-debug log for session 3c8ada8f (559s span, 67 Responses turns).

   What the debug log proves

   signal	count	meaning
   SEM/turn_started / turn_completed	67 / 67	proxy semantic stream healthy
   SEM/block_completed	212	live blocks streaming fine
   JSONL/jsonl_entries	1 (at t=0, +53)	committed transcript dead after bootstrap
   STATE/optimistic_user_add / remove	7 / 0	7 user messages sent, none ever reconciled

   Over 9 minutes / 67 turns, runtime.entries was frozen at the 53-entry bootstrap snapshot. Nothing committed.

   Breakpoint chain

   1. Root cause — the Codex rollout-file tailer goes silent after bootstrap. CodexHeadless.start() attaches tailFile() to one rollout JSONL file. This session was resumed (53 entries of prior history) → it took the findRolloutByThreadId → tailFile(rolloutPath) branch. The tailer read the 53-line bootstrapTail, then received nothing — it is pinned to a file Codex is no longer writing. On-disk evidence: ~/.codex/sessions/2026/05/16/ contains both resumed files (filename-ts ≪ mtime → Codex appended) and rollout files created 3-within-10s (the orchestration multi-Codex burst). Codex's resume behavior is inconsistent: sometimes appends, sometimes forks a new rollout file. When it forks, tailFile is stuck on the dead old path. The fallback-to-new-file-watcher only fires when findRolloutByThreadId misses — not when it hits but Codex forked. That gap is the bug.

   2. Store layer — committed channel dead ⇒ runtime.entries never grows. The 7 optimistic user rows are appended and never removed, but the path that grows the feed (session:jsonl-entries → Pass B) never fires.

   3. Renderer layer — with no committed entries, the only live source is the single currentTurn slot in foldEvent.ts. Every turn_completed archived it to history, which (pre-patch) was a content-less summary Feed never rendered. MCP/exec turns produce many short Responses turns ⇒ the single slot churns every few seconds ⇒ "feed clears after ~2 messages, conversation never grows." The user's own message appears to vanish — it is stuck in the frozen 53-entry block while the live turn below it keeps resetting.

   MCP is not the cause. It just multiplies turn count so the single-slot churn and the history cap get hit fast.

   Why the ghost system does NOT save us here

   The ghost system is, by its own design (mergedEntries.ts, docs/design/ghost-system.md), a fallback for a ~100ms JSONL-stall — not a renderer of record. It cannot cover a permanent committed-channel outage:

   • 30s orphan TTL (GHOST_ORPHAN_TTL_MS = 30000). A ghost is render-ineligible until 30s after its last update (predicate rule 2). The system assumes JSONL catches up in ~100ms. With JSONL permanently dead, every completed turn is invisible for its first 30s — and by then 5-10 more turns have churned past it.
   • Ownership gap. Predicate rule 3 hides any ghost whose turnId === currentTurnId (deferring to SemanticStreamingTurn). So a just-completed turn is owned by nobody for ~30s: SemanticStreamingTurn dropped it (currentTurn moved on), the ghost predicate hasn't adopted it yet (not orphaned).
   • Sidecar-shape filter (rule 5) actively hides assistant ghosts that are a single text block ≤200 chars — exactly the shape of short Codex commentary/replies. The design doc explicitly accepts this trade ("a real assistant turn that crashed before any JSONL write AND was a single short text block would also be hidden").
   • The ghost correctness model is "JSONL is the source of truth, ghost papers over the gap." It is built for transient failures (live-stuck mid-turn, resume-after-crash). A dead rollout tailer is a structural, permanent outage — outside the ghost system's design envelope.

   So: ghosts can't fix this because they were built to bridge 100ms, not to be the session's renderer.

   Fix direction

   • Primary (layer 1): CodexHeadless resume must not assume the findRolloutByThreadId file stays live. Run the new-file watcher concurrently on the resume path, or watch the session dir for a forked rollout, or detect "tailer silent N s while proxy is active" and re-resolve. The orchestration burst (3 rollout files in 10s) also makes tailNewRolloutFile's "first new file wins" unsafe for fresh child agents — it can latch a sibling agent's file.
   • Mitigation (layer 3): render completed semantic turns immediately with full content (full-turn history buffer rendered by Feed) instead of relying on the 30s-late, sidecar-filtered ghost path.
3. Juliusolsson05 commented on May 16, 2026

   

   Juliusolsson05

   on May 16, 2026

   OwnerAuthor

   More actions

   Correction — re-read docs/design/ghost-system.md

   My earlier comment was too dismissive of the ghost system ("outside its design envelope"). After re-reading the design doc, the precise picture:

   The ghost system WAS designed for this failure mode. Doc fallback case #1, verbatim: "JSONL stuck mid-turn. The agent process gets wedged or its writer backlogs while the proxy keeps emitting events... The ghost on disk is the only record of the turn." That is exactly this bug.

   But the doc also states the design was never finished, and that's why it doesn't save us:

   1. Phase 3 was never built. Doc: "The original design intent was for ghost to be the only live render path. SemanticStreamingTurn would be deleted... That deletion never happened." What ships is the bookkeeping half (mint/reconcile/persist) + a narrow fallback render. SemanticStreamingTurn still owns the live turn; predicate rule 3 deliberately excludes ghosts for it.

   2. The 30s TTL is a "wait and see," not a paint speed. Doc: "The TTL is not 'how fast we paint fallback' — it's 'how long we wait before concluding JSONL isn't coming.'" Healthy sessions never hit it (JSONL arrives in ~100ms). Here JSONL is dead for all 67 turns, so every turn eats the full 30s — the ghost feed is permanently 30s-lagged and gappy, not empty.

   3. Rule 5 silently drops short messages. Any assistant ghost that is a single text block ≤200 chars is classified as title-gen sidecar and hidden forever, even when orphaned. Short Codex commentary turns match that shape. The doc calls this a knowingly-accepted trade-off.

   The decisive point: the ghost system's entire correctness model is "JSONL is the source of truth; the ghost bridges until it catches up" — reconciliation, lastJsonlEntryAt, supersedure, the predicate all assume JSONL eventually arrives. This bug's root cause (the dead Codex rollout tailer, see previous comment) means JSONL arrives never. The ghost system has nothing to reconcile against, so it degrades to a 30s-lagged, short-message-filtered, half-empty feed — exactly the reported symptom.

   Conclusion: the ghost system is not the fix and was never going to be, because the cable it bridges is severed upstream of it. Fix priority order stands:

   1. Layer 1 (real fix): the Codex rollout tailer must not pin to a dead file on resume — it must follow Codex when it forks a new rollout file.
   2. Layer 3 (mitigation): render completed semantic turns immediately from a full-content history buffer, instead of relying on the 30s-lagged, sidecar-filtered ghost fallback.

   Finishing Phase 3 would not have prevented this — it would still need JSONL to reconcile against.
4. Juliusolsson05 commented on May 16, 2026

   

   Juliusolsson05

   on May 16, 2026

   OwnerAuthor

   More actions

   Debug update: Codex source + local logs point to a committed-rollout ownership bug, not an MCP tool execution bug.

   What Codex upstream does:

   • codex resume <id> is routed by the CLI into interactive TUI resume (vendor/codex-src/codex-rs/cli/src/main.rs:765, :1388).
   • TUI calls app_server.resume_thread(... target_session.thread_id) (vendor/codex-src/codex-rs/tui/src/app.rs:3724).
   • Core resume loads the selected rollout path and creates InitialHistory::Resumed (vendor/codex-src/codex-rs/core/src/thread_manager.rs:502).
   • Codex::new uses RolloutRecorderParams::resume(resumed_history.rollout_path) for resumed sessions (vendor/codex-src/codex-rs/core/src/codex.rs:1636).
   • RolloutRecorderParams::Resume opens that exact path with append mode and does not create a new session_meta (vendor/codex-src/codex-rs/rollout/src/recorder.rs:500).

   So a true Codex resume should keep appending the original rollout. If Agent Code sees a fresh rollout with a new session_meta and copied prior history, that is a fork/reconstruction path, not the durable file that a resumed pane should blindly treat as the same provider session.

   Live evidence from the affected pane:

   • Feed debug file: ~/.config/agent-code/feed-debug/3c8ada8f-c6fe-45b4-b7e1-bd49f72c29c3.jsonl.
   • Counts at the time of inspection:
     • JSONL/jsonl_entries: 1 total, the bootstrap burst only: entries +53 at 2026-05-16T15:29:08.981Z.
     • SEM/turn_started: 112.
     • SEM/block_completed: 374.
     • STATE/submit: 9.
     • STATE/optimistic_user_add: 9.
   • That means input and proxy semantics are alive, but the committed transcript channel is permanently stalled after bootstrap. Ghost rows can bridge lag; they cannot replace a dead committed channel.

   Rollout evidence:

   • Workspace state maps 3c8ada8f-c6fe-45b4-b7e1-bd49f72c29c3 to provider session 019e310d-ce49-7061-a13f-456d7dd609cd.
   • The provider rollout rollout-2026-05-16T15-50-48-019e310d-ce49-7061-a13f-456d7dd609cd.jsonl is actively growing. It contains the same current assistant/tool work that the proxy semantic stream reports, including the Codex-source investigation messages after line 3048.
   • A second same-cwd rollout exists: rollout-2026-05-16T15-36-50-c0d44dcb-3972-4c55-9a31-a242bb482e2a.jsonl. Its first line is a new session_meta for c0d44dcb..., followed by copied history from 019e310d..., and it stops around 2026-05-16T15:36:50Z. This file shape matches a fork/reconstructed history file, not true resume append.

   Root cause hypothesis now:

   Agent Code's Codex headless layer treats rollout identity too weakly. It can bootstrap from a plausible same-cwd/session-like file, while the live Codex process/proxy is actually producing the durable transcript in a different rollout. Once that happens, the renderer keeps receiving semantic/proxy events, but durable jsonl-entry events never land for the visible pane. The user sees exactly the haunted behavior: optimistic user rows appear logically, assistant reacts, then the visible feed never grows because committed entries are frozen.

   Suggested fix:

   1. Make Codex rollout attachment identity-explicit. For resume, prefer the workspace/provider session id and require the first session_meta.id to match. Do not accept a same-cwd copied-history fork as equivalent.
   2. When the selected tail is silent but proxy semantics are active and another rollout with the expected provider id is growing, switch the committed tail and emit a high-signal debug event. This is the local WIP direction in packages/codex-headless/src/CodexHeadless.ts, but it should be tightened to use provider id when available, not just cwd.
   3. For fresh sessions, stop using "first new rollout wins" as a silent invariant. Validate session_meta.cwd, capture the resulting provider id, persist it to workspace state, and log ambiguity if multiple new same-cwd rollouts appear during startup.
   4. Keep the semantic-history renderer bridge as a lag fallback, but treat it as secondary. The primary invariant must be: every live provider pane has one authoritative, validated committed rollout tail.

   This also explains why MCP exposed the issue without necessarily causing it. MCP creates longer client-tool gaps where semantic/proxy output can keep moving while committed JSONL is the thing that proves whether the transcript owner is correct.
5. Juliusolsson05 commented on May 16, 2026

   

   Juliusolsson05

   on May 16, 2026

   OwnerAuthor

   More actions

   Fix PRs open

   Layer-1 fix (the root cause — dead Codex rollout tailer):

   • codex-headless submodule: fix(codex): correct rollout tail ownership so the committed feed keeps growing codex-headless#9 — fix/codex-rollout-tail-ownership
     • Resume path follows a forked rollout, gated by cwd + copied-history lineage (opaque item-id overlap) so sibling orchestration agents in the same cwd are not latched.
     • Fresh-start path logs multi-rollout ambiguity.
     • Proxy releases the active-flow slot on response.completed so MCP follow-up turns publish.
   • agent-code parent: fix(codex): correct rollout tail ownership — feed clearing during MCP turns (#159) #160 — fix/codex-feed-clearing-mcp, bumps the submodule pointer to Track primary and active worktree context #9.

   Verified by tsc build only. Not yet exercised against a live Codex resume/fork — a live smoke test (resume a Codex pane, run an MCP-heavy turn, confirm the committed feed grows) is recommended before merge.

   The renderer-side mitigation (bounded semantic-history bridge) remains separate and is not in these PRs.
6. Juliusolsson05

   changed the title [-]Debug feed clearing and missing user rows during MCP/tool-call turns[/-] [+]Fix feed clearing and missing user rows during MCP/tool-call turns[/+] on May 16, 2026
7. Juliusolsson05

   added 3 commits that reference this issue on May 16, 2026

   fix(codex): address review — close fork-watcher race, harden lineage

   ...

   07e197b

   fix(codex): bump codex-headless to fix rollout tail ownership (#159)

   ...

   8613591
8. Juliusolsson05

   closed this as completedin #160on May 16, 2026
9. Juliusolsson05

   added 2 commits that reference this issue on May 16, 2026

   Merge pull request #160 from Juliusolsson05/fix/codex-feed-clearing-mcp

   ...

   e10c84b

   fix(codex): bump codex-headless to fix rollout tail ownership (#159)

   ...

   c9679f4