Codex subagents lose status and message tracking after server restart #15567

Activity

1. amozeo

   added

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 4, 2026
2. juliusmarminge commented on Oct 4, 2026

   

   juliusmarminge

   on Oct 4, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks for matching the native events to the stored projections, @amozeo. That made this quick to confirm. It reproduces on current main (1302ccacbd) and is the Codex counterpart of #13735, whose Claude fix doesn't touch this adapter. It isn't a duplicate of #15358, #5529, or #14630.

   What I found

   • CodexAdapterV2 keeps child registrations in subagentThreads, an in-memory map that starts empty for each provider session (around line 1668). resumeThread calls thread/resume with excludeTurns: true and keeps only thread.id and updatedAt (decodeCodexResumeMetadata), so it never rebuilds existing children.
   • After restart, a child turn/started from an unregistered native thread gets parked in pendingSubagentTurns (rememberSubagentTurnStarted). That queue is flushed only by registerSubagentThread, which runs for a new spawnAgent or a subAgentActivity of kind started. Later collabAgentToolCall status updates and other subAgentActivity kinds do nothing for unregistered children.
   • Item and turn/completed events then fail in resolveItemEventContext / activeTurns and return without projecting, so the stored subagent keeps its pre-restart status and messages. That matches your table. A child spawned after restart registers normally.
   • "Continue threads after restarts" isn't the cause, since the parent was resumed explicitly and native events are arriving. Startup recovery also isn't involved: it cancels unfinished work whose process died and correctly leaves already-completed children alone.
   • The existing test "preserves a subagent result across a trailing empty final and resume" stays within one adapter session, so it doesn't cover a process restart. The only CodexAdapterV2 change since f391794 is fix(codex): resume archived native sessions #15389 (resume archived native sessions), which doesn't rebuild this map.

   Likely fix area

   • One option: when a native thread with an existing child projection starts a turn in a new session, reattach it to the same child thread and subagent row, then flush the parked turn the way registerSubagentThread does.
   • Child and provider thread ids come from the native thread id, but the subagent node id comes from the original tool-call id. Treating it as a fresh spawnAgent would likely open a second Lineage row instead of reopening the old one, so the lookup would need to recover that original id.

   A maintainer will decide on the fix direction.
3. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   on Oct 4, 2026
4. xHeaven commented on Oct 5, 2026

   

   xHeaven

   on Oct 5, 2026

   More actions

   Confirmed the same failure on macOS with T3 desktop 0.0.46-nightly.20261005.2676 and native Codex 0.160.0.

   After desktop server startup recovery, the parent resumed and revived 12 existing native children through followup_task. Codex reported all 12 running, but T3's child status and conversation projections remained stale.

   Read-only correlation of the provider logs, Codex rollout and SQLite events gives this timeline. All timestamps are UTC on 2026-10-05:

   20:53:58.753  startup reconciliation cancels the 12 saved child records
   20:58:20.299  fresh Codex initialize for the parent
   20:58:20.345  existing parent thread/resume with excludeTurns=true
   20:59:29.367 through 21:01:49.488
                  new native turn/started for each of the 12 existing children
   21:01:53.303  native list_agents reports all 12 children running
   
   T3 durable state reconstructed at that same instant:
     12 child provider threads idle
     11 latest subagent records cancelled, 1 completed
     0 stored provider turns matching the 12 new native turn IDs
   

   All 12 child identities and their parent lineage remain present, with none archived or deleted. The retained provider log also contains new completed assistant-message items for every child, while each child's stored message timestamp predates its resumed turn. This confirms lost tracking independently of the parent's running-agent claim.

   All 12 children's saved forkedFrom.providerThreadId match the parent's current provider-thread ID. There was no replacement of the parent provider thread in this incident.

   I reviewed #15727 at ed6c95b. Its saved-child recovery on turn/started, preserved launch identities and explicit reopening appear to cover this exact path. The separate concern about recovery after a parent provider-thread replacement does not explain this case. This is a source review, not a live verification of the fix.

   One diagnostic caveat: active-only t3_thread_list is not an authoritative native-agent census. These children have no T3 run records, and its filter uses run-derived status. The evidence above instead matches native turn IDs against durable projections.

   No app state or agents were changed during the investigation. No controlled restart reproduction or live test of the proposed patch was performed. Conversation bodies, credentials and local paths are omitted.

   Prepared with Codex, GPT-6.1 Sol, through npx t3 triage and follow-up source review.