Repository navigation
[Bug]: Codex goal turns run invisibly after a server restart (timeline stuck on "Thinking") #21865
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Start a Codex thread and set a goal with
/goal …, so the goal is active and Codex continues turns on its own. - Restart the T3 server while the goal is still active. In my case the desktop app's server restarted on its own about 45 minutes after the goal was set (
server-runtime.jsonstartedAt). - Keep using the thread. I handed off to Claude for a few runs, then back to Codex with "continue".
Observed once, over ~20 hours, in a single thread. I haven't run a clean minimal repro.
Expected behavior
Goal continuation turns are attached to a run and render in the timeline. Failing that, the existing guard in the turn/started handler (orchestration-v2.codex-goal-turn-without-run: pause the goal and send turn/interrupt) stops them. Codex work shouldn't run for hours without any trace in the UI.
Actual behavior
After the restart, no Codex run in the thread gets any projected work items. The active run shows only "Working for 1h 23m" and "Thinking". Runs #16 and #17 show "Run interrupted before provider start". Run #19 stayed running for 2 hours with exactly one turn item (the user message).
Codex was working the whole time. In ~1.5 h it made 5 commits in the repo, and the provider log has ~1,200 events (commands, reasoning, agent messages, turn/started for new goal turns) on the main native thread. Subagent native threads were projected normally. Only the main thread's items were missing.
What the code and data suggest (server bundle, Codex adapter v2). This is inference; I didn't step through it in a debugger.
item/started/item/completedcallawaitActiveTurn(payload.turnId)and return silently when no context exists, so every item of an unregistered turn is dropped.- The guard in
turn/startedrequiresrootProviderThreads.has(threadId).rootProviderThreadsis in-memory only and is populated inrunGoalCommand, i.e. only when/goalruns in the current server process. After a restart it is empty, so the guard never fires, the turn falls through torememberSubagentTurnStarted, and its items are dropped. I found nocodex-goal-turn-without-runwarning in the server logs. - When T3 later sent
turn/startfor run Track per-thread terminal running state in UI and store #19, Codex returned an existing turn id (status: "inProgress",startedAt: null) that it had started on its own hours earlier.registerRootTurnwas called withwaitForNativeStart: true, but no newturn/startedfor that id arrived.orchestration_v2_projection_provider_turnshas no rows for this provider thread after run oxc stack #1.
Workaround that worked once: quit and relaunch the app. This also stops the old codex app-server. The next Codex run rendered normally (messages, commands, browser use, tasks), and /goal wasn't needed. I haven't confirmed whether later goal continuation turns in the new server process stay visible.
Possible fixes:
- Rehydrate goal state on server start or provider-thread resume (e.g.
thread/goal/getfor root provider threads), or persistrootProviderThreads. - When
turn/startreturns an already in-progress turn, register it right away instead of waiting for aturn/startedthat never comes. - At minimum, log a warning instead of silently returning when item events arrive for an unknown turn on a root provider thread.
Related, but not the same trigger: #20232 (native goal work continues behind a failed run after /goal resume), #18833, #16942.
Impact
Major degradation or frequent failure
Version or commit
v0.1.0 (stable desktop release)
Environment
macOS (Darwin 25.2.0, arm64), T3 Code desktop app with local server; Codex provider gpt-6.1-sol, Claude Agent provider claude-opus-5-5 in the same thread via context handoff
Logs or stack traces
# provider log, main native thread, run #19 start (UTC)
16:25:41.094 outgoing thread/inject_items (context handoff)
16:25:41.112 outgoing turn/start input=[{text:"continue"}]
16:25:41.115 incoming result turn.id=<T> status=inProgress startedAt=null # <T> was first started 18 h earlier
16:25:59.567 incoming item/started userMessage "continue" turnId=<T>
... ~1,200 item/* and turn/started events on the main thread over the next 1.5 h
# state DB (copy), run #19
orchestration_v2_projection_turn_items where run_id = run #19 -> 1 row (user_message)
orchestration_v2_projection_provider_turns for this provider thread -> last row from run #1
Activity
juliusmarminge commentedon Oct 19, 2026
Note
Grok responding on behalf of Julius.
Thanks for the detailed log excerpts. I checked the diagnosis against current main, and two parts don't match it.
apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:1904). It is filled every time T3 registers a root turn (CodexAdapterV2.ts:1931-1938,:2046), not only when/goalruns, so it doesn't stay empty after a restart.turn/startreturns a turn that hasn't started yet, the turn is registered right away and its items can attach to it (CodexAdapterV2.ts:1984-2024,:6225-6230). Only a Stop waits for the native start.Items for a turn T3 never registered are dropped without a log line (
CodexAdapterV2.ts:2058-2080,:4766-4769). That part matches, but it doesn't explain why your registered turn showed nothing.Since this happened once on 0.1.0, could you share whether it happens again on the latest nightly? If it does, the server log from around the restart and the first run after it would help most.