Skip to content

[Bug]: Codex goal turns run invisibly after a server restart (timeline stuck on "Thinking") #21865

Open

Description

@michal-jarczewski-deel

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Start a Codex thread and set a goal with /goal …, so the goal is active and Codex continues turns on its own.
  2. Restart the T3 server while the goal is still active. In my case the desktop app's server restarted on its own about 45 minutes after the goal was set (server-runtime.json startedAt).
  3. Keep using the thread. I handed off to Claude for a few runs, then back to Codex with "continue".

Observed once, over ~20 hours, in a single thread. I haven't run a clean minimal repro.

Expected behavior

Goal continuation turns are attached to a run and render in the timeline. Failing that, the existing guard in the turn/started handler (orchestration-v2.codex-goal-turn-without-run: pause the goal and send turn/interrupt) stops them. Codex work shouldn't run for hours without any trace in the UI.

Actual behavior

After the restart, no Codex run in the thread gets any projected work items. The active run shows only "Working for 1h 23m" and "Thinking". Runs #16 and #17 show "Run interrupted before provider start". Run #19 stayed running for 2 hours with exactly one turn item (the user message).

Codex was working the whole time. In ~1.5 h it made 5 commits in the repo, and the provider log has ~1,200 events (commands, reasoning, agent messages, turn/started for new goal turns) on the main native thread. Subagent native threads were projected normally. Only the main thread's items were missing.

What the code and data suggest (server bundle, Codex adapter v2). This is inference; I didn't step through it in a debugger.

  • item/started / item/completed call awaitActiveTurn(payload.turnId) and return silently when no context exists, so every item of an unregistered turn is dropped.
  • The guard in turn/started requires rootProviderThreads.has(threadId). rootProviderThreads is in-memory only and is populated in runGoalCommand, i.e. only when /goal runs in the current server process. After a restart it is empty, so the guard never fires, the turn falls through to rememberSubagentTurnStarted, and its items are dropped. I found no codex-goal-turn-without-run warning in the server logs.
  • When T3 later sent turn/start for run Track per-thread terminal running state in UI and store #19, Codex returned an existing turn id (status: "inProgress", startedAt: null) that it had started on its own hours earlier. registerRootTurn was called with waitForNativeStart: true, but no new turn/started for that id arrived. orchestration_v2_projection_provider_turns has no rows for this provider thread after run oxc stack #1.

Workaround that worked once: quit and relaunch the app. This also stops the old codex app-server. The next Codex run rendered normally (messages, commands, browser use, tasks), and /goal wasn't needed. I haven't confirmed whether later goal continuation turns in the new server process stay visible.

Possible fixes:

  • Rehydrate goal state on server start or provider-thread resume (e.g. thread/goal/get for root provider threads), or persist rootProviderThreads.
  • When turn/start returns an already in-progress turn, register it right away instead of waiting for a turn/started that never comes.
  • At minimum, log a warning instead of silently returning when item events arrive for an unknown turn on a root provider thread.

Related, but not the same trigger: #20232 (native goal work continues behind a failed run after /goal resume), #18833, #16942.

Impact

Major degradation or frequent failure

Version or commit

v0.1.0 (stable desktop release)

Environment

macOS (Darwin 25.2.0, arm64), T3 Code desktop app with local server; Codex provider gpt-6.1-sol, Claude Agent provider claude-opus-5-5 in the same thread via context handoff

Logs or stack traces

# provider log, main native thread, run #19 start (UTC)
16:25:41.094 outgoing thread/inject_items   (context handoff)
16:25:41.112 outgoing turn/start            input=[{text:"continue"}]
16:25:41.115 incoming result turn.id=<T>    status=inProgress startedAt=null   # <T> was first started 18 h earlier
16:25:59.567 incoming item/started userMessage "continue" turnId=<T>
...          ~1,200 item/* and turn/started events on the main thread over the next 1.5 h

# state DB (copy), run #19
orchestration_v2_projection_turn_items  where run_id = run #19  -> 1 row (user_message)
orchestration_v2_projection_provider_turns for this provider thread -> last row from run #1

Activity

  1. juliusmarminge commented on Oct 19, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Thanks for the detailed log excerpts. I checked the diagnosis against current main, and two parts don't match it.

    • The list of root provider threads is in memory only (apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:1904). It is filled every time T3 registers a root turn (CodexAdapterV2.ts:1931-1938, :2046), not only when /goal runs, so it doesn't stay empty after a restart.
    • When turn/start returns a turn that hasn't started yet, the turn is registered right away and its items can attach to it (CodexAdapterV2.ts:1984-2024, :6225-6230). Only a Stop waits for the native start.

    Items for a turn T3 never registered are dropped without a log line (CodexAdapterV2.ts:2058-2080, :4766-4769). That part matches, but it doesn't explain why your registered turn showed nothing.

    Since this happened once on 0.1.0, could you share whether it happens again on the latest nightly? If it does, the server log from around the restart and the first run after it would help most.

  2. added
    needs more infoInitial triage showed no bug. Awaiting more info
    via-triageFiled through npx t3 triage
    and removed
    needs more infoInitial triage showed no bug. Awaiting more info
    on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions