[Bug]: A scheduled run that fails is recorded as succeeded #17889

Activity

1. juliusmarminge commented on Oct 10, 2026

   

   juliusmarminge

   on Oct 10, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Thanks for the report. I checked this against main and it matches what the code does.

   Where it happens: runTask in apps/server/src/scheduledTasks/ScheduledTaskService.ts (around lines 795–860 on main). It calls threadLaunch.launch(...) for unbound tasks or threadManagement.sendToThread(..., mode: "queue") for bound ones, wrapped in Effect.exit. Right after that it sets lastRunStatus = result._tag === "Success" ? "succeeded" : "failed", and markCompleted writes that status along with lastRunAt, nextRunAt and runCount + 1.

   So "succeeded" means the command was accepted/dispatched, not that the turn reached a terminal outcome. A provider rejection such as a usage limit happens later, during the turn, so the task still records success. Nothing in this path retries or catches up a run either. The webhook path reads the same status ("The run failed to start."), so webhook delivery logs have the same blind spot.

   Possible fix (unverified):

   • Record a dispatched state (running/dispatched) at dispatch time. Then have the run's terminal outcome (turn completed / failed / interrupted, matched by the scheduled-task: commandId or messageId / scheduledTaskId) update lastRunStatus and lastRunError once it lands.
   • Or keep the current shape and add a follow-up subscription that downgrades to failed with the provider error when that turn fails.
   • Retrying or notifying on a provider-limit failure is a separate product decision. A usage limit won't clear on an immediate retry.

   Tests such as ThreadLaunchService.test.ts and ClaudeAutomaticDelivery.integration.test.ts currently assert succeeded right after dispatch, so they'd need updating.

   Possibly related: #16183, #17309, #16818, #15553, #15167, #17458. I haven't checked how much each one overlaps.
2. juliusmarminge

   added

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   on Oct 10, 2026
3. SotoAugusto commented on Oct 30, 2026

   

   SotoAugusto

   on Oct 30, 2026

   More actions

   Another case, with numbers, from T3 Code on Linux. It happened on 2026-10-05, before #18467.

   A thread-bound recurring task fired every 3 minutes. The thread had moved from Codex to Claude. Each wake started a Codex run, and each run failed in under 1 s with "Insufficient context allowance for the provider handoff".

   • 315 consecutive wakes failed, from 2026-10-05 12:37 UTC to 2026-10-06 05:02 UTC (about 16.5 h).
   • Nothing stopped the task. I found the loop later, in an audit of the run history (orchestration_v2_projection_runs, status failed).

   #18467 fixes the provider choice. The loop is still possible for any wake that fails at once: a usage limit, the handoff budget, or a missing base branch as in #16183.

   Request: after the real terminal status is recorded, pause a recurring task after N consecutive failed wakes (for example 3). Show a notification with the last error.
4. ilshat-galimov commented on Oct 30, 2026

   

   ilshat-galimov

   on Oct 30, 2026

   More actions

   Another path to the same result: an unbound task (fresh thread per run) whose run fails in workspace preparation (git worktree add failed (ref_not_found), see #16183). ThreadLaunchService.launch returns once the thread exists and preparation runs in the background (prepareInBackground, forked into preparationScope), so runTask records succeeded and advances next_run_at by a full period (weekly in my case) before preparation fails. The error is visible only inside the created thread.