Skip to content

ai: claude_stalled_no_output escapes the no-retry guard — every Claude timeout costs 3 full attempts and parks the queue #9476

Description

@JSONbored

Summary

claude --output-format json is fully buffered, so any Claude run that exceeds its effort timeout has produced zero stdout bytes when the deadline fires. The adapter therefore classifies it as claude_stalled_no_output rather than subscription_cli_timeout — and the retry-break check tests for the latter by strict string equality. The result: every timed-out Claude review is retried the full 3 times before falling back, and subscription_cli_timeout is effectively unreachable for claude-code, making #3987's fix dead code on this deployment.

Mechanism (verified at HEAD, 776c414)

resolveClaudeFirstOutputTimeoutMs defaults to 1,800,000 ms and is then clamped to timeoutMs - 1 (deliberate, #5053) — src/selfhost/ai.ts:1054-1061, :293-300, :212. Because the CLI emits nothing until it finishes, the first-output timer always fires first, at timeoutMs - 1, yielding stalledNoOutput: true and a thrown claude_stalled_no_output: ….

The break in runWorkersOpinion is:

isSubscriptionCliTimeout(error) || isRateLimitError(error) || isStructuralProviderConfigError(error)

and isSubscriptionCliTimeout is a strict equality on "subscription_cli_timeout" (src/services/ai-review.ts:1379-1381, :1636). claude_stalled_no_output never matches, so the loop retries.

Cost

At default medium effort: 3 × 180 s ≈ 9 minutes of subprocess time in the claude slot before ollama is even attempted. At the 600 s tier: 30 minutes. With QUEUE_CONCURRENCY defaulting to 8 (src/selfhost/pg-queue.ts:306-308), a Claude slowdown parks the entire queue in this state until the per-provider circuit breaker trips — and the breaker needs 3 consecutive full-length failures first (src/selfhost/ai.ts:1230-1241).

test/unit/ai-review.test.ts:3601 pins the classification but nothing pins the retry behaviour, which is why this survived.

Related, same class: the ollama HTTP call has a hard AbortSignal.timeout(120_000) and that abort is also absent from the no-retry set — so the fallback can burn 3 × 120 s on top.

Deliverables

  • Add an isNonTransientCliFailure() (or extend the existing check) covering claude_stalled_no_output and codex_stalled_no_output. These carry a : detail suffix, so match by prefix, not equality — the strict-equality assumption is the actual bug.
  • Include the ollama request-abort case in the same break.
  • Reconsider whether subscription_cli_timeout can ever be produced for a buffered-output CLI; if not, either remove it or document that it is reachable only for streaming adapters, so the next reader is not misled the same way.
  • Emit a metric distinguishing "provider timed out and we stopped" from "provider timed out and we retried", so the retry budget is observable.

Tests (must fail against current main)

  • A CLI that exceeds its timeout with no stdout ⇒ one attempt, then fallback. Assert the attempt count, not just the error classification — that gap is what let this ship.
  • Same for codex_stalled_no_output.
  • Ollama abort ⇒ no retry storm.
  • A genuinely transient failure (non-timeout) still retries as before.

Expected outcome

A provider timeout costs one attempt, not three. A Claude outage degrades to the fallback in roughly a third of the current time, and the circuit breaker trips proportionally sooner.

Metadata

Metadata

Assignees

Labels

gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions