Skip to content

fix(selfhost): stop killing legitimate slow claude-code reviews (#5053) - #5055

Merged
JSONbored merged 1 commit into
mainfrom
fix/5053-claude-firstoutput-false-positive
Jul 11, 2026
Merged

fix(selfhost): stop killing legitimate slow claude-code reviews (#5053)#5055
JSONbored merged 1 commit into
mainfrom
fix/5053-claude-firstoutput-false-positive

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Fixes fix(selfhost): claude-code fast-fail timeout false-positives on --output-format json (buffered, non-streaming) #5053live incident: AI reviews were completely broken on the self-host box (edge-nl-01) right now. Root-caused live via SSH, not guessed.
  • fix(review): claude-code subscription-CLI hangs/times out repeatedly (4,030 Sentry events, ongoing) #4994 (this session, ~2 hours before this PR) added a 30-second "first output" fast-fail deadline to claude-code, mirroring codex's existing mechanism. Codex's own code comment explains why that one is safe: "real JSONL progress from codex --json always lands on stdout." I never verified the equivalent claim for claude-code before copying the pattern — it doesn't hold.
  • claude --help documents --output-format json (what this codebase uses) as returning a "single result" — fully buffered, not streamed. Confirmed live on the box: ran the exact CLI invocation the app makes with a realistic 274KB prompt at effort:high — it took 116 seconds to complete successfully, producing zero stdout bytes for the entire run, then the full 9KB response arrived at once. A 30-second (or even 120-second) "no output yet" signal cannot distinguish that from a genuine hang for this CLI mode — it's a coin flip that fix(review): claude-code subscription-CLI hangs/times out repeatedly (4,030 Sentry events, ongoing) #4994 mis-set badly.
  • Compounding factor, also confirmed live: this box has AI_PROVIDER=claude-code with no fallback provider configured, and src/selfhost/ai.ts's per-provider circuit breaker opens after 3 consecutive failures (60s cooldown). Once every review started failing deterministically at the 30s mark, the circuit tripped repeatedly and stayed open, blocking essentially all AI reviews fleet-wide — a total outage, not just noise.

Fix

resolveClaudeFirstOutputTimeoutMs's default and ceiling now match resolveCliTimeoutFrom's own outer clamp (30 minutes) — since the call site already does Math.min(resolveClaudeFirstOutputTimeoutMs(...), timeoutMs - 1), this means the fast-fail window equals the real timeout budget by default, exactly the pre-#4994 behavior. claude_stalled_no_output now only fires when nothing arrived for the entire configured budget — a genuine hang. An operator can still opt into a shorter, riskier window via CLAUDE_AI_FIRST_OUTPUT_TIMEOUT_MS if they understand the tradeoff. codex's identical-looking mechanism is untouched — its streaming premise still holds, confirmed by its own existing code comment.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (Closes #5053).

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • test:coverage (full unsharded): not run end-to-end — ran scoped vitest --coverage for test/unit/selfhost-ai.test.ts (157 tests) and confirmed via lcov that both branches of the changed function are covered. Also confirmed against the real claude CLI on the affected box (not just unit tests) given the severity.
  • actionlint / test:workers / build:mcp / test:mcp-pack / ui:openapi:check / ui:lint / ui:typecheck / ui:build / npm audit: not run — this change touches only src/selfhost/ai.ts (existing internal timeout resolution, no new API/schema/binding/dependency surface) and its tests; no workflow, MCP, UI, or dependency-manifest surface changed.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A.)
  • UI changes use live API data or real empty/error/loading states. (N/A.)
  • Visible UI changes include a UI Evidence section. (N/A.)
  • Public docs/changelogs are updated where needed. (N/A — internal engine behavior; changelog is not edited in a normal PR.)

Notes

This is a fix for a bug my own earlier fix in this session (#4994 / PR #5013) introduced. Needs to be merged and deployed to edge-nl-01 urgently — it's currently the live production configuration causing the outage.

#4994 added a 30s "first output" fast-fail deadline to claude-code,
copying codex's own mechanism -- but codex's comment explains why
THAT one is safe ("real JSONL progress from codex --json always lands
on stdout"); claude-code's --output-format json is documented by
`claude --help` as a "single result": fully buffered, not streamed.
Confirmed live: a realistic 274KB/effort:high prompt took 116s to
complete successfully with zero stdout bytes the entire time, then
the full response arrived at once. Every review slower than 30s was
being killed and mislabeled claude_stalled_no_output.

Combined with the per-provider circuit breaker (opens after 3
consecutive failures) and a self-host box running AI_PROVIDER=
claude-code with no fallback, this caused a total AI-review outage:
every attempt failed deterministically, keeping the circuit open
fleet-wide.

resolveClaudeFirstOutputTimeoutMs's default/ceiling now make the
fast-fail window equal the full timeoutMs by default (the call site's
own Math.min(this, timeoutMs - 1) does the clamping) -- matching
pre-#4994 behavior exactly. An operator can still opt into a shorter,
riskier window via CLAUDE_AI_FIRST_OUTPUT_TIMEOUT_MS. Codex's own
mechanism is untouched -- its streaming premise still holds.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.13%. Comparing base (dc81d72) to head (6a9ad83).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5055   +/-   ##
=======================================
  Coverage   94.13%   94.13%           
=======================================
  Files         465      465           
  Lines       39542    39542           
  Branches    14431    14431           
=======================================
  Hits        37222    37222           
  Misses       1664     1664           
  Partials      656      656           
Flag Coverage Δ
shard-1 46.49% <0.00%> (-0.02%) ⬇️
shard-2 33.39% <100.00%> (-0.12%) ⬇️
shard-3 31.59% <0.00%> (+0.25%) ⬆️
shard-4 32.65% <0.00%> (-0.03%) ⬇️
shard-5 33.25% <0.00%> (-0.20%) ⬇️
shard-6 45.15% <0.00%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/selfhost/ai.ts 98.05% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 11, 2026
@loopover-orb

loopover-orb Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-11 11:06:30 UTC

2 files · no blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • AI review could not be completed: The dual-model AI review did not return a usable verdict for this change.

Review summary
AI review could not be completed for this PR head. Gittensory is holding this PR for manual review instead of relying on deterministic signals alone.

Nits — 2 non-blocking
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
  • AI review could not be completed — The gate is held for a human reviewer rather than passed automatically; it re-evaluates on the next update.
Signal Result Evidence
Code review ✅ No blockers No AI review summary
Linked issue ✅ Linked #5053
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 47 registered-repo PR(s), 39 merged, 428 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 47 PR(s), 428 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence.
Linked issue satisfaction

Addressed
The PR raises resolveClaudeFirstOutputTimeoutMs's default/ceiling to 1,800,000ms (matching the outer 30-minute clamp) so the fast-fail window equals the real timeout budget by default, directly fixing the false-positive kill of slow-but-successful buffered claude-code responses described in the issue, and also adds the regression tests the issue's checklist called out as missing. The only remainin

Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 47 PR(s), 428 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 11, 2026
@JSONbored
JSONbored merged commit ff6f4f7 into main Jul 11, 2026
19 checks passed
@JSONbored
JSONbored deleted the fix/5053-claude-firstoutput-false-positive branch July 11, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

fix(selfhost): claude-code fast-fail timeout false-positives on --output-format json (buffered, non-streaming)

1 participant