Skip to content

fix(selfhost): validate AI reviewer-provider configuration and add a failure circuit breaker - #2626

Merged
JSONbored merged 1 commit into
mainfrom
fix/ai-reviewer-provider-validation
Jul 2, 2026
Merged

fix(selfhost): validate AI reviewer-provider configuration and add a failure circuit breaker#2626
JSONbored merged 1 commit into
mainfrom
fix/ai-reviewer-provider-validation

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Advances #2540

Problem

Self-host's dual-AI review resolves up to two reviewer slots from AI_PROVIDER with no deduplication check. Listing the same provider twice collapses "two independent reviewers reaching consensus" into "one provider called twice in parallel" — routeProviders' name-keyed map collapses duplicate provider names to a single runtime instance, so a single provider's outage takes down both slots. Separately, there was no circuit breaker: a provider already known to be failing paid the full cost of a fresh attempt (a real HTTP call or CLI subprocess spawn) on every single PR review during a sustained outage.

Changes

  • resolveAiReviewerPlan (src/selfhost/ai.ts) now throws loudly at boot when the two dual-review slots it would actually use are the same provider, matching the existing fail-loud pattern already used for other self-host misconfigurations (assertNoLegacySharedAiEnv). A third+ duplicate further down an otherwise-fine provider list does not trigger it — only the two slots actually consumed matter.
  • New per-provider circuit breaker in the shared runProviderWithOtel chokepoint (used by both the fallback chain and the dual-review named-slot path): after 3 consecutive failures a provider enters a 60s cooldown; further calls during that window are skipped fast without touching the real provider. A success resets the count; a real failure rethrows unchanged so existing fallback/chain behavior is untouched.
  • New Prometheus counters: gittensory_ai_review_inconclusive_total{mode}, gittensory_ai_provider_failures_total{provider}, gittensory_ai_provider_circuit_open_total{provider}.
  • New alert rules: GittensoryAiReviewInconclusiveSpike and GittensoryAiProviderCircuitOpen in prometheus/rules/alerts.yml, following the existing absolute-increase-threshold convention.

Test plan

  • npm run typecheck clean
  • npm run test:coverage (unsharded) green, 100% branch coverage on all new/changed lines
  • npm run test:ci full local gate green
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • New tests cover: duplicate-provider throw (and the non-triggering 3rd-duplicate case), circuit opens after 3 consecutive failures, a call during cooldown skips the real provider, cooldown expiry lets calls through again, a success resets the failure count, the dual-review direct-address path shares the same breaker as the fallback chain, and a regression test for a lost-update race under concurrent same-provider failures (reproduced failing before the fix, passing after)

…failure circuit breaker

Self-host's dual-AI review resolves up to two reviewer slots from
AI_PROVIDER with no distinctness check. Listing the same provider twice
(a copy-paste mistake) silently collapses "two independent reviewers" into
"one provider called twice" -- routeProviders' name-keyed map collapses
duplicates to a single runtime instance, so a provider outage takes down
both slots at once. resolveAiReviewerPlan now throws loudly at boot when
the two slots the plan actually uses are identical, matching the existing
fail-loud pattern used for other self-host misconfigurations.

Also adds a per-provider circuit breaker: after 3 consecutive failures a
provider enters a 60s cooldown during which further calls are skipped
fast (no real HTTP/CLI cost) instead of retrying a known-broken provider
from a cold state on every single PR review. Shared by both the fallback
chain and the dual-review named-slot path via their one common call site.

New Prometheus counters (gittensory_ai_review_inconclusive_total,
gittensory_ai_provider_failures_total, gittensory_ai_provider_circuit_open_total)
plus two alert rules for a sustained inconclusive-verdict spike or an open
circuit breaker.
@dosubot dosubot Bot added the size:M label Jul 2, 2026
@loopover-orb

loopover-orb Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-02 19:39:48 UTC

5 files · no blockers · readiness 91/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • The AI review could not be completed for this change, so the gate is held for a human reviewer rather than passed automatically. It re-evaluates on the next update.

Review summary
AI review could not be completed for this PR head. Gittensory is holding this PR for manual review instead of relying on deterministic signals alone.

Nits — 1 non-blocking
  • AI review could not be completed — The gate is held for a human reviewer rather than passed automatically; it re-evaluates on the next update.
Signal Result Evidence
Code review ✅ No blockers No AI review summary
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (size label size:M; no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 64 registered-repo PR(s), 55 merged, 526 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 64 PR(s), 526 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 64 PR(s), 526 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • No action.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 2, 2026
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.05%. Comparing base (074b888) to head (2cb751f).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2626   +/-   ##
=======================================
  Coverage   96.05%   96.05%           
=======================================
  Files         234      234           
  Lines       26228    26247   +19     
  Branches     9517     9522    +5     
=======================================
+ Hits        25192    25211   +19     
  Misses        425      425           
  Partials      611      611           
Files with missing lines Coverage Δ
src/selfhost/ai.ts 98.52% <100.00%> (+0.08%) ⬆️
src/services/ai-review.ts 95.17% <100.00%> (+0.01%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant