Skip to content

feat(selfhost): add a per-analyzer circuit breaker to review-enrichment - #2624

Merged
JSONbored merged 4 commits into
mainfrom
fix/selfhost-review-enrichment-circuit-breaker
Jul 2, 2026
Merged

feat(selfhost): add a per-analyzer circuit breaker to review-enrichment#2624
JSONbored merged 4 commits into
mainfrom
fix/selfhost-review-enrichment-circuit-breaker

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Closes #2541.

review-enrichment's analyzer pipeline is correctly fail-safe with respect to the main review verdict — an enrichment timeout degrades review quality but never blocks the review. However, individual analyzers that depend on a third-party HTTP API (registry lookups, GitHub API calls, endoflife.date, etc.) had no memory of recent failures: every incoming enrichment request re-attempted a currently-unhealthy dependency from a cold state, even seconds after an identical call just timed out or errored.

What changed

  • New module review-enrichment/src/analyzer-circuit-breaker.ts: a generic, per-analyzer, in-process circuit breaker (module-level Map, no persistence layer — review-enrichment is a single long-lived process). After 3 consecutive thrown failures (a timeout counts — it's a rejection from runWithTimeout), an analyzer trips its breaker and enters a 5-minute cooldown. Mirrors the shape of src/selfhost/ai.ts's per-provider AI circuit breaker built for the same class of problem in a related roadmap issue (fix(selfhost): validate AI reviewer-provider configuration and add a failure circuit breaker #2540).
  • scheduler.ts: skipReasonForAnalyzer now checks isAnalyzerCircuitOpen first, before any other check — a circuit-open analyzer is skipped at planning time (via the exact same plan.skipped/skipReason mechanism every other skip reason already uses) regardless of whether it was explicitly requested via req.analyzers, since an explicit request can't fix a dependency that's actually down.
  • brief.ts: runAnalyzer records a circuit-breaker success after findings[name] = result (covering BOTH a clean "ok" run and a non-throwing degraded/partial one — the dependency responded, just not completely, so this is not the failure mode the breaker guards against) and records a failure at the top of the catch block (a genuine thrown error or timeout).
  • Cost-class parallelization, evaluated per the issue's own ask: verified directly against the real orchestration (for (const cost of COST_ORDER) { await runWithConcurrency(...) }) — cost classes DO run strictly sequentially, each class's bounded worker pool fully draining before the next starts. This is deliberate prioritization (cheap/certain signals collected before expensive/uncertain ones; a shrinking time budget correctly starves later classes first, never the reverse), not an oversight. Parallelizing it would sum every class's concurrency limit at once (8+3+2+1+1 = 15 simultaneous external calls on the "deep" profile instead of at most 8), spiking third-party call volume exactly when third-party health is the concern this issue is about — not a low-risk change. Documented in place in brief.ts rather than implemented, per the issue's own "or file as a documented follow-up" option.

Correctness notes

  • Generic by analyzer name, not special-cased to one specific known-flaky analyzer — applies uniformly to any future analyzer with the same shape.
  • Mutation-tested both integration points directly: removing the scheduler.ts circuit-open check and removing the brief.ts failure-recording call each caused the corresponding tests to fail as expected (a 4th call reaching the "broken" analyzer instead of being skipped), then reverted.
  • review-enrichment is a separate package (Node's built-in test runner, tsc build, its own npm test chain — build && validate:sourcemaps && metadata:check && test:node) outside src/**, so the main app's Codecov patch-coverage gate does not apply here; validated via the full local npm run rees:test (the exact command CI runs) instead.

Tests

review-enrichment/test/analyzer-circuit-breaker.test.ts (new): stays closed below the failure-streak threshold; opens and skips after 3 consecutive failures with zero further calls to the broken dependency; a timeout counts the same as a thrown error; a non-throwing partial result does NOT count as a failure; a success resets the streak so it doesn't bleed into a later, separate failure run; a circuit-expired analyzer is tried again rather than staying open forever; an explicitly-requested analyzer is still skipped while its circuit is open.

Validation

  • npm run rees:test (from repo root — the exact command CI's review-enrichment job runs: install, build, validate-sourcemaps, metadata:check, then the full node --test suite)
  • All 401 existing review-enrichment tests still pass alongside the 8 new ones
  • npm run db:migrations:check (no-op — no schema changes; this PR is entirely scoped to review-enrichment/)
  • git diff --check
  • Manual mutation testing of both integration points (see Correctness notes above)
  • Ran the new timeout test 5x in isolation to confirm no flakiness after tuning its budget to match scheduler.test.ts's own proven-stable value

Scope

  • Change is narrow and limited to the stated problem
  • No secrets, wallets, hotkeys, trust scores, or reward values added anywhere
  • No edits to site/, CNAME, **/lovable/**, or CHANGELOG.md

Safety

  • No auth/session/CORS surface touched
  • No change to the non-blocking, fail-safe contract of the enrichment service as a whole — a circuit-open analyzer degrades exactly like any other skip reason (missing input, budget-capped, etc.) that already existed

@dosubot dosubot Bot added the size:M label Jul 2, 2026
@loopover-orb

loopover-orb Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-02 21:30:23 UTC

4 files · 1 AI reviewer · no blockers · readiness 80/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
The diff adds an in-process per-analyzer circuit breaker, wires it into planning, and records success/failure outcomes around analyzer execution. The core open/cooldown/half-open behavior is coherent, and the scheduler ordering correctly avoids claiming a half-open probe for analyzers that will be skipped for unrelated input reasons. The tests exercise the important state transitions and end-to-end skip path, but the implementation relies on side effects from `isAnalyzerCircuitOpen`, which makes future call-site changes easy to get wrong.

Nits — 6 non-blocking
  • nit: `review-enrichment/src/analyzer-circuit-breaker.ts:39` makes `isAnalyzerCircuitOpen` both a predicate and a state-mutating probe claimer, so a future diagnostic or telemetry caller could accidentally consume the single half-open probe; consider naming it to reflect the claim side effect or splitting check-vs-claim APIs.
  • nit: `review-enrichment/src/brief.ts:394` adds a long design note about cost-class parallelization in the hot orchestration file even though the code path is unchanged; this would be easier to maintain as a shorter comment or a focused test/documentation note tied to the scheduler behavior.
  • nit: `review-enrichment/test/analyzer-circuit-breaker.test.ts:96` and similar tests monkey-patch `Date.now` repeatedly; a small helper would reduce the chance of a future test forgetting to restore it.
  • Rename `isAnalyzerCircuitOpen` in `review-enrichment/src/analyzer-circuit-breaker.ts:39` to something like `shouldSkipAnalyzerAndMaybeClaimProbe`, or add a side-effect-free read helper for tests/telemetry.
  • Move the cost-class scheduling rationale from `review-enrichment/src/brief.ts:394` into a scheduler-focused test or compact it to the invariant the code depends on: cost classes are awaited sequentially.
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #2541
Related work ⚠️ 3 scoped overlaps Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope ❌ 8/20 High review scope from cached public metadata (size label size:L; 1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 64 registered-repo PR(s), 55 merged, 526 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 64 PR(s), 526 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Review top overlaps.
  • Add a concise scope and risk note.
  • No action.
  • Check active issues and PRs before submitting.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

Closes #2541.

review-enrichment's analyzer pipeline is correctly fail-safe with respect to
the main review verdict, but individual analyzers with a third-party HTTP
dependency (registry lookups, GitHub API calls, endoflife.date, etc) had no
memory of recent failures -- every incoming enrichment request re-attempted
a currently-unhealthy dependency from a cold state, even seconds after an
identical call just timed out or errored.

Adds a generic, per-analyzer, in-process circuit breaker (module-level Map,
no persistence layer, matching src/selfhost/ai.ts's per-provider AI circuit
breaker for the same class of problem): after 3 consecutive THROWN failures
(a timeout counts -- it's a rejection from runWithTimeout), an analyzer is
skipped entirely at planning time for a 5-minute cooldown, falling through
the SAME plan.skipped mechanism every other skip reason already uses. A
non-throwing degraded/partial result does NOT count as a failure -- the
dependency responded, just not completely -- so it resets the streak rather
than risk tripping the breaker on a benign internal cap unrelated to
third-party health. Applies uniformly to every analyzer, including a future
one with the same shape, not special-cased to one specific name.

Also evaluated (per the issue's own ask) whether cost classes running
strictly sequentially could be parallelized. Verified directly against the
real orchestration in brief.ts (the deliberate for-loop over COST_ORDER,
each class's bounded worker pool fully draining before the next starts) --
not implemented: cost-class ordering is intentional prioritization (cheap,
certain signals collected before expensive, uncertain ones; a shrinking
time budget correctly starves later classes first, never the reverse), and
running every class concurrently would sum every class's concurrency limit
at once, spiking third-party call volume exactly when third-party health is
the concern this issue is about. Documented in place (brief.ts) rather than
implemented, per the issue's own "or file as a documented follow-up" option.
@JSONbored
JSONbored force-pushed the fix/selfhost-review-enrichment-circuit-breaker branch from 43a21cf to 4b27b5f Compare July 2, 2026 21:05
…eaker

The breaker had no half-open state: once the cooldown expired, a burst of
concurrent requests (review-enrichment serves one per in-flight PR review)
would all see the circuit as closed and all retry the same still-unhealthy
dependency at once, defeating the point of the cooldown.

isAnalyzerCircuitOpen now claims a single probe slot as a side effect the
first time it observes an expired cooldown; every other concurrent caller
sees the circuit as still open until that one probe resolves. A claimed
probe that never actually reaches the analyzer call (budget/timeout capped
in brief.ts first) is released via releaseAnalyzerCircuitProbe rather than
staying stuck forever with no outcome ever recorded.
@dosubot dosubot Bot added size:L and removed size:M labels Jul 2, 2026
JSONbored added 2 commits July 2, 2026 14:25
…streak threshold

isAnalyzerCircuitOpen claimed the half-open probe slot for ANY existing
circuit state, since cooldownUntilMs <= nowMs is also true at the initial
cooldownUntilMs === 0 (never tripped). A circuit with only 1-2 recorded
failures would spuriously skip a concurrent second caller as circuit_open,
even though the breaker never actually opened.

Gate the whole half-open branch on cooldownUntilMs !== 0, the same signal
recordAnalyzerCircuitFailure already uses to mean "reached the threshold."
…r, after every other skip reason

isAnalyzerCircuitOpen was checked FIRST, so it could claim the half-open
probe for an analyzer that's about to be skipped for a totally unrelated
reason (missing head SHA, no dependency manifest, no added lines, etc.).
Since a plan.skipped item never reaches runAnalyzer -- the only place a
claimed probe is released -- that claim would leak forever, permanently
blocking every later request from probing the analyzer again even once
healthy.

Move the circuit check to the end, after every other skip condition has
cleared, so the probe is only ever claimed when the analyzer would
otherwise actually run.
@JSONbored
JSONbored merged commit bcc19ea into main Jul 2, 2026
10 checks passed
@JSONbored
JSONbored deleted the fix/selfhost-review-enrichment-circuit-breaker branch July 2, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Development

Successfully merging this pull request may close these issues.

feat(selfhost): add a per-analyzer circuit breaker to review-enrichment

1 participant