Skip to content

feat(selfhost): add a per-analyzer circuit breaker to review-enrichment - #2632

Merged
JSONbored merged 1 commit into
mainfrom
feat/rees-analyzer-circuit-breaker
Jul 2, 2026
Merged

feat(selfhost): add a per-analyzer circuit breaker to review-enrichment#2632
JSONbored merged 1 commit into
mainfrom
feat/rees-analyzer-circuit-breaker

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Advances #2541

Problem

The review-enrichment service's analyzer pipeline is correctly fail-safe with respect to the main review — an enrichment timeout degrades review quality but never blocks the review itself. However, within the service, individual analyzers that depend on a third-party HTTP API (osv.dev, bundlephobia, endoflife.date, npm/pypi registries, deps.dev) have no memory of recent failures: every request re-attempts a currently-unhealthy endpoint from a cold state, even seconds after an identical call just timed out or errored.

Changes

  • New circuit breaker in review-enrichment/src/external-fetch.ts, wired into the shared boundedFetchText/boundedFetchStatus chokepoint (boundedFetchJson inherits it automatically via its internal boundedFetchText call). Keyed by endpointCategory. After 3 consecutive remote-health failures, further calls for that category are skipped for a 30s cooldown — no real HTTP cost — instead of retrying a known-broken endpoint on every request.
  • Narrow failure classification (isRemoteHealthFailure): only timeout, network_error, and http_error with status 403/429/5xx trip the breaker. A plain 404 does nottyposquat.ts calls boundedFetchStatus for many candidate package names where a 404 just means "this candidate doesn't exist," a legitimate negative result, not a sign the remote service is unhealthy. aborted (always caller-driven) and response_too_large/invalid_json (the remote did respond) are excluded too.
  • A circuit-open skip returns the exact same BoundedFetchFailure shape every other failure path already returns (never throws) and flows through the existing attachDiagnostics pipeline, so the service's fail-safe/non-blocking contract with the main review is unchanged — no new instrumentation needed.

Cost-class parallelization (issue's second ask)

Evaluated parallelizing analyzer cost-class execution (review-enrichment/src/brief.ts currently drains each of the 5 COST_ORDER classes — local, registry, github-light, github-heavy, tooling — strictly sequentially before starting the next) and am deferring it as a follow-up rather than implementing it here:

  • The current per-class concurrency limits were presumably tuned assuming classes run in isolation. Running all 5 concurrently would let up to the sum of all per-class limits run at once (e.g. balanced profile: 8+3+2+1+1 = 15 concurrent operations instead of a max of 8 today) — a meaningfully different resource/rate-limit profile against third-party APIs that hasn't been validated against real traffic.
  • The sequential ordering may intentionally prioritize cheap/fast classes completing reliably before spending time-budget on expensive ones under budget pressure; parallelizing removes that implicit prioritization.
  • No live production telemetry is available in this session to validate the change's safety before shipping it, unlike the circuit breaker, whose behavior is narrow and fully testable in isolation.

Recommend filing a scoped follow-up issue to trial cost-class parallelization behind a flag/profile variant rather than as a blanket change.

Test plan

  • npm run build (review-enrichment) clean
  • npm run test (review-enrichment: build + sourcemap validation + analyzer-metadata check + full node:test suite) — 402/402 passing
  • Root npm run typecheck clean
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • New tests: healthy endpoint never opens the circuit; opens after 3 consecutive network errors/500s and skips the underlying fetch during cooldown; recovers after cooldown elapses; regression — repeated plain 404s never trip the breaker (the load-bearing typosquat scenario); regression — repeated aborted results never trip the breaker; boundedFetchStatus/boundedFetchText/boundedFetchJson share one circuit per endpointCategory; two different categories never cross-contaminate; reset helper clears state between tests

Every analyzer that depends on a third-party HTTP API (osv.dev,
bundlephobia, endoflife.date, npm/pypi registries, deps.dev) re-attempts a
currently-unhealthy endpoint from a cold state on every request -- even
seconds after an identical call just timed out or errored.

Adds a circuit breaker to the shared boundedFetchText/boundedFetchStatus
chokepoint (boundedFetchJson inherits it via boundedFetchText), keyed by
endpointCategory: after 3 consecutive remote-health failures within a
window, further calls are skipped for a 30s cooldown instead of paying the
real network cost. Only failures that actually indicate the remote is
unhealthy count toward the breaker (timeout, network_error, and http_error
with status 403/429/5xx) -- a plain 404 does not, since analyzers like
typosquat.ts call boundedFetchStatus for candidate package names where a
404 is a legitimate negative result, not a service-health signal.

A circuit-open skip returns the same BoundedFetchFailure shape every other
failure path already returns (never throws), so the service's existing
fail-safe/non-blocking contract with the main review is unaffected -- it
surfaces through the same diagnostics pipeline every other failure reason
already uses.

Evaluated parallelizing analyzer cost-class execution (currently strictly
sequential in brief.ts) per the issue's second ask; deferring it as a
scoped follow-up rather than implementing here, since it would change the
effective total concurrency profile against third-party rate limits in a
way that hasn't been validated against real traffic.
@dosubot dosubot Bot added the size:L label Jul 2, 2026
@loopover-orb

loopover-orb Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-02 20:34:52 UTC

2 files · 1 AI reviewer · no blockers · readiness 73/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
The change adds an in-memory per-endpointCategory circuit breaker at the shared external-fetch chokepoint and extends tests across text/status/json callers, cooldown behavior, diagnostics, and non-tripping failures like 404s and caller aborts. The implementation preserves the existing fail-safe result shape and correctly limits breaker trips to timeout, network errors, and selected HTTP statuses. I do not see a reachable correctness break in the provided diff; the main remaining gaps are around documenting the exported test reset helper and tightening edge-case test coverage for the explicit 403/429 branches and failure-count reset semantics.

Nits — 6 non-blocking
  • nit: review-enrichment/src/external-fetch.ts:58 exports resetExternalFetchCircuitBreakerForTest from production code without a test-only naming guard or comment explaining why this export is acceptable in the runtime module surface.
  • nit: review-enrichment/test/external-fetch.test.ts:208 covers 500 as a remote-health HTTP failure and 404 as non-health, but does not directly exercise the explicit 403/429 status branches added in isRemoteHealthFailure.
  • nit: review-enrichment/test/external-fetch.test.ts:251 verifies cooldown recovery after three failures followed by success, but does not pin the intended behavior for one or two failures followed by a success resetting the consecutive failure count.
  • In review-enrichment/src/external-fetch.ts:58, add a short comment that resetExternalFetchCircuitBreakerForTest is exported only for node:test isolation, or move circuit state behind a small internal test hook pattern used elsewhere in the repo if one exists.
  • In review-enrichment/test/external-fetch.test.ts, add a table-style status test covering 403, 429, 500, and 404 so the breaker classification mirrors the documented policy exactly.
  • Readiness score is below the configured threshold — Use the readiness panel as advisory maintainer context; the score does not block this PR.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ⚠️ 3 scoped overlaps Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope ❌ 8/20 High review scope from cached public metadata (size label size:L; no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 64 registered-repo PR(s), 55 merged, 526 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 64 PR(s), 526 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Review top overlaps.
  • Add a concise scope and risk note.
  • No action.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
  • Check active issues and PRs before submitting.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. labels Jul 2, 2026
@JSONbored
JSONbored merged commit 23f3a66 into main Jul 2, 2026
10 checks passed
@JSONbored
JSONbored deleted the feat/rees-analyzer-circuit-breaker branch July 2, 2026 20:40
JSONbored added a commit that referenced this pull request Jul 2, 2026
The AI review flagged a real defect: cachedFetchLivePullRequestMergeState
/ cachedFetchLivePullRequestState / cachedFetchLivePullRequestHeadSha
each wrote through only the ONE field they cared about, but all three
share a single prStateFetchedAt freshness stamp. A live fetch from any
one of them would make the OTHER two fields look "fresh" to a
subsequent reader despite never having been fetched, so that reader
would silently return undefined for a field that was simply never
populated -- indistinguishable from a confirmed-empty GitHub value.

All three narrow live-fetchers already hit the exact same GET
/pulls/{n} endpoint, just extracting one field each, so there's no
extra API cost to fixing this: a new internal fetchAndCachePrStateFields
helper fetches the full payload once and writes mergeable_state, state,
and headSha through together (headSha omitted, not nulled, when absent,
preserving the existing PARTIAL-UPDATE CONTRACT so a prior headSha the
files cache depends on is never cleared). It also only writes when the
fetch actually succeeds, so a transient failure no longer poisons the
cache with a false "confirmed fresh" stamp the way the old per-field
writes did. The three public, uncached narrow fetchers used by the
act-boundary/gate-override callers are untouched.

Also fixes a second, separate flagged issue: `{ token: "installation-token" }`
in six of this PR's own new test fixtures tripped the deterministic
generic_secret_assignment scanner (a keyword-shaped heuristic, not a
real credential format) -- renamed to `"fake-installation-token"`,
which the scanner's own placeholder-value allowlist already recognizes,
without touching the shared scanner itself.

Rebased onto current main (renumbered migration 0093->0094 to resolve
a collision with #2616, and again to catch up with #2632). Full local
gate (test:ci, npm audit) green; no other changes.
JSONbored added a commit that referenced this pull request Jul 2, 2026
The AI review flagged a real defect: cachedFetchLivePullRequestMergeState
/ cachedFetchLivePullRequestState / cachedFetchLivePullRequestHeadSha
each wrote through only the ONE field they cared about, but all three
share a single prStateFetchedAt freshness stamp. A live fetch from any
one of them would make the OTHER two fields look "fresh" to a
subsequent reader despite never having been fetched, so that reader
would silently return undefined for a field that was simply never
populated -- indistinguishable from a confirmed-empty GitHub value.

All three narrow live-fetchers already hit the exact same GET
/pulls/{n} endpoint, just extracting one field each, so there's no
extra API cost to fixing this: a new internal fetchAndCachePrStateFields
helper fetches the full payload once and writes mergeable_state, state,
and headSha through together (headSha omitted, not nulled, when absent,
preserving the existing PARTIAL-UPDATE CONTRACT so a prior headSha the
files cache depends on is never cleared). It also only writes when the
fetch actually succeeds, so a transient failure no longer poisons the
cache with a false "confirmed fresh" stamp the way the old per-field
writes did. The three public, uncached narrow fetchers used by the
act-boundary/gate-override callers are untouched.

Also fixes a second, separate flagged issue: `{ token: "installation-token" }`
in six of this PR's own new test fixtures tripped the deterministic
generic_secret_assignment scanner (a keyword-shaped heuristic, not a
real credential format) -- renamed to `"fake-installation-token"`,
which the scanner's own placeholder-value allowlist already recognizes,
without touching the shared scanner itself.

Rebased onto current main (renumbered migration 0093->0094 to resolve
a collision with #2616, and again to catch up with #2632). Full local
gate (test:ci, npm audit) green; no other changes.
JSONbored added a commit that referenced this pull request Jul 2, 2026
The AI review flagged a real defect: cachedFetchLivePullRequestMergeState
/ cachedFetchLivePullRequestState / cachedFetchLivePullRequestHeadSha
each wrote through only the ONE field they cared about, but all three
share a single prStateFetchedAt freshness stamp. A live fetch from any
one of them would make the OTHER two fields look "fresh" to a
subsequent reader despite never having been fetched, so that reader
would silently return undefined for a field that was simply never
populated -- indistinguishable from a confirmed-empty GitHub value.

All three narrow live-fetchers already hit the exact same GET
/pulls/{n} endpoint, just extracting one field each, so there's no
extra API cost to fixing this: a new internal fetchAndCachePrStateFields
helper fetches the full payload once and writes mergeable_state, state,
and headSha through together (headSha omitted, not nulled, when absent,
preserving the existing PARTIAL-UPDATE CONTRACT so a prior headSha the
files cache depends on is never cleared). It also only writes when the
fetch actually succeeds, so a transient failure no longer poisons the
cache with a false "confirmed fresh" stamp the way the old per-field
writes did. The three public, uncached narrow fetchers used by the
act-boundary/gate-override callers are untouched.

Also fixes a second, separate flagged issue: `{ token: "installation-token" }`
in six of this PR's own new test fixtures tripped the deterministic
generic_secret_assignment scanner (a keyword-shaped heuristic, not a
real credential format) -- renamed to `"fake-installation-token"`,
which the scanner's own placeholder-value allowlist already recognizes,
without touching the shared scanner itself.

Rebased onto current main (renumbered migration 0093->0094 to resolve
a collision with #2616, and again to catch up with #2632). Full local
gate (test:ci, npm audit) green; no other changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant