Skip to content

fix(selfhost): validate AI reviewer-provider configuration and add a failure circuit breaker #2540

Description

@JSONbored

Parent: #1936

Problem

Self-host's dual-AI review path resolves up to two configured reviewer "slots" from an operator-provided provider list, with no deduplication or distinctness check. A plausible configuration mistake (the same provider name listed twice, or a copy-paste artifact) silently collapses "two independent reviewers reaching consensus" into "the same provider/model called twice in parallel," which defeats the purpose of dual-AI review and means a single provider's outage or auth failure takes down both reviewer slots simultaneously. Live error telemetry over a short window showed a correlated spike of "inconclusive verdict" and "provider exhausted" events consistent with exactly this failure mode.

Separately, there is no circuit breaker or failure-streak tracking for AI providers: every single PR review independently retries a full multi-attempt budget against a provider from a cold state, even when that same provider failed the exact same way on the previous PR reviewed seconds earlier. During a sustained provider outage, this multiplies retry cost and review latency across every PR reviewed in that window instead of degrading fast.

Requirements

  • Reject or loudly warn (fail loud at boot, matching the existing pattern used for other legacy/misconfigured env combinations) when the resolved reviewer plan would call the identical provider+model for more than one reviewer slot.
  • Add a lightweight, in-process failure-streak/cooldown per provider name so a provider already known to be failing short-circuits to a fast failure instead of repeating its full retry budget on every subsequent PR review within the cooldown window.
  • Add basic observability: a counter for inconclusive-verdict events and per-provider failure/exhaustion events, wired into the existing Prometheus/Alertmanager pattern used elsewhere in this repo.

Deliverables

  • Validation in the reviewer-plan resolution path that catches and surfaces duplicate provider+model configuration.
  • A provider-health cooldown mechanism scoped to the running process.
  • New Prometheus counters + a corresponding alert rule, following the existing alert-rule conventions in this repo.
  • Tests covering: duplicate-provider configuration is caught, a provider under cooldown short-circuits without a full retry budget, and the cooldown expires/recovers correctly.

Acceptance criteria

  • A duplicate-provider configuration is surfaced clearly at boot/startup rather than silently degrading review quality.
  • A sustained provider outage produces materially less retry cost per PR after the first failure within the cooldown window.
  • New metrics are visible and alertable using the existing observability stack.

Expected outcome

Dual-AI review actually provides two independent opinions whenever configured to, and a provider outage degrades gracefully and visibly instead of silently and repeatedly.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions