Skip to content

Add a Prometheus alert rule for prediction-calibration drift #5188

Description

@JSONbored

Context

ORB already ships a Prometheus alert rule group for AI-review-reliability / circuit-breaker conditions, and the miner-prediction side already has a calibration-metrics renderer that Prometheus can scrape. What's missing is an actual rule group in prometheus/rules/alerts.yml that fires when those calibration metrics drift beyond an acceptable tolerance -- today the metrics exist but nothing watches them.

Dependencies

Depends on #4838 (wiring the Prometheus renderer into a command so the calibration metrics are actually emitted/scraped) and #4849 (building the calibration-report join the renderer reads from) landing first. Do not start this issue until both of those have shipped -- writing an alert rule against metrics that either don't exist yet or are joined incorrectly will produce a rule that silently never fires (or fires on garbage data), which is worse than no rule at all. If you pick this up before #4838 and #4849 are merged, stop and wait; check their PR status first.

Requirements

  1. Add a new rule group to prometheus/rules/alerts.yml, following the same structural pattern (naming, for:, labels:, annotations:) as ORB's existing AI-review-reliability/circuit-breaker rule group in the same file.
  2. The rule must key off the miner-prediction calibration metric(s) exposed by the renderer landed in Wire the existing miner-prediction Prometheus renderer into a real command #4838/Build the missing calibration-report join #4849 -- do not invent new metric names; use whatever metric name(s) that renderer actually emits.
  3. Define a configurable drift tolerance threshold (e.g. via a rule expression constant or an existing Prometheus recording rule pattern already used elsewhere in the file) rather than a magic number buried inline, so the threshold is easy to tune later.
  4. Set a reasonable for: duration so the alert doesn't flap on a single noisy scrape (mirror the duration style used by the existing circuit-breaker rule).
  5. Add severity and descriptive annotations (summary/description) consistent with the existing rule group's annotation style, so the alert is actionable in Alertmanager/Grafana without needing to read the YAML.
  6. Do not modify calibration scoring logic, the renderer, or the report-join logic themselves -- this issue is alert-authoring only, consuming metrics that already exist once Wire the existing miner-prediction Prometheus renderer into a real command #4838/Build the missing calibration-report join #4849 ship.

Deliverables / Acceptance Criteria

  • New rule group added to prometheus/rules/alerts.yml for prediction-calibration drift
  • Rule follows the existing ORB AI-review-reliability/circuit-breaker rule group's structure and style
  • Threshold and for: duration are named/configurable, not unexplained inline literals
  • severity and human-readable annotations (summary + description) are present
  • Any Prometheus rule-lint/promtool validation the repo runs in CI passes on the new rule group
  • PR explicitly notes in its description that it was built against Wire the existing miner-prediction Prometheus renderer into a real command #4838/Build the missing calibration-report join #4849's shipped metric shape

Test Coverage Requirements

This PR must ship with full test coverage for every changed line and branch -- the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. Since this is a declarative Prometheus rules file rather than application code, test coverage here means: (1) a promtool test rules (or equivalent unit-test fixture the repo already uses for alerts.yml) covering both the case where calibration metrics are within tolerance (alert does NOT fire) and where they drift past the threshold (alert DOES fire), (2) an invariant test asserting the alert rule never fires when the underlying calibration metric is absent/unset (i.e. it degrades to silent, not a false positive), and (3) if any code change is needed to make the metric testable under promtool, add a regression test locking in the exact threshold behavior.

Expected Outcome

An operator running self-hosted Grafana/Prometheus will get paged (or see a firing alert) automatically when miner-prediction calibration drifts out of tolerance, instead of having to notice the drift by eyeballing a dashboard.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions