You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ORB already ships a Prometheus alert rule group for AI-review-reliability / circuit-breaker conditions, and the miner-prediction side already has a calibration-metrics renderer that Prometheus can scrape. What's missing is an actual rule group in prometheus/rules/alerts.yml that fires when those calibration metrics drift beyond an acceptable tolerance -- today the metrics exist but nothing watches them.
Dependencies
Depends on #4838 (wiring the Prometheus renderer into a command so the calibration metrics are actually emitted/scraped) and #4849 (building the calibration-report join the renderer reads from) landing first. Do not start this issue until both of those have shipped -- writing an alert rule against metrics that either don't exist yet or are joined incorrectly will produce a rule that silently never fires (or fires on garbage data), which is worse than no rule at all. If you pick this up before #4838 and #4849 are merged, stop and wait; check their PR status first.
Requirements
Add a new rule group to prometheus/rules/alerts.yml, following the same structural pattern (naming, for:, labels:, annotations:) as ORB's existing AI-review-reliability/circuit-breaker rule group in the same file.
Define a configurable drift tolerance threshold (e.g. via a rule expression constant or an existing Prometheus recording rule pattern already used elsewhere in the file) rather than a magic number buried inline, so the threshold is easy to tune later.
Set a reasonable for: duration so the alert doesn't flap on a single noisy scrape (mirror the duration style used by the existing circuit-breaker rule).
Add severity and descriptive annotations (summary/description) consistent with the existing rule group's annotation style, so the alert is actionable in Alertmanager/Grafana without needing to read the YAML.
This PR must ship with full test coverage for every changed line and branch -- the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. Since this is a declarative Prometheus rules file rather than application code, test coverage here means: (1) a promtool test rules (or equivalent unit-test fixture the repo already uses for alerts.yml) covering both the case where calibration metrics are within tolerance (alert does NOT fire) and where they drift past the threshold (alert DOES fire), (2) an invariant test asserting the alert rule never fires when the underlying calibration metric is absent/unset (i.e. it degrades to silent, not a false positive), and (3) if any code change is needed to make the metric testable under promtool, add a regression test locking in the exact threshold behavior.
Expected Outcome
An operator running self-hosted Grafana/Prometheus will get paged (or see a firing alert) automatically when miner-prediction calibration drifts out of tolerance, instead of having to notice the drift by eyeballing a dashboard.
Links & Resources
prometheus/rules/alerts.yml (existing ORB circuit-breaker rule group to mirror)
Context
ORB already ships a Prometheus alert rule group for AI-review-reliability / circuit-breaker conditions, and the miner-prediction side already has a calibration-metrics renderer that Prometheus can scrape. What's missing is an actual rule group in prometheus/rules/alerts.yml that fires when those calibration metrics drift beyond an acceptable tolerance -- today the metrics exist but nothing watches them.
Dependencies
Depends on #4838 (wiring the Prometheus renderer into a command so the calibration metrics are actually emitted/scraped) and #4849 (building the calibration-report join the renderer reads from) landing first. Do not start this issue until both of those have shipped -- writing an alert rule against metrics that either don't exist yet or are joined incorrectly will produce a rule that silently never fires (or fires on garbage data), which is worse than no rule at all. If you pick this up before #4838 and #4849 are merged, stop and wait; check their PR status first.
Requirements
Deliverables / Acceptance Criteria
Test Coverage Requirements
This PR must ship with full test coverage for every changed line and branch -- the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. Since this is a declarative Prometheus rules file rather than application code, test coverage here means: (1) a promtool test rules (or equivalent unit-test fixture the repo already uses for alerts.yml) covering both the case where calibration metrics are within tolerance (alert does NOT fire) and where they drift past the threshold (alert DOES fire), (2) an invariant test asserting the alert rule never fires when the underlying calibration metric is absent/unset (i.e. it degrades to silent, not a false positive), and (3) if any code change is needed to make the metric testable under promtool, add a regression test locking in the exact threshold behavior.
Expected Outcome
An operator running self-hosted Grafana/Prometheus will get paged (or see a firing alert) automatically when miner-prediction calibration drifts out of tolerance, instead of having to notice the drift by eyeballing a dashboard.
Links & Resources