Skip to content

Add a Prometheus alert rule for governor rate-limit/budget threshold pressure #5187

Description

@JSONbored

Context

prometheus/rules/alerts.yml already has an established GitHub-rate-limit-pressure rule for ORB, but AMS's governor has no equivalent alert for its own rate-limit/budget state. This is distinct from #5134 (persisting the governor's cross-attempt state, maintainer-only) and #4840 (alerting guidance docs, not rule definitions) — this issue is the concrete alert-rule definition, reading governor state rather than modifying it.

Dependencies

Depends on #5134 (persist governor cross-attempt state) landing first. #5134 is what would make the governor's rate-limit/budget state persisted and observable in the first place — there is no metric for this rule to alert on until that ships. Do not start this issue until #5134 lands and documents the persisted state/metric it exposes.

Requirements

  1. Add a new rule group to prometheus/rules/alerts.yml that fires when the governor's persisted rate-limit or budget state crosses a configurable threshold.
  2. Use the same severity/for/runbook-annotation format as the existing GitHub-rate-limit-pressure rule for ORB.
  3. The threshold must be configurable (not hardcoded), consistent with the pattern used by existing similar rules in the same file.
  4. Reference the metric/state exposed by Persist governor cross-attempt state (rate-limit, budget, convergence, reputation, self-plagiarism) #5134 once it lands — do not invent a metric name; use whatever Persist governor cross-attempt state (rate-limit, budget, convergence, reputation, self-plagiarism) #5134 documents/exposes.
  5. This issue reads governor state only — it must not modify governor decision logic in any way, since that logic stays maintainer-only per Persist governor cross-attempt state (rate-limit, budget, convergence, reputation, self-plagiarism) #5134.
  6. Include a runbook annotation link/text following the existing convention.

Deliverables / Acceptance Criteria

Test Coverage Requirements

This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. prometheus/rules/alerts.yml is config, not src/**, so Codecov's line/branch gate does not directly instrument it — but if the repo has a promtool test rules harness (or equivalent), this rule must ship with test cases for both sides of the threshold: (1) a rule test where the governor's persisted rate-limit/budget state stays under the configured threshold and no alert fires, (2) a rule test where the state crosses the threshold for the full for: duration and the alert fires with the expected labels/severity, and (3) an invariant test asserting this rule only reads the governor's persisted state and never triggers a write to or mutation of governor decision logic. No regression test applies since this is a new rule, not a bug fix.

Expected Outcome

Once #5134 ships persisted governor state, an operator's existing Alertmanager can immediately load this rule and get paged when the governor is approaching a rate-limit or budget ceiling, without needing to hand-author the rule themselves.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions