You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
prometheus/rules/alerts.yml already has an established GitHub-rate-limit-pressure rule for ORB, but AMS's governor has no equivalent alert for its own rate-limit/budget state. This is distinct from #5134 (persisting the governor's cross-attempt state, maintainer-only) and #4840 (alerting guidance docs, not rule definitions) — this issue is the concrete alert-rule definition, reading governor state rather than modifying it.
Dependencies
Depends on #5134 (persist governor cross-attempt state) landing first. #5134 is what would make the governor's rate-limit/budget state persisted and observable in the first place — there is no metric for this rule to alert on until that ships. Do not start this issue until #5134 lands and documents the persisted state/metric it exposes.
Requirements
Add a new rule group to prometheus/rules/alerts.yml that fires when the governor's persisted rate-limit or budget state crosses a configurable threshold.
Use the same severity/for/runbook-annotation format as the existing GitHub-rate-limit-pressure rule for ORB.
The threshold must be configurable (not hardcoded), consistent with the pattern used by existing similar rules in the same file.
promtool check rules (or equivalent CI rule-linting) passes
No changes made to governor decision logic anywhere in the codebase
Test Coverage Requirements
This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. prometheus/rules/alerts.yml is config, not src/**, so Codecov's line/branch gate does not directly instrument it — but if the repo has a promtool test rules harness (or equivalent), this rule must ship with test cases for both sides of the threshold: (1) a rule test where the governor's persisted rate-limit/budget state stays under the configured threshold and no alert fires, (2) a rule test where the state crosses the threshold for the full for: duration and the alert fires with the expected labels/severity, and (3) an invariant test asserting this rule only reads the governor's persisted state and never triggers a write to or mutation of governor decision logic. No regression test applies since this is a new rule, not a bug fix.
Expected Outcome
Once #5134 ships persisted governor state, an operator's existing Alertmanager can immediately load this rule and get paged when the governor is approaching a rate-limit or budget ceiling, without needing to hand-author the rule themselves.
Links & Resources
Target file: prometheus/rules/alerts.yml
Existing pattern to mirror: the GitHub-rate-limit-pressure rule (ORB) in the same file
Context
prometheus/rules/alerts.ymlalready has an established GitHub-rate-limit-pressure rule for ORB, but AMS's governor has no equivalent alert for its own rate-limit/budget state. This is distinct from #5134 (persisting the governor's cross-attempt state, maintainer-only) and #4840 (alerting guidance docs, not rule definitions) — this issue is the concrete alert-rule definition, reading governor state rather than modifying it.Dependencies
Depends on #5134 (persist governor cross-attempt state) landing first. #5134 is what would make the governor's rate-limit/budget state persisted and observable in the first place — there is no metric for this rule to alert on until that ships. Do not start this issue until #5134 lands and documents the persisted state/metric it exposes.
Requirements
prometheus/rules/alerts.ymlthat fires when the governor's persisted rate-limit or budget state crosses a configurable threshold.Deliverables / Acceptance Criteria
prometheus/rules/alerts.ymlfor governor rate-limit/budget threshold pressureseverity/for/runbook annotations mirror the existing GitHub-rate-limit-pressure rule's formatpromtool check rules(or equivalent CI rule-linting) passesTest Coverage Requirements
This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced.
prometheus/rules/alerts.ymlis config, notsrc/**, so Codecov's line/branch gate does not directly instrument it — but if the repo has apromtool test rulesharness (or equivalent), this rule must ship with test cases for both sides of the threshold: (1) a rule test where the governor's persisted rate-limit/budget state stays under the configured threshold and no alert fires, (2) a rule test where the state crosses the threshold for the fullfor:duration and the alert fires with the expected labels/severity, and (3) an invariant test asserting this rule only reads the governor's persisted state and never triggers a write to or mutation of governor decision logic. No regression test applies since this is a new rule, not a bug fix.Expected Outcome
Once #5134 ships persisted governor state, an operator's existing Alertmanager can immediately load this rule and get paged when the governor is approaching a rate-limit or budget ceiling, without needing to hand-author the rule themselves.
Links & Resources
prometheus/rules/alerts.yml