You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
prometheus/rules/alerts.yml already has a dead-letter-backlog rule with an established severity/for/runbook-annotation shape, but nothing alerts on portfolio-queue items that get claimed and then never completed or released. This is distinct from #4827 (the lease/timeout/reclaim CLI logic itself) and #4840 (alerting guidance docs) — neither ticket produces the concrete alert-rule definition.
Dependencies
Depends on #4827 (lease/timeout/reclaim) landing first. #4827 is what would expose a real, scrapeable metric for claimed-but-not-completed portfolio-queue item age/count — this alert rule has nothing to evaluate against until that metric exists. Do not start this issue until #4827 ships and documents the metric it exposes.
Requirements
Add a new rule group to prometheus/rules/alerts.yml for portfolio-queue items that are claimed but not completed/released past a threshold.
Mirror the existing dead-letter-backlog rule's severity/for/runbook-annotation shape and labeling conventions exactly.
The rule must load cleanly via promtool check rules (or the repo's equivalent CI rule-linting) even before an AMS scrape target is configured, so it ships as a dormant, valid rule an operator's existing Alertmanager can load whenever a scrape surface for AMS exists.
Deliverables / Acceptance Criteria
New rule group added to prometheus/rules/alerts.yml for stale/stuck claimed portfolio-queue items
severity/for/runbook annotations mirror the existing dead-letter-backlog rule's shape
This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. prometheus/rules/alerts.yml is config, not src/**, so Codecov's line/branch gate does not directly instrument it — but if the repo has a promtool test rules harness (or equivalent), this rule must ship with test cases for both sides of the threshold: (1) a unit/rule test where the claimed-item metric stays under the threshold for the full for: duration and no alert fires, (2) a rule test where the metric exceeds the threshold for the full for: duration and the alert fires with the expected labels/severity, and (3) an invariant test asserting this rule only evaluates AMS's portfolio-queue metric and never fires on an unrelated ORB queue metric with a similar name. No regression test applies since this is a new rule, not a bug fix.
Expected Outcome
Once #4827 ships a real scrape surface, an operator's existing Alertmanager can immediately load this rule and get paged when portfolio-queue items are claimed but stuck, without needing to hand-author the rule themselves.
Links & Resources
Target file: prometheus/rules/alerts.yml
Existing pattern to mirror: the dead-letter-backlog rule in the same file
Context
prometheus/rules/alerts.ymlalready has a dead-letter-backlog rule with an established severity/for/runbook-annotation shape, but nothing alerts on portfolio-queue items that get claimed and then never completed or released. This is distinct from #4827 (the lease/timeout/reclaim CLI logic itself) and #4840 (alerting guidance docs) — neither ticket produces the concrete alert-rule definition.Dependencies
Depends on #4827 (lease/timeout/reclaim) landing first. #4827 is what would expose a real, scrapeable metric for claimed-but-not-completed portfolio-queue item age/count — this alert rule has nothing to evaluate against until that metric exists. Do not start this issue until #4827 ships and documents the metric it exposes.
Requirements
prometheus/rules/alerts.ymlfor portfolio-queue items that are claimed but not completed/released past a threshold.alerts.yml.for:clause) consistent with existing similar rules in the same file.promtool check rules(or the repo's equivalent CI rule-linting) even before an AMS scrape target is configured, so it ships as a dormant, valid rule an operator's existing Alertmanager can load whenever a scrape surface for AMS exists.Deliverables / Acceptance Criteria
prometheus/rules/alerts.ymlfor stale/stuck claimed portfolio-queue itemsseverity/for/runbook annotations mirror the existing dead-letter-backlog rule's shapepromtool check rules(or equivalent CI rule-linting) passesTest Coverage Requirements
This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced.
prometheus/rules/alerts.ymlis config, notsrc/**, so Codecov's line/branch gate does not directly instrument it — but if the repo has apromtool test rulesharness (or equivalent), this rule must ship with test cases for both sides of the threshold: (1) a unit/rule test where the claimed-item metric stays under the threshold for the fullfor:duration and no alert fires, (2) a rule test where the metric exceeds the threshold for the fullfor:duration and the alert fires with the expected labels/severity, and (3) an invariant test asserting this rule only evaluates AMS's portfolio-queue metric and never fires on an unrelated ORB queue metric with a similar name. No regression test applies since this is a new rule, not a bug fix.Expected Outcome
Once #4827 ships a real scrape surface, an operator's existing Alertmanager can immediately load this rule and get paged when portfolio-queue items are claimed but stuck, without needing to hand-author the rule themselves.
Links & Resources
prometheus/rules/alerts.yml