Skip to content

Add a Prometheus alert rule for stuck/stale portfolio-queue claimed items #5186

Description

@JSONbored

Context

prometheus/rules/alerts.yml already has a dead-letter-backlog rule with an established severity/for/runbook-annotation shape, but nothing alerts on portfolio-queue items that get claimed and then never completed or released. This is distinct from #4827 (the lease/timeout/reclaim CLI logic itself) and #4840 (alerting guidance docs) — neither ticket produces the concrete alert-rule definition.

Dependencies

Depends on #4827 (lease/timeout/reclaim) landing first. #4827 is what would expose a real, scrapeable metric for claimed-but-not-completed portfolio-queue item age/count — this alert rule has nothing to evaluate against until that metric exists. Do not start this issue until #4827 ships and documents the metric it exposes.

Requirements

  1. Add a new rule group to prometheus/rules/alerts.yml for portfolio-queue items that are claimed but not completed/released past a threshold.
  2. Mirror the existing dead-letter-backlog rule's severity/for/runbook-annotation shape and labeling conventions exactly.
  3. Reference the metric exposed by Add lease/timeout/reclaim to portfolio-queue's stuck items #4827's scrape surface once it lands — do not invent a metric name; use whatever Add lease/timeout/reclaim to portfolio-queue's stuck items #4827 documents/exposes, following the metric-naming convention already established elsewhere in alerts.yml.
  4. Include a configurable threshold duration (for: clause) consistent with existing similar rules in the same file.
  5. Include a runbook annotation link/text following the existing convention.
  6. Must not touch the lease/reclaim implementation itself (Add lease/timeout/reclaim to portfolio-queue's stuck items #4827's CLI logic) — this issue is alert-rule YAML only.
  7. The rule must load cleanly via promtool check rules (or the repo's equivalent CI rule-linting) even before an AMS scrape target is configured, so it ships as a dormant, valid rule an operator's existing Alertmanager can load whenever a scrape surface for AMS exists.

Deliverables / Acceptance Criteria

Test Coverage Requirements

This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. prometheus/rules/alerts.yml is config, not src/**, so Codecov's line/branch gate does not directly instrument it — but if the repo has a promtool test rules harness (or equivalent), this rule must ship with test cases for both sides of the threshold: (1) a unit/rule test where the claimed-item metric stays under the threshold for the full for: duration and no alert fires, (2) a rule test where the metric exceeds the threshold for the full for: duration and the alert fires with the expected labels/severity, and (3) an invariant test asserting this rule only evaluates AMS's portfolio-queue metric and never fires on an unrelated ORB queue metric with a similar name. No regression test applies since this is a new rule, not a bug fix.

Expected Outcome

Once #4827 ships a real scrape surface, an operator's existing Alertmanager can immediately load this rule and get paged when portfolio-queue items are claimed but stuck, without needing to hand-author the rule themselves.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions