feat(miner): add portfolio-queue Prometheus metrics + stuck/backlog alerts - #5603
Merged
Merged
Conversation
…lerts Neither #4827 (lease/timeout/reclaim CLI logic) nor #4840 (alerting guidance docs) actually produces a scrapeable metric for claimed-but-not- completed portfolio-queue items -- this issue's own stated dependency was on the metric existing, but no portfolio-queue metrics renderer existed anywhere in the codebase yet. Adds one, mirroring event-ledger-cli.js's renderEventLedgerMetrics / the engine's renderMinerPredictionMetrics conventions (HELP/TYPE, deterministic sorted series, well-formed on an empty queue): - gittensory_miner_portfolio_queue_items{status} -- current count per status (queued/in_progress/done). - gittensory_miner_portfolio_queue_oldest_in_progress_lease_age_seconds -- age of the oldest still-in-flight claim lease, 0 when nothing is in-flight. The concrete "is anything stuck" signal. Exposed via a new `gittensory-miner queue metrics` subcommand (mirrors `gittensory-miner metrics` / `gittensory-miner ledger metrics`'s existing pattern -- local + offline, no network). Adds two Prometheus alert rules in the same dormant-until-a-scrape-target- exists posture as the existing gittensory-miner-prediction group (#5188): absent gittensory_miner_portfolio_queue_* series yield no result, so they load cleanly via selfhost:validate-observability and never false-fire on an install with no miner scrape configured. - GittensoryMinerPortfolioQueueItemStuck: oldest in-flight lease age past 1h (a generous multiple of the 30m self-heal default), sustained 15m -- the self-heal isn't running or is itself stuck. - GittensoryMinerPortfolioQueueBacklogHigh: queued backlog past 200, sustained 30m -- claiming is falling behind discovery. A dedicated fixture test (alerts-miner-portfolio-queue-backlog.test.ts, mirroring alerts-miner-prediction-calibration-drift.test.ts) pins each rule's formula/threshold/metric-names to the real renderer's exported constants, so a future rename can't silently drift the alert away from what the renderer actually produces. Closes #5186.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 7a07f93 | Commit Preview URL Branch Preview URL |
Jul 13 2026, 08:38 AM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5603 +/- ##
=======================================
Coverage 95.00% 95.00%
=======================================
Files 573 573
Lines 45600 45629 +29
Branches 14661 14661
=======================================
+ Hits 43321 43350 +29
Misses 1528 1528
Partials 751 751
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
github-project-automation
Bot
moved this to Todo
in Autonomous Miner System (Loopover AMS)
loopover-orb
Bot
added
the
gittensor:feature
github-project-automation
Bot
moved this from Todo
to Done
in Autonomous Miner System (Loopover AMS)
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gittensory-miner queue metrics, a Prometheus text-exposition endpoint for the local portfolio-queue:gittensory_miner_portfolio_queue_items{status=...}(gauge, one series per queue status) andgittensory_miner_portfolio_queue_oldest_in_progress_lease_age_seconds(gauge, age of the oldest still-claimed lease, 0 when nothing is in-flight).gittensory-miner-portfolio-queuePrometheus rule group:GittensoryMinerPortfolioQueueItemStuck(oldest in-progress lease age > 1h, sustained 15m) andGittensoryMinerPortfolioQueueBacklogHigh(queued-item count > 200, sustained 30m).gittensory-miner-predictionmetrics/alerts pattern already shipped for calibration drift: same escaping helpers, deterministic sorted output, always-emit HELP/TYPE, pure renderer (caller suppliesnowMs), and a fixture test that parses the realprometheus/rules/alerts.ymland pinsexpr/for/labels/annotationsto the renderer's exported metric-name constants (the config-side equivalent ofpromtool test rules).Scope note
#5186 lists #4827 and #5134 as dependencies and describes this as "just add an alert rule." Neither of those issues actually produced a portfolio-queue Prometheus renderer or a scrapeable metric — the only existing miner-side renderer was
renderMinerPredictionMetricsfor a different subsystem. An alert rule with no backing metric can't be validated or fire, so this PR includes the renderer +queue metricsCLI command as a prerequisite, following the same local-package-only conventionevent-ledger-cli.js'srenderEventLedgerMetricsalready established (avoids touching the separately-published engine package for a miner-only metric).Closes #5186
Test plan
npm run test:ci(unit + typecheck)npm run build:minernpm run selfhost:validate-observability(alerts.yml bracket/operator sanity check)test/unit/alerts-miner-portfolio-queue-backlog.test.ts) parses the realalerts.ymland pins both rules'expr/for/severity/annotations to the renderer's exported metric-name constantsrenderPortfolioQueueMetrics/runQueueMetricscover empty queue, multiple statuses, oldest-lease selection (including the "younger lease must not replace the running max" branch), and malformedleasedAttimestamps