Skip to content

feat(miner): add governor rate-limit/budget Prometheus metrics + pressure alerts - #5604

Merged
JSONbored merged 2 commits into
mainfrom
governor-rate-limit-budget-alert
Jul 13, 2026
Merged

feat(miner): add governor rate-limit/budget Prometheus metrics + pressure alerts#5604
JSONbored merged 2 commits into
mainfrom
governor-rate-limit-budget-alert

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Adds gittensory-miner governor metrics, a Prometheus text-exposition endpoint for the governor's persisted state (Persist governor cross-attempt state (rate-limit, budget, convergence, reputation, self-plagiarism) #5134): gittensory_miner_governor_rate_limit_remaining_ratio{scope,action_class,repo} (gauge, headroom remaining in the current write-rate-limit window as a fraction of the configured limit) and gittensory_miner_governor_cap_usage_ratio{dimension} (gauge, cumulative budget/turns/elapsed-time usage as a fraction of the configured ceiling).
  • Adds two alert rules in a new gittensory-miner-governor Prometheus rule group: GittensoryMinerGovernorRateLimitPressureHigh (remaining ratio < 10%, sustained 10m) and GittensoryMinerGovernorCapUsageHigh (cap usage ratio > 90%, sustained 10m).
  • Both gauges compose governor-state.js's already-persisted loadRateLimitState()/loadCapUsage() with the engine's existing pure calculators (evaluateLocalRateLimit, evaluateGovernorCaps) against the same default policies the production loop already falls back to when no .gittensory-ams.yml override is configured (DEFAULT_WRITE_RATE_LIMIT_POLICIES, DEFAULT_AMS_POLICY_SPEC.capLimits) — no invented thresholds. Strictly read-only: it does not gate, retry, or modify governor decision logic (governor-chokepoint.js/governor-chokepoint-persisted.js are untouched).
  • Mirrors the queue metrics/portfolio-queue-alerts pattern shipped for Add a Prometheus alert rule for stuck/stale portfolio-queue claimed items #5186 (merged in feat(miner): add portfolio-queue Prometheus metrics + stuck/backlog alerts #5603): same escaping helpers, deterministic sorted output, always-emit HELP/TYPE, and a fixture test that parses the real prometheus/rules/alerts.yml and pins expr/for/labels/annotations to the renderer's exported metric-name constants.

Scope note

#5187 depends on #5134 (governor cross-attempt state persistence, already shipped) and says to reference "the metric/state exposed by #5134" — but #5134 only persisted state to SQLite, it never shipped a Prometheus renderer for it (the only existing governor CLI surface was governor list/pause/resume/status). This PR adds the renderer as a prerequisite, following the same local-package convention already established by queue metrics/ledger metrics.

capLimits isn't itself persisted per-repo by governor-state.js (capUsage is a single global scalar row, not indexed by repo), so the cap-usage ratio uses the fleet-wide DEFAULT_AMS_POLICY_SPEC.capLimits default — the same approximation loop-cli.js itself already makes for any repo without its own .gittensory-ams.yml override.

Closes #5187

Test plan

  • npm run typecheck
  • npm run build:miner (added the new file to the package's node --check list)
  • npm run selfhost:validate-observability, npm run docs:drift-check, npm run command-reference:check, npm run miner:env-reference:check, npm run test:miner-pack
  • New fixture test (test/unit/alerts-miner-governor-rate-limit-budget-pressure.test.ts) parses the real alerts.yml and pins both rules' expr/for/severity/annotations to the renderer's exported metric-name constants, plus asserts neither rule's runbook instructs a governor decision-logic change
  • New unit tests for renderGovernorMetrics/runGovernorMetrics cover empty state, global + per-repo bucket ratios, an actionClass with no configured policy (skipped), a malformed per-repo key with no colon separator, deterministic sort order, and cap-usage ratios across all three dimensions

…sure alerts

Adds `gittensory-miner governor metrics` (rate-limit-remaining and cap-usage
gauges) and two matching Prometheus alerts, mirroring the queue-metrics
pattern shipped for #5186.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 13, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui 6df377b Commit Preview URL

Branch Preview URL
Jul 13 2026, 09:13 AM

@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.01%. Comparing base (7e55ebc) to head (6df377b).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5604   +/-   ##
=======================================
  Coverage   95.00%   95.01%           
=======================================
  Files         573      574    +1     
  Lines       45629    45684   +55     
  Branches    14661    14661           
=======================================
+ Hits        43350    43405   +55     
  Misses       1528     1528           
  Partials      751      751           
Flag Coverage Δ
shard-1 43.95% <7.27%> (-0.10%) ⬇️
shard-2 35.78% <0.00%> (+0.08%) ⬆️
shard-3 32.06% <0.00%> (-0.13%) ⬇️
shard-4 30.99% <100.00%> (-1.69%) ⬇️
shard-5 32.65% <0.00%> (+0.85%) ⬆️
shard-6 44.46% <5.45%> (+0.33%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/gittensory-miner/lib/cli.js 100.00% <ø> (ø)
...ckages/gittensory-miner/lib/governor-ledger-cli.js 98.59% <100.00%> (+0.02%) ⬆️
...kages/gittensory-miner/lib/governor-metrics-cli.js 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 13, 2026
@loopover-orb

loopover-orb Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-13 08:59:48 UTC

9 files · 2 AI reviewers · 1 blocker · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR adds a Prometheus renderer (`governor metrics` CLI subcommand) and two alert rules exposing the governor's persisted rate-limit/cap-usage state, closely mirroring the existing `queue metrics`/`ledger metrics` patterns. It composes existing pure calculators (`evaluateLocalRateLimit`, `evaluateGovernorCaps`) against the production defaults and is genuinely read-only — it doesn't touch governor decision logic. The fixture test pins alert `expr`/`for`/labels to the renderer's exported metric-name constants, and the malformed-key/unknown-actionClass edge cases are exercised.

Blockers

  • packages/gittensory-miner/lib/governor-metrics-cli.js:60 computes `remainingRatio()` from `evaluateLocalRateLimit(...).remaining`, but that value is the remaining allowance after a hypothetical next write, so an active per-repo bucket with limit 3 and count 2 exports `0` even though one write remains; change the renderer to compute current-window headroom from the bucket/count/window state directly or add an engine helper that reports pre-write remaining headroom.
Nits — 6 non-blocking
  • The 'debug leftover' flagged by the brief at governor-metrics-cli.js:158 is `console.log(...)`, which is the intended CLI output mechanism here (mirrors `runGovernorMetrics`'s sibling commands), not a stray debug statement — safe to ignore.
  • governor-metrics-cli.js:78-93 nests four levels deep (for/for/if/push) purely from two near-identical global/perRepo loops; consider extracting a shared `collectBucketRows(entries, policyMap, scope, nowMs)` helper to cut duplication and depth, per the brief's nesting flag.
  • prometheus/rules/alerts.yml is now ~743 lines; not this PR's fault alone, but worth a follow-up to split per-scrape-target files (server vs miner) as the miner section keeps growing.
  • The scope note says Add a Prometheus alert rule for governor rate-limit/budget threshold pressure #5187 only says 'reference the metric/state exposed by Persist governor cross-attempt state (rate-limit, budget, convergence, reputation, self-plagiarism) #5134' and this PR fills the gap by adding the renderer — worth double-checking with a maintainer that inventing the renderer surface (rather than just the alert rule Add a Prometheus alert rule for governor rate-limit/budget threshold pressure #5187 literally asks for) is in scope, even though the reasoning given is sound.
  • Consider factoring `collectRateLimitRows`'s global/perRepo loops (governor-metrics-cli.js:59-93) into one helper parameterized by scope to reduce duplication and nesting.
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Concerns raised — review before merging

  • packages/gittensory-miner/lib/governor-metrics-cli.js:60 computes `remainingRatio()` from `evaluateLocalRateLimit(...).remaining`, but that value is the remaining allowance after a hypothetical next write, so an active per-repo bucket with limit 3 and count 2 exports `0` even though one write remains; change the renderer to compute current-window headroom from the bucket/count/window state directly or add an engine helper that reports pre-write remaining headroom.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. packages/gittensory-miner/lib/governor-metrics-cli.js:60 computes \`remainingRatio\(\)\` from \`evaluateLocalRateLimit\(...\).remaining\`, but that value is the remaining allowance after a hypothetical next write, so an active per-repo bucket with limit 3 and count 2 exports \`0\` even though one write remains; change the renderer to compute current-window headroom from the bucket/count/window state directly or add an engine helper that reports pre-write remaining headroom.
Signal Result Evidence
Code review ❌ 1 blocker 2 reviewers, synthesized
Linked issue ✅ Linked #5187
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 44 registered-repo PR(s), 36 merged, 347 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 44 PR(s), 347 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence. LLM value judgment: moderate — The PR delivers a working, well-tested, read-only observability surface (metrics renderer + two alert rules) that directly extends #5187's scope by filling the gap left by #5134, following an established pattern exactly.
Linked issue satisfaction

Partially addressed
The PR adds the governor rate-limit/cap-usage alert group mirroring the existing severity/for/runbook pattern with configurable thresholds and test coverage, and appears read-only with respect to governor decision logic, but the thresholds (0.1 / 0.9) are hardcoded in the expr rather than templated/configurable as the issue explicitly requires, and it introduces a brand-new metric/renderer rather

Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 44 PR(s), 347 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

…adjusted remaining

evaluateLocalRateLimit's `remaining` field already accounts for one
hypothetical next write (remaining = limit - effectiveCount - 1), so a
bucket at count=2/limit=3 (one write still allowed) rendered the same
0 ratio as a fully exhausted count=3/limit=3 bucket. Recover true
headroom as `allowed ? remaining + 1 : 0` before dividing by limit.
@JSONbored

Copy link
Copy Markdown
Owner Author

Fixed in 6df377b. remaining already accounts for one hypothetical next write (limit - effectiveCount - 1), so count=2/limit=3 rendered the same 0 as a fully-exhausted count=3/limit=3 bucket. Now computed as allowed ? remaining + 1 : 0 before dividing by limit — verified live: count=2/limit=3 now renders 0.333..., count=3/limit=3 still renders 0. Added a test distinguishing the two cases plus one exercising the newly-introduced allowed branch; coverage confirmed 100% line/branch/function on the file.

@JSONbored
JSONbored merged commit 29e460e into main Jul 13, 2026
14 of 15 checks passed
@JSONbored
JSONbored deleted the governor-rate-limit-budget-alert branch July 13, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a Prometheus alert rule for governor rate-limit/budget threshold pressure

1 participant