Skip to content

feat(miner-selfimprove): render prediction-calibration Prometheus metrics - #4461

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
jeffrey701:feat/miner-prediction-metrics
Jul 9, 2026
Merged

feat(miner-selfimprove): render prediction-calibration Prometheus metrics#4461
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
jeffrey701:feat/miner-prediction-metrics

Conversation

@jeffrey701

Copy link
Copy Markdown
Contributor

Closes #4264

What

A pure Prometheus text-exposition renderer for the miner's own predicted-gate accuracy — the miner-side counterpart to the server's src/selfhost/metrics.ts registry, scoped to calibration rather than the server's queue/webhook/AI-provider metrics.

Design decision (the issue's open question)

gittensory-miner is a local CLI, not a daemon, so this is an on-demand renderer a caller prints to stdout for its own scrape/cron setup — not an HTTP-served registry. It mirrors src/selfhost/metrics.ts's metric-naming (gittensory_miner_*_total) and # HELP/# TYPE/label-escaping conventions rather than importing across the package boundary. It stays a pure, side-effect-free engine function: the caller reads the prediction-ledger (prediction-ledger.js readPredictions, #4263) and passes the rows in — no data collection of its own lives here.

Counters

  • gittensory_miner_predictions_total{conclusion="..."} — predictions recorded, one series per predicted conclusion (e.g. merge/close/hold).
  • gittensory_miner_prediction_correct_total — predictions whose realized outcome matched the prediction.
  • gittensory_miner_prediction_incorrect_total — predictions whose realized outcome differed.

The correct/incorrect counters (the confusion-matrix-shaped buckets, mirroring computeGateEval) only move for rows carrying a resolved outcome; unresolved rows count toward predictions_total only, so the surface is meaningful before outcome-pairing exists and grows once it does. Output is deterministic (conclusion series sorted) and always emits HELP/TYPE, so it's well-formed even for an empty ledger.

API (packages/gittensory-engine/src/miner-prediction-metrics.ts)

  • renderMinerPredictionMetrics(rows) => string — Prometheus exposition text.
  • MinerPredictionMetricRow{ conclusion, correct? } (a ledger row joined with its optional resolved outcome).
  • MINER_PREDICTIONS_TOTAL / MINER_PREDICTION_CORRECT_TOTAL / MINER_PREDICTION_INCORRECT_TOTAL name constants.

The counters are documented in the module header.

Files

  • packages/gittensory-engine/src/miner-prediction-metrics.ts — the renderer.
  • packages/gittensory-engine/src/index.ts — barrel re-export.
  • test/unit/miner-prediction-metrics.test.ts — tests.

Testing

npx vitest run test/unit/miner-prediction-metrics.test.ts
npm run typecheck

4/4 tests pass (empty / unresolved / mixed-correctness / label-escaping fixtures); typecheck clean. New engine file at 100% line + branch coverage (24/24, 6/6, 4/4).

…rics

A pure Prometheus text-exposition renderer for the miner's own predicted-gate
accuracy — the miner-side counterpart to the server's src/selfhost/metrics.ts
registry, scoped to calibration rather than the server's queue/webhook metrics.

Design decision (the issue's open question — CLI vs HTTP, shared vs standalone):
gittensory-miner is a local CLI, not a daemon, so this is an on-demand RENDERER
a caller prints to stdout for its own scrape/cron setup, not an HTTP-served
registry. It mirrors src/selfhost/metrics.ts's metric-naming (gittensory_miner_*
_total) and HELP/TYPE/label-escaping conventions rather than importing across
the package boundary. It stays a pure, side-effect-free engine function: the
caller reads the prediction-ledger (packages/gittensory-miner/lib/prediction-
ledger.js `readPredictions`, JSONbored#4263) and passes the rows in — no data collection
of its own.

Counters:
- gittensory_miner_predictions_total{conclusion="..."} — predictions recorded,
  one series per predicted conclusion.
- gittensory_miner_prediction_correct_total / _incorrect_total — the confusion-
  matrix-shaped correct/incorrect counts, which only move for rows carrying a
  resolved outcome (unresolved rows count toward predictions_total only), so the
  surface is meaningful before outcome-pairing exists and grows once it does.

- packages/gittensory-engine/src/miner-prediction-metrics.ts: the renderer +
  metric-name constants + the MinerPredictionMetricRow input type; the counters
  are documented in the module header.
- packages/gittensory-engine/src/index.ts: barrel re-export.
- test/unit/miner-prediction-metrics.test.ts: valid-exposition assertions over
  empty / unresolved / mixed-correctness / label-escaping fixture states.

Closes JSONbored#4264
@jeffrey701
jeffrey701 requested a review from JSONbored as a code owner July 9, 2026 19:57
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.01%. Comparing base (10b2fe5) to head (4379953).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4461   +/-   ##
=======================================
  Coverage   94.01%   94.01%           
=======================================
  Files         418      419    +1     
  Lines       37410    37434   +24     
  Branches    13675    13678    +3     
=======================================
+ Hits        35171    35195   +24     
  Misses       1583     1583           
  Partials      656      656           
Files with missing lines Coverage Δ
.../gittensory-engine/src/miner-prediction-metrics.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier. labels Jul 9, 2026
@loopover-orb

loopover-orb Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-09 20:15:42 UTC

3 files · 1 AI reviewer · no blockers · readiness 82/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This adds a small, pure, side-effect-free Prometheus exposition renderer for miner prediction-calibration counters, mirroring the naming/HELP/TYPE/escaping conventions of src/selfhost/metrics.ts, plus a barrel re-export and a focused unit test suite. The counting logic, sort-by-conclusion determinism, and label/help-text escaping are all correct and match the stated design (unresolved rows count toward predictions_total only), and the diff matches the PR description exactly with no scope creep. Closes #4264 as required, involves no schema/migration surface, and CI is fully green.

Nits — 5 non-blocking
  • packages/gittensory-engine/src/miner-prediction-metrics.ts:29,34 — the doc comments pin exact line numbers in src/selfhost/metrics.ts (':204', ':193'); these will drift out of sync as that file changes, so reference the function/const name instead of a line number.
  • The renderer has no consumer yet (by design, per the PR's stated scoping decision) — confirm in the issue thread that a follow-up PR will actually wire a CLI command to call renderMinerPredictionMetrics and print it, so feat(miner-selfimprove): calibration metrics exporter — Prometheus-style counters for prediction accuracy #4264 doesn't end up with an unused export.
  • test/unit/miner-prediction-metrics.test.ts imports across the package boundary (../../packages/gittensory-engine/src/index) rather than being co-located inside packages/gittensory-engine — confirm this matches this repo's established test-placement convention for engine modules.
  • Consider naming the constant metric names via a shared list/tuple (similar to MINER_TELEMETRY_EVENT_TYPES) if more counters are expected to land soon, to keep the sorted-iteration pattern reusable.
  • If a CLI entry point consuming this renderer is imminent, link it in this PR's description (even as 'follow-up: #____') to make the issue-closure story clearer to reviewers.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #4264
Related work ⚠️ 2 scoped overlaps Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope ❌ 8/20 High review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 97 registered-repo PR(s), 40 merged, 4 issue(s).
Contributor context ✅ Confirmed Gittensor contributor jeffrey701; Gittensor profile; 97 PR(s), 4 issue(s).
Gate result ✅ Passing No configured blocker found.
Linked issue satisfaction

Partially addressed
The PR delivers a well-structured, pure Prometheus renderer with the correct metric names, HELP/TYPE/escaping conventions, and solid fixture tests (empty, unresolved, mixed correct/incorrect), fulfilling most of the issue's formatting and testing asks. However, the issue explicitly requires the exporter to read from the local prediction-ledger's storage as its data source (e.g. via a CLI command o

Review context
  • Author: jeffrey701
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 97 PR(s), 4 issue(s).
  • Related work: Titles/paths share 6 meaningful terms. (issue #4261, issue #4307)
  • Related work: Titles/paths share 8 meaningful terms. (issue #4261, issue #4308)
Contributor next steps
  • Review top overlaps.
  • Add a concise scope and risk note.
  • Check active issues and PRs before submitting.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gittensory approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit da9952f into JSONbored:main Jul 9, 2026
10 checks passed
@loopover-orb loopover-orb Bot added gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. and removed gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier. labels Jul 9, 2026
@JSONbored JSONbored added gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier. and removed gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(miner-selfimprove): calibration metrics exporter — Prometheus-style counters for prediction accuracy

2 participants