Skip to content

feat(miner-selfimprove): calibration metrics exporter — Prometheus-style counters for prediction accuracy #4264

Description

@JSONbored

src/selfhost/metrics.ts is this repo's canonical in-process Prometheus text-format registry (used by the self-host observability stack — Grafana dashboards scrape it): counters (incr(name, labels?, by?), monotonic), gauges sampled at scrape time (gauge/gaugeVector), histograms (observe), a DEFAULT_METRIC_META table registering # HELP/# TYPE lines per metric name (src/selfhost/metrics.ts:36-141), and a renderMetrics() that emits the full exposition-format text (src/selfhost/metrics.ts:274-315). Metric names follow a gittensory_<area>_<unit>_total/_seconds convention throughout.

This item is the miner-side counterpart, scoped to calibration/prediction accuracy rather than the server's queue/webhook/AI-provider metrics. It's naturally downstream of the local prediction-ledger item in this same batch — the exporter's job is to turn ledger rows into counters: predictions made (by conclusion: merge/close/hold), and — once a prediction can be paired with a realized outcome — correct vs incorrect counts per the same confusion-matrix shape computeGateEval already uses server-side (src/review/parity.ts:33-58: mergeConfirmed/mergeFalse/closeConfirmed/closeFalse).

One open design question this item should resolve rather than this issue prescribing it: the self-host metrics.ts pattern is built for a long-running server process scraped over HTTP (GET /metrics); gittensory-miner is a local CLI, not (currently) a daemon. The exporter likely needs to either (a) render Prometheus text format on demand via a new CLI command (e.g. gittensory-miner metrics printing to stdout, for an operator's own scrape/cron setup) rather than serving it over HTTP itself, or (b) reuse src/selfhost/metrics.ts's registry primitives directly if that's reachable from packages/ without a layering violation — it may not be, in which case a small standalone counter/render implementation mirroring the same conventions is the right call, not a shared import. Either is acceptable; keep the metric naming convention (gittensory_miner_*_total, HELP/TYPE lines) consistent with the existing self-host metrics either way.

Deliverables

  • A counters module (CLI-rendered or HTTP-served per the design question above) exposing at minimum: gittensory_miner_predictions_total (labeled by conclusion), and once the prediction ledger can be paired with outcomes, gittensory_miner_prediction_correct_total/gittensory_miner_prediction_incorrect_total (or an equivalent confusion-matrix-shaped counter set mirroring computeGateEval's buckets).
  • Prometheus text-exposition-format output (# HELP/# TYPE lines, same convention as src/selfhost/metrics.ts:203-219's pushMetricMeta/escapeHelpText) — reuse the same escaping/formatting rules rather than reinventing them, even if the underlying registry is a separate small implementation.
  • Reads from the local prediction-ledger item's storage (this same batch) as its data source — no new data collection of its own.
  • Tests asserting the rendered text format is valid Prometheus exposition syntax for a handful of fixture ledger states (empty, some predictions with no resolved outcome yet, a mix of correct/incorrect).
  • Document (in the module or a short README section) which counters exist and what each measures, since this is a new metric surface a future dashboard will depend on.

References

  • src/selfhost/metrics.ts:1-35 (registry primitives: counters/gauges/gaugeVector/histograms)
  • src/selfhost/metrics.ts:36-141 (DEFAULT_METRIC_META, naming convention)
  • src/selfhost/metrics.ts:203-219 (pushMetricMeta/escapeHelpText, HELP/TYPE formatting to mirror)
  • src/selfhost/metrics.ts:274-315 (renderMetrics, exposition-format render)
  • src/review/parity.ts:33-58 (GateEvalRow — the confusion-matrix bucket shape to mirror for correct/incorrect counters)
  • The local prediction-ledger item in this batch (packages/gittensory-miner/lib/prediction-ledger.js, once filed/built — this exporter's data source)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.gittensor:priorityMaintainer-selected Gittensor priority — scores a 1.5x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions