From 5ab5a4edfe3b9313f544d69ae2c3b8eb64ac778f Mon Sep 17 00:00:00 2001 From: JSONbored <49853598+JSONbored@users.noreply.github.com> Date: Mon, 13 Jul 2026 02:06:47 -0700 Subject: [PATCH] docs(miner): document opt-in Prometheus textfile-collector export Adds scripts/export-miner-prometheus-textfile.sh, wiring the miner's four existing `X metrics` CLI commands into node_exporter's textfile collector pattern (fail-open per family), and documents it in packages/gittensory-miner/docs/observability.md. --- .../gittensory-miner/docs/observability.md | 34 +++++++++++++++ scripts/export-miner-prometheus-textfile.sh | 43 +++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 scripts/export-miner-prometheus-textfile.sh diff --git a/packages/gittensory-miner/docs/observability.md b/packages/gittensory-miner/docs/observability.md index ad23a1f64d..e641105573 100644 --- a/packages/gittensory-miner/docs/observability.md +++ b/packages/gittensory-miner/docs/observability.md @@ -58,3 +58,37 @@ directory. To visualize AMS activity, add a dashboard JSON there — or import o (**Dashboards → Import**) — and point its panels at the `AMS Attempt Log` / `AMS Prediction Ledger` datasources above. Panels should query only the redacted reporting schema (e.g. `SELECT * FROM attempt_log_events`), never a `payload_json`/`reason` column — the exporter drops both, so a panel referencing them returns no such column. + +## Prometheus metrics (opt-in) + +The two sections above cover Grafana-via-SQLite (historical attempt/prediction rows). Separately, the miner CLI +exposes four Prometheus text-exposition documents for live counters/gauges — a completely independent, +**off-by-default** mechanism: nothing in the miner package runs these on a schedule or opens a port itself. + +| Command | Metric family | +| ------------------------------------ | ------------------------------------------------------------------------------------ | +| `gittensory-miner metrics` | Prediction-calibration counters (`gittensory_miner_prediction*_total`) | +| `gittensory-miner queue metrics` | Portfolio-queue backlog + lease-age gauges (`gittensory_miner_portfolio_queue*`) | +| `gittensory-miner ledger metrics` | Event-ledger counters (`gittensory_miner_events_total`) | +| `gittensory-miner governor metrics` | Write-rate-limit + cap-usage pressure gauges (`gittensory_miner_governor*`) | + +None of these is a long-running HTTP server — Prometheus can't scrape a one-shot CLI command directly. To opt in, +wire [`scripts/export-miner-prometheus-textfile.sh`](../../../scripts/export-miner-prometheus-textfile.sh) into +your own cron/systemd timer alongside [node_exporter's textfile +collector](https://github.com/prometheus/node_exporter#textfile-collector): the script runs all four commands and +atomically writes their concatenated output to `$GITTENSORY_MINER_PROMETHEUS_TEXTFILE` (default +`/var/lib/node_exporter/textfile_collector/gittensory_miner.prom`), the standard directory node_exporter's +textfile collector watches. Point `GITTENSORY_MINER_BIN` at the miner binary if it isn't on `PATH`. + +A broken/corrupt local store for one family (e.g. the portfolio queue) never blocks the other three — that +family's metrics are simply omitted from the file for that run (logged to stderr), not the whole export. + +```sh +# crontab -e +*/5 * * * * GITTENSORY_MINER_CONFIG_DIR=/data/miner sh /path/to/gittensory/scripts/export-miner-prometheus-textfile.sh +``` + +Then point your own `prometheus.yml` at node_exporter as usual — no changes to this repo's `prometheus/` config +are needed. See [`prometheus/rules/alerts.yml`](../../../prometheus/rules/alerts.yml)'s +`gittensory-miner-prediction` / `gittensory-miner-portfolio-queue` / `gittensory-miner-governor` rule groups for +alert rules that already target these exact metric names. diff --git a/scripts/export-miner-prometheus-textfile.sh b/scripts/export-miner-prometheus-textfile.sh new file mode 100644 index 0000000000..28122145ea --- /dev/null +++ b/scripts/export-miner-prometheus-textfile.sh @@ -0,0 +1,43 @@ +#!/bin/sh +set -eu + +# Miner Prometheus textfile export (#4839): the miner CLI already emits four Prometheus text-exposition +# documents -- `gittensory-miner metrics` (prediction calibration), `queue metrics` (portfolio-queue), +# `ledger metrics` (event ledger), and `governor metrics` (rate-limit/cap-usage pressure) -- but none of them +# is a long-running HTTP server Prometheus can scrape directly; each is a one-shot CLI command. This script +# bridges the two with the standard node_exporter "textfile collector" pattern: run all four, concatenate their +# output, and atomically write the result to a .prom file node_exporter's own textfile collector picks up on +# its next scrape. +# +# Entirely OPT-IN, per this issue's own boundary: nothing in the miner package invokes this script itself -- a +# self-hoster wires it into their own cron/systemd timer alongside node_exporter (see docs/observability.md). +# AMS's zero-infra "laptop mode" is completely unaffected if this script is never run. +# +# Fail-open per metric family, mirroring export-ams-reporting-db.sh's philosophy: a broken/corrupt local store +# for ONE subsystem (e.g. the portfolio queue) must not take down the other three families' metrics. A failing +# family is skipped (its own stderr flows through unredirected, e.g. into the cron/systemd journal) rather than +# aborting the whole export -- Prometheus treats an absent series as "no data", not an error, so a partial +# export is strictly better than a stale-forever or entirely-missing one. + +MINER_BIN="${GITTENSORY_MINER_BIN:-gittensory-miner}" +OUT_FILE="${GITTENSORY_MINER_PROMETHEUS_TEXTFILE:-/var/lib/node_exporter/textfile_collector/gittensory_miner.prom}" +TMP_FILE="${OUT_FILE}.tmp" + +mkdir -p "$(dirname "$OUT_FILE")" +: >"$TMP_FILE" + +export_family() { + label="$1" + shift + if ! "$MINER_BIN" "$@" >>"$TMP_FILE"; then + echo "[miner-prometheus-textfile:$label] export failed, this family's metrics are omitted from $OUT_FILE" >&2 + fi +} + +export_family "prediction-calibration" metrics +export_family "portfolio-queue" queue metrics +export_family "event-ledger" ledger metrics +export_family "governor" governor metrics + +mv "$TMP_FILE" "$OUT_FILE" +echo "[miner-prometheus-textfile] wrote $OUT_FILE"