Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions packages/gittensory-miner/docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,37 @@ directory. To visualize AMS activity, add a dashboard JSON there — or import o
(**Dashboards → Import**) — and point its panels at the `AMS Attempt Log` / `AMS Prediction Ledger` datasources
above. Panels should query only the redacted reporting schema (e.g. `SELECT * FROM attempt_log_events`), never a
`payload_json`/`reason` column — the exporter drops both, so a panel referencing them returns no such column.

## Prometheus metrics (opt-in)

The two sections above cover Grafana-via-SQLite (historical attempt/prediction rows). Separately, the miner CLI
exposes four Prometheus text-exposition documents for live counters/gauges — a completely independent,
**off-by-default** mechanism: nothing in the miner package runs these on a schedule or opens a port itself.

| Command | Metric family |
| ------------------------------------ | ------------------------------------------------------------------------------------ |
| `gittensory-miner metrics` | Prediction-calibration counters (`gittensory_miner_prediction*_total`) |
| `gittensory-miner queue metrics` | Portfolio-queue backlog + lease-age gauges (`gittensory_miner_portfolio_queue*`) |
| `gittensory-miner ledger metrics` | Event-ledger counters (`gittensory_miner_events_total`) |
| `gittensory-miner governor metrics` | Write-rate-limit + cap-usage pressure gauges (`gittensory_miner_governor*`) |

None of these is a long-running HTTP server — Prometheus can't scrape a one-shot CLI command directly. To opt in,
wire [`scripts/export-miner-prometheus-textfile.sh`](../../../scripts/export-miner-prometheus-textfile.sh) into
your own cron/systemd timer alongside [node_exporter's textfile
collector](https://github.com/prometheus/node_exporter#textfile-collector): the script runs all four commands and
atomically writes their concatenated output to `$GITTENSORY_MINER_PROMETHEUS_TEXTFILE` (default
`/var/lib/node_exporter/textfile_collector/gittensory_miner.prom`), the standard directory node_exporter's
textfile collector watches. Point `GITTENSORY_MINER_BIN` at the miner binary if it isn't on `PATH`.

A broken/corrupt local store for one family (e.g. the portfolio queue) never blocks the other three — that
family's metrics are simply omitted from the file for that run (logged to stderr), not the whole export.

```sh
# crontab -e
*/5 * * * * GITTENSORY_MINER_CONFIG_DIR=/data/miner sh /path/to/gittensory/scripts/export-miner-prometheus-textfile.sh
```

Then point your own `prometheus.yml` at node_exporter as usual — no changes to this repo's `prometheus/` config
are needed. See [`prometheus/rules/alerts.yml`](../../../prometheus/rules/alerts.yml)'s
`gittensory-miner-prediction` / `gittensory-miner-portfolio-queue` / `gittensory-miner-governor` rule groups for
alert rules that already target these exact metric names.
43 changes: 43 additions & 0 deletions scripts/export-miner-prometheus-textfile.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#!/bin/sh
set -eu

# Miner Prometheus textfile export (#4839): the miner CLI already emits four Prometheus text-exposition
# documents -- `gittensory-miner metrics` (prediction calibration), `queue metrics` (portfolio-queue),
# `ledger metrics` (event ledger), and `governor metrics` (rate-limit/cap-usage pressure) -- but none of them
# is a long-running HTTP server Prometheus can scrape directly; each is a one-shot CLI command. This script
# bridges the two with the standard node_exporter "textfile collector" pattern: run all four, concatenate their
# output, and atomically write the result to a .prom file node_exporter's own textfile collector picks up on
# its next scrape.
#
# Entirely OPT-IN, per this issue's own boundary: nothing in the miner package invokes this script itself -- a
# self-hoster wires it into their own cron/systemd timer alongside node_exporter (see docs/observability.md).
# AMS's zero-infra "laptop mode" is completely unaffected if this script is never run.
#
# Fail-open per metric family, mirroring export-ams-reporting-db.sh's philosophy: a broken/corrupt local store
# for ONE subsystem (e.g. the portfolio queue) must not take down the other three families' metrics. A failing
# family is skipped (its own stderr flows through unredirected, e.g. into the cron/systemd journal) rather than
# aborting the whole export -- Prometheus treats an absent series as "no data", not an error, so a partial
# export is strictly better than a stale-forever or entirely-missing one.

MINER_BIN="${GITTENSORY_MINER_BIN:-gittensory-miner}"
OUT_FILE="${GITTENSORY_MINER_PROMETHEUS_TEXTFILE:-/var/lib/node_exporter/textfile_collector/gittensory_miner.prom}"
TMP_FILE="${OUT_FILE}.tmp"

mkdir -p "$(dirname "$OUT_FILE")"
: >"$TMP_FILE"

export_family() {
label="$1"
shift
if ! "$MINER_BIN" "$@" >>"$TMP_FILE"; then
echo "[miner-prometheus-textfile:$label] export failed, this family's metrics are omitted from $OUT_FILE" >&2
fi
}

export_family "prediction-calibration" metrics
export_family "portfolio-queue" queue metrics
export_family "event-ledger" ledger metrics
export_family "governor" governor metrics

mv "$TMP_FILE" "$OUT_FILE"
echo "[miner-prometheus-textfile] wrote $OUT_FILE"