Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions alertmanager/alertmanager.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Alertmanager configuration for the gittensory self-host stack (#980 observability).
# Alertmanager configuration for the loopover self-host stack (#980 observability).
# Schema: Alertmanager v0.27+ (UTF-8 matchers; `matchers:` as a list of strings).
# Mounted at /etc/alertmanager/alertmanager.yml in the alertmanager container.
#
Expand All @@ -8,7 +8,7 @@
# means `docker compose --profile observability up -d` always comes up green.
#
# THIS IS A REQUIRED STEP for any deployment you expect to run unattended, not an
# optional one — see https://gittensory.aethereal.dev/docs/self-hosting-operations
# optional one — see https://loopover.ai/docs/self-hosting-operations
# (apps/loopover-ui/src/routes/docs.self-hosting-operations.tsx), "Alerting — required
# for a 24/7 deployment" section, for the fastest verified path (a Discord webhook via
# webhook_url_file, uncomment the block below) and for the Grafana "Alerts" row you can
Expand Down
2 changes: 1 addition & 1 deletion grafana/dashboards/ai-usage.json
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@
"url": "/d/loopover-miner-usage/loopover-miner-usage",
"icon": "external link",
"targetBlank": false,
"tooltip": "AMS's own coding-agent usage dashboard -- a separate local CLI (gittensory-miner) from every provider above, tracked on its own dashboard since it's a structurally different local-SQLite source, not another row in ai_usage_events."
"tooltip": "AMS's own coding-agent usage dashboard -- a separate local CLI (loopover-miner) from every provider above, tracked on its own dashboard since it's a structurally different local-SQLite source, not another row in ai_usage_events."
}
],
"panels": [
Expand Down
2 changes: 1 addition & 1 deletion grafana/dashboards/miner-usage.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"from": "now-7d",
"to": "now"
},
"description": "Per-provider coding-agent (claude-cli/codex-cli/agent-sdk) attempt outcomes, cost, and token usage for the Autonomous Miner System (AMS) -- a separate local CLI (gittensory-miner) from ORB, this dashboard's own AI usage. Reads the redacted attempt_outcome_summary rows exported from AMS's local attempt-log ledger (see packages/loopover-miner/docs/observability.md) via the AMS Attempt Log SQLite datasource -- never the miner's live LOOPOVER_MINER_CONFIG_DIR ledgers, and never scraped from a Prometheus metrics path. All three coding-agent providers share one dashboard (a $provider template variable, not one dashboard per provider) since AMS's local ledger has far lower data volume than ORB's cloud reviewers. tokens_used is honestly 0/NULL until some future coding-agent driver reports real token usage (#5395) -- never fabricated.",
"description": "Per-provider coding-agent (claude-cli/codex-cli/agent-sdk) attempt outcomes, cost, and token usage for the Autonomous Miner System (AMS) -- a separate local CLI (loopover-miner) from ORB, this dashboard's own AI usage. Reads the redacted attempt_outcome_summary rows exported from AMS's local attempt-log ledger (see packages/loopover-miner/docs/observability.md) via the AMS Attempt Log SQLite datasource -- never the miner's live LOOPOVER_MINER_CONFIG_DIR ledgers, and never scraped from a Prometheus metrics path. All three coding-agent providers share one dashboard (a $provider template variable, not one dashboard per provider) since AMS's local ledger has far lower data volume than ORB's cloud reviewers. tokens_used is honestly 0/NULL until some future coding-agent driver reports real token usage (#5395) -- never fabricated.",
"annotations": {
"list": []
},
Expand Down
4 changes: 2 additions & 2 deletions grafana/dashboards/resource-hub.json
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
"gridPos": { "h": 13, "w": 12, "x": 12, "y": 0 },
"options": {
"mode": "markdown",
"content": "## 📊 Dashboards\n- **[Upstream PRs & issues (GitHub)](/d/loopover-github)** — live, accurate census + open-PR triage (GitHub API).\n- **[Reviews & PRs (maintainer)](/d/loopover-maintainer)** — loopover's own review activity + reviewed-PR log.\n- **[AI usage](/d/loopover-ai-usage)** — durable cross-provider ai_usage_events (filterable by provider/feature/model), live Prometheus counters, and Claude Code's own OTEL session telemetry, all in one place.\n- **[LoopOver (infra)](/d/loopover-selfhost)** — queue, jobs, HTTP, GitHub API cache/rate limits.\n- **[GPU metrics](/d/loopover-gpu)** — utilization/VRAM for a self-hosted Ollama GPU box.\n- **[Infra health](/d/loopover-infra-health)** — host CPU/mem/disk/network (node-exporter), per-container resource usage (cAdvisor), Redis, Qdrant, and whether the observability stack itself is up.\n- **[REES (review-enrichment)](/d/loopover-rees)** — request outcomes/latency and per-analyzer run/timeout/degrade rates for the optional `--profile rees` service.\n- **[Browserless (visual review)](/d/loopover-browserless)** — queue depth, concurrency, and rejection/error/timeout rate for the optional `--profile visual-review` screenshot service.\n- **[Sentry issues](/d/loopover-sentry)** — recent unresolved issues, top issues by event count, and error-volume trend, queried live from Sentry (`scripts/setup-sentry-datasource.sh`). The plain link below still opens Sentry itself for actions this read-only view can't do (resolving/assigning issues).\n\n## 📈 Metrics & logs\n- **Prometheus** — [targets](http://localhost:9090/targets) · [graph](http://localhost:9090)\n- **Alertmanager** — [alerts](http://localhost:9093)\n- **Loki** — query in [Explore](/explore) (pick the *Loki* datasource), e.g. `{compose_service=\"loopover\"}`\n- **Sentry** — release/source-map enriched errors. Edit the dashboard link if your project URL differs.\n\n## 🩺 Quick health checks\n| What | Where |\n|---|---|\n| App serving | `GET /ready` → 200 |\n| AI wired | boot log `selfhost_ai_provider` |\n| Embeds wired | boot log `selfhost_embed_provider` |\n| Vectors wired | boot log `selfhost_vectorize` |\n| Token spend | **[AI usage](/d/loopover-ai-usage)** dashboard |\n\n## 📚 Docs\n- [Maintainer self-hosting](https://gittensory.aethereal.dev/docs/maintainer-self-hosting) — setup, configuration, AI, REES, RAG, operations, and troubleshooting."
"content": "## 📊 Dashboards\n- **[Upstream PRs & issues (GitHub)](/d/loopover-github)** — live, accurate census + open-PR triage (GitHub API).\n- **[Reviews & PRs (maintainer)](/d/loopover-maintainer)** — loopover's own review activity + reviewed-PR log.\n- **[AI usage](/d/loopover-ai-usage)** — durable cross-provider ai_usage_events (filterable by provider/feature/model), live Prometheus counters, and Claude Code's own OTEL session telemetry, all in one place.\n- **[LoopOver (infra)](/d/loopover-selfhost)** — queue, jobs, HTTP, GitHub API cache/rate limits.\n- **[GPU metrics](/d/loopover-gpu)** — utilization/VRAM for a self-hosted Ollama GPU box.\n- **[Infra health](/d/loopover-infra-health)** — host CPU/mem/disk/network (node-exporter), per-container resource usage (cAdvisor), Redis, Qdrant, and whether the observability stack itself is up.\n- **[REES (review-enrichment)](/d/loopover-rees)** — request outcomes/latency and per-analyzer run/timeout/degrade rates for the optional `--profile rees` service.\n- **[Browserless (visual review)](/d/loopover-browserless)** — queue depth, concurrency, and rejection/error/timeout rate for the optional `--profile visual-review` screenshot service.\n- **[Sentry issues](/d/loopover-sentry)** — recent unresolved issues, top issues by event count, and error-volume trend, queried live from Sentry (`scripts/setup-sentry-datasource.sh`). The plain link below still opens Sentry itself for actions this read-only view can't do (resolving/assigning issues).\n\n## 📈 Metrics & logs\n- **Prometheus** — [targets](http://localhost:9090/targets) · [graph](http://localhost:9090)\n- **Alertmanager** — [alerts](http://localhost:9093)\n- **Loki** — query in [Explore](/explore) (pick the *Loki* datasource), e.g. `{compose_service=\"loopover\"}`\n- **Sentry** — release/source-map enriched errors. Edit the dashboard link if your project URL differs.\n\n## 🩺 Quick health checks\n| What | Where |\n|---|---|\n| App serving | `GET /ready` → 200 |\n| AI wired | boot log `selfhost_ai_provider` |\n| Embeds wired | boot log `selfhost_embed_provider` |\n| Vectors wired | boot log `selfhost_vectorize` |\n| Token spend | **[AI usage](/d/loopover-ai-usage)** dashboard |\n\n## 📚 Docs\n- [Maintainer self-hosting](https://loopover.ai/docs/maintainer-self-hosting) — setup, configuration, AI, REES, RAG, operations, and troubleshooting."
}
},

Expand All @@ -77,7 +77,7 @@
"gridPos": { "h": 4, "w": 24, "x": 0, "y": 13 },
"options": {
"mode": "markdown",
"content": "## 🤖 AMS — Autonomous Miner System\n- **[Observing your miner](https://github.com/JSONbored/gittensory/blob/main/packages/loopover-miner/docs/observability.md)** — AMS is a separate, local `gittensory-miner` CLI a dual-role operator may run alongside ORB on the same box. This guide is the AMS observability entry point: it covers pointing Grafana at the redacted AMS ledger datasources and loading its Grafana dashboard, separate from the ORB review-service observability above."
"content": "## 🤖 AMS — Autonomous Miner System\n- **[Observing your miner](https://github.com/JSONbored/gittensory/blob/main/packages/loopover-miner/docs/observability.md)** — AMS is a separate, local `loopover-miner` CLI a dual-role operator may run alongside ORB on the same box. This guide is the AMS observability entry point: it covers pointing Grafana at the redacted AMS ledger datasources and loading its Grafana dashboard, separate from the ORB review-service observability above."
}
}
]
Expand Down
2 changes: 1 addition & 1 deletion grafana/provisioning/datasources/ams-ledgers.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# AMS (gittensory-miner) redacted SQLite reporting datasources (#5184).
# AMS (loopover-miner) redacted SQLite reporting datasources (#5184).
#
# Grafana must not mount the miner's live LOOPOVER_MINER_CONFIG_DIR ledgers. The attempt log and prediction
# ledger contain free-form reasons, payload JSON, repo/target identifiers, readiness scores, and blocker/warning
Expand Down
6 changes: 3 additions & 3 deletions prometheus/prometheus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ alerting:
- targets: ["alertmanager:9093"]

scrape_configs:
- job_name: gittensory
- job_name: loopover
static_configs:
- targets: ["loopover:8787"]
metrics_path: /metrics
Expand All @@ -39,7 +39,7 @@ scrape_configs:

# Backup freshness from the read-only backup-exporter sidecar. The metrics exist only when the backup
# profile is active, which keeps backup alerts opt-in with the backup feature.
- job_name: gittensory-backup
- job_name: loopover-backup
# The exporter is a tiny BusyBox httpd wrapper around Prometheus text output. Prometheus v3 rejects
# blank Content-Type responses unless the scrape protocol is explicit.
fallback_scrape_protocol: PrometheusText0.0.4
Expand Down Expand Up @@ -92,7 +92,7 @@ scrape_configs:

# browserless queue/concurrency/rejection/error metrics, translated from its own JSON /metrics by the
# browserless-exporter sidecar (--profile visual-review, #5368). Same BusyBox-httpd-wrapper shape as
# gittensory-backup above, so it needs the same explicit scrape protocol fallback.
# loopover-backup above, so it needs the same explicit scrape protocol fallback.
- job_name: browserless
fallback_scrape_protocol: PrometheusText0.0.4
static_configs:
Expand Down
30 changes: 15 additions & 15 deletions prometheus/rules/alerts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
#
# Every rule below is grounded ONLY in metrics the loopover app actually exports at
# GET /metrics, plus the synthetic `up` metric Prometheus emits per scrape target. The
# exceptions are the final three groups -- gittensory-miner-prediction (#5188),
# gittensory-miner-portfolio-queue (#5186), and gittensory-miner-governor (#5187) -- which
# target the gittensory-miner's OWN separate scrape surface (see each group's own comment
# exceptions are the final three groups -- loopover-miner-prediction (#5188),
# loopover-miner-portfolio-queue (#5186), and loopover-miner-governor (#5187) -- which
# target the loopover-miner's OWN separate scrape surface (see each group's own comment
# for why it still ships here and stays dormant until a miner scrape target exists).
#
# Thresholds are sane defaults for a SMALL single-host self-host. Tune the numbers in
Expand All @@ -30,7 +30,7 @@ groups:
# `up` is 0 when Prometheus cannot scrape /metrics (process down, crash-loop,
# network partition, or wrong port). Scoped to our job so node/other targets
# don't trip this. 2m absorbs a single missed scrape + a restart.
expr: up{job="gittensory"} == 0
expr: up{job="loopover"} == 0
for: 2m
labels:
severity: critical
Expand Down Expand Up @@ -621,7 +621,7 @@ groups:
runbook: "Check CLOUDFLARE_D1_MONITOR_API_TOKEN is still valid and CLOUDFLARE_D1_MONITOR_ACCOUNT_ID/DATABASE_ID are correct. Tail logs for level=error event=d1_size_probe_error."

# ── Miner prediction-calibration drift (#5188) ────────────────────────────
# UNLIKE every group above, this rule targets the gittensory-MINER's own scrape surface, not the
# UNLIKE every group above, this rule targets the loopover-MINER's own scrape surface, not the
# loopover server's GET /metrics. The miner is a local CLI (not a daemon), so an operator renders
# renderMinerPredictionMetrics (packages/loopover-engine/src/miner-prediction-metrics.ts, #4264, wired
# into a command in #4838 over the calibration-report join built in #4849) to a file/pushgateway their
Expand All @@ -630,7 +630,7 @@ groups:
# the 0/0 case undefined), so it loads cleanly via `selfhost:validate-observability` and never false-fires
# on an install that has no miner scrape configured yet -- exactly like loopover-d1-storage stays silent
# without the D1 probe.
- name: gittensory-miner-prediction
- name: loopover-miner-prediction
rules:
- alert: LoopOverMinerPredictionCalibrationDrift
# Calibration = how often the miner's predicted gate conclusion matched the realized outcome. This is
Expand Down Expand Up @@ -661,17 +661,17 @@ groups:
labels:
severity: warning
annotations:
summary: "gittensory miner prediction calibration has drifted"
summary: "loopover miner prediction calibration has drifted"
description: "{{ $value | humanizePercentage }} of the miner's recently resolved gate predictions were incorrect over the last 6h (sustained 30m). Predicted-gate accuracy has drifted out of tolerance."
runbook: "Compare loopover_miner_prediction_correct_total vs loopover_miner_prediction_incorrect_total and review the miner's own calibration report (`gittensory-miner calibration`). Sustained drift usually means the predicted-gate heuristics need recalibration against the repo(s) the miner now targets, or a recent upstream gate-behavior change the predictor hasn't caught up to."
runbook: "Compare loopover_miner_prediction_correct_total vs loopover_miner_prediction_incorrect_total and review the miner's own calibration report (`loopover-miner calibration`). Sustained drift usually means the predicted-gate heuristics need recalibration against the repo(s) the miner now targets, or a recent upstream gate-behavior change the predictor hasn't caught up to."

# ── Miner portfolio-queue backlog health (#5186) ──────────────────────────
# Same DORMANT-until-a-scrape-target-exists posture as gittensory-miner-prediction above: an operator runs
# `gittensory-miner queue metrics` (packages/loopover-miner/lib/portfolio-queue-cli.js's
# Same DORMANT-until-a-scrape-target-exists posture as loopover-miner-prediction above: an operator runs
# `loopover-miner queue metrics` (packages/loopover-miner/lib/portfolio-queue-cli.js's
# renderPortfolioQueueMetrics) to a file/pushgateway their Prometheus scrapes. Absent
# loopover_miner_portfolio_queue_* series simply yield no result, so this loads cleanly via
# `selfhost:validate-observability` and never false-fires on an install with no miner scrape configured.
- name: gittensory-miner-portfolio-queue
- name: loopover-miner-portfolio-queue
rules:
- alert: LoopOverMinerPortfolioQueueItemStuck
# The concrete "is a claimed item actually stuck" signal, distinct from #4827 (the lease/timeout/reclaim
Expand All @@ -686,9 +686,9 @@ groups:
labels:
severity: warning
annotations:
summary: "gittensory miner has a claimed portfolio-queue item stuck in progress"
summary: "loopover miner has a claimed portfolio-queue item stuck in progress"
description: "The oldest in-flight portfolio-queue claim lease is {{ $value | printf \"%.0f\" }}s old (sustained 15m), well past the 30m default reclaim window. A crashed/killed attempt process's lease isn't being automatically reclaimed."
runbook: "Run `gittensory-miner queue list --json` to find the stuck item, and `gittensory-miner queue claim-batch` (or a scheduled reclaim sweep) to trigger the lease-expiry self-heal (portfolio-queue-expiry.js's sweepStuckItems). If it stays stuck after that, release it manually with `gittensory-miner queue release <owner/repo> <identifier>`."
runbook: "Run `loopover-miner queue list --json` to find the stuck item, and `loopover-miner queue claim-batch` (or a scheduled reclaim sweep) to trigger the lease-expiry self-heal (portfolio-queue-expiry.js's sweepStuckItems). If it stays stuck after that, release it manually with `loopover-miner queue release <owner/repo> <identifier>`."

- alert: LoopOverMinerPortfolioQueueBacklogHigh
# Standing size of the queued (not yet claimed) backlog, the miner counterpart of
Expand All @@ -700,7 +700,7 @@ groups:
labels:
severity: warning
annotations:
summary: "gittensory miner portfolio-queue backlog above 200"
summary: "loopover miner portfolio-queue backlog above 200"
description: "{{ $value | printf \"%.0f\" }} portfolio-queue items are queued (sustained 30m) -- claiming/attempting is falling behind discovery."
runbook: "Check whether `queue next` / `loop` is running at all (a laptop-mode miner only drains the queue while actively invoked), or whether WIP caps (#4850's --global-wip/--per-repo-wip) are set low enough to bottleneck throughput relative to discovery volume."

Expand All @@ -711,7 +711,7 @@ groups:
# series simply yield no result. Read-only: these rules alert on the governor's already-persisted rate-limit
# (#5134/write-rate-limit.ts) and cap-usage (#5134/budget-cap.ts) state; they do not gate, retry, or modify
# governor decision logic in any way (governor-chokepoint.js/governor-chokepoint-persisted.js are untouched).
- name: gittensory-miner-governor
- name: loopover-miner-governor
rules:
- alert: LoopOverMinerGovernorRateLimitPressureHigh
# remaining_ratio is 1 for an empty bucket and 0 once a write-rate-limit bucket (global or per-repo,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ describe("LoopOverMinerGovernorRateLimitPressureHigh alert (#5187)", () => {

it("lives in its own miner-scoped rule group, separate from the loopover server groups", () => {
const group = alertsDoc.groups.find((g) => g.rules.some((r) => r.alert === rule.alert));
expect(group?.name).toBe("gittensory-miner-governor");
expect(group?.name).toBe("loopover-miner-governor");
});

it("keys off the real renderer's rate-limit-remaining gauge, not an invented metric name", () => {
Expand Down Expand Up @@ -84,7 +84,7 @@ describe("LoopOverMinerGovernorCapUsageHigh alert (#5187)", () => {

it("lives in the same miner-scoped governor rule group", () => {
const group = alertsDoc.groups.find((g) => g.rules.some((r) => r.alert === rule.alert));
expect(group?.name).toBe("gittensory-miner-governor");
expect(group?.name).toBe("loopover-miner-governor");
});

it("keys off the real renderer's cap-usage gauge, thresholded at 90%", () => {
Expand Down
Loading
Loading