fix(observability): restore historical continuity in the Self-Host dashboard - #6779
Merged
Merged
Conversation
…shboard The gittensory->loopover rebrand renamed every emitted Prometheus metric's __name__ (e.g. gittensory_queue_pending -> loopover_queue_pending), not just a job/instance label. Prometheus treats a __name__ change as a completely distinct time series -- it does not retroactively merge old and new names -- so every panel querying only the new loopover_* names went dark for anything before the rename, even though the old gittensory_* series are still fully present and queryable (90d retention, nothing was ever deleted). Confirmed live: 118 gittensory_* metric names still hold real historical data in Prometheus (weeks of it for some), completely invisible to every panel in this dashboard. Wraps each panel query's bare metric/range-vector reference in an `or` fallback against its gittensory_* counterpart -- e.g. `loopover_queue_pending` -> `(loopover_queue_pending or gittensory_queue_pending)`, and for rate()/increase() (which require a literal range-vector selector, not an arbitrary sub-expression) duplicates the whole function call per name and unions the two resulting instant vectors instead. 55 of the 60 metric names referenced in this dashboard have a real gittensory_* historical counterpart; the other 5 are genuinely new (post-rename) metrics with nothing to recover, left as-is. Verified against live Prometheus: the transformed queries now return continuous data spanning the full pre-rename / transitional / post-rename range instead of cutting off at the rename boundary.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
loopover-orb
Bot
added
the
gittensor:bug
Merged
24 tasks
JSONbored
added a commit
that referenced
this pull request
Jul 17, 2026
…ssertions-6779 test(selfhost): sync grafana dashboard assertions with #6779's gittensory_ fallback exprs
Merged
3 tasks
Merged
3 tasks
thomasalvaedison7777-lgtm
pushed a commit
to thomasalvaedison7777-lgtm/loopover
that referenced
this pull request
Jul 17, 2026
…s gittensory_ fallback exprs PR JSONbored#6779 changed most Prometheus expr query strings in grafana/dashboards/gittensory.json to add a legacy-metric fallback (e.g. "loopover_x" became "(loopover_x or gittensory_x)"), a dual-read migration for the old gittensory_-prefixed metric names alongside the new loopover_-prefixed ones. That change to the dashboard JSON was intentional and correct, but test/unit/selfhost-grafana-dashboard.test.ts was never updated to match, so its exact-string expr/legendFormat assertions started failing on main. Updates the 25 affected assertions across 6 describe/it blocks (GitHub response cache counters, Orb zero-safe panels, Postgres/backup freshness, maintenance admission deferrals, self-host runtime-drift signals, and backlog-vs-fresh-intake lane fairness) to match the dashboard JSON's current, shipped expr values. No behavior change -- grafana/dashboards/gittensory.json is untouched.
thomasalvaedison7777-lgtm
pushed a commit
to thomasalvaedison7777-lgtm/loopover
that referenced
this pull request
Jul 17, 2026
…tinuity query rewrite JSONbored#6779 wrapped 71 panel queries in this dashboard with an (new_metric or gittensory_old_metric) union for historical Prometheus continuity across the rebrand, but didn't update this test file's exact target.expr string assertions to match -- 25 of them were pinned to the pre-JSONbored#6779 query text and have been silently failing since that PR merged.
thomasalvaedison7777-lgtm
pushed a commit
to thomasalvaedison7777-lgtm/loopover
that referenced
this pull request
Jul 17, 2026
…hboard JSONbored#5522's hard-cutover rename of the gittensory_ metric prefix to loopover_ left this dashboard's 9 loopover_ai_* queries with no fallback to their pre-rebrand names, so every panel here only ever showed data recorded after that cutover -- confirmed live, both metric names have real historical series in Prometheus going back well past the cutover. Applies the same (loopover_x or gittensory_x) union fix already shipped for grafana/dashboards/gittensory.json (JSONbored#6779/JSONbored#6787), including that fix's own lesson: a label matcher like {provider="codex"} must bind to each side of the union individually, never to the closing paren of the union as a whole -- verified all 9 rewritten queries directly against live Prometheus before deploying. Deployed to edge-nl-01 and restarted Grafana to force a clean reprovision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
__name__(e.g.gittensory_queue_pending->loopover_queue_pending), not just a job/instance label. Prometheus treats a__name__change as a completely distinct time series — it does not retroactively merge old and new names — so every panel querying only the newloopover_*names went dark for anything before the rename, even though the oldgittensory_*series are still fully present and queryable (90d retention, nothing was ever deleted).gittensory_*metric names still hold real historical data in Prometheus (weeks of it for some), completely invisible to every panel in this dashboard.orfallback against itsgittensory_*counterpart — e.g.loopover_queue_pending->(loopover_queue_pending or gittensory_queue_pending), and forrate()/increase()(which require a literal range-vector selector, not an arbitrary sub-expression) duplicates the whole function call per name and unions the two resulting instant vectors instead. 55 of the 60 metric names referenced in this dashboard have a realgittensory_*historical counterpart; the other 5 are genuinely new (post-rename) metrics with nothing to recover, left as-is.Note: this dashboard's own filename (
gittensory.json) and internaljob=references to the legacy label are intentionally left alone in this PR — that's part of a separate, broader "eliminate remaining gittensory branding" pass, not bundled here to keep this fix narrow and reviewable.Validation
npm run selfhost:validate-observability— dashboards and alert rules validgit diff --check— cleansrc/**, no Codecov patch-coverage obligation for this file