Context
GittensoryHighJobFailureRatio in prometheus/rules/alerts.yml (~line 46) is documented as measuring the fraction of processed jobs that failed, but its expr is sum(rate(gittensory_jobs_failed_total[10m])) / sum(rate(gittensory_jobs_processed_total[10m])) > 0.10.
gittensory_jobs_processed_total increments only on success (src/selfhost/pg-queue.ts, src/selfhost/sqlite-queue.ts), while gittensory_jobs_failed_total is a disjoint counter for any processing failure. So the expr computes failed / success, not failed / (failed + success). At a true 50% failure rate the expr evaluates to 1.0, not 0.5 — the alert fires far earlier than its stated 10% threshold implies, and its humanizePercentage description misreports the real failure rate to whoever gets paged.
The Grafana "Job Failure Rate" panel (grafana/dashboards/gittensory.json) already uses the correct failed / (processed + failed + epsilon) formula for the equivalent _persisted_total metrics, so the page and the dashboard currently disagree on the same underlying signal.
Requirements
- Change the alert's denominator to
sum(rate(gittensory_jobs_failed_total[10m])) + sum(rate(gittensory_jobs_processed_total[10m])), matching the dashboard panel's formula.
- Re-validate the 0.10 threshold makes sense under the corrected (true-percentage) semantics.
- Add/adjust a test or fixture that pins the corrected PromQL expression if this repo has rule-linting/unit coverage for
alerts.yml.
Acceptance criteria
- The alert and the Grafana panel agree on the formula for the same conceptual signal.
promtool check rules (or the repo's equivalent) still passes.
Parent: #1667
Context
GittensoryHighJobFailureRatioinprometheus/rules/alerts.yml(~line 46) is documented as measuring the fraction of processed jobs that failed, but its expr issum(rate(gittensory_jobs_failed_total[10m])) / sum(rate(gittensory_jobs_processed_total[10m])) > 0.10.gittensory_jobs_processed_totalincrements only on success (src/selfhost/pg-queue.ts,src/selfhost/sqlite-queue.ts), whilegittensory_jobs_failed_totalis a disjoint counter for any processing failure. So the expr computesfailed / success, notfailed / (failed + success). At a true 50% failure rate the expr evaluates to 1.0, not 0.5 — the alert fires far earlier than its stated 10% threshold implies, and itshumanizePercentagedescription misreports the real failure rate to whoever gets paged.The Grafana "Job Failure Rate" panel (
grafana/dashboards/gittensory.json) already uses the correctfailed / (processed + failed + epsilon)formula for the equivalent_persisted_totalmetrics, so the page and the dashboard currently disagree on the same underlying signal.Requirements
sum(rate(gittensory_jobs_failed_total[10m])) + sum(rate(gittensory_jobs_processed_total[10m])), matching the dashboard panel's formula.alerts.yml.Acceptance criteria
promtool check rules(or the repo's equivalent) still passes.Parent: #1667