Skip to content

fix(selfhost): correct GittensoryHighJobFailureRatio alert formula #3892

Description

@JSONbored

Context

GittensoryHighJobFailureRatio in prometheus/rules/alerts.yml (~line 46) is documented as measuring the fraction of processed jobs that failed, but its expr is sum(rate(gittensory_jobs_failed_total[10m])) / sum(rate(gittensory_jobs_processed_total[10m])) > 0.10.

gittensory_jobs_processed_total increments only on success (src/selfhost/pg-queue.ts, src/selfhost/sqlite-queue.ts), while gittensory_jobs_failed_total is a disjoint counter for any processing failure. So the expr computes failed / success, not failed / (failed + success). At a true 50% failure rate the expr evaluates to 1.0, not 0.5 — the alert fires far earlier than its stated 10% threshold implies, and its humanizePercentage description misreports the real failure rate to whoever gets paged.

The Grafana "Job Failure Rate" panel (grafana/dashboards/gittensory.json) already uses the correct failed / (processed + failed + epsilon) formula for the equivalent _persisted_total metrics, so the page and the dashboard currently disagree on the same underlying signal.

Requirements

  • Change the alert's denominator to sum(rate(gittensory_jobs_failed_total[10m])) + sum(rate(gittensory_jobs_processed_total[10m])), matching the dashboard panel's formula.
  • Re-validate the 0.10 threshold makes sense under the corrected (true-percentage) semantics.
  • Add/adjust a test or fixture that pins the corrected PromQL expression if this repo has rule-linting/unit coverage for alerts.yml.

Acceptance criteria

  • The alert and the Grafana panel agree on the formula for the same conceptual signal.
  • promtool check rules (or the repo's equivalent) still passes.

Parent: #1667

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions