Skip to content

site(status): uptime and incident history — sample the status board so 'has it been healthy?' has a source #9985

Description

@JSONbored

The second slice of #9747. #9983 ships the live board; this adds the history half.

Why it needs its own change

/v1/public/service-status answers "is each component healthy right now?" from Alertmanager's active alerts. It cannot answer "has it been?", because there is no source:

  • Alertmanager's /api/v2/alerts returns active alerts only — resolved ones are gone.
  • There is no Prometheus on the box; metrics flow through the otel-collector.
  • There is no status-sample table.

Deriving an uptime figure from a single live sample would be fabrication, so #9983 declined to publish one rather than publishing a number that looks measured.

Shape

  • A service_status_samples table: (component, status, sampled_at), one row per component per tick.
  • Written from the existing cron, not a new scheduler — same cadence the rest of the periodic work already runs on.
  • /v1/public/service-status gains uptime (24h / 7d / 30d) and incidents, both derived from contiguous non-operational runs of samples. That is what makes incidents appear without manual posting, per site(status): public service-status surface — ORB and AMS fleet heartbeats and uptime #9747's acceptance.

Things to get right

  • History starts at deploy. A 30-day window with three days of samples must say so rather than reporting 100% over the days it has. An uptime figure whose window silently shrinks to the data available is the same class of false-green site(status): public service-status endpoint, sourced from the existing Alertmanager #9983 exists to avoid.
  • unknown samples are not operational. A tick where the source could not be read must not count toward uptime in either direction; it is unmeasured time and should be reported as coverage, not folded into the percentage.
  • Retention. Samples are high-volume and low-value once aggregated — they need a retention rule alongside the other tables in src/db/retention.ts, and a rollup if the 30-day window is to survive it.

Acceptance

  • Uptime and incident history are published per component, derived from samples rather than posted by hand.
  • A window with partial coverage states its coverage rather than implying a full window.
  • Unmeasured ticks are excluded from the percentage and reported separately.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions