Context
Found during the 2026-07-06 incident audit (parent: #1667). edge-us-01's system clock silently drifted ~3 minutes off true time because its configured NTP server was a dead single point of failure — chronyc sources showed Reach: 0 (zero successful syncs) the entire time, with no redundant fallback server. This broke GitHub App JWT auth ("Bad credentials") for a window, contributing to today's outage. Already remediated today as an immediate fix: replaced the dead single source with time.cloudflare.com (confirmed reliable) plus a pool.ntp.org fallback pool; chrony is now actively synced.
Grepped the entire repo (case-insensitive) for "ntp"/"chrony" and confirmed there is genuinely zero clock-drift awareness anywhere in the codebase, Prometheus config, Grafana dashboards, or self-host docs. Nothing would have surfaced the drift before JWT auth started failing.
Requirements
- Add a lightweight clock-skew probe: compare the self-host process clock against a trusted external time source (e.g. the
Date response header from a GitHub API call already being made) and expose the delta as a gauge.
- Alert when drift exceeds a threshold that would plausibly break JWT auth (well under the 3 minutes seen in this incident — GitHub App JWTs have limited skew tolerance).
- Document in
docker-compose.yml/self-host setup docs that operators should configure redundant NTP sources, not a single one — today's incident was caused specifically by having only one, dead server configured.
- Consider surfacing this as a
captureError when a JWT-mint attempt fails with an auth error AND clock skew is simultaneously elevated, to make the causal link explicit.
Deliverables
- New gauge (e.g.
gittensory_clock_skew_seconds), sampled periodically, registered in src/selfhost/metrics.ts.
prometheus/rules/alerts.yml: new alert rule.grafana/dashboards/gittensory.json: new panel (System Health row).- Self-host docs/
docker-compose.yml: guidance on redundant NTP configuration.
Expected outcome
Clock drift that would break GitHub App JWT auth is visible in Grafana and alertable within minutes of drifting, instead of being discovered only after JWT auth starts failing with an opaque "Bad credentials" error.
Context
Found during the 2026-07-06 incident audit (parent: #1667). edge-us-01's system clock silently drifted ~3 minutes off true time because its configured NTP server was a dead single point of failure —
chronyc sourcesshowedReach: 0(zero successful syncs) the entire time, with no redundant fallback server. This broke GitHub App JWT auth ("Bad credentials") for a window, contributing to today's outage. Already remediated today as an immediate fix: replaced the dead single source withtime.cloudflare.com(confirmed reliable) plus apool.ntp.orgfallback pool; chrony is now actively synced.Grepped the entire repo (case-insensitive) for "ntp"/"chrony" and confirmed there is genuinely zero clock-drift awareness anywhere in the codebase, Prometheus config, Grafana dashboards, or self-host docs. Nothing would have surfaced the drift before JWT auth started failing.
Requirements
Dateresponse header from a GitHub API call already being made) and expose the delta as a gauge.docker-compose.yml/self-host setup docs that operators should configure redundant NTP sources, not a single one — today's incident was caused specifically by having only one, dead server configured.captureErrorwhen a JWT-mint attempt fails with an auth error AND clock skew is simultaneously elevated, to make the causal link explicit.Deliverables
gittensory_clock_skew_seconds), sampled periodically, registered insrc/selfhost/metrics.ts.prometheus/rules/alerts.yml: new alert rule.grafana/dashboards/gittensory.json: new panel (System Health row).docker-compose.yml: guidance on redundant NTP configuration.Expected outcome
Clock drift that would break GitHub App JWT auth is visible in Grafana and alertable within minutes of drifting, instead of being discovered only after JWT auth starts failing with an opaque "Bad credentials" error.