orb(observability): clock-skew is never sampled in broker mode and reads a hard 0, while the staleness gauge built to catch exactly that is wired to nothing

[JSONbored]

Problem

The clock-drift detector built after the #3811 outage never takes a sample on the deployment topology
that runs today, and reads a hard 0 — indistinguishable from "clock is fine".

recordClockSkewFromResponse has exactly one call site: src/github/app.ts:224, inside
requestInstallationTokenWithJwt. In broker mode mintInstallationToken returns from the broker branch
first (src/github/app.ts:241):

if (isOrbBrokerMode(env)) { … return brokered.token; }

A brokered self-host holds no App private key, so the JWT path is unreachable by construction.
lastSkewSeconds stays at its initialiser 0 (src/selfhost/clock-skew.ts:10) and
clockSkewSecondsSample() returns it verbatim (:33) — "0 until the first successful sample".

Both alert rules are abs(loopover_clock_skew_seconds) > 60 / > 120
(prometheus/rules/alerts.yml:552, :564). abs(0) satisfies neither, ever. The Grafana panel
(grafana/dashboards/selfhost.json:2771) asserts perfect clock sync on a host whose clock was never
measured.

The companion gauge loopover_clock_skew_sample_age_seconds — added in #7000 precisely so "a stale
reading is distinguishable from a fresh one", and correctly returning -1 when never sampled — is
referenced by zero alert rules and zero dashboard panels (grep clock_skew_sample_age prometheus/ grafana/ → no hits).

Impact

A placeholder satisfies the health check for its own absence — the same shape as #9008. Drift then
corrupts everything time-derived (cached.expiresAtMs > Date.now() token-grace decisions, run_after
scheduling, cooldown/sinceIso windows, Redis lock TTL reasoning) with no signal, which is the exact
edge-us-01 incident class the module was built for.

Requirements

• Sample skew from any GitHub response Date header, including the broker path
  (fetchBrokeredInstallationToken is an HTTPS response and carries one), rather than only the local JWT
  mint.
• Return a -1/NaN sentinel from clockSkewSecondsSample() until the first real sample, so 0 stops
  meaning "unmeasured".
• Add an alert rule on loopover_clock_skew_sample_age_seconds == -1 or > <threshold> and put it on the
  dashboard.
• Audit the other -1-sentinel gauges for the same never-alerted shape (the relay-drain gauge in orb(observability): the relay-drain staleness alarm is suppressed by its own worst failure — 'never drained' reads -1, and a deploy silences a firing alert #9128 is
  a confirmed sibling).

Test Coverage Requirements

99%+ patch coverage, branch-counted; both arms (sampled / never sampled) and the broker-path sample.

Links & Resources

• src/selfhost/clock-skew.ts ~10, ~33; src/github/app.ts ~224, ~241-288; src/server.ts ~895-898;
  prometheus/rules/alerts.yml ~552, ~564; grafana/dashboards/selfhost.json ~2771
• obs(selfhost): detect and alert on NTP/system clock drift #3811 (the originating incident), clock-skew gauge has no staleness/freshness signal #7000 (added the unused staleness gauge), orb(observability): the relay-drain staleness alarm is suppressed by its own worst failure — 'never drained' reads -1, and a deploy silences a firing alert #9128 (same sentinel shape),
  orb(review): forceAiReview does not bypass the AI-review lock — the re-run button is silently inert behind an orphaned lock (root cause of #9000) #9008 (placeholder suppressing its own alarm)

maintainer-only — observability correctness.

[linear-code]

JSO-6328