Skip to content

clock-skew gauge has no staleness/freshness signal #7000

Description

@JSONbored

Context

src/selfhost/clock-skew.ts's clockSkewSecondsSample() feeds the loopover_clock_skew_seconds gauge (server.ts:830) directly from module-level lastSkewSeconds, which is only updated when a GitHub response's Date header is observed (piggybacking on JWT-authenticated installation-token mint calls, per this module's own header comment about the #3811 incident this check exists to catch). If token-minting activity stalls for any reason (e.g. a long-lived cached or broker-provided token means no fresh mint call happens for a while), the gauge keeps reporting an old sample as if it were current — an operator watching this metric has no way to distinguish "clock is fine, just sampled a while ago" from "clock is fine right now."

src/selfhost/d1-size-probe.ts already has an established, deliberate convention for exactly this class of problem: a -1 sentinel value distinguishing "probe disabled, never sampled yet, or data unavailable" from a real reading (see its own comments around lines 201, 232-235, 249-253, matching loopover_host_load_avg1_per_core's same convention).

Requirements

  • Track the timestamp (or age in seconds) of the last successful clock-skew sample alongside lastSkewSeconds, updated in the same place recordClockSkewFromResponse currently updates lastSkewSeconds.
  • Expose that staleness signal as a new function (e.g. clockSkewSampleAgeSeconds()) mirroring clockSkewSecondsSample()'s existing shape, following the -1-sentinel convention from d1-size-probe.ts for the "never sampled yet" case.
  • Wire the new signal into a companion gauge in server.ts alongside the existing loopover_clock_skew_seconds (follow whatever naming convention the existing gauge registration there uses, e.g. loopover_clock_skew_sample_age_seconds).
  • Do not change recordClockSkewFromResponse's existing skew-calculation logic or resetClockSkewForTest's reset behavior beyond also resetting the new staleness state.

Deliverables

  • New staleness/age signal exported from clock-skew.ts
  • Companion gauge registered in server.ts
  • resetClockSkewForTest resets the new state too
  • Test asserting the age signal starts at the "never sampled" sentinel and updates correctly after a sample

Test Coverage Requirements

99%+ Codecov patch coverage on every changed line and branch, plus the regression test above.

Expected Outcome

An operator watching the clock-skew gauge in Grafana can distinguish a fresh reading from a stale one, instead of an old sample silently looking current indefinitely.

Links & Resources

src/selfhost/d1-size-probe.ts:201,232-235,249-253 — the -1-sentinel freshness convention to follow. src/selfhost/clock-skew.ts — the module to extend. server.ts:830 — the existing gauge registration site.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions