feat(selfhost): add clock-skew sample-age staleness gauge (#7000)

[jeffrey701]

Summary

Adds a companion loopover_clock_skew_sample_age_seconds gauge so an operator watching the clock-skew metric can tell a fresh reading from a stale one. loopover_clock_skew_seconds only refreshes when a GitHub App JWT-mint call observes a Date header; if token-mint activity stalls (e.g. a long-lived cached or broker-provided token means no fresh mint happens for a while), the existing gauge keeps reporting an old sample as if it were current — with no way to distinguish "clock is fine, just sampled a while ago" from "clock is fine right now."

• src/selfhost/clock-skew.ts: track the wall-clock time of the last successful sample alongside lastSkewSeconds, updated in the same place recordClockSkewFromResponse sets the skew; expose clockSkewSampleAgeSeconds() returning the age in seconds, or a -1 sentinel when no sample has landed yet — the same "never sampled" convention d1-size-probe.ts / loopover_host_load_avg1_per_core already use. resetClockSkewForTest resets the new state too. The existing skew-calculation logic is unchanged (the local timestamp is captured once and reused for both the skew and the sample time).
• src/server.ts: register the companion gauge next to loopover_clock_skew_seconds.
• src/selfhost/metrics.ts: declare the new gauge's HELP/TYPE metadata.

Why

An old sample silently looking current defeats the point of the drift check (#3811): a dead NTP source could leave the clock drifting while the gauge shows a stale-but-fine reading. The age signal makes staleness observable, mirroring the freshness sentinel the D1 probe already established for exactly this class of problem.

Validation

• npm run typecheck, npm run selfhost:validate-observability, npm run docs:drift-check — all green.
• test/unit/clock-skew.test.ts extended to 100% line + branch coverage on the changed module: the age starts at the -1 never-sampled sentinel, reads 0 right after a sample and grows as time passes (fake timers), stays anchored to the last GOOD sample when a response is ignored (a missing/invalid Date header must not reset the sample time), and returns to -1 after reset.

Closes #7000

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.74%. Comparing base (15715dc) to head (c99b8b9).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7032   +/-   ##
=======================================
  Coverage   93.74%   93.74%           
=======================================
  Files         691      691           
  Lines       68658    68664    +6     
  Branches    18760    18761    +1     
=======================================
+ Hits        64361    64367    +6     
  Misses       3302     3302           
  Partials      995      995           

Flag	Coverage Δ
shard-1	43.42% <100.00%> (-0.45%)	⬇️
shard-2	36.90% <14.28%> (+<0.01%)	⬆️
shard-3	32.96% <14.28%> (-0.11%)	⬇️
shard-4	34.56% <14.28%> (+0.60%)	⬆️
shard-5	31.72% <71.42%> (-0.55%)	⬇️
shard-6	46.07% <14.28%> (+0.27%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/selfhost/clock-skew.ts	100.00% <100.00%> (ø)
src/selfhost/metrics.ts	100.00% <ø> (ø)

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-17 19:04:04 UTC}

4 files · 1 AI reviewer · no blockers · readiness 83/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
Adds a companion loopover_clock_skew_sample_age_seconds gauge that tracks the wall-clock time of the last successful clock-skew sample and exposes its age, using the same -1 never-sampled sentinel convention as the existing D1 probe. The implementation is correct: the timestamp is captured once and reused for both the skew and sample-age calculations, a missing/invalid Date header correctly leaves the previous sample untouched, and resetClockSkewForTest properly clears the new state. Wiring is complete across the gauge registration in server.ts and the HELP/TYPE metadata in metrics.ts, and the added tests exercise the never-sampled sentinel, growth over time, ignored-response non-advancement, and reset behavior.

Nits — 3 non-blocking

• The JSDoc reference `{@​link d1DatabaseSizeBytesSample}` in clock-skew.ts is a cross-file link that may not resolve depending on the TSDoc/typedoc config; verify it renders correctly if docs are generated from this comment.
• Consider whether `clockSkewSampleAgeSeconds()` should floor/round its return value like `clockSkewSecondsSample()` implicitly does via the header-diff math, for consistent dashboard formatting with `d1` probe conventions.
• No changes needed to `d1-size-probe.ts` — confirmed it's included only as reference for the -1 sentinel convention, not part of this diff.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#7000
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	❌ 8/20	High review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 145 registered-repo PR(s), 70 merged, 32 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	jeffrey701; Gittensor profile; 145 PR(s), 32 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: minor

Linked issue satisfaction

Addressed
The diff adds lastSkewSampleAtMs tracked in recordClockSkewFromResponse, exposes clockSkewSampleAgeSeconds() with the -1 never-sampled sentinel, resets it in resetClockSkewForTest, wires a companion loopover_clock_skew_sample_age_seconds gauge in server.ts with metadata, and includes tests covering the sentinel and age-growth behavior — matching every requirement and deliverable in the issue.

Review context

• Author: jeffrey701
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 145 PR(s), 32 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Add a concise scope and risk note.
• Then work through the remaining 2 steps in the Signals table above.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.