Skip to content

calibration: nightly config-drift sentinel — cron wire + one-shot stale-config alert #8213

Description

@JSONbored

Problem

Epic #8211 track A. With the pure drift evaluator available, nothing runs it: stale-config detection must ride the existing scheduled-tick machinery and alert exactly once per detected drift episode, or it becomes either silent or noisy.

Requirements

⚠️ Required pattern. Mirror runScheduledKnobLoosening (src/services/knob-loosening-run.ts) verbatim: fail-safe tick (a thrown evaluation is logged + swallowed), ONE structured error-level alert on the Workers-Logs + Sentry notify path with the ev sub-field per knob, and the whole surface flag-gated OFF by default behind its own truthy-string wrangler var (CONFIG_DRIFT_SENTINEL_ENABLED) so the deploy is byte-identical until opted in.

  • Per live registry knob per tick: resolve the LIVE value (override ?? shipped, same read as consumption), run the drift evaluator over the knob's 90-day corpus, and alert only on a TIGHTER-dominating result (looser duplicates the loosening loop's own surfacing — suppress, with a debug log).
  • Episode dedup: persist the last-alerted drift fingerprint (knob + direction + dominating value) in system_flags; re-alert only when the fingerprint changes or clears. No re-alerting every tick on the same standing drift.
  • Operator surface: extend GET /v1/internal/calibration/knobs entries with the current drift report (null when clean) — visibility must not require the flag, mirroring the calibration: operator visibility for applied loosenings — live floor + history on the internal calibration surface #8161 posture.
  • cf-typegen + test-helper env plumbing for the new var, per the standing checklist.

Deliverables

  • Scheduled wire + alert + fingerprint dedup, 100% branch on all touched src
  • Knobs-endpoint drift field + tests
  • Docs: drift-sentinel section on the backtest-calibration page

Links & Resources

#8211 (epic), the pure-evaluator sub-issue (blocked-by), src/services/knob-loosening-run.ts, src/queue/job-dispatch.ts (satisfaction-floor-loosening case)

Boundaries

ALERT-ONLY authority: the sentinel never writes a knob value — a tighter setting taking effect remains a human decision (or a future issue with its own approval gate). Flag default false.

maintainer-only — scheduled-authority wiring.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions