Skip to content

engine(calibration): phase7-calibration-loop hardcodes prOutcomeMetric.fresh to true instead of computing it #10317

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

packages/loopover-engine/src/phase7-calibration-loop.ts's computePhase7CalibrationLoop builds two
CalibrationSourceMetric objects, one per calibration source. historicalReplayMetric.fresh (around
line 410) is a real computation, gated through isHistoricalReplayRunFresh against
observedAt/config.replayFreshnessMaxAgeHours. prOutcomeMetric.fresh (line 374) is a hardcoded
constant:

const prOutcomeMetric: CalibrationSourceMetric = {
  source: "pr_outcome",
  accuracy: prOutcomeDerived?.accuracy ?? null,
  sampleSize: prOutcomeDerived?.sampleSize ?? 0,
  observedAt: normalizeObservedAt(input.prOutcome?.observedAt),
  fresh: true,
};

fresh is always true regardless of observedAt — even when observedAt is null because no
timestamp was supplied at all. CalibrationSourceMetric.fresh is a single shared field used
identically for both sources by calibration-dashboard.ts's sourceRow (lines 42-48), which renders
detail: n=${sampleSize} · ${fresh ? "fresh" : "stale"} for both the "Historical replay" and "PR
outcome" dashboard rows. The practical effect: the "PR outcome" row can never show "stale" no matter
how old the underlying pr_outcome sample actually is, silently misleading whoever reads the
dashboard about how current that number is. test/unit/phase7-calibration-loop.test.ts (grepped for
"fresh") never asserts a value for prOutcomeMetric.fresh — the constant has zero test coverage of
its own correctness.

Requirements

  • Compute prOutcomeMetric.fresh the same way historicalReplayMetric.fresh is computed: reuse
    isHistoricalReplayRunFresh (or extract a source-agnostic freshness helper if that reads better —
    the exported/internal function name is up to the implementation, but the freshness semantics must
    be identical for both sources: same max-age comparison against config.replayFreshnessMaxAgeHours,
    same handling of a missing/unparseable observedAt as NOT fresh) against
    prOutcomeMetric.observedAt and the same now/config inputs historicalReplayMetric.fresh
    already uses.
  • A null observedAt (no timestamp supplied for the pr_outcome source) must resolve to
    fresh: false, matching how isHistoricalReplayRunFresh already treats a missing/invalid
    observedAt for the historical-replay source.
  • Do not change prOutcomeDerived, accuracy, or sampleSize computation, and do not change
    historicalReplayMetric's own freshness logic.

Deliverables

  • prOutcomeMetric.fresh is true when input.prOutcome.observedAt is within
    config.replayFreshnessMaxAgeHours of now, and false when it is older, using the same
    comparison historicalReplayMetric.fresh uses.
  • prOutcomeMetric.fresh is false when input.prOutcome is present but its observedAt is
    null/missing/unparseable (matching the historical-replay source's existing handling of the
    equivalent case).
  • calibration-dashboard.ts's sourceRow rendering of the "PR outcome" row's freshness detail
    reflects the newly-computed value (no changes needed to calibration-dashboard.ts itself if
    the fix is confined to phase7-calibration-loop.ts — verify this is the case as part of this
    issue, and if it is not, fix whatever additional wiring is needed to make the dashboard
    actually display the real freshness).

All three Deliverables are required in this one PR — there is no narrower scope for this issue.

Test Coverage Requirements

packages/loopover-engine/src/** is measured by Codecov via two separate uploads whose hits are
unioned
— root test/** AND packages/loopover-engine/test/**. Add regression tests to
packages/loopover-engine/test/phase7-calibration-loop.test.ts (this file's existing test location)
asserting prOutcomeMetric.fresh for: a recent observedAt (within the max-age window) → true; a
stale observedAt (older than the max-age window) → false; and a null/missing observedAt
false. Also add or extend a test in test/unit/calibration-dashboard.test.ts (root-level)
confirming the "PR outcome" dashboard row renders "stale" when prOutcomeMetric.fresh is false
(previously impossible to test meaningfully, since the value was always true). Target 100% branch
coverage of the new freshness computation.

Expected Outcome

The calibration dashboard's "PR outcome" row can genuinely show "stale," matching the accuracy the
"Historical replay" row already has, so an operator reading the dashboard gets a truthful signal
about how current the pr_outcome calibration sample actually is.

Links & Resources

  • packages/loopover-engine/src/phase7-calibration-loop.ts (line 374 prOutcomeMetric.fresh,
    isHistoricalReplayRunFresh at line 293, historicalReplayMetric construction around line 410)
  • packages/loopover-engine/src/calibration-dashboard.ts (lines 42-48 sourceRow)
  • packages/loopover-engine/test/phase7-calibration-loop.test.ts
  • test/unit/calibration-dashboard.test.ts
  • Milestone: Miner Wave 4.6 — AMS Hardening Round 3

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions