Skip to content

feat(miner): wire the historical-replay harness into the live Phase 7 calibration loop #3014

Description

@JSONbored

Context

Part of the internal historical-replay calibration harness roadmap. The freeze/snapshot mechanism, leakage-safe task generation, objective-anchor scoring, and pairwise-judge scoring layer are each internally useful, but none of them change miner behavior until the composite calibration score they produce is actually consumed by the live Phase 7 calibration loop alongside, or in place of, the current passive pr_outcome signal. This is the highest-leverage issue in the epic: it is the piece that turns the harness from an internal research exercise into something that actually gates and changes miner behavior safely.

Goal

Wire the historical-replay composite calibration score into the live Phase 7 calibration loop as a first-class input, and gate future autonomy-level increases on it.

Requirements

  • Add the historical-replay composite score as an input to the existing Phase 7 calibration pipeline alongside the current pr_outcome-derived signal, not as a silent replacement.
  • Expose a tracked calibration-accuracy metric derived from the combined signal, anchored explicitly against the documented 62% baseline so regressions and improvements are visible over time.
  • Define an explicit numeric threshold on this metric that must be crossed before any autonomy-level increase is permitted, and enforce it as a hard precondition in the autonomy-level change path rather than an advisory note.
  • Schedule or trigger historical-replay runs on a cadence sufficient to keep the composite score fresh relative to pr_outcome's natural arrival rate.
  • Record provenance per calibration data point (historical-replay vs. live pr_outcome) so the tracked metric can be broken down by source.
  • Add an explicit fallback/hold behavior if the replay harness is unavailable or degraded: autonomy-level gating must fail closed, not silently fall back to pr_outcome-only without flagging it.

Deliverables

  • Historical-replay composite score integrated as an input to the Phase 7 calibration pipeline
  • Tracked calibration-accuracy metric anchored against the 62% baseline, broken down by signal source
  • Explicit threshold enforced as a hard precondition on autonomy-level increases
  • Scheduled/triggered replay run cadence
  • Fail-closed behavior when the replay harness is degraded or unavailable, with an explicit flag rather than silent fallback

Expected outcomes

  • Autonomy-level increases are blocked whenever the tracked calibration-accuracy metric is below the configured threshold
  • Calibration-accuracy metric is visible broken down by historical-replay vs. live pr_outcome source

Part of the historical-replay calibration roadmap: #2994

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:priorityMaintainer-selected Gittensor priority — scores a 1.5x multiplier.

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions