Skip to content

feat(miner): leakage-safe task generation from real historical outcomes #3011

Description

@JSONbored

Context

Part of the internal historical-replay calibration harness roadmap. A frozen snapshot at commit T is only useful for calibration if there is a well-defined, real, subsequent history to score against, and if nothing in the frozen context lets the miner infer that future history by pattern-matching text rather than reasoning. This issue defines how freeze points are selected and how the frozen context is scrubbed before it is handed to the replay pipeline.

Goal

Select calibration-worthy freeze points and scrub forward references so replay cannot cheat off future information embedded in the frozen context.

Requirements

  • Define selection criteria for a freeze point T: minimum prior commit history depth, and a minimum number of subsequent revealed commits/PRs after T to serve as scorable ground truth.
  • Scrub the frozen context of forward references: issue/PR numbers and links, deep-links, and raw commit SHAs that only resolve to something created after T.
  • Mix freeze-point selection across two pools: repos with activity recent to the calibration run's own model training cutoff, and older or lower-traffic repos, to reduce the chance a judge or planner has memorized the actual outcome. Record which pool each freeze point was drawn from so downstream scoring can be sliced by pool.
  • Store the revealed post-T history (the ground truth) separately from the frozen pre-T snapshot so the replay pipeline never has both in scope at once.
  • Add a lint/check that fails a freeze-point candidate if scrubbing cannot remove all detected forward references.

Deliverables

  • Freeze-point selection function with configurable minimum prior-history and minimum revealed-history thresholds
  • Forward-reference scrubber for issue/PR links, deep-links, and raw SHAs, applied to all frozen-context text
  • Recency/older-repo pool mixing with pool provenance recorded per freeze point
  • Separation of frozen snapshot storage from revealed-ground-truth storage
  • Unit tests covering a freeze point with unscrubbable references (must be rejected) and one with none (must pass)

Expected outcomes

  • No frozen-context sample handed to a replay run contains a resolvable forward reference
  • Freeze-point pool is measurably mixed across recency bands, not concentrated in one

Part of the historical-replay calibration roadmap: #2994

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions