Skip to content

feat(miner): objective-anchor scoring for replayed plans against revealed history #3012

Description

@JSONbored

Context

Part of the internal historical-replay calibration harness roadmap. Once a replay run produces a plan and PR against a frozen snapshot, half of the calibration score should come from a deterministic, structural comparison, not an LLM judgment call, so the score has a stable, auditable floor. This issue is that structural half: compare what the miner's replayed output targeted against what the revealed post-T history shows actually changed.

Goal

Produce a deterministic structural score comparing the replayed plan/PR's targets against the real revealed changes.

Requirements

  • Extract structural features from the miner's replayed plan/PR: modules or paths touched, and a classification of change kind (feature, fix, refactor, docs, etc.), reusing the same classification the miner already applies to its own submissions where available.
  • Extract the same structural features from the revealed post-T history (the actual commits/PRs that followed T).
  • Define a deterministic comparison function producing a numeric objective-anchor score from the overlap/divergence between replayed-target features and revealed-actual features.
  • Ensure the comparison has no LLM call in its path; it must be reproducible byte-for-byte given the same two feature sets.
  • Handle the case where the revealed history touches zero overlapping modules with the replayed plan (score floor, not an error).
  • Log the intermediate feature extraction for both sides so a low score is auditable without re-running the comparison.

Deliverables

  • Structural feature extractor for replayed plan/PR (modules touched, change kind)
  • Structural feature extractor for revealed post-T history
  • Deterministic objective-anchor scoring function with no LLM dependency
  • Audit log of extracted features for both sides per scored run
  • Unit tests covering full overlap, zero overlap, and partial-module-path overlap cases

Expected outcomes

  • Objective-anchor score is identical across repeated runs on the same inputs
  • Score computation completes without any external model call

Part of the historical-replay calibration roadmap: #2994

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions