Context
Part of the internal historical-replay calibration harness roadmap. Once a replay run produces a plan and PR against a frozen snapshot, half of the calibration score should come from a deterministic, structural comparison, not an LLM judgment call, so the score has a stable, auditable floor. This issue is that structural half: compare what the miner's replayed output targeted against what the revealed post-T history shows actually changed.
Goal
Produce a deterministic structural score comparing the replayed plan/PR's targets against the real revealed changes.
Requirements
- Extract structural features from the miner's replayed plan/PR: modules or paths touched, and a classification of change kind (feature, fix, refactor, docs, etc.), reusing the same classification the miner already applies to its own submissions where available.
- Extract the same structural features from the revealed post-T history (the actual commits/PRs that followed T).
- Define a deterministic comparison function producing a numeric objective-anchor score from the overlap/divergence between replayed-target features and revealed-actual features.
- Ensure the comparison has no LLM call in its path; it must be reproducible byte-for-byte given the same two feature sets.
- Handle the case where the revealed history touches zero overlapping modules with the replayed plan (score floor, not an error).
- Log the intermediate feature extraction for both sides so a low score is auditable without re-running the comparison.
Deliverables
Expected outcomes
- Objective-anchor score is identical across repeated runs on the same inputs
- Score computation completes without any external model call
Part of the historical-replay calibration roadmap: #2994
Context
Part of the internal historical-replay calibration harness roadmap. Once a replay run produces a plan and PR against a frozen snapshot, half of the calibration score should come from a deterministic, structural comparison, not an LLM judgment call, so the score has a stable, auditable floor. This issue is that structural half: compare what the miner's replayed output targeted against what the revealed post-T history shows actually changed.
Goal
Produce a deterministic structural score comparing the replayed plan/PR's targets against the real revealed changes.
Requirements
Deliverables
Expected outcomes
Part of the historical-replay calibration roadmap: #2994