Context
Part of the internal historical-replay calibration harness roadmap. The objective-anchor score captures structural overlap but not judgment quality, for example whether the miner's replayed reasoning correctly anticipated the real fix's approach even when file-level overlap is partial. This issue adds an LLM-judged half of the score, with an explicit defense against order-swap presentation bias, and combines both halves into one composite calibration score.
Goal
Add an LLM pairwise-judge score, defended against presentation-order bias, and combine it with the objective-anchor score into one composite calibration score.
Requirements
- Present the judge with the miner's replayed reasoning/output and the real revealed outcome as a pair, run once in each presentation order (A-then-B and B-then-A).
- Count a pairwise judgment as valid only if the verdict is stable across both orderings; discard and flag order-unstable judgments rather than averaging them in.
- Define the judge's scoring rubric to be scoped to plan/reasoning quality relative to the revealed outcome, not a restatement of the objective-anchor comparison.
- Combine the validated pairwise-judge score with the objective-anchor score (from the objective-anchor scoring issue) into a single composite calibration score, with the combination weights configurable rather than hardcoded.
- Track and expose the order-instability rate (fraction of judgments discarded for flipping under swap) as its own metric, not silently dropped.
- Cap per-run judge cost/latency with a bounded retry policy for instability, rather than swapping indefinitely.
Deliverables
Expected outcomes
- Order-unstable judgments are excluded from the composite score rather than averaged in
- Composite calibration score is available per replay run alongside its two component scores
Part of the historical-replay calibration roadmap: #2994
Context
Part of the internal historical-replay calibration harness roadmap. The objective-anchor score captures structural overlap but not judgment quality, for example whether the miner's replayed reasoning correctly anticipated the real fix's approach even when file-level overlap is partial. This issue adds an LLM-judged half of the score, with an explicit defense against order-swap presentation bias, and combines both halves into one composite calibration score.
Goal
Add an LLM pairwise-judge score, defended against presentation-order bias, and combine it with the objective-anchor score into one composite calibration score.
Requirements
Deliverables
Expected outcomes
Part of the historical-replay calibration roadmap: #2994