Skip to content

feat(miner): pairwise-judge scoring layer with order-swap bias defense #3013

Description

@JSONbored

Context

Part of the internal historical-replay calibration harness roadmap. The objective-anchor score captures structural overlap but not judgment quality, for example whether the miner's replayed reasoning correctly anticipated the real fix's approach even when file-level overlap is partial. This issue adds an LLM-judged half of the score, with an explicit defense against order-swap presentation bias, and combines both halves into one composite calibration score.

Goal

Add an LLM pairwise-judge score, defended against presentation-order bias, and combine it with the objective-anchor score into one composite calibration score.

Requirements

  • Present the judge with the miner's replayed reasoning/output and the real revealed outcome as a pair, run once in each presentation order (A-then-B and B-then-A).
  • Count a pairwise judgment as valid only if the verdict is stable across both orderings; discard and flag order-unstable judgments rather than averaging them in.
  • Define the judge's scoring rubric to be scoped to plan/reasoning quality relative to the revealed outcome, not a restatement of the objective-anchor comparison.
  • Combine the validated pairwise-judge score with the objective-anchor score (from the objective-anchor scoring issue) into a single composite calibration score, with the combination weights configurable rather than hardcoded.
  • Track and expose the order-instability rate (fraction of judgments discarded for flipping under swap) as its own metric, not silently dropped.
  • Cap per-run judge cost/latency with a bounded retry policy for instability, rather than swapping indefinitely.

Deliverables

  • Pairwise judge invocation run in both presentation orders per sample
  • Order-swap stability check that discards and flags unstable verdicts
  • Composite scoring function combining objective-anchor and validated pairwise-judge scores with configurable weights
  • Order-instability rate tracked and surfaced as a metric
  • Unit tests covering a stable-verdict pair, an order-flipping pair, and the retry/cap boundary

Expected outcomes

  • Order-unstable judgments are excluded from the composite score rather than averaged in
  • Composite calibration score is available per replay run alongside its two component scores

Part of the historical-replay calibration roadmap: #2994

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions