Skip to content

benchmark: realized-history ground truth — reversal-aware outcomes per work unit #9261

Description

@JSONbored

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 2 names realized history as ground truth. This issue derives it: for each work unit in a snapshot, what the maintainer actually did within the prediction horizon — the labels every proposal is scored against.

Requirements

  1. Ground truth from realized outcomes, in the same action vocabulary as benchmark: maintainer action-space schema + proposal validator #9260's schema, so proposal and outcome are directly comparable.
  2. Reuse the established provenance rules, do not invent new ones. The existing corpus builder already resolves human-override and reversal provenance (reversal_reopened / reversal_reverted / reversal_superseded); this must follow the same rules so benchmark ground truth and internal backtest ground truth can never disagree about the same event.
  3. Reversal-aware: a merge later reverted is not scored as a clean merge. The realized outcome is the settled state at horizon end, with the reversal recorded — mirroring how public-rule-precision.ts already treats reversals as the thing that makes precision honest.
  4. Explicit unresolved class: a work unit with no maintainer action within the horizon is unresolved and is excluded from the scoring denominator, never silently counted as a correct abstention or an incorrect prediction. An unresolved rate that climbs is itself a signal the horizon is wrong.
  5. Pure core + IO wrapper, 100% branch coverage, including the reversal and unresolved paths.

Deliverables

  • Realized-outcome extractor (pure core + CLI) emitting a labeled ground-truth set keyed to a snapshot's work units
  • Invariant test: ground truth for a given (snapshot, horizon) is stable regardless of when extraction runs, provided the horizon has elapsed
  • Tests covering reversal-aware settlement and the unresolved exclusion

Expected outcome

A stable, provenance-consistent label set that makes an agent's proposals scoreable — and that cannot drift from how the rest of the system already understands "what actually happened."

References

Sub-issue of #9216. Provenance rules: src/review/public-rule-precision.ts, packages/loopover-engine/src/calibration/backtest-corpus.ts. Depends on #9259 (snapshots) and #9260 (action vocabulary). Epic #8534.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions