You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 2 names realized history as ground truth. This issue derives it: for each work unit in a snapshot, what the maintainer actually did within the prediction horizon — the labels every proposal is scored against.
Reuse the established provenance rules, do not invent new ones. The existing corpus builder already resolves human-override and reversal provenance (reversal_reopened / reversal_reverted / reversal_superseded); this must follow the same rules so benchmark ground truth and internal backtest ground truth can never disagree about the same event.
Reversal-aware: a merge later reverted is not scored as a clean merge. The realized outcome is the settled state at horizon end, with the reversal recorded — mirroring how public-rule-precision.ts already treats reversals as the thing that makes precision honest.
Explicit unresolved class: a work unit with no maintainer action within the horizon is unresolved and is excluded from the scoring denominator, never silently counted as a correct abstention or an incorrect prediction. An unresolved rate that climbs is itself a signal the horizon is wrong.
Pure core + IO wrapper, 100% branch coverage, including the reversal and unresolved paths.
Deliverables
Realized-outcome extractor (pure core + CLI) emitting a labeled ground-truth set keyed to a snapshot's work units
Invariant test: ground truth for a given (snapshot, horizon) is stable regardless of when extraction runs, provided the horizon has elapsed
Tests covering reversal-aware settlement and the unresolved exclusion
Expected outcome
A stable, provenance-consistent label set that makes an agent's proposals scoreable — and that cannot drift from how the rest of the system already understands "what actually happened."
References
Sub-issue of #9216. Provenance rules: src/review/public-rule-precision.ts, packages/loopover-engine/src/calibration/backtest-corpus.ts. Depends on #9259 (snapshots) and #9260 (action vocabulary). Epic #8534.
Context
Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 2 names realized history as ground truth. This issue derives it: for each work unit in a snapshot, what the maintainer actually did within the prediction horizon — the labels every proposal is scored against.
Requirements
reversal_reopened/reversal_reverted/reversal_superseded); this must follow the same rules so benchmark ground truth and internal backtest ground truth can never disagree about the same event.public-rule-precision.tsalready treats reversals as the thing that makes precision honest.unresolvedand is excluded from the scoring denominator, never silently counted as a correct abstention or an incorrect prediction. An unresolved rate that climbs is itself a signal the horizon is wrong.Deliverables
Expected outcome
A stable, provenance-consistent label set that makes an agent's proposals scoreable — and that cannot drift from how the rest of the system already understands "what actually happened."
References
Sub-issue of #9216. Provenance rules:
src/review/public-rule-precision.ts,packages/loopover-engine/src/calibration/backtest-corpus.ts. Depends on #9259 (snapshots) and #9260 (action vocabulary). Epic #8534.