You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 2 of that issue is explicit that scoring reuses the existing confusion-matrix and Pareto-floor primitives rather than growing a parallel scoring system. This issue is that adapter plus the multi-class handling the existing binary primitives do not cover.
Requirements
Reuse, do not reimplement.scoreBacktest and compareBacktestScores (packages/loopover-engine/src/calibration/) stay the scoring core. A second scoring implementation is precisely the drift this issue exists to prevent — if a new primitive is genuinely needed, it is added there and shared, not forked here.
Multi-class over a closed action set: the existing primitives are binary (positive class reversed). Proposals span five actions, so define per-action one-vs-rest scoring, then the aggregate. Document the aggregation choice (macro vs micro) and why — they answer different questions about a benchmark, and picking one silently would be the kind of unexamined default this project's scoring discipline exists to avoid.
Pareto floor applies to agent comparison too: an agent that gains on one action while regressing another does not get a free pass, matching compareBacktestScores's existing "any regressed axis ⇒ regressed" verdict.
Pure, deterministic, 100% branch coverage, including every null-metric arm.
Deliverables
Benchmark scorer built on the existing primitives, in packages/loopover-engine/src/calibration/
A recorded decision on the aggregation choice, in the module's own header comment
Unit tests at 100% branch coverage, including a test asserting the scorer and the internal backtest agree on an equivalent binary case (the anti-drift guard)
Expected outcome
Benchmark scores that are mechanically the same kind of number as the internal backtest's, so a claim about one is a claim about the other.
References
Sub-issue of #9216. Primitives: backtest-score.ts, backtest-compare.ts. Semantics: #9215. Depends on #9260, #9261. Epic #8534.
Context
Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 2 of that issue is explicit that scoring reuses the existing confusion-matrix and Pareto-floor primitives rather than growing a parallel scoring system. This issue is that adapter plus the multi-class handling the existing binary primitives do not cover.
Requirements
scoreBacktestandcompareBacktestScores(packages/loopover-engine/src/calibration/) stay the scoring core. A second scoring implementation is precisely the drift this issue exists to prevent — if a new primitive is genuinely needed, it is added there and shared, not forked here.reversed). Proposals span five actions, so define per-action one-vs-rest scoring, then the aggregate. Document the aggregation choice (macro vs micro) and why — they answer different questions about a benchmark, and picking one silently would be the kind of unexamined default this project's scoring discipline exists to avoid.coverage = decided / (decided + abstained), abstentions never folded into errors,nullnever rendered as0below the sample floor.compareBacktestScores's existing "any regressed axis ⇒ regressed" verdict.null-metric arm.Deliverables
packages/loopover-engine/src/calibration/Expected outcome
Benchmark scores that are mechanically the same kind of number as the internal backtest's, so a claim about one is a claim about the other.
References
Sub-issue of #9216. Primitives:
backtest-score.ts,backtest-compare.ts. Semantics: #9215. Depends on #9260, #9261. Epic #8534.