Skip to content

benchmark: proposal scorer — multi-class over the existing confusion-matrix + Pareto primitives #9262

Description

@JSONbored

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 2 of that issue is explicit that scoring reuses the existing confusion-matrix and Pareto-floor primitives rather than growing a parallel scoring system. This issue is that adapter plus the multi-class handling the existing binary primitives do not cover.

Requirements

  1. Reuse, do not reimplement. scoreBacktest and compareBacktestScores (packages/loopover-engine/src/calibration/) stay the scoring core. A second scoring implementation is precisely the drift this issue exists to prevent — if a new primitive is genuinely needed, it is added there and shared, not forked here.
  2. Multi-class over a closed action set: the existing primitives are binary (positive class reversed). Proposals span five actions, so define per-action one-vs-rest scoring, then the aggregate. Document the aggregation choice (macro vs micro) and why — they answer different questions about a benchmark, and picking one silently would be the kind of unexamined default this project's scoring discipline exists to avoid.
  3. Coverage and abstention per Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215: coverage = decided / (decided + abstained), abstentions never folded into errors, null never rendered as 0 below the sample floor.
  4. Pareto floor applies to agent comparison too: an agent that gains on one action while regressing another does not get a free pass, matching compareBacktestScores's existing "any regressed axis ⇒ regressed" verdict.
  5. Pure, deterministic, 100% branch coverage, including every null-metric arm.

Deliverables

  • Benchmark scorer built on the existing primitives, in packages/loopover-engine/src/calibration/
  • A recorded decision on the aggregation choice, in the module's own header comment
  • Unit tests at 100% branch coverage, including a test asserting the scorer and the internal backtest agree on an equivalent binary case (the anti-drift guard)

Expected outcome

Benchmark scores that are mechanically the same kind of number as the internal backtest's, so a claim about one is a claim about the other.

References

Sub-issue of #9216. Primitives: backtest-score.ts, backtest-compare.ts. Semantics: #9215. Depends on #9260, #9261. Epic #8534.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions