calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus

[JSONbored]

Context

packages/loopover-engine/src/calibration/backtest-corpus.ts's BacktestCase[] (#8083) is a labeled history: "this rule fired against this target, a human later said reversed/confirmed." There is no way yet to ask "if a DIFFERENT/proposed version of this rule had been run against the same targets, would it have gotten more of them right?" This issue adds the pure scoring function that answers that, replaying a caller-supplied classifier over the corpus and comparing against the real labels — mirrors src/review/auto-tune.ts's GateEvalRow confusion-matrix shape (wouldMerge/mergeConfirmed/mergeFalse/decided/mergePrecision), but at a backtest-replay grain instead of a live-eval grain.

Requirements

• Add a new file packages/loopover-engine/src/calibration/backtest-score.ts.
• Export a type:

  // Convention: "reversed" is the positive class. A classifier that correctly predicts a case's real
  // label of "reversed" (i.e. correctly identifies that the rule's original firing was WRONG) is a true
  // positive. This is a deliberate, non-obvious choice — keep this comment attached to the type.
  export type BacktestScoreReport = {
    ruleId: string;
    caseCount: number;
    truePositive: number;
    falsePositive: number;
    trueNegative: number;
    falseNegative: number;
    precision: number | null;
    recall: number | null;
  };

• Export a function:

  export function scoreBacktest(
    ruleId: string,
    cases: readonly BacktestCase[],
    classify: (backtestCase: BacktestCase) => "reversed" | "confirmed",
  ): BacktestScoreReport

  • Runs classify(case) for every case in cases whose ruleId matches the function's ruleId argument, compares the result to case.label, and accumulates the four confusion-matrix counts (classify result "reversed" + label "reversed" → truePositive; "reversed" + "confirmed" → falsePositive; "confirmed" + "confirmed" → trueNegative; "confirmed" + "reversed" → falseNegative).
  • precision = truePositive / (truePositive + falsePositive), null when that sum is 0. recall = truePositive / (truePositive + falseNegative), null when that sum is 0. This is the same "null when the denominator is 0, never coerced to 0 or 1" discipline as RulePrecisionReport.precision in signal-tracking.ts — read that field's doc comment before starting and follow the same pattern.
  • Cases whose ruleId does not match the function's ruleId argument are excluded from every count (not scored at all) — mirrors the same defensive filtering computeRulePrecision does for overrides.
  • caseCount is the number of cases actually scored (i.e., after the ruleId filter above), not cases.length.
  • Pure — no IO, no randomness, no wall-clock reads. classify must be a plain synchronous function (not async) — every case must be scorable without I/O, so a caller can score thousands of historical cases against a fast, in-memory candidate rule implementation.

Deliverables

• packages/loopover-engine/src/calibration/backtest-score.ts with BacktestScoreReport and scoreBacktest as specified above.
• packages/loopover-engine/test/backtest-score.test.ts covering: an all-correct classifier produces precision: 1, recall: 1; an all-wrong classifier produces precision: 0, recall: 0 (with the correct false-positive/false-negative counts, not just the aggregate numbers); a mixed classifier produces the expected four counts; an empty cases array produces all counts at 0 and precision/recall both null; cases with a non-matching ruleId are excluded from every count including caseCount.
• Add export * from "./calibration/backtest-score.js"; to packages/loopover-engine/src/index.ts, on its own new line immediately after the export * from "./calibration/backtest-corpus.js"; line added by calibration: pure BacktestCase corpus builder from RuleFiredEvent/HumanOverrideEvent pairs #8083.

Test Coverage Requirements

99%+ patch coverage (branch-counted) — packages/loopover-engine follows this repo's standard Codecov gate. Cover all four confusion-matrix branches plus both zero-denominator (null precision, null recall) paths as explicit, separate test cases — do not rely on one mixed fixture to implicitly cover all of them.

Expected Outcome

Given a corpus and any candidate classifier function, produce an objective, reproducible confusion-matrix score — the primitive a Pareto-floor comparison (a follow-up issue) and the eventual advisory CI check both consume.

Links & Resources

• Parent epic: Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082
• Depends on: calibration: pure BacktestCase corpus builder from RuleFiredEvent/HumanOverrideEvent pairs #8083
• src/review/auto-tune.ts (GateEvalRow, the confusion-matrix shape this mirrors — read it before starting)
• packages/loopover-engine/src/calibration/signal-tracking.ts (RulePrecisionReport.precision, the null-discipline to mirror)

[linear-code]

JSO-5583