You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
BacktestScoreReport (#8085) and BacktestComparison (#8086) are both plain data — there is no human-readable rendering yet. This is the "receipt" a maintainer (and, per the parent epic's proposal, eventually an advisory CI comment) reads directly, so it needs to be a deterministic pure function producing stable Markdown, not ad-hoc console logging.
Requirements
Add a new file packages/loopover-engine/src/calibration/backtest-report.ts.
Outputs Markdown containing the rule ID, caseCount, all four confusion-matrix counts (truePositive, falsePositive, trueNegative, falseNegative), and precision/recall.
Output layout (table vs. bullet list) is the implementer's choice, but must be covered by a snapshot-style test asserting the exact rendered string for at least one fixture.
renderBacktestComparison:
Outputs Markdown containing: the rule ID; every axis listed in regressedAxes under a clearly labeled "Regressed" section (or heading); every axis listed in improvedAxes under a clearly labeled, visually separate "Improved" section; and a closing line stating the verdict in words.
When verdict === "regressed", the closing line's text must contain the literal word REGRESSED (case-sensitive) and must state that the change should not be merged (e.g. Verdict: REGRESSED — do not merge) — this exact wording requirement exists so a future automated consumer (a follow-up CI-wiring issue) can reliably detect the regressed case by string match without re-implementing the comparison logic.
A regressed axis must never appear in the improved section and vice versa — a reader must never have to cross-reference the raw regressedAxes/improvedAxes arrays to know which is which from the rendered output alone.
When regressedAxes is empty, the output must not contain a "Regressed" section listing any axis (an empty section header with no items is acceptable, or omitting the section entirely — either is fine, but it must not read as if something regressed).
Both functions are pure (string in, string out; no IO, no wall-clock reads) and must produce byte-identical output for byte-identical input.
Deliverables
packages/loopover-engine/src/calibration/backtest-report.ts with both functions as specified above.
packages/loopover-engine/test/backtest-report.test.ts covering: a report with non-null precision/recall renders both numbers; a report with nullprecision/recall renders N/A for each, not 0; a "regressed" comparison's output contains the literal word REGRESSED; a comparison with a non-empty improvedAxes and empty regressedAxes renders as improved with no regressed axis claimed; calling either function twice with the same input produces identical output strings (byte-identical determinism).
99%+ patch coverage (branch-counted), including both the null and non-null precision/recall render paths and all three verdict values ("improved", "regressed", "unchanged").
Expected Outcome
Any BacktestScoreReport or BacktestComparison can be turned into readable Markdown with one function call — the last piece the maintainer-only advisory CI wiring (a follow-up issue) needs before it can post anything human-readable.
Context
BacktestScoreReport(#8085) andBacktestComparison(#8086) are both plain data — there is no human-readable rendering yet. This is the "receipt" a maintainer (and, per the parent epic's proposal, eventually an advisory CI comment) reads directly, so it needs to be a deterministic pure function producing stable Markdown, not ad-hoc console logging.Requirements
packages/loopover-engine/src/calibration/backtest-report.ts.renderBacktestScoreReport:caseCount, all four confusion-matrix counts (truePositive,falsePositive,trueNegative,falseNegative), andprecision/recall.nullprecision or recall as the literal stringN/A— never as0, the wordnull, or an empty cell. This mirrors the null-is-not-zero discipline established inBacktestScoreReportitself (calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085).renderBacktestComparison:regressedAxesunder a clearly labeled "Regressed" section (or heading); every axis listed inimprovedAxesunder a clearly labeled, visually separate "Improved" section; and a closing line stating theverdictin words.verdict === "regressed", the closing line's text must contain the literal wordREGRESSED(case-sensitive) and must state that the change should not be merged (e.g.Verdict: REGRESSED — do not merge) — this exact wording requirement exists so a future automated consumer (a follow-up CI-wiring issue) can reliably detect the regressed case by string match without re-implementing the comparison logic.regressedAxes/improvedAxesarrays to know which is which from the rendered output alone.regressedAxesis empty, the output must not contain a "Regressed" section listing any axis (an empty section header with no items is acceptable, or omitting the section entirely — either is fine, but it must not read as if something regressed).Deliverables
packages/loopover-engine/src/calibration/backtest-report.tswith both functions as specified above.packages/loopover-engine/test/backtest-report.test.tscovering: a report with non-nullprecision/recallrenders both numbers; a report withnullprecision/recallrendersN/Afor each, not0; a"regressed"comparison's output contains the literal wordREGRESSED; a comparison with a non-emptyimprovedAxesand emptyregressedAxesrenders as improved with no regressed axis claimed; calling either function twice with the same input produces identical output strings (byte-identical determinism).export * from "./calibration/backtest-report.js";topackages/loopover-engine/src/index.ts, on its own new line immediately after theexport * from "./calibration/backtest-compare.js";line added by calibration: Pareto-floor comparator between two BacktestScoreReports #8086.Test Coverage Requirements
99%+ patch coverage (branch-counted), including both the
nulland non-null precision/recall render paths and all threeverdictvalues ("improved","regressed","unchanged").Expected Outcome
Any
BacktestScoreReportorBacktestComparisoncan be turned into readable Markdown with one function call — the last piece the maintainer-only advisory CI wiring (a follow-up issue) needs before it can post anything human-readable.Links & Resources