Skip to content

feat(calibration): render backtest score/comparison reports as Markdown - #8120

Closed
kai392 wants to merge 1 commit into
JSONbored:mainfrom
kai392:feat/critical-issue-backtest-report
Closed

feat(calibration): render backtest score/comparison reports as Markdown#8120
kai392 wants to merge 1 commit into
JSONbored:mainfrom
kai392:feat/critical-issue-backtest-report

Conversation

@kai392

@kai392 kai392 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add packages/loopover-engine/src/calibration/backtest-report.ts with renderBacktestScoreReport and renderBacktestComparison: the deterministic, pure Markdown "receipt" renderers for BacktestScoreReport (calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085) and BacktestComparison (calibration: Pareto-floor comparator between two BacktestScoreReports #8086) — byte-identical output for byte-identical input, no IO, no wall-clock reads.
  • renderBacktestScoreReport renders the rule ID, caseCount, all four confusion-matrix counts, and precision/recall as a Markdown table, with null ratios rendered as the literal N/A — never 0, the word null, or an empty cell (the null-is-not-zero discipline from calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085), pinned by an exact-string snapshot test.
  • renderBacktestComparison renders clearly separated Regressed/Improved sections (an empty side renders (none), so an empty regression list can never read as if something regressed) and a closing verdict line; the regressed line is the pinned literal Verdict: REGRESSED — do not merge, so the follow-up CI-wiring consumer can detect the regressed case by string match without re-implementing the comparison.
  • Barrel export added to packages/loopover-engine/src/index.ts on its own line immediately after the backtest-compare.js line, per the issue's deliverable.

Closes #8088

Test plan

  • packages/loopover-engine/test/backtest-report.test.ts (node:test, per the issue's deliverable): exact-string snapshot of a non-null report; N/A (never 0/null/empty) for null ratios; the literal REGRESSED wording on a regressed comparison; improved comparison claims no regressed axis; unchanged comparison lists no axis on either side; byte-identical determinism for both renderers
  • test/unit/backtest-report-engine.test.ts (vitest, importing the engine source): same cases — this is what exercises the .ts for codecov/patch, since @loopover/engine imports resolve to dist
  • Local coverage on the new file: 6/6 lines, 8/8 branches (both null-render sides, all three verdict arms, both empty/non-empty section sides)
  • npm run test --workspace @loopover/engine green; npm run typecheck green
  • CI validate

@kai392
kai392 requested a review from JSONbored as a code owner July 22, 2026 23:07
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#8116 was opened first, which links to close the same issue.

@JSONbored JSONbored closed this Jul 22, 2026
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.01%. Comparing base (e2d8ef8) to head (c774db9).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #8120       +/-   ##
===========================================
+ Coverage   64.35%   92.01%   +27.65%     
===========================================
  Files         759      760        +1     
  Lines       77327    77335        +8     
  Branches    23376    23380        +4     
===========================================
+ Hits        49766    71162    +21396     
+ Misses      23344     5061    -18283     
+ Partials     4217     1112     -3105     
Flag Coverage Δ
shard-1 57.41% <0.00%> (+3.43%) ⬆️
shard-2 51.10% <0.00%> (?)
shard-3 54.45% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...loopover-engine/src/calibration/backtest-report.ts 100.00% <100.00%> (ø)

... and 438 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

calibration: render a backtest score/comparison report as Markdown

3 participants