Skip to content

calibration: render a backtest score/comparison report as Markdown #8088

Description

@JSONbored

Context

BacktestScoreReport (#8085) and BacktestComparison (#8086) are both plain data — there is no human-readable rendering yet. This is the "receipt" a maintainer (and, per the parent epic's proposal, eventually an advisory CI comment) reads directly, so it needs to be a deterministic pure function producing stable Markdown, not ad-hoc console logging.

Requirements

  • Add a new file packages/loopover-engine/src/calibration/backtest-report.ts.
  • Export two functions:
    export function renderBacktestScoreReport(report: BacktestScoreReport): string
    export function renderBacktestComparison(comparison: BacktestComparison): string
  • renderBacktestScoreReport:
    • Outputs Markdown containing the rule ID, caseCount, all four confusion-matrix counts (truePositive, falsePositive, trueNegative, falseNegative), and precision/recall.
    • Render null precision or recall as the literal string N/A — never as 0, the word null, or an empty cell. This mirrors the null-is-not-zero discipline established in BacktestScoreReport itself (calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085).
    • Output layout (table vs. bullet list) is the implementer's choice, but must be covered by a snapshot-style test asserting the exact rendered string for at least one fixture.
  • renderBacktestComparison:
    • Outputs Markdown containing: the rule ID; every axis listed in regressedAxes under a clearly labeled "Regressed" section (or heading); every axis listed in improvedAxes under a clearly labeled, visually separate "Improved" section; and a closing line stating the verdict in words.
    • When verdict === "regressed", the closing line's text must contain the literal word REGRESSED (case-sensitive) and must state that the change should not be merged (e.g. Verdict: REGRESSED — do not merge) — this exact wording requirement exists so a future automated consumer (a follow-up CI-wiring issue) can reliably detect the regressed case by string match without re-implementing the comparison logic.
    • A regressed axis must never appear in the improved section and vice versa — a reader must never have to cross-reference the raw regressedAxes/improvedAxes arrays to know which is which from the rendered output alone.
    • When regressedAxes is empty, the output must not contain a "Regressed" section listing any axis (an empty section header with no items is acceptable, or omitting the section entirely — either is fine, but it must not read as if something regressed).
  • Both functions are pure (string in, string out; no IO, no wall-clock reads) and must produce byte-identical output for byte-identical input.

Deliverables

  • packages/loopover-engine/src/calibration/backtest-report.ts with both functions as specified above.
  • packages/loopover-engine/test/backtest-report.test.ts covering: a report with non-null precision/recall renders both numbers; a report with null precision/recall renders N/A for each, not 0; a "regressed" comparison's output contains the literal word REGRESSED; a comparison with a non-empty improvedAxes and empty regressedAxes renders as improved with no regressed axis claimed; calling either function twice with the same input produces identical output strings (byte-identical determinism).
  • Add export * from "./calibration/backtest-report.js"; to packages/loopover-engine/src/index.ts, on its own new line immediately after the export * from "./calibration/backtest-compare.js"; line added by calibration: Pareto-floor comparator between two BacktestScoreReports #8086.

Test Coverage Requirements

99%+ patch coverage (branch-counted), including both the null and non-null precision/recall render paths and all three verdict values ("improved", "regressed", "unchanged").

Expected Outcome

Any BacktestScoreReport or BacktestComparison can be turned into readable Markdown with one function call — the last piece the maintainer-only advisory CI wiring (a follow-up issue) needs before it can post anything human-readable.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions