Skip to content

calibration: pure BacktestCase corpus builder from RuleFiredEvent/HumanOverrideEvent pairs #8083

Description

@JSONbored

Context

packages/loopover-engine/src/calibration/signal-tracking.ts (shipped in #7982) defines RuleFiredEvent, HumanOverrideEvent, and computeRulePrecision(ruleId, fired, overrides), which pairs fired events with overrides by matching ruleId (see the internal overrideMatchesRule helper) to compute an aggregate precision number. There is no equivalent that keeps each PAIRED case as an individual labeled record — which is what a backtest needs: a list of concrete "this rule fired against this target, and a human later said it was right/wrong" cases, replayable against a different candidate rule/classifier later (see the parent epic for why).

⚠️ Required pattern. This file must be pure TypeScript with no IO, no DB access, and no imports from src/, packages/loopover-miner, or any host adapter — only the existing RuleFiredEvent/HumanOverrideEvent types from signal-tracking.ts. A PR that adds any database, environment, or network access to this file does NOT satisfy this issue.

Requirements

  • Add a new file packages/loopover-engine/src/calibration/backtest-corpus.ts.
  • Export a type BacktestCase:
    export type BacktestCase = {
      ruleId: string;
      targetKey: string;
      outcome: string;
      label: "reversed" | "confirmed";
      firedAt: string;
      decidedAt: string;
      metadata?: Record<string, unknown>;
    };
    • outcome comes from the matching RuleFiredEvent.outcome.
    • label comes from the matching HumanOverrideEvent.verdict.
    • firedAt is the RuleFiredEvent.occurredAt; decidedAt is the HumanOverrideEvent.occurredAt.
    • metadata is the fired event's metadata field, omitted entirely from the object (not set to undefined) when the fired event has none — the same optional-property discipline RuleFiredEvent itself already uses.
  • Export a pure function:
    export function buildBacktestCorpus(
      ruleId: string,
      fired: readonly RuleFiredEvent[],
      overrides: readonly HumanOverrideEvent[],
    ): BacktestCase[]
    • Matching rule: a RuleFiredEvent and HumanOverrideEvent pair into one BacktestCase when both have ruleId equal to the function's ruleId argument AND the same targetKey.
    • A fired event with no matching override is excluded from the result (not included as an unlabeled case) — mirrors computeRulePrecision's "only the decided ones count" discipline (read that function's own doc comment in signal-tracking.ts before starting).
    • When a targetKey has multiple override events for the same rule (re-fired and re-judged more than once), pair each fired event with the override whose occurredAt is closest in time after that specific fired event's occurredAt; if none strictly follows it, fall back to the most recent override by occurredAt. Document this pairing rule in a doc comment above the function, in the same style as computeRulePrecision's own doc comment. Do not produce duplicate BacktestCases for the same fired event.
    • Only events where event.ruleId equals the function's ruleId argument are considered — mirror overrideMatchesRule's existing one-line filter (event.ruleId === ruleId) with a comment noting it mirrors that helper in signal-tracking.ts. Do not modify signal-tracking.ts to export overrideMatchesRule — this issue is additive-only in a new file.

Deliverables

  • packages/loopover-engine/src/calibration/backtest-corpus.ts with the BacktestCase type and buildBacktestCorpus function as specified above.
  • packages/loopover-engine/test/backtest-corpus.test.ts covering: no-override fired events excluded; a single fired+override pair produces one correctly-labeled case; multiple overrides for one target pair with the correct (nearest-following, else most-recent) override; events for a different ruleId are ignored; empty input arrays produce an empty array.
  • Add export * from "./calibration/backtest-corpus.js"; to packages/loopover-engine/src/index.ts, on its own new line immediately after the existing export * from "./calibration/signal-tracking.js"; line (line 165 as of this writing) — do not add it anywhere else in the file.

Test Coverage Requirements

99%+ patch coverage (branch-counted), this repo's standard gate — packages/loopover-engine is measured by Codecov exactly like the rest of the repo (see the already-merged packages/loopover-engine/test/signal-tracking.test.ts from #7982 as the precedent for this exact directory). Cover both branches of every conditional described above: has-override vs. no-override; single vs. multiple overrides (including the "no override strictly follows" fallback branch); matching vs. non-matching ruleId; present vs. absent metadata.

Expected Outcome

A reusable, pure function exists that turns the calibration module's raw event history into a labeled corpus of "rule fired, human later said right/wrong" cases — the direct input the backtest scorer (a follow-up issue in this epic) needs. No behavior change to any existing consumer; this is purely additive.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions