calibration: deterministic seeded held-out/visible split of the backtest corpus

[JSONbored]

Context

A held-out/visible split — evaluate against a visible set for local iteration, keep a separate held-out set for the real gate check — prevents a rule fix from being hand-tuned to just the specific incidents already known about. packages/loopover-engine/src/calibration/backtest-corpus.ts's BacktestCase[] (#8083) has no such split today. packages/loopover-engine/src/miner/deny-hook-synthesis.ts already has a deterministic-hash pattern in this same package (createHash("sha256").update(...).digest("hex"), see its digest local variable) — this issue reuses that exact hashing approach for a deterministic corpus partition.

Requirements

• Add a new file packages/loopover-engine/src/calibration/backtest-split.ts.
• Export a function:

  export function splitBacktestCorpus(
    cases: readonly BacktestCase[],
    heldOutFraction: number,
    seed: string,
  ): { visible: BacktestCase[]; heldOut: BacktestCase[] }

  • Throws a plain Error if heldOutFraction is not within the inclusive range [0, 1] — the thrown message must include the invalid value.
  • For each case, compute createHash("sha256").update(`${seed}:${case.ruleId}:${case.targetKey}`).digest("hex") (createHash from node:crypto, same algorithm as deny-hook-synthesis.ts's own usage — do not use a different hash algorithm). Take the first 8 hex characters of the digest, parse them as a base-16 integer (parseInt(hex8, 16)), and divide by 0xffffffff to get a value in [0, 1). A case is assigned to heldOut when that value is strictly less than heldOutFraction; otherwise it goes to visible.
  • Determinism (must be covered by a test, not just asserted in prose): calling this function twice with the same cases, heldOutFraction, and seed must produce byte-identical output, including the relative order of cases within each output array — preserve each case's original position from the input cases array within its assigned bucket; do not sort or shuffle.
  • A case's split assignment must depend only on (seed, ruleId, targetKey) — never on its position in the input array, never on cases.length — so a corpus that grows over time doesn't reshuffle which already-processed cases were previously held out. This is the entire reason to hash the content instead of, e.g., taking every Nth case or using an index-based random split.
  • Pure — no IO, no Math.random(), no wall-clock reads.

Deliverables

• packages/loopover-engine/src/calibration/backtest-split.ts with splitBacktestCorpus as specified above.
• packages/loopover-engine/test/backtest-split.test.ts covering: heldOutFraction: 0 → every case in visible, none in heldOut; heldOutFraction: 1 → every case in heldOut; the same cases/heldOutFraction/seed called twice → byte-identical output (the determinism requirement above); a different seed with the same cases and a fractional heldOutFraction (e.g. 0.5) produces a different split for at least one case, using a fixture with enough cases (at least ~10) to make that reliably true rather than flaky; an out-of-range heldOutFraction (e.g. -0.1 and separately 1.5) throws in both directions; original per-bucket ordering is preserved for a fixture where at least one case lands in each bucket.

Test Coverage Requirements

99%+ patch coverage (branch-counted), including the throw branch (both the below-0 and above-1 cases).

Expected Outcome

A reproducible way to carve any backtest corpus into a visible-for-iteration slice and a held-out slice, without needing a second, separately-maintained "private" corpus.

Links & Resources

• Parent epic: Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082
• Depends on: calibration: pure BacktestCase corpus builder from RuleFiredEvent/HumanOverrideEvent pairs #8083
• packages/loopover-engine/src/miner/deny-hook-synthesis.ts (the createHash pattern this reuses — read the surrounding function for the exact usage before starting)

[linear-code]

JSO-5585