benchmark: anti-overfit controls — seeded held-out, per-repo splits, submission caps, rotation

[JSONbored]

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 3 of that issue: a public benchmark with a leaderboard creates a direct incentive to overfit, and an overfit-to-the-leaderboard agent is worse than useless because it looks best exactly when it generalizes worst. The internal backtest gate already enforces this discipline; this issue applies the same controls to external submissions, plus the ones only an adversarial public benchmark needs.

Requirements

1. Seeded held-out split, reusing splitBacktestCorpus — assignment is sha256(seed:ruleId:targetKey)-derived, so it is deterministic, unguessable without the seed, and identical across runs. Do not invent a second split mechanism.
2. Held-out results are not published per-submission. A leaderboard that reports held-out scores on every attempt converts the held-out set into a visible set within a few dozen submissions. Publish visible-split scores; report held-out only on a fixed cadence or at evaluation close.
3. Per-repo splits: an agent must not train on repo A's history and be scored on repo A's future alone. Cross-repo generalization is the property worth measuring; document the split policy explicitly.
4. Submission caps per (agent, benchmark window), with the rationale recorded: unbounded resubmission against a fixed corpus is gradient descent on the test set by brute force.
5. Rotation policy: how and when snapshots/windows are retired and replaced, so a benchmark that has been public for a year is not still the scoring basis.

Deliverables

• Anti-overfit controls implemented against the existing seeded-split primitive
• A recorded decision on each of: publication cadence for held-out scores, per-repo split policy, submission cap, rotation cadence — each with its reasoning, since these are the knobs an adversary optimizes against
• Tests covering split determinism, cap enforcement, and that held-out membership is not inferable from published output

Expected outcome

A benchmark whose leaderboard position means "generalizes," not "has submitted the most times."

References

Sub-issue of #9216. Primitive: packages/loopover-engine/src/calibration/backtest-split.ts. Existing discipline: .github/workflows/calibration-advisory.yml. Epic #8534.

[linear-code]

JSO-6427