You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parent: #8828 (Phase 2 — regression gate). ORB's decision policy ships without an evaluation gate, in a repo that enforces 99% patch coverage on everything else. The existing threshold/logic backtests (#8082 epic) replay rule changes; this extends that into a versioned golden corpus gating every gate-affecting change.
CI job on gate/prompt/ruleset/config changes (path-filtered like backtest-logic-check): replay the corpus, fail on any known-bad case flipping to merge, and fail on merge-precision regression beyond a stated tolerance.
Corpus additions follow the loop: production failure → label → corpus entry; document the workflow in the backtest-calibration docs page.
Tiered: fast subset on every PR; full corpus on model-version or prompt-template changes (the highest-risk uncontrolled inputs).
Test Coverage Requirements
99%+ patch coverage, branch-counted, on the runner and corpus loader; the corpus format gets a schema validation test.
Expected Outcome
No gate-affecting change ships without demonstrating it does not reintroduce a known failure — the same discipline the repo already applies to code, applied to judgment.
Context
Parent: #8828 (Phase 2 — regression gate). ORB's decision policy ships without an evaluation gate, in a repo that enforces 99% patch coverage on everything else. The existing threshold/logic backtests (#8082 epic) replay rule changes; this extends that into a versioned golden corpus gating every gate-affecting change.
Requirements
Test Coverage Requirements
99%+ patch coverage, branch-counted, on the runner and corpus loader; the corpus format gets a schema validation test.
Expected Outcome
No gate-affecting change ships without demonstrating it does not reintroduce a known failure — the same discipline the repo already applies to code, applied to judgment.