Context
Parent: #8828 (Phase 3 — the mechanism that actually delivers "99.5%"). With labels (audit + holdout sub-issues) and a confidence score (confidence sub-issue), set the act/hold threshold with a finite-sample, distribution-free guarantee: P(correct | acted) ≥ 1−α with confidence 1−δ, via fixed-sequence testing (Learn then Test; Trust or Escalate Theorem 1).
Requirements
- Calibration procedure: sweep the confidence threshold λ on the labeled calibration set; exact binomial upper confidence bound on empirical risk; pick the smallest λ whose bound stays ≤ α for all larger λ. α and δ are config-as-code (defaults α=0.005, δ=0.05).
- SEPARATE thresholds per arm (Neyman–Pearson): the merge arm bound tighter than the close arm (defaults: 0.002 merge / 0.015 close), because a wrong merge costs more than a wrong close. Publish the resulting per-arm coverage.
- Rolling recalibration (cron tick, mirroring the self-tune rollup) on a windowed calibration set; per-repo where sample size permits, global fallback otherwise. Distribution shift is the known limitation — recalibration cadence and window are config.
- Decisions below λ̂ → hold with a reason code naming the calibrated-abstention source, so held contributors see why.
- The published accuracy surface (Phase-1 sub-issue) reports "≥(1−α) guaranteed at N% coverage" alongside the empirical figures.
- Integration guard: if the calibration set is under-powered (< the rule-of-three minimum for α), the system refuses to claim the guarantee and says so on the surface — never silently degrades.
Test Coverage Requirements
99%+ patch coverage, branch-counted; the threshold-selection procedure is pure with property tests (monotonicity of the bound sweep, under-powered refusal arm, both NP arms).
Expected Outcome
The 99.5% goal becomes a provable statement about acted decisions at a stated coverage — with the honest outcome possible that coverage is low, which quantifies exactly how much the judge must improve.
Context
Parent: #8828 (Phase 3 — the mechanism that actually delivers "99.5%"). With labels (audit + holdout sub-issues) and a confidence score (confidence sub-issue), set the act/hold threshold with a finite-sample, distribution-free guarantee: P(correct | acted) ≥ 1−α with confidence 1−δ, via fixed-sequence testing (Learn then Test; Trust or Escalate Theorem 1).
Requirements
Test Coverage Requirements
99%+ patch coverage, branch-counted; the threshold-selection procedure is pure with property tests (monotonicity of the bound sweep, under-powered refusal arm, both NP arms).
Expected Outcome
The 99.5% goal becomes a provable statement about acted decisions at a stated coverage — with the honest outcome possible that coverage is low, which quantifies exactly how much the judge must improve.