Context
Parent: #8828 (Phase 2 — labels). The selective-labels problem: when ORB closes a PR we never observe whether merging would have been fine, so close-precision is estimated only on the closes humans happened to contest. ORB is deterministic, so off-policy estimators (IPS et al.) are undefined — randomization is the only fix (Lakkaraju et al., KDD 2017).
Requirements
- Route a small random fraction ε (config-as-code, default 3-5%) of would-auto-close PRs to HOLD + human adjudication instead of auto-close. The manifest schema gains the knob; per-repo override supported; 0 disables.
- Log the randomization: the drawn value, ε at draw time, and the counterfactual action, persisted with the decision so propensities exist for any future estimator.
- The held PRs' human adjudications land in the same
decision_audit_labels store as the audit-sampling sub-issue, tagged as holdout-sourced (unbiased for close-precision, unlike the noticed-reversal stream).
- Deterministic under an injected RNG seed for tests; the draw must happen AFTER the close decision is final so it cannot influence the decision itself.
- Surface held-out PRs to the operator (existing approval-queue surface if compatible, else the recap).
Test Coverage Requirements
99%+ patch coverage, branch-counted; a regression test that ε=0 is byte-identical to today's behavior.
Expected Outcome
The only unbiased close-precision estimate this system can have, plus logged propensities that make counterfactual evaluation well-defined for the first time.
Context
Parent: #8828 (Phase 2 — labels). The selective-labels problem: when ORB closes a PR we never observe whether merging would have been fine, so close-precision is estimated only on the closes humans happened to contest. ORB is deterministic, so off-policy estimators (IPS et al.) are undefined — randomization is the only fix (Lakkaraju et al., KDD 2017).
Requirements
decision_audit_labelsstore as the audit-sampling sub-issue, tagged as holdout-sourced (unbiased for close-precision, unlike the noticed-reversal stream).Test Coverage Requirements
99%+ patch coverage, branch-counted; a regression test that ε=0 is byte-identical to today's behavior.
Expected Outcome
The only unbiased close-precision estimate this system can have, plus logged propensities that make counterfactual evaluation well-defined for the first time.