Skip to content

calibration: randomized ε-holdout on would-close PRs with propensity logging #8831

Description

@JSONbored

Context

Parent: #8828 (Phase 2 — labels). The selective-labels problem: when ORB closes a PR we never observe whether merging would have been fine, so close-precision is estimated only on the closes humans happened to contest. ORB is deterministic, so off-policy estimators (IPS et al.) are undefined — randomization is the only fix (Lakkaraju et al., KDD 2017).

Requirements

  • Route a small random fraction ε (config-as-code, default 3-5%) of would-auto-close PRs to HOLD + human adjudication instead of auto-close. The manifest schema gains the knob; per-repo override supported; 0 disables.
  • Log the randomization: the drawn value, ε at draw time, and the counterfactual action, persisted with the decision so propensities exist for any future estimator.
  • The held PRs' human adjudications land in the same decision_audit_labels store as the audit-sampling sub-issue, tagged as holdout-sourced (unbiased for close-precision, unlike the noticed-reversal stream).
  • Deterministic under an injected RNG seed for tests; the draw must happen AFTER the close decision is final so it cannot influence the decision itself.
  • Surface held-out PRs to the operator (existing approval-queue surface if compatible, else the recap).

Test Coverage Requirements

99%+ patch coverage, branch-counted; a regression test that ε=0 is byte-identical to today's behavior.

Expected Outcome

The only unbiased close-precision estimate this system can have, plus logged propensities that make counterfactual evaluation well-defined for the first time.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions