The risk_control_insufficient audit event tells the operator how many labels an arm still needs. For a repo that already has plenty of labels, it reports a tiny number and sends the operator off collecting labels that will not help.
The bug
calibrateActThreshold (src/review/risk-control.ts) returns have from two different branches with two different meanings:
const needed = minimumCalibrationLabels(alpha, delta);
if (pairs.length < needed)
return { status: "insufficient_labels", needed, have: pairs.length, ... }; // (A) total pairs
...
for (const lambda of candidates) { ... lastTestedN = n; ... }
return { status: "insufficient_labels", needed, have: lastTestedN, ... }; // (B) n at the LAST tested lambda
- (A) fires when there genuinely are not enough labels.
have = total usable pairs. Correct and actionable.
- (B) fires when there ARE enough labels but no threshold achieves the error bound.
have = the size of the final, most-conservative surviving stratum — a residual that can be arbitrarily small and says nothing about label supply.
recalibrateArm renders both through one string (src/review/risk-control-wire.ts):
detail: `${scope}: cannot certify α=${alpha} — ${result.have} usable label(s) of ${result.needed} needed`
Live proof (edge-nl-01, 2026-07-26 recalibration)
| arm |
audit message said |
actual usable pairs (verified by running the real join) |
close:jsonbored/loopover |
"2 usable label(s) of 59 needed" |
61 |
close:jsonbored/metagraphed |
"9 usable label(s) of 59 needed" |
138 |
close:jsonbored/awesome-claude |
"27 usable label(s) of 59 needed" |
27 ✓ (branch A, correct) |
I ran loadCalibrationPairs's exact join against the live DB: metagraphed 138 pairs, loopover 61, awesome-claude 27 — all 226 with a non-null aiConfidence. The global arm reconciles perfectly (totalPairs 226, nAtLambda 129, coverage 129/226 = 0.5708 = the published 57.1%), confirming the join and the data are sound. Only the per-repo have is wrong.
Why it matters
The label-collection burn-down for epic #8828 is driven by exactly this message. It currently tells you loopover needs 57 more labels when loopover already has 61 — its problem is that the error rate is too high at every candidate threshold, which needs a completely different response (investigate close precision, or accept a weaker α), not more labels. Two of the three repos are misreported, and the more labels a repo accumulates the more misleading the number gets.
The published global guarantee is not affected — it is on the calibrated path and reconciles exactly.
Fix
- Distinguish the two outcomes in the type:
insufficient_labels (branch A, have = total pairs) vs a new no_certifiable_threshold (branch B, carrying totalPairs, bestN, and the best achieved upper bound).
- Give branch B its own message, e.g.
"{scope}: {totalPairs} labels available but no threshold achieves α={alpha} (best upper bound {x} at λ={y}, n={n})".
- Emit a distinct
event_type for B so the burn-down dashboard cannot conflate "needs labels" with "needs better precision".
Acceptance
- A repo with ample labels that cannot certify reports its true label count and says the threshold, not the sample, is the blocker.
Refs #8828, #8835, #9003.
The
risk_control_insufficientaudit event tells the operator how many labels an arm still needs. For a repo that already has plenty of labels, it reports a tiny number and sends the operator off collecting labels that will not help.The bug
calibrateActThreshold(src/review/risk-control.ts) returnshavefrom two different branches with two different meanings:have= total usable pairs. Correct and actionable.have= the size of the final, most-conservative surviving stratum — a residual that can be arbitrarily small and says nothing about label supply.recalibrateArmrenders both through one string (src/review/risk-control-wire.ts):detail: `${scope}: cannot certify α=${alpha} — ${result.have} usable label(s) of ${result.needed} needed`Live proof (edge-nl-01, 2026-07-26 recalibration)
close:jsonbored/loopoverclose:jsonbored/metagraphedclose:jsonbored/awesome-claudeI ran
loadCalibrationPairs's exact join against the live DB: metagraphed 138 pairs, loopover 61, awesome-claude 27 — all 226 with a non-nullaiConfidence. The global arm reconciles perfectly (totalPairs226,nAtLambda129, coverage 129/226 = 0.5708 = the published 57.1%), confirming the join and the data are sound. Only the per-repohaveis wrong.Why it matters
The label-collection burn-down for epic #8828 is driven by exactly this message. It currently tells you loopover needs 57 more labels when loopover already has 61 — its problem is that the error rate is too high at every candidate threshold, which needs a completely different response (investigate close precision, or accept a weaker α), not more labels. Two of the three repos are misreported, and the more labels a repo accumulates the more misleading the number gets.
The published global guarantee is not affected — it is on the
calibratedpath and reconciles exactly.Fix
insufficient_labels(branch A,have= total pairs) vs a newno_certifiable_threshold(branch B, carryingtotalPairs,bestN, and the best achieved upper bound)."{scope}: {totalPairs} labels available but no threshold achieves α={alpha} (best upper bound {x} at λ={y}, n={n})".event_typefor B so the burn-down dashboard cannot conflate "needs labels" with "needs better precision".Acceptance
Refs #8828, #8835, #9003.