Skip to content

orb(risk-control): 'N usable labels of 59 needed' is wrong for 2 of 3 repos — have field carries two different meanings, misdirecting the label burn-down #9048

Description

@JSONbored

The risk_control_insufficient audit event tells the operator how many labels an arm still needs. For a repo that already has plenty of labels, it reports a tiny number and sends the operator off collecting labels that will not help.

The bug

calibrateActThreshold (src/review/risk-control.ts) returns have from two different branches with two different meanings:

const needed = minimumCalibrationLabels(alpha, delta);
if (pairs.length < needed)
  return { status: "insufficient_labels", needed, have: pairs.length, ... };   // (A) total pairs
...
for (const lambda of candidates) { ... lastTestedN = n; ... }
return { status: "insufficient_labels", needed, have: lastTestedN, ... };      // (B) n at the LAST tested lambda
  • (A) fires when there genuinely are not enough labels. have = total usable pairs. Correct and actionable.
  • (B) fires when there ARE enough labels but no threshold achieves the error bound. have = the size of the final, most-conservative surviving stratum — a residual that can be arbitrarily small and says nothing about label supply.

recalibrateArm renders both through one string (src/review/risk-control-wire.ts):

detail: `${scope}: cannot certify α=${alpha}${result.have} usable label(s) of ${result.needed} needed`

Live proof (edge-nl-01, 2026-07-26 recalibration)

arm audit message said actual usable pairs (verified by running the real join)
close:jsonbored/loopover "2 usable label(s) of 59 needed" 61
close:jsonbored/metagraphed "9 usable label(s) of 59 needed" 138
close:jsonbored/awesome-claude "27 usable label(s) of 59 needed" 27 ✓ (branch A, correct)

I ran loadCalibrationPairs's exact join against the live DB: metagraphed 138 pairs, loopover 61, awesome-claude 27 — all 226 with a non-null aiConfidence. The global arm reconciles perfectly (totalPairs 226, nAtLambda 129, coverage 129/226 = 0.5708 = the published 57.1%), confirming the join and the data are sound. Only the per-repo have is wrong.

Why it matters

The label-collection burn-down for epic #8828 is driven by exactly this message. It currently tells you loopover needs 57 more labels when loopover already has 61 — its problem is that the error rate is too high at every candidate threshold, which needs a completely different response (investigate close precision, or accept a weaker α), not more labels. Two of the three repos are misreported, and the more labels a repo accumulates the more misleading the number gets.

The published global guarantee is not affected — it is on the calibrated path and reconciles exactly.

Fix

  1. Distinguish the two outcomes in the type: insufficient_labels (branch A, have = total pairs) vs a new no_certifiable_threshold (branch B, carrying totalPairs, bestN, and the best achieved upper bound).
  2. Give branch B its own message, e.g. "{scope}: {totalPairs} labels available but no threshold achieves α={alpha} (best upper bound {x} at λ={y}, n={n})".
  3. Emit a distinct event_type for B so the burn-down dashboard cannot conflate "needs labels" with "needs better precision".

Acceptance

  • A repo with ample labels that cannot certify reports its true label count and says the threshold, not the sample, is the blocker.

Refs #8828, #8835, #9003.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions