fix(review): correct risk-control calibration and the published guarantee's fleet integrity - #9228
Merged
Merged
Conversation
…calibration delta (#9048, #9066) calibrateActThreshold returned "insufficient_labels" from two branches with different "have" semantics: a genuine label shortfall (have = total pairs) and a residual high-confidence stratum too small to certify despite ample total labels (have = that stratum's size). The latter now returns a distinct "no_certifiable_threshold" status carrying totalPairs/bestN/bestLambda/bestUpperBound, rendered through its own message and a distinct risk_control_no_certifiable_threshold audit event so the label burn-down no longer conflates "needs labels" with "needs a better error rate". Separately, the ascending-lambda scan reported whichever of K observed-confidence candidates passed first without correcting for testing K of them — the advertised 1-delta confidence overstated what the scan actually delivered. Each candidate is now tested at a Bonferroni-split delta/K, so the certified lambda is valid at the full, originally-advertised delta regardless of which candidate passes. Chosen over a true fixed-sequence rewrite as the smaller, safer diff against the existing ascending scan. Also adds AND dr2.action = dal.verdict to the calibration join (latent-risk hardening: today's data is not mis-joined, but a later HOLD/MERGE record on the same PR could otherwise shadow the acted CLOSE record a label adjudicates) and tags each calibration pair's provenance (backfilled vs live, via the backfill's configDigest sentinel) so a published guarantee can later say how much of its evidence is reconstructed history.
…ction, pool decisionAccuracy (#9068) orb_signals ingest stored any object-shaped risk_control payload verbatim, so a registered instance with a stale or misconfigured alpha (or an outright refused calibration) could reach the public guarantee unchecked. handleOrbIngest now runs the payload through validateCalibrationPayload (status === "calibrated", alpha/ lambda/coverage in range, nAtLambda clearing the zero-error floor for its own alpha/delta) before it reaches orb_risk_control_arms at all. gamingPatternFlags compared each eligible instance against the fleet median, which is structurally unfireable below 3 eligible instances (an instance IS the median at n=1) and can never flag "low reversal" once the fleet's own reversal-rate median is exactly 0 (a common, healthy-fleet case) since a fraction of zero can never be undercut. computeFleetAnalytics now gates detection on eligible.length >= 3 (surfaced via the new gamingDetectionEligible field) and falls back to an absolute reversal-rate floor when the fleet median is zero. fleet.decisionAccuracy published the per-instance MEDIAN while accuracyCiPct (public-stats.ts) is a Wilson interval over the POOLED counts — different estimands that only coincide at equal per-instance volumes. fleet.decisionAccuracy now publishes the pooled proportion directly (the same population the interval describes); the per-instance median survives as the new decisionAccuracyMedian diagnostic field. The underlying per-instance last-writer-wins fleet-key bug this issue also describes was already fixed by #9177 (orb_risk_control_arms is keyed per instance_id/arm and public-stats already aggregates across registered instances at read time) — this change covers the remaining validation-before-publish and detection-floor gaps.
…date it before serving (#9050) readGuarantee published "coveragePct" as if it were a share of all closes, adjacent to a different fleetAccuracy.coveragePct that IS a share of all decided signals -- one word, two denominators. The guarantee's own field is actually the share of the arm's AI-JUDGED sub-population the threshold covers (loadCalibrationPairs can only join a confidence to decisions an AI-judgment blocker ran on, a minority of real closes). Renamed to aiJudgedCoveragePct and the homepage string now names the sub-population explicitly instead of leaving a bare percentage next to its sibling. readGuarantee also now re-validates every stored orb_risk_control_arms row through risk-control.ts's validateCalibrationPayload (defense in depth alongside the ingest-side check from the companion #9068 fix) and walks all registered rows for an arm in nAtLambda-descending order instead of trusting only the top one, so a single malformed or stale peer can no longer hide a good row behind it. Each calibration pair now carries whether it's backfilled (the 2026-07 calibration-corpus backfill's configDigest sentinel) or live; calibrateActThreshold surfaces the split as backfilledPairs, and the public guarantee renders it as backfilledPct so a guarantee resting mostly on reconstructed history says so. Regenerated apps/loopover-ui/public/openapi.json for the schema rename/nullability.
…ions npm run ui:lint's format:check caught unformatted lines from the #9050 fleetAccuracy.guaranteed rename in proof-of-power-stats-model.ts and its new test.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | b89464d | Commit Preview URL Branch Preview URL |
Jul 27 2026, 08:06 AM |
Bundle ReportChanges will increase total bundle size by 952 bytes (0.01%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
❌ 4 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
loopover-orb Bot
pushed a commit
that referenced
this pull request
Jul 28, 2026
…thhold the pooled count at n=2 (#9168) (#9574) `fleetAccuracy` had a per-instance volume floor (MIN_DECIDED) but nothing bounding the NUMBER of instances. At the one registered instance live today it publishes one operator's own outcomes under fleet framing, next to a risk-control guarantee calibrated by that same instance -- which invites a reader to treat one party's self-report as two independent sources. #9228 already fixed the detector half of this issue (gamingFlagsCaught is null, not 0, below GAMING_MIN_ELIGIBLE). This is the framing half, which did not land. WHAT CHANGES - FLEET_FRAMING_MIN_INSTANCES, deliberately defined AS GAMING_MIN_ELIGIBLE: the n at which a median becomes robust to a single bad contributor is the same n at which "this far above the median" becomes satisfiable. Pinning them together means a reader never sees fleet framing next to "the detector could not run". - A `basis` discriminator on the published block: "fleet" or "single_instance_self_report". The numbers are real and stay published at every n -- only the claim about what they are changes. - The pooled `decidedCount` is withheld (null) at exactly 1 < instanceCount < floor. The count is a plain SUM, and this deployment's own volume is already public via byProject, so at n=2 a reader recovers the OTHER instance's decision volume by subtraction. For a hosted tenant that volume is a business metric -- how many PRs they ship, how many get closed -- and not ours to publish. At n=1 there is nothing to subtract; at n >= floor the sum no longer isolates anyone. RATES are safe at every n, since a proportion carries no volume, so only the count is withheld. - The median-robustness comment in analytics.ts claimed robustness unconditionally. Corrected to state the n it actually requires, since that claim was the reason the median was chosen and it was doing no work at n=1. - The DB-failure fallback fails closed (fleetFramingEligible: false) with the rest of that path -- if the fleet tables cannot be read we certainly cannot claim a fleet. The OpenAPI schema is a hand-maintained zod mirror, so `basis` and the now-nullable `decidedCount` are added there too, and apps/loopover-ui/public/openapi.json regenerated -- otherwise the spec silently drifts from the response it documents. NOT DECIDED HERE: whether a hosted tenant's outcomes should enter the public aggregate at all, and if so opt-in or opt-out. That is a product/privacy call, not a code one. This change is safe under either answer -- it withholds the leaky figure by default rather than presuming consent -- and the issue records the question. Tests: the outlier and gaming thresholds had never been exercised against multi-instance fixtures, which #9168 called out. Now covered at n=1, n=2, n=3, with the eligible-vs-registered distinction (volume and registration both gate the floor) and the fail-closed path. 0 uncovered changed lines or branches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four related fixes to the published statistical guarantee subsystem:
insufficient_labelsoutcomes (genuinely too few labels vs. ample labels but no certifiable threshold) into distinct outcomes with distinct messages and event types, so the label burn-down dashboard stops conflating "needs labels" with "needs better precision."gamingFlagsCaughtcomputation to require enough eligible instances, and publishes the pooled proportion asdecisionAccuracy's point estimate instead of conflating it with the cross-instance median.Test plan
npx tsc --noEmit -p tsconfig.json --incremental false— cleanmain(picked up fix(orb): installation/permission/config integrity — suspension backfill, broker identity, numeric env validation, migration content hash #9222 — no conflicts, re-verified)Closes #9048
Closes #9050
Closes #9066
Closes #9068