fix(orb): publish decision-grounded accuracy instead of the reversal rate - #8822
Merged
Conversation
…rate
The homepage 'Decision accuracy' tile published 1 - reversalRate, which
overstates the gate's real accuracy two independent ways:
1. DENOMINATOR: reversalRate divides by every fleet signal including
'hold' verdicts. A hold is a deferral to a human, not a decision
that can be right or wrong. Holds were ~36% of live fleet signals,
dragging the rate toward zero and the published number toward 100
regardless of how the gate actually performed.
2. NUMERATOR: it counted only explicit reversal markers, so outright
mispredictions never registered at all -- on the live fleet, 213
PRs where the gate said merge and the PR ended up closed, plus 21
where it said close and the PR merged, were entirely invisible.
Measured on live fleet data the two differ by ~6 points: the retired
formula published 99.6% where the gate's real decision accuracy is
93.6% (3,783 of 4,043 merge/close calls confirmed by outcome).
Advances #8820.
- analytics: InstanceMetrics/FleetAnalytics gain decisionAccuracy =
(mergeConfirmed + closeConfirmed) / (merge + close verdicts), null
for a holds-only instance; fleet value is the median, matching every
other fleet aggregate
- public-stats: publish it as fleetAccuracy.accuracyPct
- UI: the hint states what the number measures rather than naming the
retired formula
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 7df6643 | Commit Preview URL Branch Preview URL |
Jul 26 2026, 09:51 AM |
Bundle ReportChanges will increase total bundle size by 35 bytes (0.0%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Owner
Author
|
Correction to the 93.6% figure in this PR's description. The scoring fix here is right and stands — accuracy must score merge/close decisions confirmed by outcome, not The Measured across that class (n=210): 84 were bot-closed, 59 have their verdict timestamped after the close. The remaining 126 were human-closed despite a merge verdict — those are genuine disagreements and a real signal. So the honest reading today is a range, roughly 93.6%–95.6%, with 93.6% as the lower bound. A single trustworthy figure needs #8823 (reversals under-counted → biases up) and #8825 (contradictory verdicts → bias down) both fixed. This PR remains the correct denominator/numerator definition; the value it reports gets trustworthy once those land. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8822 +/- ##
==========================================
- Coverage 93.83% 92.28% -1.55%
==========================================
Files 800 800
Lines 79831 79832 +1
Branches 24196 24197 +1
==========================================
- Hits 74908 73676 -1232
- Misses 3558 5092 +1534
+ Partials 1365 1064 -301
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the homepage "Decision accuracy" tile measure what its label claims. Advances #8820.
Why — the number was structurally inflated, not just missing data
The tile published
1 − reversalRate. That formula overstates the gate's accuracy two independent ways:holdverdicts. A hold is a deferral to a human — not a decision that can be right or wrong. On the live fleet 2,234 of 6,277 signals (~36%) are holds, dragging the reversal rate toward zero and the published accuracy toward 100 regardless of how the gate actually performed.mergeand the PR ended up closed, plus 21 where it saidcloseand the PR merged — 234 wrong calls, entirely invisible.Measured on the real fleet confusion matrix (all three repos ORB runs on):
That ~6-point gap is real errors, not rounding, and it matches the maintainer's direct experience of reversing decisions that the metric never reflected.
How
analytics.ts:InstanceMetrics/FleetAnalyticsgaindecisionAccuracy = (mergeConfirmed + closeConfirmed) / (merge + close verdicts);nullfor a holds-only instance (a genuine "no signal yet"). The fleet value is the median, matching every other fleet aggregate.reversalRateis kept — the anti-farming detector legitimately uses it — with its doc corrected to say it spans all signals including holds.public-stats.ts: publishdecisionAccuracyasfleetAccuracy.accuracyPct. Null still falls back to the own-ledger number exactly as before.reversal-groundedformula.No schema or export change — this is a scoring fix over data already collected.
Verification
decisionAccuracywhere1 − reversalRatewould report a flat 100%.Follow-up (not in this PR)
pr_outcomeis missing for 40 of 66 PRs carrying areversal_supersededmarker — those PRs are bot-closed on GitHub but never got an outcome row, so they're excluded from the fleet export entirely (inner join). Fixing that will lower the published number further toward its true value. Filing separately.