Skip to content

fix(review): public-stats accuracyPct denominator includes Orb-fleet merged/closed but numerator (reversed) stays own-ledger-only #7449

Description

@JSONbored

Context

src/review/public-stats.ts's getPublicStats computes the homepage's global accuracyPct
(line 401) as accuracyPct(totals.merged, totals.closed, totals.reversed). totals.reversed is
accumulated only from loopover's own allowlisted repos (line 358), but totals.merged/
totals.closed are folded with the registered Orb fleet's aggregate counts first (lines 381-382,
via getOrbGlobalStats, which — per this file's own comment at line 378 — "has no reversal concept
at all"). accuracyPct's formula (line 151) is 1 - min(1, reversed / (merged + closed)). Because
the denominator (merged + closed) grows with the whole registered fleet while the numerator
(reversed) stays fixed to a handful of own-ledger repos, the published accuracy percentage trends
toward 100% as more self-hosted installations register, independent of the fleet's actual
merge/close reversal behavior — which this number is supposed to represent.

This file's header comment is otherwise very thorough about reasoning through this fold's
tradeoffs (explicitly noting reversals stay own-ledger-only), but doesn't address this specific
downstream skew on the published accuracyPct value.

Requirements

  • The publicly-displayed global accuracyPct must not be computable in a way where its numerator
    (reversed count) and denominator (merged+closed count) are drawn from structurally different,
    independently-growing populations. Acceptable fixes include (pick one, and document the choice
    in a code comment next to accuracyPct's call site):
    1. Compute the global accuracyPct from own-ledger totals only (i.e. call accuracyPct with the
      pre-fold ownLedger-scoped merged/closed, not the fleet-folded totals.merged/totals.closed),
      leaving the Orb-fleet merged/closed counts to inflate only the raw reviewed/handled/
      minutesSaved totals as they do today (which have no numerator/denominator mismatch problem);
    2. Or, if getOrbGlobalStats can be extended to also report a fleet-wide reversal count, fold
      that into totals.reversed alongside merged/closed so numerator and denominator scale
      together.
      Do not silently drop the Orb-fleet fold from merged/closed/reviewed/handled entirely —
      those aggregate totals are working as intended per the file's own comments; only the
      accuracyPct computation's numerator/denominator mismatch needs fixing.
  • Do not change accuracyPct's own formula/signature, byProject's per-project accuracyPct
    values (each project's is already computed from same-scope merged/closed/reversed — verify this
    stays true), or any other field in the returned stats object.

Deliverables

  • The global totals.accuracyPct computation reworked so its numerator and denominator are
    drawn from the same population (own-ledger-only per option 1, or fleet-wide-both per option 2)
  • A code comment at the call site explaining which option was chosen and why, matching this
    file's existing house style of documenting non-obvious metric-composition choices
  • Regression test: a scenario with zero own-ledger reversals but a large getOrbGlobalStats
    merged/closed count no longer produces accuracyPct trending toward 100 purely from the
    fleet fold — assert the value reflects only the reversal-comparable population chosen above
  • Existing tests for getPublicStats's other fields (reviewed, handled, minutesSaved,
    per-project accuracyPct) continue to pass unchanged

Test Coverage Requirements

This repo's Codecov patch gate is 99%+ hard (branch-counted) on every changed line/branch in
src/**. The new regression test must exercise the real getPublicStats function (mocking
getOrbGlobalStats and the own-ledger disposition query) end-to-end, not just unit-test
accuracyPct in isolation with hand-picked numbers.

Expected Outcome

The homepage's publicly-displayed global accuracy percentage can no longer be pulled toward 100%
purely as an artifact of more self-hosted installations registering into the Orb fleet — it either
reflects the same own-ledger population its numerator has always used, or fleet-wide reversal data
that scales with the same denominator.

Links & Resources

src/review/public-stats.ts:151-163 (accuracyPct), :358 (own-ledger totals.reversed
accumulation), :376-401 (the Orb-fleet fold and the final accuracyPct call site),
getOrbGlobalStats (src/orb/outcomes.ts, confirm whether a reversal-count extension is already
tracked elsewhere before choosing option 2 above).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions