Skip to content

ORB fairness analytics: fix broken public accuracy metric, add fairness report + contributor trust profiles #7567

Description

@JSONbored

Summary

Gittensor miners have raised fairness concerns about ORB's review/gate decisions. This tracks three related pieces of work to both prove the gate is fair in aggregate and build the internal tooling to keep improving it over time.

Scope

  1. Bugfix: the public "Decision accuracy" homepage stat was structurally unable to detect a merge later undone by a separate revert PR (a merged PR's state can never read as open again), and was scoped only to a frozen snapshot of the maintainer's own repos rather than the live self-hosted fleet. Fixes reversal detection to reuse the existing correct detector, and blends in live fleet-wide accuracy.
  2. Public fairness report: a /fairness page (linked from the homepage accuracy tile) showing the 8-week accuracy trend, per-repo breakdown, and anti-gaming-flags-caught count. Aggregate counts only — no PR content, no contributor identities.
  3. Internal contributor trust profiles: per-login, per-repo gate-decision accuracy and fairness-outlier detection (flags favorable and unfavorable deviation, never asserting fault), composed with per-repo submission counts and moderation-violation history into one profile. Internal/bearer-gated only, never public. Includes a backfill mechanism to reconstruct historical per-contributor gate history, and config-as-code control (global .loopover.yml toggle + per-repo .loopover.yml opt-out of participation).

Validation

  • npm run typecheck, npm run db:schema-drift:check, npm run db:migrations:check, npm run docs:drift-check, npm run manifest:drift-check, npm run ui:openapi:check, npm run ui:openapi:settings-parity all green
  • New/changed unit + integration tests, 100% branch coverage on every new module
  • npm --workspace @loopover/engine run test, engine/driver/live-gate parity contract tests all green

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions