Skip to content

feat(stats): review-accuracy as a historical trend, not just a live snapshot #4447

Description

@JSONbored

Part of #4445.

Context

Review-decision accuracy is already computed and already public — computeGateEval (src/review/parity.ts:88-169) joins every gate decision (the predicted merge/close/hold verdict) against the realized pr_outcome once a PR actually closes, reversal-discounted via recordReversalSignals (src/review/outcomes-wire.ts:402-471, catching both a contributor reopening a bot-closed PR and someone reverting a bot-merged PR). This already powers the homepage's live "Decision accuracy %" (shipped via #1059/#1567) and an existing auto-tune circuit breaker (runSelfTuneBreaker, src/review/outcomes-wire.ts:499-573) that self-tightens per-repo if precision drops below a floor.

What's missing is history. The current public number is a live snapshot computed at request time — there is no persisted time series, so there's no way to show (or even internally verify) whether accuracy is improving, stable, or degrading over weeks/months. This is distinct from #4261/#2352, which track the miner's own self-review calibration (packages/gittensory-engine/src/phase7-calibration-loop.ts, a different subsystem, different audience) — this issue is about the maintainer-facing review engine's own accuracy trend.

Requirements

  • Add a periodic (weekly, matching the existing rollup cadence precedent) snapshot job that calls computeGateEval and persists the result — combined precision, per-verdict-type precision, sample size, and the reversal count/rate — into a new small rollup table (or extends the pr_handling_stats rollup already described in feat(stats): public proof-of-power — lifetime PRs-handled + accuracy counter on homepage hero #1059's deliverables, if that table already exists with room for these fields — check before adding a new one).
  • Never let a low-sample-size week produce a misleading precision number: apply the same kind of "insufficient history" guard feat(signals): add queue pressure and review velocity trend windows #110 already uses for its trend windows, and consider carrying forward computeGateEval's existing sample-size field into the guard condition.
  • Extend GET /v1/public/stats with the accuracy trend (reuse the existing response shape, per the parent epic's guidance).
  • Add a trend chart to the homepage//stats page alongside the existing live accuracy number — the live number stays as the "right now" headline, the trend is the new "is this improving" story.
  • Document the accuracy definition publicly and precisely (reversal-grounded: a decision is "correct" unless a human later overrides/reverts it) — feat(stats): public proof-of-power — lifetime PRs-handled + accuracy counter on homepage hero #1059 already required this for the live number; make sure the trend's methodology is described identically so the two never appear inconsistent to a reader.

Deliverables

  • Weekly accuracy snapshot persisted (precision by verdict type, sample size, reversal rate)
  • Insufficient-history state handled explicitly, not silently rendered as a real number
  • GET /v1/public/stats extended with the accuracy trend; OpenAPI regenerated
  • Homepage//stats page shows an accuracy-over-time chart alongside the existing live number
  • Public methodology write-up matches the existing reversal-grounded definition exactly
  • Tests: snapshot computation, low-sample guard, public endpoint shape

Expected outcome

"Decision accuracy %" stops being a single live number and becomes a real, honest trend line the community (and we, internally) can watch over time — reusing the accuracy computation that already exists rather than inventing a second one.

References

Effort

M

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions