Skip to content

docs(fairness): the automation rate is published without a methodology note #10065

Description

@JSONbored

The second sub-issue of #9726, which asks to "publish the series on /fairness with a methodology note". The series ships; the note does not.

State

  • Computed and persisted from decision_recordssrc/review/automation-rate.ts. Done.
  • Published at /v1/public/stats under automationRate, and rendered on /fairness with a weekly table. Done.
  • Methodology notefairness-methodology.mdx does not mention automation at all. Missing.

Why the gap matters more here than for the sibling metrics

Every other figure on /fairness has its definition written down. This one is the metric the 97% target is stated against, and its definition contains three choices a reader cannot guess and would plausibly assume the other way:

  1. Non-deciding verdicts are excluded from both halves, not counted as automated. action also carries label, update_branch, approve and the error classes; counting those would let work the gate never ruled on read as an automated decision.
  2. A PR that was held and later merged is manual. Scoring the final disposition instead would let the rate be inflated by holding everything and merging it by hand.
  3. Three human signals count, not just hold — a named reevaluation_actor, and a reevaluation_reason of maintainer_request, also make a PR manual.

There is also a backfill horizon the page surfaces as a per-week basis field without explaining it anywhere. The re-evaluation provenance fields arrived in migration 0204 on 2026-07-29; weeks before that can only observe hold, so they can only under-count manual work and therefore only over-state automation. A reader comparing this quarter to last needs to know that, and right now the only place it is written down is a source comment.

Acceptance

  • The methodology page documents the buckets, the three deliberate choices above, and the horizon — with the same numbers and language the code uses.
  • /fairness's automation section links to it, as its sibling sections already link to their definitions.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions