Skip to content

orb(recap): apply the gate-precision MIN_SAMPLE floor to the recap's aggregate false-positive rates #9691

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

services/gate-precision.ts sets one rule for the gate false-positive rate and states why, at src/services/gate-precision.ts:99: "Null below the min sample — a 1-of-1 'false positive' is noise, not a precision signal." MIN_SAMPLE = 5 (:22); both perGateType[].falsePositiveRate and overall.falsePositiveRate (:109) are null below it.

buildMaintainerRecap (src/services/maintainer-recap.ts:87) folds those per-repo reports into a cross-repo RecapReport and applies that rule inconsistently inside one function:

  • Per-repo path — floored. toRecapCohortCounts (:80-82) copies overall.falsePositiveRate straight out of the GatePrecisionReport, so repos[i].cohorts.*.gateFalsePositiveRate correctly reads null below the floor.
  • Aggregate path — not floored. :145-146 re-derives it as totals.blocked > 0 ? Math.round((totals.gateFalsePositives / totals.blocked) * 100) / 100 : null, and the cohort aggregates at :153 and :158 use the same > 0 guard.

The rendered digest then contradicts itself. Two repos, each blocked: 1 / blockedThenMerged: 1 (aggregate blocked: 2) produce, in one document: summary (:164) "Gate false-positive rate: 100% (2/2 block(s) later merged)."; Totals (:203, :226) "- Gate false positives: 2/2 (100%)"; and Gate-outcomes section (src/services/maintainer-recap-gate-outcomes.ts:62, :66) "False-positive rate: n/a (fewer than 5 blocks in the last 7 day(s))".

The maintainer reads "the gate is 100% wrong" and "not enough data to say" about the same window, from the same totals struct. maintainer-recap-gate-outcomes.ts:13-14 documents itself as mirroring gate-precision's floor — it is the only consumer of totals that actually does. MIN_SAMPLE = 5 is a private const duplicated in two files (gate-precision.ts:22, maintainer-recap-gate-outcomes.ts:14), which is what let a third consumer be written without it.

Requirements

  • Export the floor from src/services/gate-precision.ts as export const MIN_GATE_PRECISION_SAMPLE = 5;, replacing that file's private MIN_SAMPLE at line 22.
  • src/services/maintainer-recap-gate-outcomes.ts imports it and deletes its duplicate at line 14; its rendered "fewer than N blocks" copy must use the imported value.
  • buildMaintainerRecap (:145-146) returns null for totals.gateFalsePositiveRate when totals.blocked < MIN_GATE_PRECISION_SAMPLE, else the existing 2-dp ratio. The existing blocked === 0 case is subsumed.
  • The cohort aggregates at :153 and :158 apply the same floor against their own blocked denominator, not the blended one.
  • The rate string at :162-165 keeps exactly its two existing arms; the null arm's existing wording now also covers the below-floor case. Do not add a third arm.
  • RecapReport.totals.gateFalsePositiveRate stays typed number | null; no src/types.ts or src/openapi/schemas.ts change is needed or permitted here.
  • Existing tests asserting the un-floored value must be updated, not deleted: test/unit/maintainer-recap.test.ts:149-153 asserts miner: { blocked: 2, ... gateFalsePositiveRate: 0.5 } and human: { blocked: 4, ... 0.25 } — both below the floor, both must become null. Sweep the file for siblings in that shape.

⚠️ Required pattern: mirror buildGateOutcomesRecapSection at src/services/maintainer-recap-gate-outcomes.ts:58-62 — a single blocked >= MIN_GATE_PRECISION_SAMPLE guard producing number | null, raw counts still always reported. It does NOT satisfy this issue to leave MIN_SAMPLE duplicated in three files and only patch maintainer-recap.ts; to suppress the rendered lines while leaving totals.gateFalsePositiveRate numerically un-floored; to introduce a new, different floor constant; or to change GatePrecisionReport's own semantics.

Deliverables

  • MIN_GATE_PRECISION_SAMPLE is exported from src/services/gate-precision.ts and is the only 5 noise-floor literal across gate-precision.ts, maintainer-recap-gate-outcomes.ts, maintainer-recap.ts (grep-verifiable).
  • buildMaintainerRecap over two repos each blocked 1 / blockedThenMerged 1 returns totals.gateFalsePositiveRate === null, asserted by a new named regression case in test/unit/maintainer-recap.test.ts.
  • Aggregate blocked === 5, gateFalsePositives === 1 returns 0.2 (floor inclusive at 5), asserted by a new case.
  • totals.cohorts.miner.gateFalsePositiveRate / .human. are each null when that cohort's own blocked is below the floor even though the blended blocked is above it — new case with a mixed fixture.
  • formatMaintainerRecap over a below-floor report renders - Gate false positives: 2/2 (n/a) and the summary's not-enough-blocked arm, with no percentage anywhere the Gate-outcomes section says n/a — new case in test/unit/maintainer-recap-format.test.ts.
  • test/unit/maintainer-recap.test.ts:149-153 and any sibling below-floor assertion updated to the floored expectation.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example adding the shared constant and the gate-outcomes import without changing maintainer-recap.ts:145-146/:153/:158 — does not resolve this issue.

Test Coverage Requirements

99%+ Codecov patch coverage, branch-counted, on src/**. All three touched files are inside coverage.include, so both arms of every changed conditional need a test: the below-floor (null) arm and the at-or-above arm, for the blended rate, each cohort rate, and the rendered string. The below-floor blended case must be a named regression test. Measure coverage unsharded with npm run test:coverage.

Expected Outcome

A digest can no longer state a gate false-positive percentage in its summary/Totals while its own Gate-outcomes section says the sample is too small to judge. The 5-block noise floor lives in one place, so the next consumer of RecapReport.totals inherits it.

Links & Resources

src/services/gate-precision.ts:22, :99, :109; src/services/maintainer-recap.ts:80-82, :145-146, :153, :158, :162-165, :203, :226; src/services/maintainer-recap-gate-outcomes.ts:13-14, :58-67; test/unit/maintainer-recap.test.ts, test/unit/maintainer-recap-format.test.ts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions