Skip to content

maintainer: fleet-wide anti-farming detector — flag a miner/fleet gaming the calibration score #2350

Description

@JSONbored

Once miners self-review against the predicted-gate and (via the Orb fleet-telemetry path already shipped for the review-stack) an anonymized accuracy signal reaches a central collector, a bad actor could try to game the aggregate accuracy number — e.g. by only submitting trivially-safe PRs to inflate merge-precision, or by farming duplicate-issue claims across many identities to win the isDuplicateClusterWinnerByClaim election disproportionately. This issue builds the detector that flags that pattern for a human, mirroring the existing self-host src/selfhost/orb-collector.ts anonymized/HMAC'd/reversal-aware export design so the detector itself never needs raw PII, source diffs, or shared credentials to do its job.

This is squarely the safety-governor/anti-farming chokepoint the phase brief calls out by name — maintainer-only, no exceptions.

Deliverables

  • A fleet-side aggregation job reading the anonymized Orb export stream (repo/PR HMAC'd identifiers, verdict, outcome, reversal flag — never raw content) and computing per-anonymized-actor submission-pattern statistics (volume vs. merge-precision vs. reversal-rate, and duplicate-claim-election win-rate skew)
  • A threshold-based flag (not an automatic action) that surfaces "this anonymized actor's pattern looks like gaming" to a human dashboard/alert — detection only, no automatic penalty or block
  • Explicit non-goals documented: this does not deanonymize, does not auto-ban, does not touch the live gate — it is a human-facing signal only
  • Tests: a normal-distribution fixture produces no flag; an inflated-trivial-volume fixture flags; a duplicate-claim-election-skew fixture flags; the detector never emits an identity, only an opaque anonymized handle

References

  • src/selfhost/orb-collector.ts (235 lines) — the anonymized/HMAC'd/reversal-aware export pattern this detector consumes and whose privacy guarantees it must preserve
  • src/signals/duplicate-winner.ts (60 lines) — isDuplicateClusterWinnerByClaim, the election logic whose win-rate distribution this detector watches for skew
  • migrations/0067_orb_pr_outcomes.sql — the outcome shape the fleet aggregation reads

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainer-onlyOwner-only work — yields no Gittensor points.

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions