Skip to content

feat(review): fleet-wide gaming-pattern detector for the self-host orb fleet (#2350) - #5083

Merged
JSONbored merged 1 commit into
mainfrom
feat/orb-fleet-farming-detector-2350
Jul 11, 2026
Merged

feat(review): fleet-wide gaming-pattern detector for the self-host orb fleet (#2350)#5083
JSONbored merged 1 commit into
mainfrom
feat/orb-fleet-farming-detector-2350

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Extends computeFleetAnalytics (src/orb/analytics.ts) with gamingPatternFlags: a more targeted detection signal than the existing single-metric outlier check. An instance whose decided-PR volume is unusually high (>2x fleet median), whose merge-precision is unusually high (>fleet median + 0.25), and whose reversal-rate is unusually low (<0.5x fleet median) — all three simultaneously — fits the "mass-submitting only trivially-safe PRs to inflate merge-precision" signature the issue describes. High precision alone isn't suspicious (could be a genuinely careful team); combined with abnormal volume and suspiciously few reversals, it is.
  • Surfaced read-only on the existing operator dashboard tile set and the operator-only MCP gittensory_get_fleet_analytics tool summary. Detection only — nothing acts on it automatically, nothing touches the live gate.
  • instanceId is the same opaque, HMAC-derived handle already used throughout this pipeline (orb-collector.ts) — never a login or anything more identifying.

Scope (verified against the actual pipeline before implementing)

  • Instance-level, not per-miner. The fleet pipeline carries no per-actor identity by deliberate, repeatedly-documented design — review_audit has no login column, and predicted_gate_calibration_ledger is explicitly documented as never-exported, citing this issue by number as the reason why. A genuine per-miner detector would require adding a new anonymized per-actor signal to the export pipeline, a separate privacy-sensitive design decision that deserves its own focused issue/PR rather than a rushed addition here.
  • "Duplicate-claim-election win-rate skew" is not implemented. isDuplicateClusterWinnerByClaim's outcome is never persisted anywhere in this pipeline — only the losing side of a duplicate cluster produces a finding (duplicate_pr_risk), with no cluster id and no actor linkage. There is no winner data to measure a win-rate from, and a per-instance duplicate_risk rate would measure something different (how often this instance's own PRs lose a local collision) than "identities farming wins." No proxy is implemented — a misleading one would be worse than none.

Given these two structural gaps, this uses "Advances #2350" rather than closing it.

Test plan

  • npm run typecheck clean
  • test/unit/orb-analytics.test.ts — 14 new tests: normal-distribution fixture (no flag), inflated-trivial-volume fixture (flags, with exact expected numbers), high-precision-alone / high-volume-alone / high-volume-and-precision-but-normal-reversal (each individually insufficient), null-mergePrecision instance never flags even alongside a real farmer, no-merge-verdict-at-all fleet, unregistered instance never flags, below-MIN_DECIDED instance never flags, empty store, DB-error fail-safe
  • test/unit/operator-dashboard.test.ts — 2 new tests: empty-fleet tile shows "no gaming pattern detected", populated fixture surfaces the flagged instance by name
  • Measured coverage directly (not whole-file): src/orb/analytics.ts scoped to its own test file — 100% statements/branches/functions/lines
  • Confirmed the literal word "farming" cannot appear anywhere in the operator dashboard payload (this repo's own FORBIDDEN_EXPORT_TERMS check, mirroring src/signals/redaction.ts's public-safe vocabulary) — renamed the field/labels to "gaming-pattern" throughout; "anti-farming" stays in source comments only, never serialized

…b fleet (#2350)

Extends computeFleetAnalytics (src/orb/analytics.ts) with a more targeted
detection signal than the existing single-metric outlier check: an instance
whose decided-PR volume is unusually high, whose merge-precision is
unusually high, AND whose reversal-rate is unusually low -- all three
simultaneously -- fits the exact "mass-submitting only trivially-safe PRs to
inflate merge-precision" signature the issue describes. A high-precision
instance alone isn't suspicious (could be a genuinely careful team); combined
with abnormal volume and suspiciously few reversals, it is.

Detection only: gamingPatternFlags is a new read-only field surfaced on the
existing operator dashboard tile set and the operator-only MCP
gittensory_get_fleet_analytics tool summary -- nothing acts on it
automatically, nothing here touches the live gate, and instanceId is the
same opaque HMAC-derived handle already used throughout this pipeline
(orb-collector.ts), never a login or anything more identifying.

Scope note (see the module's own doc comment): this flags a self-hosted
INSTANCE, never an individual miner -- the fleet pipeline carries no
per-actor identity by deliberate, repeatedly-documented design
(review_audit has no login column; predicted_gate_calibration_ledger is
explicitly never-exported, citing this issue as the reason why). A genuine
per-miner detector would require adding a new anonymized per-actor signal to
the export pipeline, a separate privacy-sensitive design decision deserving
its own focused issue.

Also out of scope: "duplicate-claim-election win-rate skew"
(isDuplicateClusterWinnerByClaim) is not implemented. Its outcome is never
persisted anywhere in this pipeline -- only the losing side of a duplicate
cluster produces a finding, with no cluster id and no actor linkage, so
there is no winner data to measure a win-rate from. No proxy is implemented;
a misleading one would be worse than none.

Advances #2350 (does not close it -- the win-rate-skew and per-miner
sub-deliverables remain genuinely unbuildable from data that exists today)
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.20%. Comparing base (74afe7d) to head (53f761e).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5083   +/-   ##
=======================================
  Coverage   94.20%   94.20%           
=======================================
  Files         470      470           
  Lines       39705    39719   +14     
  Branches    14489    14492    +3     
=======================================
+ Hits        37404    37418   +14     
  Misses       1645     1645           
  Partials      656      656           
Flag Coverage Δ
shard-1 46.40% <93.33%> (+<0.01%) ⬆️
shard-2 33.62% <66.66%> (+0.05%) ⬆️
shard-3 31.36% <93.33%> (+0.03%) ⬆️
shard-4 32.63% <26.66%> (-0.16%) ⬇️
shard-5 33.47% <20.00%> (-0.10%) ⬇️
shard-6 45.10% <26.66%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/mcp/server.ts 95.62% <ø> (ø)
src/orb/analytics.ts 100.00% <100.00%> (ø)
src/services/operator-dashboard.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 1822fca into main Jul 11, 2026
17 checks passed
@JSONbored
JSONbored deleted the feat/orb-fleet-farming-detector-2350 branch July 11, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

maintainer: fleet-wide anti-farming detector — flag a miner/fleet gaming the calibration score

1 participant