feat(review): fleet-wide gaming-pattern detector for the self-host orb fleet (#2350) - #5083
Merged
Merged
Conversation
…b fleet (#2350) Extends computeFleetAnalytics (src/orb/analytics.ts) with a more targeted detection signal than the existing single-metric outlier check: an instance whose decided-PR volume is unusually high, whose merge-precision is unusually high, AND whose reversal-rate is unusually low -- all three simultaneously -- fits the exact "mass-submitting only trivially-safe PRs to inflate merge-precision" signature the issue describes. A high-precision instance alone isn't suspicious (could be a genuinely careful team); combined with abnormal volume and suspiciously few reversals, it is. Detection only: gamingPatternFlags is a new read-only field surfaced on the existing operator dashboard tile set and the operator-only MCP gittensory_get_fleet_analytics tool summary -- nothing acts on it automatically, nothing here touches the live gate, and instanceId is the same opaque HMAC-derived handle already used throughout this pipeline (orb-collector.ts), never a login or anything more identifying. Scope note (see the module's own doc comment): this flags a self-hosted INSTANCE, never an individual miner -- the fleet pipeline carries no per-actor identity by deliberate, repeatedly-documented design (review_audit has no login column; predicted_gate_calibration_ledger is explicitly never-exported, citing this issue as the reason why). A genuine per-miner detector would require adding a new anonymized per-actor signal to the export pipeline, a separate privacy-sensitive design decision deserving its own focused issue. Also out of scope: "duplicate-claim-election win-rate skew" (isDuplicateClusterWinnerByClaim) is not implemented. Its outcome is never persisted anywhere in this pipeline -- only the losing side of a duplicate cluster produces a finding, with no cluster id and no actor linkage, so there is no winner data to measure a win-rate from. No proxy is implemented; a misleading one would be worse than none. Advances #2350 (does not close it -- the win-rate-skew and per-miner sub-deliverables remain genuinely unbuildable from data that exists today)
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5083 +/- ##
=======================================
Coverage 94.20% 94.20%
=======================================
Files 470 470
Lines 39705 39719 +14
Branches 14489 14492 +3
=======================================
+ Hits 37404 37418 +14
Misses 1645 1645
Partials 656 656
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This was referenced Jul 11, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
computeFleetAnalytics(src/orb/analytics.ts) withgamingPatternFlags: a more targeted detection signal than the existing single-metric outlier check. An instance whose decided-PR volume is unusually high (>2x fleet median), whose merge-precision is unusually high (>fleet median + 0.25), and whose reversal-rate is unusually low (<0.5x fleet median) — all three simultaneously — fits the "mass-submitting only trivially-safe PRs to inflate merge-precision" signature the issue describes. High precision alone isn't suspicious (could be a genuinely careful team); combined with abnormal volume and suspiciously few reversals, it is.gittensory_get_fleet_analyticstool summary. Detection only — nothing acts on it automatically, nothing touches the live gate.instanceIdis the same opaque, HMAC-derived handle already used throughout this pipeline (orb-collector.ts) — never a login or anything more identifying.Scope (verified against the actual pipeline before implementing)
review_audithas no login column, andpredicted_gate_calibration_ledgeris explicitly documented as never-exported, citing this issue by number as the reason why. A genuine per-miner detector would require adding a new anonymized per-actor signal to the export pipeline, a separate privacy-sensitive design decision that deserves its own focused issue/PR rather than a rushed addition here.isDuplicateClusterWinnerByClaim's outcome is never persisted anywhere in this pipeline — only the losing side of a duplicate cluster produces a finding (duplicate_pr_risk), with no cluster id and no actor linkage. There is no winner data to measure a win-rate from, and a per-instanceduplicate_riskrate would measure something different (how often this instance's own PRs lose a local collision) than "identities farming wins." No proxy is implemented — a misleading one would be worse than none.Given these two structural gaps, this uses "Advances #2350" rather than closing it.
Test plan
npm run typecheckcleantest/unit/orb-analytics.test.ts— 14 new tests: normal-distribution fixture (no flag), inflated-trivial-volume fixture (flags, with exact expected numbers), high-precision-alone / high-volume-alone / high-volume-and-precision-but-normal-reversal (each individually insufficient), null-mergePrecision instance never flags even alongside a real farmer, no-merge-verdict-at-all fleet, unregistered instance never flags, below-MIN_DECIDEDinstance never flags, empty store, DB-error fail-safetest/unit/operator-dashboard.test.ts— 2 new tests: empty-fleet tile shows "no gaming pattern detected", populated fixture surfaces the flagged instance by namesrc/orb/analytics.tsscoped to its own test file — 100% statements/branches/functions/linesFORBIDDEN_EXPORT_TERMScheck, mirroringsrc/signals/redaction.ts's public-safe vocabulary) — renamed the field/labels to "gaming-pattern" throughout; "anti-farming" stays in source comments only, never serialized