feat(review): discount reverted/reopened outcomes in the gate-eval fold - #4169
Conversation
computeGateEval's mergePrecision/closePrecision treats every PR as equally weighted, so a miner (or the fleet) could game the accuracy number with high volumes of barely-passing PRs that later get reverted. Adds weightedMergeConfirmed/weightedCloseConfirmed alongside the existing raw counts, discounting a merge/close later marked reversal_reverted/ reversal_reopened by REVERSAL_DISCOUNT_WEIGHT (a hardcoded, documented constant -- not runtime-tunable, per the issue's auditability requirement). The denominator (wouldMerge/wouldClose) is unchanged; only the credit for an outcome that didn't hold up is discounted. Raw fields are byte-identical to today for every existing consumer -- this is purely additive. Does not change what auto-tune.ts reads; that re-verification is explicit follow-up work per the issue, not bundled here.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4169 +/- ##
=======================================
Coverage 93.73% 93.74%
=======================================
Files 387 387
Lines 36304 36312 +8
Branches 13298 13301 +3
=======================================
+ Hits 34031 34039 +8
Misses 1617 1617
Partials 656 656
🚀 New features to boost your workflow:
|
|
Warning 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏸️ Gittensory review result - manual review recommendedReview updated: 2026-07-08 09:18:20 UTC
⏸️ Suggested Action - Manual Review
Review summary Nits — 5 non-blocking
Concerns raised — review before merging
Review context
Contributor next steps
Signal definitions
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
#2348) (#5088) planAutoTune/shouldAutoClear and their close-side mirrors now engage and clear on GateEvalRow's reversal-discounted weightedMergePrecision / weightedClosePrecision (already computed by parity.ts's computeGateEval, PR #4169) instead of the raw precision fields. A high volume of later- reverted merges can no longer keep the raw number artificially healthy to dodge the breaker. The raw fields are preserved on AutoTuneAction / CloseAutoTuneAction for log continuity and surfaced alongside the weighted number in the alert message. computeTuningRecommendations (the separate advisory-only tuning surface) is unchanged. evalRowFromCalibration's synthetic rows (selftune-wire.ts, sourced from agent_recommendation_outcomes) carry no reversal signal, so their weighted fields mirror the raw ones by construction.
Summary
computeGateEval'smergePrecision/closePrecisiontreats every PR as equally weighted — a miner (or the fleet) could game the accuracy number by producing high volumes of barely-passing PRs that later get reverted, since a later revert never affects the "was this prediction correct" measurement.weightedMergeConfirmed/weightedCloseConfirmed/weightedMergePrecision/weightedClosePrecisiontoGateEvalRow, additive alongside the existing raw fields — no existing field's meaning or value changes for any current consumer.reversal_reverted/reversal_reopened(the existing signalrecordReversalSignalsinsrc/review/outcomes-wire.tsalready writes) gets its credit toward the weighted-CONFIRMED bucket discounted byREVERSAL_DISCOUNT_WEIGHT— a new, hardcoded, documented module constant (currently0, full discount), not read from env/config, per the issue's explicit "not silently tunable at runtime" requirement.wouldMerge/wouldClose— how many merge/close predictions were made) is unchanged; only the credit for an outcome that didn't hold up is discounted. This meansweightedMergePrecision <= mergePrecisionalways, andweightedClosePrecision <= closePrecisionalways.revCTE (SELECT DISTINCT target_id FROM review_audit WHERE event_type IN ('reversal_reverted', 'reversal_reopened')) LEFT JOINed into the existinggd ⨝ poquery, withreversedadded as an extraGROUP BYdimension so the JS fold can apply the discount per-cell.Scope
CONTRIBUTING.md.auto-tune.ts).Validation
git diff --checknpm run typechecknpm run docs:drift-check— unaffected (no*GateMode/flag/command surface touched; this is a scoring-definition constant, not a config field).npx vitest run test/unit/parity.test.ts— 36/36 passing (30 pre-existing + 6 new). All 30 pre-existing tests pass unchanged — old test fixtures have noreversedfield at all, soc.reversedisundefined(falsy) and every existing fixture naturally falls through to unweighted behavior, proving byte-identical backward compatibility rather than just asserting it.wouldMerge/mergeConfirmed/mergePrecisionstay untouched; the same for a reversed (reopened) close; a reversedmergeFalse/closeFalsecell contributing nothing to either weighted bucket (reversal only ever discounts an already-CONFIRMED credit, never creates negative credit); weighted precisions stayingnull(not0/0) with no merge/close predictions at all; and a pinned-value test onREVERSAL_DISCOUNT_WEIGHTitself so a future silent formula change shows up as a failing test, not a silent behavior shift.src/review/parity.ts: 100% lines, 99.07% statements, 97.61% branches — the two remaining "uncovered" spots (lines with pre-existing/* v8 ignore next */pragmas) are insidecomputeGateParity, a function this PR does not touch (confirmed viagit diff— zero changes in that function's body), so they don't affect this PR's patch coverage.npm run test:workers,build:mcp,test:mcp-pack,ui:openapi:check,ui:lint,ui:typecheck,ui:build,npm audit— not re-run locally (no worker/MCP/OpenAPI/UI-component code touched); CI runs them authoritatively.Safety
Notes
src/review/auto-tune.ts'splanAutoTune/planCloseAutoTuneagainst the new weighted fields. They continue reading the rawmergePrecision/closePrecisionfields unchanged in this PR — switching the auto-tune circuit-breaker to the weighted definition is a deliberate, separate maintainer decision (the issue explicitly calls out that downstream consumers "must be re-verified against the new definition before cutover"), not something to fold silently into the fold-logic change itself.computeGateEval(e.g.src/services/review-recap.ts) keeps its historical (raw, unweighted) meaning unless a future change explicitly switches it to readweightedMergePrecision/weightedClosePrecisioninstead. This PR does not flip that switch anywhere.REVERSAL_DISCOUNT_WEIGHT = 0(full discount) was chosen over a partial-credit value as the most defensible, auditable default: it makes gaming via later-reverted volume completely unprofitable rather than merely more expensive, with no magic partial-credit number to justify. The constant's own doc comment notes how to change it (bump the value + comment, never make it runtime-configurable) if the maintainer later wants partial credit instead.