Skip to content

feat(review): record MCP predicted-gate verdicts to review_audit and measure predicted-vs-live gate agreement #4516

Description

@JSONbored

Context

A maintainer review-stack × Autonomous Miner System (AMS) integration audit (2026-07-09) found that the whole self-review premise ("the predicted gate runs the exact same engine the maintainer PR pipeline runs") is never actually checked empirically — predicted-vs-live gate agreement is completely unmeasured.

Evidence

  • src/mcp/server.ts:2879-2939computePredictedGateVerdict/predictGate/explainGateDisposition never call any review_audit write.
  • The only two review_audit gate_decision/pr_outcome writers are src/review/parity-wire.ts:156 (recordNativeGateDecision) and src/review/outcomes-wire.ts:229 — both keyed to a real project#pr@sha target, and neither is ever called from the MCP prediction path.
  • src/review/parity.tscomputeGateParity/computeGateEval (the confusion-matrix / cutover-readiness engine) already exists and is used at ~10 real call sites, but is documented and scoped specifically for the one-time reviewbot→gittensory-native migration and live-gate accuracy — never invoked for a "predicted" writer source.
  • packages/gittensory-engine/src/predicted-gate.ts:33-35 — the "it runs the EXACT same engine" claim is real for the code path, but is an unenforced claim with respect to actual outcome agreement.

Why this matters for the Autonomous Miner System

Phase 7 calibration (the Phase7CalibrationLoopResult contract, the documented 0.62 historical baseline, REVERSAL_DISCOUNT_WEIGHT) has no review-stack-side ground truth for "was this specific miner prediction later confirmed by the real gate" — only whatever the miner's own local, unverifiable prediction-ledger self-reports. Once the still-open local iterate-loop (#2333/#2334) starts making autonomous submit-or-iterate decisions based on its own self-review, the review stack itself should be able to independently answer "how often does our own predicted gate agree with our own live gate," rather than relying entirely on miner-side self-reporting.

Deliverables

  • Record every MCP predict_gate call as a review-stack-owned row (a new source, e.g. source: "predicted"), correlated by repo + login + time-window (note: predictGateShape currently has no PR-number field — it's an explicit pre-PR-existence dry run — so this needs a correlation-key design rather than reusing recordNativeGateDecision's exact project#pr@sha keying)
  • Extend computeGateEval/computeGateParity (or a sibling function) to join a "predicted" row against the eventual real gate_decision/pr_outcome for the same repo+login+PR once it materializes
  • Surface a predicted-vs-live agreement rate as its own metric, distinct from the existing reviewbot-migration parity use case
  • Tests: agreement/disagreement join logic, correlation-key matching when a PR number is supplied later, no regression to the existing migration-parity use of computeGateEval

Surfaced by a maintainer review-stack × AMS integration audit (2026-07-09). Related: #4257 (a dev-time golden-fixture cross-check between the two engine copies — a different, static mechanism than this issue's live/production reconciliation), #2349 (personalized gate-prediction tuning, which would consume this data once it exists), #7 below (a login-keyed persistence layer for the same underlying gap, framed around data ownership rather than measurement).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.orbGittensory Orb related - maintainer self-hosting analytics.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions