Skip to content

fix(review): reconcile AI verdicts against hard structured state before rendering #2996

Description

@JSONbored

Context

We have already had to patch, as a one-off, a real production incident where the review disposition recommended approve/merge while a hard gate blocker was still active -- the AI verdict and the structured gate state disagreed, and the AI's language won out in what got rendered. That fix addressed the specific shape of the bug it was found in, but the underlying failure mode -- an AI-generated verdict rendered without being checked against known hard structured facts -- is not closed as a class. It can recur in a new shape: a duplicate/linked PR that should suppress this one, a linked issue that has since been unlinked or closed, or a different blocker type than the one already patched.

src/services/ai-review.ts owns verdict rendering (see consensusDefectOf and the combine-strategy resolution around resolveAiReviewPlan) and already has access to the structured signals the gate consults. src/signals/duplicate-winner.ts encodes the linked-issue-claim precedence logic for duplicate clusters, and is one example of a structured fact source this reconciliation layer needs to cross-check against.

Goal

Add a deterministic reconciliation pass that runs on every AI-generated verdict before it is rendered, checking it against known hard structured state and correcting or flagging contradictions.

Requirements

  • Run the reconciliation pass unconditionally between AI verdict generation and rendering in src/services/ai-review.ts, not as an opt-in flag -- this closes a failure class, not a feature.
  • Check the verdict against, at minimum: active gate blockers, duplicate/linked PR state per src/signals/duplicate-winner.ts semantics, and linked-issue state (open/closed/unlinked).
  • On contradiction (e.g., verdict text implies approve/merge while a hard blocker is active), the deterministic structured state wins: either rewrite the rendered verdict to reflect the hard state, or suppress the contradictory language and attach an explicit reconciliation-flag finding -- never let the AI's contradictory language reach the rendered output unmodified.
  • Log every reconciliation intervention (contradiction detected + correction applied) to the existing audit/observability path so interventions are auditable and their frequency can be tracked over time.
  • Treat this as gate/verdict-adjacent and safety-sensitive: no new external dependencies, fail closed (block/flag rather than silently pass through) if the structured-state lookup itself errors.
  • Add a regression test reproducing the original incident's shape (approve/merge verdict text alongside an active blocker) plus at least one new shape (duplicate/linked-PR contradiction, unlinked-issue contradiction) to prove the guard generalizes rather than re-patching the single known case.

Deliverables

  • Deterministic reconciliation pass wired unconditionally into the verdict-rendering path in src/services/ai-review.ts
  • Contradiction checks against gate blockers, duplicate/linked-PR state, and linked-issue state
  • Audit logging for every detected contradiction and applied correction
  • Fail-closed behavior on structured-state lookup errors
  • Regression tests for the original incident shape plus at least one additional contradiction shape

Expected outcomes

  • Zero rendered verdicts contradicting known hard structured state, verified by the regression suite covering multiple contradiction shapes
  • Every reconciliation intervention is visible in audit logs, not just the original patched case
  • The original incident's fix becomes a special case of this general guard rather than a standalone patch

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions