Skip to content

fix(review): stop labeling an advisory-only AI consensus defect a "Blocker" - #2673

Merged
JSONbored merged 1 commit into
mainfrom
fix/ai-consensus-defect-advisory-labeling
Jul 3, 2026
Merged

fix(review): stop labeling an advisory-only AI consensus defect a "Blocker"#2673
JSONbored merged 1 commit into
mainfrom
fix/ai-consensus-defect-advisory-labeling

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Investigated #2592 end-to-end (supersedes #2002):

  • aiReviewGateMode (src/rules/advisory.ts) already exists and already provides a real, config-driven way for a maintainer to make an AI consensus defect actually block merge — block mode promotes ai_consensus_defect/ai_review_split into the gate's own hard blockers. It's off by default.
  • The check-run conclusion that actually drives GitHub's merge decision (gateEvaluation.conclusion, posted via the check-run) is completely independent of deriveUnifiedStatus's comment-only rendering — confirmed by tracing both paths in src/queue/processors.ts.
  • The real, narrower bug: runGittensoryAiReview (src/queue/processors.ts:5378) unconditionally pushes an ai_consensus_defect finding into advisory.findings whenever the two AI reviewers agree on a critical defect, regardless of aiReviewGateMode. The unified-comment bridge (consensusDefectFromFindings in src/review/unified-comment-bridge.ts) unconditionally recovered it into the comment's "Blockers" section. Net effect: by default (mode unset/advisory), a PR's check-run reports success (auto-merges) while the SAME PR's own comment lists a "critical defect" under "Blockers" — internally contradictory.

Decision

Chose option 3 from the issue (confirm the gate's block/advisory decision is correct as designed) — aiReviewGateMode: block already exists for maintainers who want this to be real; a blanket default change (option 1) risks fleet-wide false-positive closes, and a comment-only veto (option 2, downgrading deriveUnifiedStatus to "held") would be worse than today given the check-run/comment split: the comment would claim manual review is needed while the check-run still silently auto-merges.

Plus a scoped accuracy fix, distinct from the three enumerated options: only surface the AI consensus defect under "Blockers" when the gate itself actually promoted it (present in gate.blockers, i.e. aiReviewGateMode: "block" is configured). Otherwise fold it into the non-blocking Nits section, clearly labeled (advisory only — not configured to block merge), so the comment never claims a merge is blocked when it will not be. The gate's own decision logic is completely untouched.

Closes #2592

Test plan

  • npm run typecheck
  • npx vitest run test/unit/unified-comment-bridge.test.ts — 59/59 pass, including 3 updated tests (now pass gateBlockers to keep asserting the real-blocker path) and 3 new tests covering the advisory-only path (routes to Nits, survives as the sole reviewer-side content, coexists with an unrelated real gate blocker)
  • npx vitest run test/unit/queue.test.ts — 368/368 pass unmodified (full processors.ts wiring unaffected)
  • npm run test:ci (full local gate, unsharded)
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • 100% branch coverage on every changed line

…ocker"

Investigated #2592 end-to-end: the gate already has a real, config-driven
way to make an AI consensus defect actually block merge
(aiReviewGateMode: "block", src/rules/advisory.ts) -- off by default. The
real bug is narrower: runGittensoryAiReview unconditionally pushes
ai_consensus_defect into advisory.findings whenever the two AI reviewers
agree on a critical defect, REGARDLESS of aiReviewGateMode, and the
unified-comment bridge unconditionally recovered it into the "Blockers"
section. So by default (mode unset/advisory), a PR could show a check-run
conclusion of "success" (auto-merging) while its own comment listed a
"critical defect" under "Blockers" -- confusing at best, actively
misleading at worst, since the check-run conclusion (which drives GitHub's
real merge decision) is completely independent of this comment-only
rendering path.

Chose option 3 from the issue (confirm the gate's block/advisory decision
is correct as designed -- a blanket default change risks fleet-wide false-
positive closes, and a comment-only "held" veto would be worse than today
given the check-run/comment split: the comment would claim manual review
is needed while the check-run still auto-merges), plus a scoped accuracy
fix: only surface the defect under "Blockers" when the gate itself
promoted it (present in gate.blockers, i.e. aiReviewGateMode: "block" is
actually configured). Otherwise fold it into the non-blocking Nits,
clearly labeled "advisory only — not configured to block merge", so the
comment never claims a merge is blocked when it will not be.

Closes #2592
@dosubot dosubot Bot added the size:S label Jul 3, 2026
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.10%. Comparing base (642b12d) to head (a75f5b0).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2673   +/-   ##
=======================================
  Coverage   96.10%   96.10%           
=======================================
  Files         237      237           
  Lines       26538    26542    +4     
  Branches     9624     9625    +1     
=======================================
+ Hits        25505    25509    +4     
  Misses        424      424           
  Partials      609      609           
Files with missing lines Coverage Δ
src/review/unified-comment-bridge.ts 96.29% <100.00%> (+0.14%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit b3244d1 into main Jul 3, 2026
11 checks passed
@JSONbored
JSONbored deleted the fix/ai-consensus-defect-advisory-labeling branch July 3, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(review): decide whether AI-judgment blockers should gate the verdict, not just annotate it

1 participant