Skip to content

orb(ai-review): a model that declared it could NOT review the diff can still emit blockers that close the PR #9087

Description

@JSONbored

src/services/ai-review.ts ~911-928:

if (assessment === INCOHERENT_DIFF_ASSESSMENT) {
  if (valueAssessment && valueAssessment.rationale.length >= SCOPE_RECLASSIFY_MIN_RATIONALE_CHARS) {
    return { assessment: SCOPE_MISMATCH_ASSESSMENT, blockers, nits, suggestions, inlineFindings, confidence, valueAssessment };

The system prompt (~99) instructs the model to bail with INCOHERENT_DIFF_ASSESSMENT and return empty blockers when the diff is unusable. A model that bails, violates that instruction, and happens to write a ≥40-character rationale has its blockers promoted into a usable review.

Under combine: "single" — which our live config uses (claude-code,ollama = single + fallback) — combineReviews (~2333-2342) turns any lone blocker into a full ai_consensus_defect: severity critical, published as "AI reviewers agree on a likely critical defect" (src/queue/ai-review-orchestration.ts ~758) for a one-reviewer run. Under aiReviewGateMode: block that is a close.

Before the #8789 reclassification this produced ai_review_inconclusive → neutral hold, which is the correct fail-closed answer.

Fix

  1. On reclassification, keep the valueAssessment (the point of fix(ai-review): the incoherent-diff bail misfires on title-vs-diff scope mismatch, turning reviewable PRs into inconclusive holds #8789) but drop blockers — a model that said it could not read the diff has not earned blocker authority over it.
  2. Fix the plural copy when plannedReviewerCount === 1 — see orb(ai-review): 'AI reviewers agree' is never verified — consensus fires whenever BOTH have any blocker, not the same one (427 closes) #9074, where the same string asserts agreement that was never checked.

Related, same file

plannedReviewerCount (~2788) is computed and has no consumer. Under consensus, a 2→1 degradation correctly fails closed to inconclusive (~2386) — but under synthesis + either, missing > 0 with one flagging reviewer returns a full-strength defect (~2361-2366) with nothing recording that only one reviewer ran. Emit loopover_ai_review_reviewer_degraded_total{planned,actual} and put plannedReviewerCount in the decision record so calibration can segment single-reviewer blocks.

Refs #9074, #8789, #8833.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions