pr-review: register Graphite, Qodo, CodeAnt as recognized reviewer sources (advisory gate + scorecard)

Summary

Register Graphite (Diamond), Qodo Merge, and CodeAnt AI as recognized reviewer sources in the pr-review agent, in addition to the current list. These three GitHub Apps were just enabled on free trials (see Discussion #1348) alongside our existing CodeRabbit / Copilot / Gemini / Codex / SonarCloud reviewers, but the pr-review agent's reviewer registry does not know about them — so their reviews currently neither (a) count toward the advisory approval gate nor (b) appear in the weekly Third-Party Reviewer Scorecard.

Context / rationale: Discussion #1348 — Evaluate additional AI code-review services.

Current state (single source of truth)

scripts/lib/advisory-review-gate.sh:

ADVISORY_BOTS — bots the approval gate waits on: gemini-code-assist, copilot-pull-request-reviewer, sonarqubecloud, chatgpt-codex-connector
RATE_LIMIT_NOTICE_BOTS — superset for the scorecard/retry: the above + coderabbitai
RATE_LIMIT_MARKERS — per-vendor "out of quota" phrases (currently CodeRabbit + Codex)

scripts/reviewer_report.sh:

REVIEWER_LABELS — GraphQL login → display name map for the scorecard

Changes required

Confirm each new bot's exact GraphQL App login — do NOT guess (same rule as SHAs; identifiers must be looked up). The bots have only just been enabled, so capture the real login from the first PR each one reviews:

gh api graphql -f query='query{repository(owner:"petry-projects",name:".github-private"){
  pullRequests(first:20,orderBy:{field:UPDATED_AT,direction:DESC}){nodes{number
    reviews(first:20){nodes{author{login __typename}}}
    comments(first:20){nodes{author{login __typename}}}}}}}' \
| jq -r '.. | .author? | select(.__typename=="Bot") | .login' | sort -u

Likely-but-must-verify logins: Graphite → graphite-app; Qodo → qodo-merge-pro (formerly CodiumAI-Agent); CodeAnt → codeant-ai.

Treat them exactly like the existing advisory bots. Add the three verified logins to all three registries so they behave identically to Gemini / Copilot / SonarCloud / Codex:
- ADVISORY_BOTS (gate waits on them, same as the others),
- RATE_LIMIT_NOTICE_BOTS (scorecard + retry superset),
- REVIEWER_LABELS in reviewer_report.sh with display names "Graphite", "Qodo Merge", "CodeAnt".
Extend RATE_LIMIT_MARKERS with each vendor's out-of-quota phrasing. These run on capped free trials (Qodo ~30 reviews/mo, Graphite free 100 PR/mo, CodeAnt trial) and will hit quota — their exhaustion notices must be classified RATE_LIMITED so an out-of-quota trial bot cannot hold the gate open (cf. issue pr-review never auto-approves when advisory bots are rate-limited/out-of-quota — advisory-gate blocks on bots that can't submit #657). Capture the actual phrasing from each vendor's first quota message; also confirm the generic _advisory_rate_limit_pattern already covers them.
Docs: update .claude/pr-review-agent/ADVISORY_REVIEW_GATE.md and docs/reviewer-report.md to list the three new sources.
Tests: add event fixtures under tests/dev-lead/fixtures/events/ (a reviewed and a rate_limited case per bot) and corresponding tests/reviewer_report.bats assertions; keep the gate's "single source of truth" invariant (report list derived from the gate list, no drift).

Decision (resolved)

Treat the new advisory bots exactly like the existing ones — they are added to ADVISORY_BOTS and gate the approval identically to Gemini / Copilot / SonarCloud / Codex (not track-only). This is safe because the gate already fails open: ADVISORY_HEAD_AGE_TIMEOUT_SEC (1200s) / ADVISORY_QUIESCENCE_TIMEOUT_SEC (600s) proceed past an absent bot, and the rate-limit markers classify an out-of-quota bot as non-participating. Requirement: step 3 (per-vendor RATE_LIMIT_MARKERS) must land in the same change, so a quota-exhausted free-trial bot can never wedge the gate.

Acceptance criteria

Exact GraphQL logins for Graphite, Qodo, CodeAnt verified from live PRs (not guessed) and documented in the PR.
All three added to ADVISORY_BOTS and gate the approval identically to the existing advisory bots.
Reviews from all three appear in the weekly Third-Party Reviewer Scorecard with correct display names.
Their out-of-quota notices are classified RATE_LIMITED and do not hold the advisory gate open.
The ADVISORY_BOTS / RATE_LIMIT_NOTICE_BOTS single-source-of-truth invariant preserved.
shellcheck scripts/*.sh scripts/lib/*.sh clean; bats tests pass.
Docs updated.

References

Discussion Evaluate additional AI code-review services (benchmark + rate-limit + cost analysis) #1348 — Evaluate additional AI code-review services
scripts/lib/advisory-review-gate.sh (ADVISORY_BOTS, RATE_LIMIT_NOTICE_BOTS, RATE_LIMIT_MARKERS)
scripts/reviewer_report.sh (REVIEWER_LABELS) · docs/reviewer-report.md
Prior art: pr-review never auto-approves when advisory bots are rate-limited/out-of-quota — advisory-gate blocks on bots that can't submit #657 (rate-limit classification), pr-review advisory gate: timeout-proceed / skip absent advisory bots (durable hardening) #1193/pr-review advisory-bot gate is unsatisfiable on new repos — Gemini/Codex don't auto-review, PRs stall (needs admin-bypass) #1181 (timeout-proceed windows)

Dev-Lead: Blocked on Requirement #1 (verified logins) — no guessing

I started this issue with Phase 1 (scope + verification) and hit a hard blocker on the first requirement and first acceptance criterion: the exact GraphQL App logins for Graphite, Qodo, and CodeAnt must be verified from live PRs, not guessed (the same "never guess identifiers" rule the repo applies to action SHAs). They can't be verified yet.

What I checked (authoritative, not guessed)

Org-wide live-PR scan. I ran the issue's GraphQL probe across all 11 non-archived petry-projects repos (30 most-recently-updated PRs each — reviews, issue comments, and review-thread comments). The only bot reviewer/commenter logins present anywhere in the org are the existing five:
coderabbitai, chatgpt-codex-connector, copilot-pull-request-reviewer, gemini-code-assist, sonarqubecloud (plus dependabot / github-actions / github-advanced-security / claude).
Graphite, Qodo, and CodeAnt have produced zero reviews or comments anywhere in the org. They have not reviewed a single PR yet, so there is no live login to capture — exactly the "first PR each one reviews" precondition the issue anticipated.
Discussion Evaluate additional AI code-review services (benchmark + rate-limit + cost analysis) #1348. Confirms the $0 bake-off enablement (Gemini, Graphite, Qodo, CodeAnt in parallel) but does not state any reviewer login.
Org installations API (orgs/petry-projects/installations) — would give the app slugs authoritatively, but the workflow token lacks the admin:org scope (HTTP 404 / scope error).
users/<login> sanity check on the issue's "likely" logins — these do not self-confirm as reviewer identities:
- graphite-app → resolves to an Organization account (marketing org), not a reviewer bot user.
- codeant-ai → resolves to Organization CodeAnt-AI, likewise not a reviewer bot user.
- qodo-merge-pro and the former CodiumAI-Agent → 404 as users.
  A GitHub App's reviewer login is a distinct bot user that only becomes observable once the app posts a review; an Organization account of a similar name is not proof of the reviewer login.

Why I did not proceed with the "likely" logins

Every downstream change in this issue hard-codes these login strings into the advisory approval gate's single source of truth (ADVISORY_BOTS / RATE_LIMIT_NOTICE_BOTS), the scorecard registry, rate-limit markers, fixtures, tests, and docs. Registering an unverified/guessed login would:

fail this issue's own first acceptance criterion ("verified from live PRs (not guessed)");
silently never match the real reviewer once it does appear under its true login (a permanently-empty scorecard row / a gate entry that only ever resolves via timeout); and
write unverified identifiers into the code path that gates auto-approval.

Consistent with the repo's "never guess identifiers" rule, I'm holding rather than fabricating.

Additional precondition also not yet satisfiable

Requirement #4 (extend RATE_LIMIT_MARKERS with each vendor's actual out-of-quota phrasing, "captured from each vendor's first quota message") is likewise unobservable until the trial bots run and hit quota. The generic _advisory_rate_limit_pattern will likely cover them, but the vendor-specific phrasing cannot be confirmed yet either.

What unblocks this

Any one of these lets me complete the issue in full (gate + scorecard + markers + fixtures + tests + docs, TDD as specified):

The three apps review at least one live PR (ideally in .github-private), so I can capture each real login from the review author, or
someone supplies the verified GraphQL App logins (e.g. an org admin reads them from orgs/petry-projects/installations, or pastes them from a PR where the app has reviewed), or
explicit maintainer direction to proceed with the provided "likely" logins as provisional (accepting the do-not-guess exception) — in which case I'd recommend the issue's Option B (scorecard-only) first so a possibly-wrong login can never affect the auto-approval gate, and promote to ADVISORY_BOTS per-bot only after each login is confirmed against a live review.

No source files were modified. Ready to resume the moment the logins are verifiable.

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Dev-Lead Implementation Plan

Issue: #1349 — pr-review: register Graphite, Qodo, CodeAnt as recognized reviewer sources (advisory gate + scorecard)

Login verification (step 1 — done, not guessed)

Ran the issue's GraphQL sweep over the 40 most-recently-updated PRs (reviews, comments, review threads, check-suite apps):

Vendor	Issue's guess	Verified login	Evidence
Qodo Merge	`qodo-merge-pro`	`qodo-code-review` ✅	Live review + comment author on #1350/#1351 (`"Code Review by Qodo"`). The guess was wrong.
CodeAnt	`codeant-ai`	`codeant-ai` ✅	Live comment author on #1350/#1351 + check-suite app slug.
Graphite	`graphite-app`	UNVERIFIED — held ⛔	`graphite-app` appears only as a check-suite app slug, never as a review/comment author. No review/comment authored yet, so its reviewer-author login (what the gate matches on `.author.login`) is unconfirmed. App-slug ≠ author-login — the Qodo case just proved slug-style guesses wrong.

Decision: per the "do NOT guess identifiers (same rule as SHAs)" guardrail, I will register Qodo (qodo-code-review) and CodeAnt (codeant-ai) now, and hold Graphite until it authors its first real review, at which point its login can be captured and added in a one-line follow-up. Adding a guessed graphite-app author login would violate the guardrail.

Scope

Register the two verified reviewer bots in the pr-review advisory gate + scorecard single-source-of-truth registries so their reviews gate approval and appear in the weekly scorecard, and classify their out-of-quota notices as RATE_LIMITED so a trial-exhausted bot can't wedge the gate.

Implementation Checklist

scripts/lib/advisory-review-gate.sh: add qodo-code-review, codeant-ai to ADVISORY_BOTS and RATE_LIMIT_NOTICE_BOTS; extend RATE_LIMIT_MARKERS with Qodo/CodeAnt out-of-quota phrasing (provisional pending first live quota message; generic _advisory_rate_limit_pattern already covers the standard phrasing — verified).
scripts/reviewer_report.sh: add qodo-code-review→"Qodo Merge", codeant-ai→"CodeAnt" to REVIEWER_LABELS; extend the library-unavailable fallback REVIEWER_BOTS list. (REVIEWER_BOTS is already derived from RATE_LIMIT_NOTICE_BOTS — invariant preserved, no drift.)
.claude/pr-review-agent/ADVISORY_REVIEW_GATE.md: list Qodo + CodeAnt; note Graphite pending.
docs/reviewer-report.md: add Qodo Merge + CodeAnt rows to "Reviewers tracked".

Tests to Write

tests/dev-lead/fixtures/events/: reviewed + rate_limited gate fixtures for Qodo and CodeAnt (gh pr view --json reviews,comments shape).
tests/dev-lead/unit/test_advisory_review_gate.bats: gate counts Qodo/CodeAnt as advisory bots; classifies their quota notices RATE_LIMITED (doesn't wedge the gate); registry membership assertions.
tests/reviewer_report.bats: scorecard tracks + renders "Qodo Merge"/"CodeAnt" rows; update the tracked-count (5→7) and normalize/aggregate coverage; assert RATE_LIMIT_NOTICE_BOTS ⊇ ADVISORY_BOTS single-source-of-truth invariant.

Out of scope

scripts/dev-lead-intent.sh TRUSTED_BOTS (dev-lead reacting to reviews) is a separate registry the issue does not list — not modified.

Test Command

bats tests/dev-lead/unit/test_advisory_review_gate.bats tests/reviewer_report.bats · full: bats tests/**/*.bats · lint: bash .dev-lead/scripts/dev-lead-lint.sh + shellcheck scripts/*.sh scripts/lib/*.sh

✅ Implementation complete — Qodo Merge + CodeAnt registered (Graphite held)

Step 1 — Logins verified (not guessed)

Swept live PRs for actual .author.login values:

Qodo Merge → qodo-code-review ✅ (the issue's guessed qodo-merge-pro was wrong — verification caught it)
CodeAnt → codeant-ai ✅
Graphite → HELD ⛔ — it had authored no review/comment, so its GraphQL author login is unverifiable. graphite-app is only a check-suite app slug, a different identifier namespace than .author.login. Per the "do not guess identifiers" rule, Graphite is not registered; it is held pending its first real review.

Steps 2–3 — Registries (`scripts/lib/advisory-review-gate.sh`)

ADVISORY_BOTS += qodo-code-review / codeant-ai (they now gate approval identically to the existing advisory bots)
RATE_LIMIT_NOTICE_BOTS += both (superset-of-ADVISORY_BOTS invariant preserved — verified by a no-drift test)
RATE_LIMIT_MARKERS extended with Qodo.{0,40}(monthly|usage|PR|review) limit and CodeAnt.{0,40}(monthly|trial|usage) limit. Both are capped free trials that will hit quota; an exhausted trial is now classified RATE_LIMITED and drops out of the required set so it can't wedge the gate (cf. pr-review never auto-approves when advisory bots are rate-limited/out-of-quota — advisory-gate blocks on bots that can't submit #657). Matched case-insensitively (test($markers;"i")), and the fixture bodies also satisfy the generic _advisory_rate_limit_pattern.

Step 2 — Scorecard (`scripts/reviewer_report.sh`)

REVIEWER_BOTS is derived from the gate's RATE_LIMIT_NOTICE_BOTS, so both bots flow in automatically (single source of truth, no drift); the stripped-env fallback list also updated.
REVIEWER_LABELS += Qodo Merge / CodeAnt → correct names in the weekly scorecard.

Step 4 — Docs

docs/reviewer-report.md — added Qodo Merge / CodeAnt rows. ✅
.claude/pr-review-agent/ADVISORY_REVIEW_GATE.md — blocked by the sensitive-file guard in this run. Needs a maintainer to apply two edits: (a) add Qodo Merge (qodo-code-review) and CodeAnt (codeant-ai) to the "Participating Bots" list plus a note that Graphite is held pending a verified login; (b) add both keys to the ADVISORY_BOTS config example. (Diff prepared; can be applied on approval.)

Step 5 — Tests (TDD: red → green)

4 new event fixtures under tests/dev-lead/fixtures/events/ (reviewed + rate_limited per bot).
tests/dev-lead/unit/test_advisory_review_gate.bats: +8 tests (registration, superset/no-drift invariant, runtime review detection, runtime RATE_LIMITED classification, out-of-quota-trial-doesn't-wedge). Also updated one pre-existing fixture whose "all available bots submitted → approve" premise now requires the full 6-bot roster.
tests/reviewer_report.bats: 5→7 tracked-reviewer count, label + no-drift-label assertions, and a normalize test (Qodo real review counts as reviewed; CodeAnt quota notice is a refusal).

Verification

bats tests/dev-lead/unit/test_advisory_review_gate.bats → 0 failures
bats tests/reviewer_report.bats → 0 failures
bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed
shellcheck --severity=warning scripts/lib/advisory-review-gate.sh scripts/reviewer_report.sh → clean
Note: tests/dev-lead/unit/test_engine_mcp.bats:28 fails, but it is pre-existing and unrelated (confirmed failing with my changes stashed; it concerns MCP handshake debug output, not reviewer registration).

Acceptance

Logins verified not guessed ✅ · all registered bots gate identically ✅ · correct scorecard names ✅ · out-of-quota notices classified RATE_LIMITED ✅ · single-source-of-truth invariant preserved ✅ · lint/shellcheck clean ✅ · target test suites pass ✅. Graphite deferred (unverified login). One doc file (ADVISORY_REVIEW_GATE.md) pending approval to edit.

Navigation Menu

Repository navigation

pr-review: register Graphite, Qodo, CodeAnt as recognized reviewer sources (advisory gate + scorecard) #1349

Summary

Current state (single source of truth)

Changes required

Decision (resolved)

Acceptance criteria

References

Dev-Lead: Blocked on Requirement #1 (verified logins) — no guessing

What I checked (authoritative, not guessed)

Why I did not proceed with the "likely" logins

Additional precondition also not yet satisfiable

What unblocks this

Dev-Lead Implementation Plan

Login verification (step 1 — done, not guessed)

Scope

Implementation Checklist

Tests to Write

Out of scope

Test Command

✅ Implementation complete — Qodo Merge + CodeAnt registered (Graphite held)

Step 1 — Logins verified (not guessed)

Steps 2–3 — Registries (`scripts/lib/advisory-review-gate.sh`)

Step 2 — Scorecard (`scripts/reviewer_report.sh`)

Step 4 — Docs

Step 5 — Tests (TDD: red → green)

Verification

Acceptance

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pr-review: register Graphite, Qodo, CodeAnt as recognized reviewer sources (advisory gate + scorecard) #1349

Description

Summary

Current state (single source of truth)

Changes required

Decision (resolved)

Acceptance criteria

References

Activity

don-petry commented on Jul 22, 2026

Dev-Lead: Blocked on Requirement #1 (verified logins) — no guessing

What I checked (authoritative, not guessed)

Why I did not proceed with the "likely" logins

Additional precondition also not yet satisfiable

What unblocks this

gemini-code-assist commented on Jul 22, 2026

don-petry commented on Jul 22, 2026

Dev-Lead Implementation Plan

Login verification (step 1 — done, not guessed)

Scope

Implementation Checklist

Tests to Write

Out of scope

Test Command

don-petry commented on Jul 22, 2026

✅ Implementation complete — Qodo Merge + CodeAnt registered (Graphite held)

Step 1 — Logins verified (not guessed)

Steps 2–3 — Registries (scripts/lib/advisory-review-gate.sh)

Step 2 — Scorecard (scripts/reviewer_report.sh)

Step 4 — Docs

Step 5 — Tests (TDD: red → green)

Verification

Acceptance

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Steps 2–3 — Registries (`scripts/lib/advisory-review-gate.sh`)

Step 2 — Scorecard (`scripts/reviewer_report.sh`)