Evaluate additional AI code-review services (benchmark + rate-limit + cost analysis)

[don-petry]

Evaluating additional AI code-review services

Context. We run CodeRabbit (Free) for PR review and have cancelled GitHub Copilot Pro (effective immediately — bottom-cluster review quality, and its June-2026 credit-metered model penalizes our volume). That leaves only CodeRabbit Free, rate-limited to 1 review/hour, as coverage — so replacing it with adequate paid review is urgent. This thread evaluates alternatives against the independent withmartian Code Review Bench, our real usage, and each tool's throughput limits (the metric that actually binds us).

Our usage profile: high-volume single contributor

• Active accounts: 2 — don-petry (only human who authors code) and donpetry-bot. We are a 1-seat customer everywhere.
• Volume: .github-private alone saw 223 PRs in the last 30 days (~7–8/day), much of it agent-generated and bursty.

Why this is the whole story: most rate limits are enforced per developer. A normal team spreads PR load across many seats, so no single developer's cap ever binds. We can't — all 223 PRs/mo funnel through one rate bucket. Deciding factors, in order: (1) throughput/rate limits, (2) pricing model (flat vs metered), (3) benchmark quality. Per-seat price is nearly irrelevant at 1 seat; per-review / per-user metering is the enemy.

Rate limits & throughput — the binding constraint

Tool	Throughput limit (relevant to us)	Verdict for 1 seat @ 223 PRs/mo
CodeRabbit Free	1 PR review / hour / dev	❌ Current bottleneck
CodeRabbit Pro	5 PR/hr/dev, rolling, no monthly cap + Fair-Use throttle >95th pct	✅ Steady OK; ⚠️ bursts >5/hr queue
CodeRabbit Pro+	10 PR/hr/dev	✅ Burst headroom
Gemini Code Assist	33 PR reviews/day (free), ≥100/day (paid) — per GitHub-app install, daily (not per-dev-hr, not per-user)	✅✅ 8/day fits inside the free tier; single-seat doesn't hurt it
CodeAnt AI	Unlimited reviews (flat seat)	✅ throughput, ❌ quality (see below)
Graphite Diamond	Seat-based, no per-review meter (free tier alone does 100 PR/mo)	✅ Should absorb 223/mo
cubic	LOC-metered: 40k LOC/mo (Team) / 100k (Pro)	⚠️ likely needs $99 Pro
Cursor Bugbot	No fixed cap, but usage-billed ~$1–1.50/run (switched from flat $40 seat at June-2026 renewal) + requires Cursor Teams seat	❌ cost scales with every PR
Qodo Teams	~20 PRs/user/month ("unlimited" is a reverting promo)	❌ 1 seat ≠ 223/mo (~11 seats ≈ $330)
Greptile	50 reviews/seat then $1/review (~268 at our volume)	❌ cost cliff (~$248/mo)

Benchmark quality (withmartian Code Review Bench) — read with throughput fit

Tool	Quality (F1)	Fit for our shape
Qodo	~64% (top)	❌ per-user metered
cubic	~62%	⚠️ LOC-metered ($99)
CodeAnt AI	~52% on paper — but weak real-world review quality in our judgment	⚠️ down-weighted
CodeRabbit	~51% (best recall)	✅ 5–10/hr
Cursor Bugbot	not on Martian leaderboard (strong ~80% on a separate CVE bench); treat as mid	❌ usage cost
Gemini Code Assist	below the leading cluster / noisier (exact F1 only on live dashboard)	✅ throughput+cost, ⚠️ quality
Greptile	top cluster	❌ cost

The tension: the highest-F1 tools (Qodo, cubic) are exactly the ones whose metering punishes single-seat high volume. The tools that fit our shape (CodeRabbit, Gemini, CodeAnt) sit lower on quality. So the play is one strong paid reviewer + a cheap high-throughput second pass, not a single premium tool.

Projected monthly cost — our profile (1 seat, ~223 PRs/mo)

Tool	Est. monthly for us	Notes
Gemini Code Assist	$0 (free tier) / $19 Standard	33 reviews/day free covers us; likely already inside our Google Pro spend
CodeRabbit Pro	$24 (annual) / $30 m2m	5/hr may queue bursts
CodeRabbit Pro+	$48	10/hr — buy if Pro bursts queue
CodeAnt AI	~$10–24	Unlimited, but quality-questioned
Graphite Diamond	~$40	Flat seat, no meter
cubic	$30 → likely $99	LOC cap pushes to Pro
Cursor Bugbot	~$243–375	Cursor seat (~$20–40) + ~$1–1.50 × ~223 runs
Qodo Teams	~$330	≈11 seats to reach 223/mo
Greptile	~$248	per-review cost cliff

Free tiers — enable & test now (a $0 bake-off)

Most of these have a free path, so we can run several as optional reviewers in parallel on live PRs and judge real-world quality at $0 — including the top-F1 tools we ruled out on price (Qodo, cubic), whose quality is still worth sampling.

Tool	Free access	What you get free	Good enough to test quality?
Gemini Code Assist	✅ Perpetual	33 PR reviews/day	✅✅ covers our full volume
Graphite Diamond	✅ Perpetual	up to 100 PR reviews/mo	✅✅ ~half our volume, no meter
CodeRabbit	✅ Perpetual	1 review/hr/dev (already enabled)	✅ baseline
Qodo Merge	✅ Perpetual	30 PR reviews/mo (org-wide)	✅ sample the ~64% F1 leader
CodeAnt AI	✅ Perpetual	up to 5 users, full engine; free for OSS	✅ sample (quality-questioned)
Greptile	✅ Perpetual	PR summaries only, 4 reviews/hr	⚠️ shallow, summaries not full review
cubic	⚠️ Trial / Starter	~20 reviews/mo, no CLI	⚠️ small sample of the ~62% F1 tool
Cursor Bugbot	⚠️ 14-day trial only	full features, time-boxed	⚠️ trial window only

Suggested test plan: enable the perpetual-free GitHub apps in parallel for a ~2-week window (Gemini, Graphite, Qodo, CodeAnt alongside our existing CodeRabbit) on live PRs, compare real comment quality and noise, then keep the winners. Reserve the time-boxed trials (cubic, Cursor Bugbot) for a deliberate week only if the free bake-off is inconclusive. Caveat: multiple reviewers on one PR adds comment noise — expect it during the window and disable the losers promptly.

GitHub Copilot Pro — cancelled

Cancelled immediately ($10/mo saved). Bottom-cluster F1 and, since June 2026, credit-metered review ($15 pool) our volume would exhaust. Losing completions is an accepted trade-off.

Recommendation (revised for rate limits, quality, and recent Cursor pricing)

0. Enable the perpetual free tiers now (Gemini, Graphite, Qodo, CodeAnt) as optional parallel reviewers — a $0 live bake-off (see section above) to validate real-world quality before we commit spend.
1. Upgrade CodeRabbit Free → Pro ($24/mo) now — immediate must-do, 1/hr → 5/hr, no migration, best-recall of the fit-tools. Move to Pro+ ($48, 10/hr) if agent bursts queue.
2. Add Gemini Code Assist as the high-throughput second pass — free tier ($0), likely already covered by our Google Pro. Its 33-reviews/day, per-install quota is the only option that ignores our single-seat problem entirely. Use it as a complementary net, accepting it's the noisier/lower-quality reviewer.
3. Deprioritize CodeAnt — its unlimited-flat pricing fit our shape, but review quality is too weak to be our quality anchor. Gemini fills the "cheap high-throughput" slot better at ~$0.
4. Do NOT add Cursor Bugbot at our volume — its June-2026 switch to usage-based billing (~$1–1.50/run) turns 223 PRs/mo into ~$243–375/mo, a Greptile-class cost cliff. Only reconsider if we adopt Cursor Teams seats for the editor anyway and hard-cap Bugbot spend.
5. Skip Qodo/Greptile (metering); treat cubic as premium-only (budget $99 if we want its ~62% F1).

Best-fit stack: CodeRabbit Pro (quality anchor) + Gemini Code Assist free (throughput net) ≈ $24/mo (→ $43 if we take Gemini Standard, → $48 if CodeRabbit Pro+). Net change after the $10 Copilot saving: +$14–38/mo, for two reviewers that between them cover both quality (CodeRabbit recall) and unlimited throughput (Gemini per-install daily quota).

---

List prices, approximate; rate-limit numbers from vendor docs as of July 2026. Confirm: Gemini GitHub-app quota under our Google plan (and reliability — some reports of ~30-review quota flakiness), CodeRabbit Pro vs Pro+ burst behavior, and Cursor Bugbot's per-run rate before any Cursor commitment.

[don-petry]

Graphite, Qodo, and CodeAnt enabled. Free trials started on all including existing CodeRabbit.

[don-petry]

Enablement tracked in #1349 — configures the pr-review agent to recognize Graphite, Qodo, and CodeAnt as reviewer sources (advisory gate + Third-Party Reviewer Scorecard), including verifying each bot's exact App login and adding per-vendor rate-limit markers so quota-capped trials can't wedge the approval gate.