PR Review Agent — failures detected 2026-05-22 #359

PR Review Agent — Health Report

Generated: 2026-05-22 | Workflow: pr-review.yml | Repo: petry-projects/.github-private

1. Executive Summary

Status: BLOCKING
Period: 2026-05-21T18:53Z – 2026-05-22T05:20Z (≈10.5 hours)
Result: 10 of 100 runs failed (10%); 11 cancelled; 60 successful

Key findings:

• DON_PETRY_BOT_GH_PAT is missing the repo OAuth scope — 4 runs fail immediately at auth validation; this was always broken but the newly added scope-check step surfaced it
• All three review engines (Claude → Gemini → Copilot) exhaust their quotas in sequence — 6 runs reach review-batch.sh but post 0 reviews and exit 1
• Copilot is the final fallback but also fails (exit code 1 at gh copilot call), leaving no working engine once daily limits are hit

Action required: Regenerate DON_PETRY_BOT_GH_PAT with repo scope added, and investigate why gh copilot exits 1 when Claude and Gemini are both rate-limited.

2. Failure Breakdown

3. Error Patterns

Pattern A — Missing repo scope (runs #1766, #1761, #1758, #1753)

Exact error:

GH_TOKEN is missing required scope: repo
##[error]Process completed with exit code 1.

Step: "Install review engine CLIs" — the scope-validation block added to the install step:

required_scopes=(repo read:org)
for required_scope in "${required_scopes[@]}"; do
  if ! grep -qE "(^|[[:space:]])${required_scope}([[:space:]]|$)" <<< "$normalized_scopes"; then
    echo "::error::GH_TOKEN is missing required scope: ${required_scope}"
    exit 1
  fi
done

Root cause: DON_PETRY_BOT_GH_PAT (token github_pat_11CDFSYKQ0w50IHawIgb0w_...) was created without the repo scope. Earlier runs never caught this because the previous workflow version ran gh auth status for display only, without scope validation. The new scope-checking code (visible in the current pr-review.yml) surfaces the pre-existing token misconfiguration immediately. All 4 affected runs targeted PR #353 and failed before any review work started.

Pattern B — Engine rate limit cascade (runs #1752, #1751, #1734, #1731, #1712, #1709)

Step 1 — Claude limit (tier 1):

[tier1] usage/rate limit detected on stdout — exiting with code 2 for engine fallback
limit message: You've hit your limit · resets 12:10am (UTC)
##[warning]Claude rate limit hit — switching to Gemini engine for remaining PRs

Step 2 — Gemini limit (tier 1, after fallback):

TerminalQuotaError: You have exhausted your daily quota on this model.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 0, model: gemini-2.0-flash
##[warning]Gemini rate limit hit — switching to Copilot engine for remaining PRs

Step 3 — Copilot failure (final fallback):

##[warning]triage exited with code 1
triage stderr:     [copilot] calling gh copilot (model=openai/o4-mini, timeout=300s, flags=--deny-tool *)
##[error]cascade failed at tier 1 (triage process exit 1) for <PR URL>
##[error]Review failed for <PR URL> (exit code 1)
Summary: 0 reviews posted, 0 no-ops skipped, 1 failures, 2 engine fallback(s) to gemini, copilot

Root cause: The Gemini free-tier quota (generate_content_free_tier_requests) is exhausted daily. Claude's OAuth rate limit (resets 12:10am UTC / 7:10pm UTC) also fires. When both are exhausted, review-batch.sh falls through to Copilot. Copilot exits 1 immediately — the stderr only shows the invocation line ([copilot] calling gh copilot ...) with no further output, suggesting gh copilot either lacks authorization under COPILOT_GITHUB_TOKEN or the built-in extension is not functioning with the provided token.

4. Token Scope Analysis

Scopes currently present on DON_PETRY_BOT_GH_PAT:

• read:org — implied present (the scope check exits on first failure, repo is checked first; read:org was never reported missing)
• Unknown additional scopes (token output is masked)

Scopes missing or insufficient:

Recommendations:

5. Recommendations

1. Regenerate DON_PETRY_BOT_GH_PAT with repo scope

• What: Go to GitHub → donpetry-bot account → Settings → Developer Settings → Personal Access Tokens. Regenerate (or create new) PAT with at minimum: repo, read:org. Update the DON_PETRY_BOT_GH_PAT secret in petry-projects/.github-private repository settings.
• Why: Missing repo scope causes 100% failure for all runs that reach the new scope-validation code. Without this scope the bot cannot read PR details across repos or write PR review comments.
• Expected impact: Eliminates Pattern A failures entirely (4 of 10 failures resolved).
• Urgency: [CRITICAL] — causes immediate exit before any review work begins.

2. Investigate and fix Copilot fallback engine

• What: In scripts/review-batch.sh (and the underlying triage invocation), verify that COPILOT_GITHUB_TOKEN has a valid Copilot subscription, gh copilot extension is installed, and the model openai/o4-mini is accessible via GitHub Models API under that token. Add explicit error capture to log the full stderr of the gh copilot call rather than truncating at the invocation line.
• Why: All 6 rate-limit failures reached the Copilot fallback and still exited 1. A working Copilot fallback would have salvaged reviews during Claude/Gemini quota exhaustion. Currently the 3-engine cascade provides no actual resilience because the last rung always fails.
• Expected impact: Converts Pattern B from "0 reviews posted" to "review posted via Copilot" during quota exhaustion windows.
• Urgency: [HIGH] — rate limits will recur daily; without a working final fallback, the cascade is useless.

3. Upgrade Gemini API from free tier to paid tier

• What: Configure GOOGLE_API_KEY to use a paid Gemini API project with billing enabled (not the free-tier generate_content_free_tier_requests quota). Update GOOGLE_API_KEY secret accordingly.
• Why: Free-tier quota (limit: 0 after daily cap) exhausts within a day of heavy PR activity, triggering fallback to Copilot. With a paid tier, Gemini becomes a reliable second engine.
• Expected impact: Reduces likelihood of Copilot being reached; fewer overall cascade failures.
• Urgency: [HIGH] — free-tier Gemini quota appears to hit zero daily.

4. Add read:org scope verification to the PAT after fix

• What: After regenerating the PAT, run gh auth status manually on the bot account and confirm read:org appears in the token scopes. The scope loop only checks one at a time and exits on first failure, so read:org absence was masked by the repo failure.
• Why: read:org is required by list-prs.sh to enumerate open PRs across the org. Without it, batch (non-event-triggered) runs will fail at the enumeration step after the auth step passes.
• Expected impact: Prevents a second category of failures from emerging once repo is fixed.
• Urgency: [MEDIUM] — secondary issue, will only surface after repo is resolved.

5. Pin Gemini to a paid model or use a non-free-tier quota metric

• What: In scripts/review-batch.sh (or wherever the Gemini engine is configured), switch from gemini-2.0-flash (free tier) to a model that draws from paid quota. Alternatively, ensure the API key is associated with a project that has Vertex AI or Gemini API billing enabled.
• Why: Error explicitly states generate_content_free_tier_requests, limit: 0 — this is a hard daily cap that cannot be worked around without upgrading.
• Expected impact: Eliminates daily Gemini quota exhaustion.
• Urgency: [MEDIUM] — supplements recommendation Add @claude delegation, auto-merge, and rebase handling #3; needed if the API key itself can't be changed.

6. Health Score

Health: 4/10 — Token misconfiguration causes immediate failures for one PR class, and all three fallback engines exhausting quota simultaneously leaves the other failure class with no recovery path.