PR Review Agent — failures detected 2026-05-13

[github-actions]

1. Executive Summary

Status: DEGRADED
Period: 2026-05-12 10:40 UTC – 2026-05-13 08:07 UTC
Result: 3 of 14 runs failed (21%)

Key findings:

• Root cause of all 3 failures: Copilot fallback engine crashes with error: Invalid command format — the old gh copilot suggest -p "$(cat <file>)" invocation is incompatible with modern gh copilot CLI; the session-abort circuit-breaker misclassifies this as a rate limit, skipping 19–22 remaining PRs per run
• Amplifier: Gemini fallback is completely unavailable (GOOGLE_API_KEY unset, @google/gemini-cli not installed), so the only fallback after Claude rate-limits is the broken Copilot engine
• Trigger: Claude Opus 4.7 (used for single-pass approval confirmation) hits rate limits during busy windows, forcing the fallback chain that leads to session abort
• Compounding defect: Even in successful runs, the rubber-duck reviewer (run_duck via Copilot) silently fails with the same Invalid command format error — it is non-fatal only because the deep-review result is used alone
• Pending fix exists but not merged: PR fix: replace gh copilot suggest with GitHub Models REST API #151 (claude/issue-147-20260512-1802) replaces all three gh copilot suggest calls with GitHub Models REST API — the fix is in review while the breakage continues in production on main at SHA 22e46ea7f67a57d9bf3de1c7dd656667751c11bf

Action required: Merge PR #151 immediately to restore the Copilot fallback engine and eliminate session aborts.

---

2. Failure Breakdown

Failure Category	Affected Runs	Example Error Message
CLI breaking change — gh copilot suggest -p invalid flag	#720, #721, #722 (100% of failures)	error: Invalid command format. Did you mean: copilot -i "suggest -p # Tier 1: Triage...
Engine rate limit — Claude Opus 4.7 quota exhausted	#720, #721, #722 (trigger for fallback)	[approve] rate limit detected — exiting with code 2 for engine fallback
Missing fallback — Gemini unavailable	#720, #721, #722 (amplifier)	Claude rate limit hit but Gemini fallback unavailable (CLI not installed or GOOGLE_API_KEY missing) — falling through to Copilot
Session abort — no viable fallback engine	#720, #721, #722 (terminal failure)	Rate limit hit on copilot engine, no fallback available for …/pull/151
Silent rubber-duck failure (non-fatal)	#720, #721 (also in successful runs)	[tier2] rubber duck did not produce valid JSON — continuing with deep review only

---

3. Error Patterns

Pattern A — Copilot Invalid command format (primary failure)

Exact error (from scripts/engine.sh, run_triage/run_duck copilot branch):

error: Invalid command format.

Did you mean: copilot -i "suggest -p # Tier 1: Triage
...
For non-interactive mode, use the -p or --prompt option.
Try 'copilot --help' for more information.

Step: Review each PR (cascade) → bash scripts/review-batch.sh → scripts/review-one-pr.sh → run_triage()/run_duck() in scripts/engine.sh

Root cause: Three call sites in scripts/engine.sh invoke:

timeout "$TRIAGE_TIMEOUT_SEC" gh copilot suggest -p "$(cat "$prompt_file")"

Modern gh copilot no longer accepts -p as a flag for suggest. Additionally, the prompt content begins with # Tier 1: Triage — the # is parsed as a flag prefix, producing Invalid command format. The non-zero exit is then processed by is_rate_limited() in review-one-pr.sh, which matches on the error: prefix and classifies it as a rate limit (exit code 2). This triggers the rate-limit fallback path, and since Gemini is also unavailable, the session aborts.

Pattern B — Gemini fallback unavailable (amplifier)

Exact error (from scripts/review-batch.sh):

##[warning]Claude rate limit hit but Gemini fallback unavailable
(CLI not installed or GOOGLE_API_KEY missing) — falling through to Copilot
##[warning]Gemini rate limit hit — switching to Copilot engine for remaining PRs

Root cause: GOOGLE_API_KEY is empty in every run (GOOGLE_API_KEY: in env dumps). The @google/gemini-cli is installed as best-effort (npm install -g @google/gemini-cli || true) during the Claude engine path, but without an API key, the CLI cannot function. The second warning (Gemini rate limit hit) is emitted immediately after because the code re-enters the fallback chain with REVIEW_ENGINE=gemini and that also exits 2 — directing traffic to Copilot.

Pattern C — Session abort (terminal)

Exact error:

##[error]Rate limit hit on copilot engine, no fallback available for
https://github.com/petry-projects/.github-private/pull/151
##[error]Session aborted early after failure on …/pull/151
(rate-limit on fallback engine). Skipped 21 remaining candidate(s);
will retry on next scheduled run.
Summary: 0 reviews posted, 2 no-ops skipped, 1 failures, 1 engine fallback(s)
to copilot (processed 3/24 candidates) [SESSION ABORTED EARLY]

Root cause: The session-abort guard in scripts/review-batch.sh treats any engine returning exit 2 (rate-limit) when that engine is already the fallback as unrecoverable. Because Copilot exits 2 due to the Invalid command format being misclassified as a rate limit, the guard fires and halts all remaining PR processing.

---

4. Token Scope Analysis

Scopes currently present (from gh auth status in all run logs):

notifications, read:audit_log, read:discussion, read:org,
read:project, repo, workflow

Assessment: Token scopes are not the cause of any observed failure. The repo scope covers PR reads/writes; read:org covers org membership queries; workflow covers the dispatch mechanism. No permission-denied or 403 errors appear in any log.

Scope	Status	Notes
repo	Present	Sufficient for PR reads, comments, approvals
read:org	Present	Sufficient for org-member checks
workflow	Present	Required for repository_dispatch triggers
notifications	Present	Not required but harmless
read:audit_log	Present	Not required but harmless
read:discussion	Present	Not required but harmless
read:project	Present	Not required but harmless

Recommendation: No token scope changes required. The failures are entirely CLI-compatibility and API-key configuration issues.

---

5. Recommendations

1. Merge PR #151 immediately

• What: Merge petry-projects/.github-private PR fix: replace gh copilot suggest with GitHub Models REST API #151 (fix: replace gh copilot suggest with GitHub Models REST API)
• Files changed: scripts/engine.sh (replaces 3× gh copilot suggest -p with copilot_chat() REST API calls), scripts/review-batch.sh (adds Copilot pre-flight smoke test), tests/test_copilot_chat.sh (new unit tests)
• Why: Eliminates the Invalid command format error at all three Copilot call sites, restores the Copilot fallback engine, and prevents future session aborts from this cause
• Expected impact: All three failure modes (Pattern A, B terminal, C) resolved; Copilot fallback becomes functional
• Urgency: CRITICAL — this is the direct cause of 100% of workflow failures and session aborts skipping 19–22 PRs per affected run

2. Set GOOGLE_API_KEY secret or remove Gemini from the fallback chain

• What: Either add GOOGLE_API_KEY as a repository secret in petry-projects/.github-private (Settings → Secrets → Actions), or accept that Gemini is permanently unavailable and document that Copilot is the sole fallback
• Files: pr-review.yml already wires GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}; the secret just needs to be populated
• Why: Without Gemini, the fallback chain collapses to Claude → (missing) → Copilot. Any Claude rate limit immediately stresses the one remaining fallback. Once PR fix: replace gh copilot suggest with GitHub Models REST API #151 is merged, Copilot will work, but Gemini as a middle tier would prevent Copilot from being the sole safety net
• Expected impact: Three-tier fallback restored; Claude rate-limit events no longer put the entire session at single-point-of-failure risk
• Urgency: HIGH — reduces blast radius of Claude rate limits after the critical fix is applied

3. Fix the rate-limit detector to distinguish CLI errors from actual rate limits

• What: In scripts/review-one-pr.sh (or wherever is_rate_limited() is defined), tighten the pattern match so that error: Invalid command format is not classified as a rate limit. It should only match on HTTP 429 responses or known rate-limit message strings, not on arbitrary error: prefixes
• Why: The misclassification of CLI errors as rate limits is what converts a recoverable tool error into a session-aborting fallback cascade. Even after PR fix: replace gh copilot suggest with GitHub Models REST API #151 fixes the Copilot invocation, future CLI errors could still trigger false rate-limit exits if the detector remains too broad
• Expected impact: CLI errors fail gracefully (skipping that PR) rather than aborting the entire session
• Urgency: HIGH — defensive hardening; PR fix: replace gh copilot suggest with GitHub Models REST API #151 partially addresses this for Copilot but the underlying detector logic may affect other engines

4. Install @google/gemini-cli unconditionally and validate GOOGLE_API_KEY at startup

• What: Move npm install -g @google/gemini-cli out of the || true best-effort block in the Claude engine case and make it a required install when GOOGLE_API_KEY is set; add a startup check that emits a clear ::warning:: if the key is absent. PR fix: add Gemini pre-flight check and engine-availability job summary #150 partially addresses this (engine availability summary) — ensure it is also merged
• Files: .github/workflows/pr-review.yml (Install review engine CLIs step), scripts/validate-engines.sh (added by PR fix: add Gemini pre-flight check and engine-availability job summary #150)
• Why: Silent Gemini absence is only discovered mid-run when Claude rate-limits, at which point it is too late to take corrective action
• Expected impact: Operators are warned at run start rather than discovering Gemini is missing during a live rate-limit event
• Urgency: MEDIUM — operational visibility improvement; reduces surprise during incidents

5. Update actions/cache@v4 to a Node.js 24-compatible version before June 2, 2026

• What: In .github/workflows/pr-review.yml, pin actions/cache to a release that supports Node.js 24 (e.g. actions/cache@v4.2.0 or later that has the Node 24 runtime)
• Why: Current actions/cache@v4 (SHA 0057852b) will be forced to Node.js 24 on 2026-06-02, and will be removed from runners on 2026-09-16. The warning appears in every run log
• Expected impact: Eliminates deprecation warnings; prevents potential cache step breakage post-June 2026
• Urgency: LOW — no current functional impact; deadline is June 2, 2026

---

6. Health Score

Health: 5/10 — Copilot fallback is broken by a known CLI incompatibility (fix is in PR #151, unmerged), causing complete session aborts that skip 19–22 PRs whenever Claude rate-limits, while the Gemini middle-tier fallback is also absent; successful runs proceed normally but without a functional rubber-duck reviewer.

[don-petry]

All 5 recommendations resolved:

• ✅ PR fix: replace gh copilot suggest with GitHub Models REST API #151 merged (Copilot REST API fix + is_cli_error() rate-limit detector hardening)
• ✅ GOOGLE_API_KEY secret set and verified (dry-run confirmed no Gemini-unavailable warnings)
• ✅ PR fix: add Gemini pre-flight check and engine-availability job summary #150 merged (Gemini pre-flight check + engine-availability job summary); rebased to resolve conflict with PR fix: replace gh copilot suggest with GitHub Models REST API #151
• ✅ actions/cache already on v5