PR Review Agent — failures detected 2026-09-11 #1756

Convergence latency (deterministic)

Workflow duration across 264 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.

PR Review Agent — Health Report

Workflow: pr-review-trigger.yml | Repo: petry-projects/.github-private
Generated: 2026-09-11

1. Executive Summary

Status: DEGRADED
Period: 2026-09-10T11:51Z – 2026-09-11T09:58Z
Result: 10 of 162 actionable runs failed (6.2%) — 75 additional runs cancelled (28.8% of total fetched)

Key findings:

• Systematic 1-failure-per-event-batch pattern: each failure is immediately preceded/followed by 6–7 concurrent successes at the same timestamp, indicating a per-event race condition or first-attempt transient failure rather than random noise
• Early cluster of 3 failures (#74629, #74638, #74639 at 11:51–11:59 UTC) coincided with a mass cancellation storm (~50 cancellations 12:00–12:30 UTC), suggesting a release/deployment event on Sep 10 that flooded the trigger queue
• No log content was returned for the failed runs — the "Failed Run Logs" section in the input was empty; all pattern analysis below is derived from run timestamps and statistical structure alone

Action required: Retrieve and inspect logs for at least runs #74890, #74883, and #74629 to confirm root cause before applying fixes.

2. Failure Breakdown

Note: Exact error messages are unavailable — the failed run logs were not included in the input. The categories below are inferred from timestamp patterns and workflow structure.

3. Error Patterns

Pattern A — Recurring single failure per concurrent batch (7 runs)

Every failure from #74759 onward occurs as the last or straggling run in a group of 6–8 runs that all fire within 60–90 seconds of each other. The successful runs complete first; the lone failure trails by 5–15 seconds. This is consistent with:

• A GitHub API rate-limit hit by the first job in the batch that clears before the others start, or inversely a last-writer-wins race where the trailing run collides with a state written by the earlier successful runs (e.g., duplicate review guard in review-one-pr.sh returning non-zero on second attempt for the same SHA)
• The check_suite: completed trigger fanning out one run per check suite result per open PR — the highest-numbered run (latest timestamp, trailing) may be the one hitting the idempotency guard or rate limit

Pattern B — Sep 10 ~11:51–12:30 UTC burst (3 failures + ~50 cancellations)

Three consecutive failures (#74629, #74638, #74639) were followed immediately by a cascade of cancellations. This pattern matches a major branch activity spike (likely a release cut or force-push to pr-review/v1-next) that:

1. Flooded the trigger queue with dozens of synchronize or check_suite events
2. Caused the concurrency group to cancel in-flight runs as newer runs queued
3. Left 3 runs in a partially-initialized state that failed before the cancel signal arrived

Root cause confidence without logs: LOW. These are the two most plausible hypotheses given the data; direct log inspection is required to confirm.

4. Token Scope Analysis

The workflow source grants the called reusable:

Scopes not declared but potentially needed:

Caveat: Without gh auth status output from the failed runs, it is not possible to confirm whether a missing scope is the actual failure cause. No 401/403 pattern is visible in the timing data alone.

5. Recommendations

1. [CRITICAL — prerequisite] Retrieve failed run logs before making any other changes.

   • Run: gh run view 74890 --log-failed, gh run view 74629 --log-failed, gh run view 74638 --log-failed
   • Why: All pattern analysis in this report is inferred; the actual error could be a broken flag, an expired token, or a new CLI incompatibility. No fix should be applied without confirming the root cause.
   • Expected impact: Converts this DEGRADED report into a confirmed diagnosis.
   • Urgency: CRITICAL

2. [HIGH] Investigate the 1-failure-per-batch pattern for an idempotency guard false-positive.

   • File: scripts/review-one-pr.sh (or engine.sh) — check the logic that skips already-reviewed SHAs
   • Why: The trailing run in each batch (highest run number, latest timestamp) may be hitting a guard that was written by an earlier successful sibling run for the same PR head SHA. If the guard returns exit 1 rather than exit 0 for "already reviewed," it will register as a failure instead of a skip.
   • Expected impact: Eliminates the 7 post-12:00 recurring failures if confirmed.
   • Urgency: HIGH

3. [HIGH] Add a concurrency group with cancel-in-progress: false safeguard for the trailing run.

   • File: .github/workflows/pr-review-trigger.yml
   • Why: If the concurrency group cancels the trailing run mid-execution (after it starts but before it writes idempotency state), it can register as a failure. Using cancel-in-progress: false for the review job ensures in-flight reviews complete rather than getting interrupted.
   • Expected impact: Reduces the cancellation-adjacent failures; may not affect all 10 if cause is elsewhere.
   • Urgency: HIGH

4. [MEDIUM] Add retry logic or graceful exit on rate-limit responses in engine.sh / review-one-pr.sh.

   • If log inspection confirms GitHub API rate-limiting as the cause of the first-attempt failures, add exponential backoff around gh api calls.
   • Expected impact: Makes the workflow self-healing under burst conditions like the Sep 10 release event.
   • Urgency: MEDIUM

5. [MEDIUM] Investigate the Sep 10 12:00 UTC cancellation storm root cause.

   • Run gh run list --workflow=pr-review-trigger.yml --limit=50 --json conclusion,createdAt,event for the 11:50–12:30 UTC window to identify which event type caused the burst.
   • Why: 50+ cancellations in 30 minutes indicates either a runaway trigger (e.g., a check_suite loop) or a large batch push. If it was a release cut, this is expected; if not, the trigger fan-out may need a filter condition.
   • Expected impact: Prevents future cancellation storms from masking real failures.
   • Urgency: MEDIUM

6. [LOW] Add workflow_run conclusion filter or check_suite app filter to reduce noise.

   • File: .github/workflows/pr-review-trigger.yml, on.check_suite block
   • Why: check_suite: completed fires for every CI app completion (not just the ones relevant to PR review), contributing to the high run volume and increasing the surface area for collisions.
   • Expected impact: Reduces total run volume, lowers cancellation rate, makes signal-to-noise cleaner.
   • Urgency: LOW

6. Health Score

Health: 6.5/10 — Systematic 1-failure-per-batch pattern needs log confirmation and likely an idempotency guard fix, but the majority of reviews are completing successfully.