Repository navigation
PR Review Agent — failures detected 2026-09-10 #1748
Description
Convergence latency (deterministic)
Workflow duration across 73 completed run(s) in the window — computed by pr_review_health.sh, not the model.
| Workflow | Runs | min | p50 | p95 | max |
|---|---|---|---|---|---|
pr-review-trigger.yml |
73 | 5s | 35s | 4m56s | 32m35s |
Run-outcome mix by triggering event
pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).
| Event | success | cancelled | skipped | failure | total | cancel % |
|---|---|---|---|---|---|---|
repository_dispatch |
37 | 3 | 0 | 4 | 44 | 6% |
workflow_dispatch |
21 | 0 | 0 | 8 | 29 | 0% |
| TOTAL | 58 | 3 | 0 | 12 | 73 | 4% |
Sweep hit-rate (deterministic)
How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.
| Metric | Value |
|---|---|
| Scheduled sweep ticks | 7 |
| Ticks that re-dispatched a review (hits) | 7 |
| Sweep hit-rate | 100% |
Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.
PR Review Agent — Health Report
Workflow: pr-review-trigger.yml | Repo: petry-projects/.github-private
Generated: 2026-09-10 | Window: 2026-09-09 → 2026-09-10
1. Executive Summary
Status: DEGRADED
Period: 2026-09-09T11:52Z → 2026-09-10T09:34Z
Result: 12 of 73 runs failed (16.4%) — 3 additional runs cancelled
Key findings:
- No error log content was captured — the "Failed Run Logs" section in the report input is empty; all pattern analysis below is derived solely from run timestamps and ordinal positions within trigger batches
- Failures exhibit a repeating nth-in-batch pattern: in 6 of 7 trigger groups, exactly 1 of ~8 concurrent runs fails, consistently at the 4th or final position in the burst — strongly suggests a per-event concurrency or API-slot exhaustion, not a random infrastructure fault
- A dense failure cluster at 2026-09-09T11:52–12:18Z produced 4 failures and 3 cancellations across ~25 minutes, indicating a deeper or longer-lasting disruption during that window
Action required: Retrieve and inspect the actual step-level logs for at least one failed run (e.g. #74626, #74598) to confirm root cause before any remediation; the nth-in-batch pattern points to a concurrency/rate-limit issue but cannot be confirmed without log evidence.
2. Failure Breakdown
Note: No log text was present in the failed run log section. The table below categorises failures by the most probable cause given the timing pattern. Root causes are unconfirmed until logs are retrieved.
| Failure Category | Affected Runs | Example Error Message |
|---|---|---|
| Engine rate limit / concurrency slot exhaustion (inferred) | #74582, #74590, #74598, #74606, #74618, #74626 (6 isolated nth-in-batch failures) | (no log captured — inferred from 4th/last-in-burst position in every group) |
| Sustained disruption / unknown (cluster) | #74559, #74560, #74573, #74575, #74577, #74578 (6 failures in 12:13–12:18 cluster) | (no log captured) |
| Cancelled / pre-empted | #74555, #74568, #74574 | (no log captured) |
| Other / unknown | — | (insufficient data) |
3. Error Patterns
Pattern A — Nth-in-batch failure (repeating across 6 independent trigger groups)
Observed in runs: #74582, #74590, #74598, #74606, #74618, #74626
Timing signature per group:
Group example (2026-09-09 19:25–19:30):
#74587 success 19:25:28
#74588 success 19:25:45
#74589 success 19:26:02
#74590 FAILURE 19:26:12 ← 4th in burst, ~44s after group start
#74591 success 19:30:47 ← new sub-wave, all succeed
...
This pattern repeats identically in groups at 16:28, 21:54, 00:02, 04:48, and 09:33. In two groups (04:48 and 09:33) the failure falls at position 8 (last) rather than position 4, but remains the final run of the burst.
Probable root cause: The reusable pr-review.yml@pr-review/v1-next enforces a concurrency gate or the upstream Claude API / GitHub token has a per-minute or per-burst request budget. The first N-1 runners acquire a slot; the Nth is rejected or rate-limited. Without logs this cannot be confirmed — the failure could equally be an idempotency guard that errors instead of exiting cleanly when it detects a "work already claimed" state.
Step suspected: The early evaluation or idempotency step inside pr-review.yml (likely engine.sh or review-one-pr.sh based on workflow comments), not the trigger stub itself (the stub has no logic beyond uses: delegation).
Pattern B — Dense cluster (2026-09-09 11:52–12:18Z)
Affected runs: #74555 (cancelled), #74559, #74560 (failures), #74568 (cancelled), #74573, #74574 (cancelled), #74575, #74577, #74578 (failures)
11:52 #74554 success
11:54 #74555 CANCELLED
11:56 #74556–#74558 success
11:56 #74559 FAILURE
11:56 #74560 FAILURE ← two consecutive failures <20s apart
11:57 #74561 success
11:59 #74562–#74566 success
12:03 #74567 success
12:03 #74568 CANCELLED
12:03 #74569 success
12:05 #74570 success
12:07 #74571–#74572 success
12:13 #74573 FAILURE
12:13 #74574 CANCELLED
12:13 #74575 FAILURE ← back-to-back failures with cancellation
12:16 #74576 success
12:18 #74577 FAILURE
12:18 #74578 FAILURE ← two more simultaneous failures
Probable root cause: A transient event — possible release-cut activity (the workflow comments reference scripts/cut-release.sh and tag movements for the pr-review/v1-next channel), a GitHub API degradation, or a brief token expiry. The back-to-back cancellations (#74555, #74568, #74574) alongside failures suggest the reusable workflow ref (@pr-review/v1-next) was temporarily unavailable or the runner pool was exhausted. Recovery was gradual, consistent with a tag/ref propagation delay rather than a hard auth failure.
4. Token Scope Analysis
Derived from workflow source only — no gh auth status output was present in the provided logs.
| Scope / Permission | Status | Source |
|---|---|---|
contents: read |
Present (declared in trigger stub) | pr-review-trigger.yml permissions block |
pull-requests: write |
Present (declared) | pr-review-trigger.yml permissions block |
checks: read |
Present (declared) | pr-review-trigger.yml permissions block |
secrets: inherit |
Present — all org/repo secrets forwarded to reusable | pr-review-trigger.yml job block |
| Claude API key / engine token | Unknown — passed via secrets: inherit; not visible in source |
Inferred from agent function |
read:org |
Unknown — not visible in trigger stub | May be required by pr-review.yml reusable for org-level PR enumeration |
Recommendations on scope:
pull-requests: writeis declared but grants no more than the calling job has — if the reusable requires additional scopes, they must be elevated at this stub level.- Cannot confirm missing scopes without log output from a failed run (
gh auth statusor a 403/401 error line). Retrieve logs before adjusting permissions.
5. Recommendations
1. Retrieve step-level logs for a representative failed run [CRITICAL]
- What: Run
gh run view 74598 --log-failed(or any of the isolated nth-in-batch failures) andgh run view 74578 --log-failed(cluster failure) inpetry-projects/.github-private. - Why: Every finding in this report is pattern-inferred. Without error text, no remediation can be targeted correctly. This is a prerequisite for all other recommendations.
- Expected impact: Identifies actual error class (rate limit, auth, concurrency, ref-not-found) within minutes.
- Urgency: CRITICAL — all other recommendations are blocked on this.
2. Investigate the 11:52–12:18Z cluster for correlation with a release-cut or tag move [HIGH]
- What: Run
git log --oneline --after="2026-09-09T11:45Z" --before="2026-09-09T12:25Z"and checkscripts/cut-release.shexecution history (workflow runs for any release/tag workflow in that window). - Why: The density and pattern of failures + cancellations in that window is consistent with the
pr-review/v1-nextref being in-flight during a tag promotion, causinguses: ...@pr-review/v1-nextto resolve to an unstable or missing ref. - Expected impact: If confirmed, adding a brief stabilisation delay or a ref-existence check before the reusable call would prevent recurrence.
- Urgency: HIGH — if this is the cause, it will recur on every release cut.
3. Add a concurrency group to the trigger stub to prevent nth-in-batch failures [HIGH]
- What: Add to
pr-review-trigger.yml:(Useconcurrency: group: pr-review-${{ github.event.pull_request.number || github.event.check_suite.head_sha || github.run_id }} cancel-in-progress: false
cancel-in-progress: trueonly if idempotency is guaranteed inside the reusable.) - Why: If Pattern A is caused by multiple trigger events (e.g.
check_suite+pull_request_review+synchronize) firing simultaneously for the same PR, a concurrency group serialises them and avoids the slot-exhaustion failure. Without logs this is speculative but the pattern strongly suggests it. - Expected impact: Would eliminate the 6 isolated nth-in-batch failures (82% of the non-cluster failures) if the root cause is concurrent same-PR runs.
- Urgency: HIGH (pending log confirmation — do not add blindly if idempotency in the reusable cannot handle
cancel-in-progress: falsegracefully).
4. Surface the health report pipeline's log-capture step [MEDIUM]
- What: The report generation pipeline that produced this input did not populate the "Failed Run Logs" section. Ensure the log-fetching step (likely
gh run view --log-failed) is not silently failing or being truncated before the report template is populated. - Why: A health report with no log content cannot drive actionable triage. The analysis is entirely blind.
- Expected impact: Future reports will contain actual error text, making root-cause identification immediate rather than requiring a separate manual investigation step.
- Urgency: MEDIUM — affects observability quality, not the workflow itself.
5. Confirm pr-review/v1-next ref stability SLA in the ring-0 rollout process [LOW]
- What: Review
docs/initiatives/agentic-release-strategy.md §5(referenced in workflow header) to confirm whether a ref-availability window is defined between tag creation and callers being expected to resolve it. - Why: If the cluster failure was ref-propagation lag, the process docs should specify a minimum stabilisation delay after
cut-release.shruns before ring-0 begins receiving live traffic. - Expected impact: Prevents future cluster-failures on every release cut.
- Urgency: LOW — process hardening; not blocking.
6. Health Score
Health: 6/10 — Workflow is operational but a repeating 16% failure rate with an unresolved cluster event and no captured error logs leaves the true root cause undiagnosed.
Activity