Skip to content

PR Review Agent — failures detected 2026-09-10 #1748

Closed

Description

@github-actions

Convergence latency (deterministic)

Workflow duration across 73 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Workflow Runs min p50 p95 max
pr-review-trigger.yml 73 5s 35s 4m56s 32m35s

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Event success cancelled skipped failure total cancel %
repository_dispatch 37 3 0 4 44 6%
workflow_dispatch 21 0 0 8 29 0%
TOTAL 58 3 0 12 73 4%

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Metric Value
Scheduled sweep ticks 7
Ticks that re-dispatched a review (hits) 7
Sweep hit-rate 100%

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.

PR Review Agent — Health Report

Workflow: pr-review-trigger.yml | Repo: petry-projects/.github-private
Generated: 2026-09-10 | Window: 2026-09-09 → 2026-09-10


1. Executive Summary

Status: DEGRADED
Period: 2026-09-09T11:52Z → 2026-09-10T09:34Z
Result: 12 of 73 runs failed (16.4%) — 3 additional runs cancelled

Key findings:

  • No error log content was captured — the "Failed Run Logs" section in the report input is empty; all pattern analysis below is derived solely from run timestamps and ordinal positions within trigger batches
  • Failures exhibit a repeating nth-in-batch pattern: in 6 of 7 trigger groups, exactly 1 of ~8 concurrent runs fails, consistently at the 4th or final position in the burst — strongly suggests a per-event concurrency or API-slot exhaustion, not a random infrastructure fault
  • A dense failure cluster at 2026-09-09T11:52–12:18Z produced 4 failures and 3 cancellations across ~25 minutes, indicating a deeper or longer-lasting disruption during that window

Action required: Retrieve and inspect the actual step-level logs for at least one failed run (e.g. #74626, #74598) to confirm root cause before any remediation; the nth-in-batch pattern points to a concurrency/rate-limit issue but cannot be confirmed without log evidence.


2. Failure Breakdown

Note: No log text was present in the failed run log section. The table below categorises failures by the most probable cause given the timing pattern. Root causes are unconfirmed until logs are retrieved.

Failure Category Affected Runs Example Error Message
Engine rate limit / concurrency slot exhaustion (inferred) #74582, #74590, #74598, #74606, #74618, #74626 (6 isolated nth-in-batch failures) (no log captured — inferred from 4th/last-in-burst position in every group)
Sustained disruption / unknown (cluster) #74559, #74560, #74573, #74575, #74577, #74578 (6 failures in 12:13–12:18 cluster) (no log captured)
Cancelled / pre-empted #74555, #74568, #74574 (no log captured)
Other / unknown — (insufficient data)

3. Error Patterns

Pattern A — Nth-in-batch failure (repeating across 6 independent trigger groups)

Observed in runs: #74582, #74590, #74598, #74606, #74618, #74626

Timing signature per group:

Group example (2026-09-09 19:25–19:30):
  #74587  success  19:25:28
  #74588  success  19:25:45
  #74589  success  19:26:02
  #74590  FAILURE  19:26:12   ← 4th in burst, ~44s after group start
  #74591  success  19:30:47   ← new sub-wave, all succeed
  ...

This pattern repeats identically in groups at 16:28, 21:54, 00:02, 04:48, and 09:33. In two groups (04:48 and 09:33) the failure falls at position 8 (last) rather than position 4, but remains the final run of the burst.

Probable root cause: The reusable pr-review.yml@pr-review/v1-next enforces a concurrency gate or the upstream Claude API / GitHub token has a per-minute or per-burst request budget. The first N-1 runners acquire a slot; the Nth is rejected or rate-limited. Without logs this cannot be confirmed — the failure could equally be an idempotency guard that errors instead of exiting cleanly when it detects a "work already claimed" state.

Step suspected: The early evaluation or idempotency step inside pr-review.yml (likely engine.sh or review-one-pr.sh based on workflow comments), not the trigger stub itself (the stub has no logic beyond uses: delegation).


Pattern B — Dense cluster (2026-09-09 11:52–12:18Z)

Affected runs: #74555 (cancelled), #74559, #74560 (failures), #74568 (cancelled), #74573, #74574 (cancelled), #74575, #74577, #74578 (failures)

11:52  #74554  success
11:54  #74555  CANCELLED
11:56  #74556–#74558  success
11:56  #74559  FAILURE
11:56  #74560  FAILURE   ← two consecutive failures <20s apart
11:57  #74561  success
11:59  #74562–#74566  success
12:03  #74567  success
12:03  #74568  CANCELLED
12:03  #74569  success
12:05  #74570  success
12:07  #74571–#74572  success
12:13  #74573  FAILURE
12:13  #74574  CANCELLED
12:13  #74575  FAILURE   ← back-to-back failures with cancellation
12:16  #74576  success
12:18  #74577  FAILURE
12:18  #74578  FAILURE   ← two more simultaneous failures

Probable root cause: A transient event — possible release-cut activity (the workflow comments reference scripts/cut-release.sh and tag movements for the pr-review/v1-next channel), a GitHub API degradation, or a brief token expiry. The back-to-back cancellations (#74555, #74568, #74574) alongside failures suggest the reusable workflow ref (@pr-review/v1-next) was temporarily unavailable or the runner pool was exhausted. Recovery was gradual, consistent with a tag/ref propagation delay rather than a hard auth failure.


4. Token Scope Analysis

Derived from workflow source only — no gh auth status output was present in the provided logs.

Scope / Permission Status Source
contents: read Present (declared in trigger stub) pr-review-trigger.yml permissions block
pull-requests: write Present (declared) pr-review-trigger.yml permissions block
checks: read Present (declared) pr-review-trigger.yml permissions block
secrets: inherit Present — all org/repo secrets forwarded to reusable pr-review-trigger.yml job block
Claude API key / engine token Unknown — passed via secrets: inherit; not visible in source Inferred from agent function
read:org Unknown — not visible in trigger stub May be required by pr-review.yml reusable for org-level PR enumeration

Recommendations on scope:

  • pull-requests: write is declared but grants no more than the calling job has — if the reusable requires additional scopes, they must be elevated at this stub level.
  • Cannot confirm missing scopes without log output from a failed run (gh auth status or a 403/401 error line). Retrieve logs before adjusting permissions.

5. Recommendations

1. Retrieve step-level logs for a representative failed run [CRITICAL]

  • What: Run gh run view 74598 --log-failed (or any of the isolated nth-in-batch failures) and gh run view 74578 --log-failed (cluster failure) in petry-projects/.github-private.
  • Why: Every finding in this report is pattern-inferred. Without error text, no remediation can be targeted correctly. This is a prerequisite for all other recommendations.
  • Expected impact: Identifies actual error class (rate limit, auth, concurrency, ref-not-found) within minutes.
  • Urgency: CRITICAL — all other recommendations are blocked on this.

2. Investigate the 11:52–12:18Z cluster for correlation with a release-cut or tag move [HIGH]

  • What: Run git log --oneline --after="2026-09-09T11:45Z" --before="2026-09-09T12:25Z" and check scripts/cut-release.sh execution history (workflow runs for any release/tag workflow in that window).
  • Why: The density and pattern of failures + cancellations in that window is consistent with the pr-review/v1-next ref being in-flight during a tag promotion, causing uses: ...@pr-review/v1-next to resolve to an unstable or missing ref.
  • Expected impact: If confirmed, adding a brief stabilisation delay or a ref-existence check before the reusable call would prevent recurrence.
  • Urgency: HIGH — if this is the cause, it will recur on every release cut.

3. Add a concurrency group to the trigger stub to prevent nth-in-batch failures [HIGH]

  • What: Add to pr-review-trigger.yml:
    concurrency:
      group: pr-review-${{ github.event.pull_request.number || github.event.check_suite.head_sha || github.run_id }}
      cancel-in-progress: false
    (Use cancel-in-progress: true only if idempotency is guaranteed inside the reusable.)
  • Why: If Pattern A is caused by multiple trigger events (e.g. check_suite + pull_request_review + synchronize) firing simultaneously for the same PR, a concurrency group serialises them and avoids the slot-exhaustion failure. Without logs this is speculative but the pattern strongly suggests it.
  • Expected impact: Would eliminate the 6 isolated nth-in-batch failures (82% of the non-cluster failures) if the root cause is concurrent same-PR runs.
  • Urgency: HIGH (pending log confirmation — do not add blindly if idempotency in the reusable cannot handle cancel-in-progress: false gracefully).

4. Surface the health report pipeline's log-capture step [MEDIUM]

  • What: The report generation pipeline that produced this input did not populate the "Failed Run Logs" section. Ensure the log-fetching step (likely gh run view --log-failed) is not silently failing or being truncated before the report template is populated.
  • Why: A health report with no log content cannot drive actionable triage. The analysis is entirely blind.
  • Expected impact: Future reports will contain actual error text, making root-cause identification immediate rather than requiring a separate manual investigation step.
  • Urgency: MEDIUM — affects observability quality, not the workflow itself.

5. Confirm pr-review/v1-next ref stability SLA in the ring-0 rollout process [LOW]

  • What: Review docs/initiatives/agentic-release-strategy.md §5 (referenced in workflow header) to confirm whether a ref-availability window is defined between tag creation and callers being expected to resolve it.
  • Why: If the cluster failure was ref-propagation lag, the process docs should specify a minimum stabilisation delay after cut-release.sh runs before ring-0 begins receiving live traffic.
  • Expected impact: Prevents future cluster-failures on every release cut.
  • Urgency: LOW — process hardening; not blocking.

6. Health Score

Health: 6/10 — Workflow is operational but a repeating 16% failure rate with an unresolved cluster event and no captured error logs leaves the true root cause undiagnosed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions