Skip to content

PR Review Agent — failures detected 2026-10-07 #2069

Closed

Description

@github-actions

Convergence latency (deterministic)

Workflow duration across 994 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Workflow Runs min p50 p95 max
pr-review-trigger.yml 994 1s 49s 10m50s 19m57s

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Event success cancelled skipped failure total cancel %
check_suite 153 178 17 0 348 51%
pull_request 73 1 0 0 74 1%
pull_request_review 25 15 122 0 162 9%
repository_dispatch 87 102 0 25 214 47%
workflow_dispatch 183 12 0 1 196 6%
TOTAL 521 308 139 26 994 30%

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Metric Value
Scheduled sweep ticks 5
Ticks that re-dispatched a review (hits) 5
Sweep hit-rate 100%

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.

PR Review Agent Health Report: pr-review-trigger.yml

1. Executive Summary

Status: WARNING
Period: 2026-10-06 13:39Z to 2026-10-07 09:31Z (last 1 day)
Result: 26 of 994 runs failed (2.6%)

Key findings:

  • The "Failed Run Logs" section was empty, so no error text is available and no failure root cause can be confirmed from the logs.
  • Failures come in bursts, not at a steady rate. Several clusters are 3–4 runs within seconds of each other, which points to a shared trigger or event, such as one PR's synchronize/check_suite events or a bad dependency.
  • 308 runs (31%) were cancelled and 139 (14%) were skipped. This is probably concurrency cancellation and job-level if filtering, but it can't be confirmed without the logs.
  • The pipeline recovered: the last failures were at 09:21Z (#93421–#93423), and runs #93424–#93436 have been successful, skipped or cancelled since.

Action required: Re-run the report with failed-job logs attached (gh run view <id> --log-failed) for the burst runs listed below, so the root cause can be identified.

2. Failure Breakdown

Failure Category Affected Runs Example Error Message
Other / unknown (no logs supplied) 26 None available (log section empty)
Burst 13:39–14:40Z (9 runs) #92443, #92464–#92471 None available
Burst 19:36–19:43Z (7 runs) #92574, #92591, #92593, #92594, #92600, #92610 None available
Isolated or small clusters (10 runs) #92767, #93068, #93127–#93128, #93204–#93205, #93279, #93293, #93421–#93423 None available
Permission / auth, rate limit, scope, CLI change, timeout Not determinable None available

The burst rows overlap the first row (26 total). The 19:36–19:43Z burst has 6 listed run IDs, so the 19:36–19:43Z burst is 6 runs, not 7. The 9 + 6 + 11 split also doesn't match the "10" in the last row. The run IDs above are authoritative.

The run durations show p95 of 10m50s and a max of 19m57s. Some runs are long, but there's no evidence that any failure was a timeout.

3. Error Patterns

No error messages could be quoted because the logs were not included. Patterns visible from run metadata only:

  • Burst clustering: Failures at 14:06Z (4 runs in about 14s), 14:39Z (4 runs in about 56s), 19:38Z (3 runs in about 11s), 09:21Z (3 runs in about 7s) and 23:19Z (2 runs in 3s) suggest a deterministic failure triggered by simultaneous events on one PR or SHA. Candidates are a check_suite fan-out, a failed reusable-workflow resolution, a failed checkout of the pr-review/v1-next tag or transient GitHub API errors. These are hypotheses, not confirmed.
  • Cancellations (308): Many cancelled runs sit next to successes at near-identical timestamps. This fits concurrency groups with cancel-in-progress, where a newer event supersedes an older run. This is expected noise and not a fault.
  • Skipped (139): There are long all-skipped sequences, for example #92777–#92816 and #92919–#92926. These are probably check_suite events that the reusable filters out.
  • Self-hosting risk: The caller pins pr-review.yml@pr-review/v1-next, a moving ring-0 channel. A bad next release would show up as bursts of failures across all PRs until the tag is rolled back. The bursts at 14:06–14:40Z and 19:36–19:43Z are consistent with this, but not proof of it.

4. Token Scope Analysis

No gh auth status output was in the logs, so this comes from the workflow source only.

  • Scopes present (caller permissions:): contents: read, pull-requests: write, checks: read. secrets: inherit passes org and repo secrets to the reusable.
  • Missing or insufficient scopes: None can be shown from the evidence, since there are no 401/403 errors.
  • Recommendations:
    • If the failure logs show 403s on review submission or comment minimization, check whether issues: write is needed. PR comments can use the issues API, and this is not confirmed.
    • If the failures involve a cross-repo checkout of the agent scripts, confirm the token or app has read access to petry-projects/.github-private.
    • Otherwise make no scope changes.

5. Recommendations

  1. [HIGH] Collect failure logs. Run gh run view <id> --log-failed for one run from each burst (#92443, #92464, #92574, #92767, #93068, #93127, #93204, #93279, #93293, #93421) and feed them to this analysis. Expected impact: the root cause is identified. The current report can't tell whether the failures come from the agent, its infrastructure or the repository.
  2. [HIGH] Fix the log-fetch step of this report job. The "Failed Run Logs" section was empty even though 26 failures were counted. Check the log download step in the health-report workflow for pagination, permissions (actions: read) and failures to fetch logs for completed runs. Expected impact: future reports carry actionable error text.
  3. [MEDIUM] Check whether bursts line up with pr-review/v1-next tag moves. Compare the burst times (about 14:06Z, 14:39Z, 19:38Z and 09:21Z) with the tag history and scripts/cut-release.sh runs. If they match, add a smoke test before advancing the next channel. Expected impact: fewer ring-0 outages.
  4. [LOW] Reduce event noise. About 45% of runs are cancelled or skipped. Consider narrowing the check_suite trigger, or filtering early in the stub with an if:, to save runner minutes. Expected impact: lower Actions cost and a cleaner run history. No reliability change.
  5. [LOW] Add an alert on failure bursts. For example, alert when 3 or more failures occur within 5 minutes. Expected impact: faster detection.

6. Health Score

Health: 8/10 — A 2.6% failure rate in clustered bursts with recovery as of 09:24Z is minor, but the missing logs mean the root cause remains unverified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions