Skip to content

[aw] Agent Job Health Monitor timed out #54094

Description

@github-actions

Workflow Failure

Workflow: Agent Job Health Monitor
Branch: main
Run: https://github.com/github/gh-aw/actions/runs/32312306604

Warning

Task Could Not Be Completed: The agent reported that the task could not be performed due to an infrastructure or tool failure.

Reasons:

  • The Agent Job Health Monitor completed its full analysis (Phases 0-6) successfully, but the create_discussion safe-output was consumed by an earlier throwaway invocation used only to test shell-quoting mechanics in this sandbox (title 'test title', body 'line one/line two/line three'), before the real report was ready to send. The safe-outputs tool enforces a 1-per-run cap on create_discussion, so the real, fully-drafted report below could not be published this run. Please disregard any discussion titled 'test title' created by this run and treat this report_incomplete as the actual output for 2026-08-19. No new issue was warranted this cycle (all Novel failure clusters are single-workflow, single-run, below the 2-workflow/10%-of-runs threshold), so no create_issue call was skipped as a result of this error.

    Summary

Warning

Fleet-wide agent-job failure rate more than doubled day-over-day: 9.38% today vs 3.33% yesterday (+6.0pp, not yet over the 10pp hard-regression threshold, but worth watching). The increase is not a broad regression — it is concentrated in a handful of already-tracked chronic offenders, chiefly the Copilot CLI startup-crash cluster covered by P0 issue #54009.

  • Window: last 24 full hours, 2026-08-18 to 2026-08-19 (UTC)
  • Total agentic runs: 341 (342 total minus 1 non-agentic Super Linter Report run, which is excluded)
  • Distinct workflows observed: 116
  • Agent-job failures: 32, across 23 distinct workflows
  • Run-weighted fleet failure rate: 32 / 341 = 9.38%
  • Workflow-weighted median rate: 0.0% (93 of 116 workflows, 80%, had zero agent-job failures)
  • Workflow-weighted mean rate: ~14% (pulled up by a few low-volume, high-rate workflows, e.g. Code Scanning Fixer 3/3 runs = 100%)
  • Tracked failures: 29 of 32 (91%), already covered by an open issue
  • Novel failures: 3 of 32 (9%), no matching open issue, none crosses the 2-workflow / 10%-of-runs bar for a new issue

Top failing step: Execute GitHub Copilot CLI (12 of 32 failures, 37.5%), consistent with #54009's Copilot CLI pre-network crash finding.

Tracked failures (29 of 32)

Ponytail Reviewer x5 (#54009), Design Decision Gate x3 (#53619/#54013), Code Scanning Fixer x3 (#54018), AI Moderator x2 (#54069), and 18 more single-failure workflows each already covered by an open issue (#54085 Daily BYOK Ollama Test, #54087 Daily Go Test Parallelizer, #54083 Duplicate Code Detector, #54070 Daily Cache Strategy Analyzer, #54056 Linter Miner, #54049/#54050 Slide Deck Maintainer, #54021 Architecture Guardian, #54020 Daily Fact, #54019 Daily Malicious Code Scan Agent, #54016 CLI Consistency Checker, #54017 CI Optimization Coach, #54000/#54001 Deep Report, #53421/#53405 Avenger, #54072 aw Failure Investigator, plus PR Sous Chef's recurring auto-filed issues, and Daily Max Ai Credits Test also under #54009).

Novel failure clusters (3, none crossing the issue-creation threshold)

A: daily-experiment-report, Copilot CLI crash matching the #54009 pattern but not yet listed in that issue - run 32230488509. B: Daily Container Image Security Scan, failing step is 'Enforce critical vulnerability and license gates' (a gate/policy check, NOT the Copilot CLI, which succeeded) - run 32220139949 - distinct root cause, needs its own follow-up if it recurs. C: Daily AgentRx Trace Optimizer, genuine Execute Claude Code CLI crash, possibly related to #53935 or #52459 but not confirmed - run 32220130823.

Schedule heartbeat

No confirmed blind spots. Avenger's low run count (7 of ~24 possible) is expected self-throttling (skips when CI passing). Bot Detection and Auto-Triage Issues show zero runs in the logs aggregation tool despite being confirmed active via gh workflow list / gh run list - this is a logs-aggregation coverage gap, not a fleet health issue. Bulk gh api repos/github/gh-aw/actions/workflows failed with a network error twice in this environment, so only the highest-frequency schedule workflows were spot-checked rather than a full sweep of all ~150 schedule-triggered workflows.

Trend

2026-08-18: 300 runs, 10 failures, 3.33%, 8 tracked, 2 novel. 2026-08-19: 341 runs, 32 failures, 9.38%, 29 tracked, 3 novel. +6.05pp day-over-day, below the 10pp hard-regression threshold but the largest single-day jump on record so far.

Recommendations

  1. Prioritize [aw-failures] [P0] Log-capture blindness regressed — GH Copilot CLI failures (Ponytail Reviewer, Daily Max Ai Credits Test) invi [Content truncated due to length] #54009 (P0) - largest cluster (37.5% of failures), add daily-experiment-report to its affected-workflow list.
  2. Investigate Cluster B (Daily Container Image Security Scan gate failure) as its own issue if it recurs - not an engine crash.
  3. Cross-check Cluster C (Daily AgentRx Trace Optimizer) against the [aw-failures] [P0] Cloud Hypervisor guest network never comes up — full egress blackout kills sandbox runs across engines #53935 audit-diff method before opening a new issue.
  4. Fix the logs aggregation tool's coverage gap for Bot Detection (active/successful but absent from the dataset).
  5. No new issue filed this cycle - all clusters are tracked or below the issue-creation bar.

References: run 32249649808 (Ponytail Reviewer, #54009 cluster), run 32230488509 (daily-experiment-report, untracked #54009-pattern instance), run 32220139949 (Daily Container Image Security Scan, distinct gate-check failure).

This is a structured incompletion signal (report_incomplete), not a real task outcome. Any other safe outputs emitted alongside this signal (e.g., comments) describe the failure state, not a completed review or action.

Agent Timed Out: The agent job exceeded the maximum allowed execution time (30 minutes).

To increase the timeout, add or update the timeout-minutes setting in your workflow's frontmatter:

---
timeout-minutes: 40
---

Action Required

Assign this issue to an agent to debug and fix the issue.

Optimize token consumption

This failure was triggered by a guardrail limit (max-turns / timeout). Use this prompt with any coding agent (GitHub Copilot, Claude, Gemini, etc.) to analyze token usage and reduce costs:

Optimize the agentic workflow token consumption using https://raw.githubusercontent.com/github/gh-aw/main/optimize.md

The workflow run is at https://github.com/github/gh-aw/actions/runs/32312306604
Debug with any coding agent

Use this prompt with any coding agent (GitHub Copilot, Claude, Gemini, etc.):

Debug the agentic workflow failure using https://raw.githubusercontent.com/github/gh-aw/main/debug.md

The failed workflow run is at https://github.com/github/gh-aw/actions/runs/32312306604
Manually invoke the agent

Debug this workflow failure using your favorite Agent CLI and the agentic-workflows prompt.

Tip

Stop reporting this workflow as a failure

To stop a workflow from creating failure issues, set report-failure-as-issue: false in its frontmatter:

safe-outputs:
  report-failure-as-issue: false

Generated from Agent Job Health Monitor · 477.6 AIC ·

  • expires on Aug 20, 2026, 12:02 PM UTC

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions