[workflow-analysis] Weekly Workflow Analysis: AI Moderator hangs and PR-bot cluster failures burn 25%+ of CI runs

[github-actions[bot]]

Weekly Workflow Analysis (2026-08-24 → 2026-08-31)

Sampled 300 unique runs across 297 workflows (daily snapshots per day plus a recent-hours deep dive, since the repo runs too high a volume — roughly 400+ runs/day — for exhaustive per-run collection in one pass). Overall failure rate in the sample: 25.7% (77/300), well above what a healthy fleet should show. Two failure modes dominate, and both are infra/driver-level, not agent reasoning errors.

Key issues

1. AI Moderator is effectively broken (94% failure, 16/17 sampled runs) and burns huge CI time doing nothing. Failing runs show Turns=0, TokenUsage=0, yet ran for 3.9–7.5 hours each before failing — the agent never starts, the job just hangs until timeout. Sampled runs alone consumed 1,805 action-minutes (~30 hours) for essentially zero agent work. This is the single highest-value fix this week.

   • Example: §33215924002 — 7.5h, 0 turns, failed.

2. A shared cluster of PR-review bots (Ponytail Reviewer, PR Code Quality Reviewer, Matt Pocock Skills Reviewer, Impeccable Skills Reviewer, Test Quality Sentinel) fail together on the same PR events, in two distinct patterns:

   • Fast-fail (~2s): all four fail within the same 2-second window on pull_request events, 0 turns — looks like a shared precondition/gate step erroring immediately.
   • Long-hang (~2.6h): the same four bots hung for 2.6 hours simultaneously on another PR (§33342582122, §33342582184) before failing.
   • Combined cluster: 72 sampled runs, 23 failures (32%), 1,188 action-minutes burned. Since all five bots share this signature, the root cause is likely a common included component (shared MCP server/config) rather than five independent bugs.

3. Driver-level CLI crashes (Execute Claude Code CLI / Execute GitHub Copilot CLI step itself failing, not the agent's reasoning) are the dominant failure kind in the sample (driver_exit ≫ agent_logic: 25 vs 4 occurrences in one batch). Confirmed on Avenger (§32906916840) and Design Decision Gate (§32787203288).

4. Daily BYOK Ollama Test: 100% failure (4/4 sampled), always at the Execute GitHub Copilot CLI step (§32784702987) — this integration looks fully broken, not flaky.

Reliability by workflow (sample)

Workflow	Failure rate	Sampled runs
AI Moderator	94%	17
Avenger	100%	5
Daily BYOK Ollama Test	100%	4
Daily Model Inventory Checker	60%	5
Design Decision Gate 🏗️	50%	14
Test Quality Sentinel	36%	14
Ponytail Reviewer / PR Code Quality Reviewer	33%	15 each
Impeccable Skills Reviewer	31%	13
Matt Pocock Skills Reviewer	27%	15
PR Sous Chef (highest volume: 48 runs)	4%	48

PR Sous Chef is worth calling out positively — by far the highest-volume workflow in the sample and one of the most reliable (4% failure).

View Details — resource usage

• Sample totals: 33.7M tokens, 13,646 AIC (cost units), 5,472 action-minutes, avg run duration 17.7 min, max 450 min (7.5h, the AI Moderator hang above).
• Heaviest single-run token consumers: Terminal Stylist (2.2M tokens / 11m), Daily Evals Feature Report (1.5M / 17.9m), PR Sous Chef (1.45M / 8.5m), Daily Observability Report for AWF Firewall and MCP Gateway (recurs 5× in top consumers, avg 765k tokens/run — worth checking whether its scope can be narrowed).
• Engine mix in sample: codex 31, pi 26, copilot 23, claude 16 (many runs lack an explicit engine tag in the log schema).

Next actions

1. Investigate why AI Moderator's agent process never starts (0 turns) yet the job runs for hours — check for a hung setup step, blocked network call, or missing timeout on an early step. This is the highest-impact fix (30+ CI-hours/week wasted at current sample rate).
2. Find the shared component behind the Ponytail/PR Code Quality/Matt Pocock/Impeccable/Test Quality Sentinel cluster's simultaneous failures — likely a common included MCP server or shared workflow snippet.
3. Add/verify step-level timeouts on the agent job across workflows so a stuck CLI can't run for hours before being killed — this alone would cap the damage from issue rejig docs #1 even before root cause is found.
4. Re-evaluate or disable Daily BYOK Ollama Test, which has a 100% failure rate in the sample and appears to test a non-functional integration.

References:

• §33215924002 — AI Moderator, 7.5h hang, 0 turns
• §33342582122 — Matt Pocock Skills Reviewer, 2.6h hang cluster
• §32784702987 — Daily BYOK Ollama Test, CLI execution failure

Generated by 🔍 Weekly Workflow Analysis · claude · agent · 234.7 AIC · ⌖ 7.34 AIC · ⊞ 5K · ◷

• expires on Sep 1, 2026, 1:51 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-01T09:51:34.467Z.

Closed by Workflow