Agent Performance Report - Week of 2026-08-20

[github-actions[bot]]

Executive Summary

• Agents analyzed: 66 active workflows (of 286 total registered); 20 with ≥2 runs in the (partial, ~12h-covered) 24h window
• Total outputs reviewed: 319 safe outputs (lower-bound estimate; collection stopped after 180/window runs due to a log-pagination timeout)
• Overall run success rate: 84.4% today vs. 95% (Aug 17) and 97% (Aug 16) — a real regression, though partly explained by incomplete log coverage inflating the failure share
• Top performers (by proxy quality — success rate, output volume, cost efficiency): Issue Monster, PR Sous Chef, PR Triage Agent
• Bottom performers (sustained plateau, per prompt-improvement initiative): Design Decision Gate 🏗️, Matt Pocock Skills Reviewer, Impeccable Skills Reviewer, Code Scanning Fixer, Daily Go Test Parallelizer

Data caveat: metrics/latest.json collection is explicitly marked partial — only 11.7h of the 24h window (180/?? runs) was captured before a tool timeout. Treat all counts below as lower bounds, not exact totals. metrics/daily/summary.json for Jan 2026 is unavailable ("GitHub API not accessible"), so week-over-week averages before Aug 2026 are not computable from this snapshot.

Performance Rankings (proxy score: success_rate×70 + min(outputs,20) − cost/run penalty)

Top Performing Agents 🏆

1. Issue Monster — 4/4 runs successful (100%), 24 outputs, $15.28 total cost (lowest cost-per-output in the set) — efficient, high-output, no failures today.
2. PR Sous Chef — 39/39 runs successful (100%), 95 comments added, $569 total cost across the highest run volume — consistent and highly active; no failures.
3. PR Triage Agent — 3/3 successful, 79 outputs — very high output-to-run ratio.
4. PR Code Quality Reviewer — 7/8 (87.5%), 18 outputs, lowest $/output among PR reviewers ($3.16/output).

Agents Needing Improvement 📉 (bottom-10 audit per Prompt Improvement Initiative)

Agent	Runs	Success%	Outputs	Cost	Prompt Deficiency Classification	Evidence
Code Scanning Fixer	2	0%	0	$75.35	Low-actionability outputs; missing self-assessment loop	2/2 runs failed, 0 outputs, longest avg duration (1572s) of any workflow sampled — prompt has no failure-recovery or partial-progress checkpoint, so a hang/timeout produces zero value rather than a partial diagnostic comment.
Design Decision Gate 🏗️	10	70%	6	$490.58	Generic task framing; stale tool guidance	3/10 runs failed (30% AR), $49/run average — among the highest cost-per-run of any agent while producing only 6 outputs total. Named in shared-alerts.md as a "redesign/deprecation candidate" across multiple prior runs (persistent, unresolved). Complex 500+ line prompt with heavy steps: prefetch logic (see design-decision-gate.md lines 82-150) suggests brittleness is baked into the harness, not just the LLM turn — a redesign-vs-deprecation decision is overdue.
Matt Pocock Skills Reviewer	10	90%	25	$378.66	Stale examples/tool guidance; scope creep risk	Highest cost among PR review agents ($15.15/run) for comparable output volume to Impeccable and PR Code Quality Reviewer, which cost far less per run ($28/run and $7/run respectively). The prompt embeds a full fallback-triage heuristic and delegates to a sub-agent (pr-triage) — dual decision paths increase both cost and drift risk; the fallback table (lines 139-150) duplicates logic already in the sub-agent, which is unnecessary complexity for a review workflow.
Impeccable Skills Reviewer	9	89%	8	$252.18	Generic task framing; low-actionability outputs	Nearly identical harness structure to Matt Pocock Reviewer (same pr-review-base.md, pr-diff-data-fetch.md imports) but produces 8 outputs vs. Matt Pocock's 25 for similar run count — the "select the most relevant installed Impeccable skills" instruction (line 111) is vague ("If no external skills are installed, perform a normal high-signal review") with no concrete skill catalogue or decision table like Matt Pocock's, so reviews likely default to generic commentary.
[aw] Failure Investigator (6h)	2	50%	2	$83.39	Missing self-assessment loop	1/2 runs failed; a "failure investigator" agent failing half the time on its own runs is a credibility risk — no evidence of a retry/backoff or self-check step before escalation.
Daily Go Test Parallelizer	3	33%	1	$110.72	Low-actionability outputs; stale tool guidance	2/3 runs failed, only 1 output produced across all 3 runs — highest failure rate of any multi-run agent this window.
AI Moderator	12	92%	0	$121.82	Missing self-assessment loop	11/12 runs succeed but 0 safe outputs across 12 runs — "success" here likely means "ran without crashing," not "did anything useful." This mirrors the persistent historical pattern noted in agent-performance-latest.md (Codex engine failures, no safe outputs) — worth confirming whether this is expected (no moderation needed) or a silent regression.
Test Quality Sentinel	10	90%	2	$134.72	Low-actionability outputs	2 outputs across 10 runs ($67/output) — very low output density relative to cost; prompt likely needs a sharper "when to act vs. noop" rubric.
Ponytail Reviewer	8	87.5%	8	$204.94	Generic task framing	$25.6/run for 1 output/run average — mid-pack cost efficiency with no distinguishing skill differentiation visible in the naming/scope compared to the other 4 PR-review agents running near-identical harnesses.
Auto-Triage Issues	3	67%	3	$6.07	Missing self-assessment loop	1/3 runs failed — small sample but worth watching; low cost limits blast radius.

Explicit Redesign-vs-Deprecation Candidates (per initiative directive)

• Design Decision Gate — 30% AR this window, previously flagged repeatedly (shared-alerts.md history: "100% AR... Redesign/deprecation candidate", "No issue filed (not new)"). Recommendation: redesign. The prefetch steps: block is doing heavy imperative work (300+ lines including .design-gate.yml handling, ADR threshold logic) that duplicates what the LLM turn should reason about — this is an architecture-fit problem, not a prompt-wording problem. A scoped rewrite separating "gate decision" (cheap, deterministic) from "ADR drafting" (LLM) would likely cut both cost and failure rate.
• Matt Pocock Skills Reviewer — Not failing acutely (90% success) but highest-cost PR reviewer for its output tier; historically flagged as a deprecation candidate in agent-performance-latest.md when 100% AR. Current data doesn't support deprecation (it's now the top output producer of the 4 PR-review agents), but the dual-path triage (sub-agent + inline fallback table) should be simplified — recommend consolidate/simplify, not deprecate.
• Impeccable Skills Reviewer — 89% success but only 8 outputs vs. its near-identical sibling Matt Pocock's 25. Recommend redesign of the skill-selection instructions to match Matt Pocock's concrete change-type table, or consider deprecating in favor of Matt Pocock if a follow-up run confirms sustained low output density with no differentiated skill value.

Prioritized Prompt-Improvement Backlog

Priority	Agent	Fix	Expected Impact
P0	Design Decision Gate	Redesign: split deterministic gate-check steps from LLM ADR-drafting; add explicit timeout/partial-progress path	Cut 30% AR toward <10%; reduce $49/run average cost
P1	Impeccable Skills Reviewer	Add a concrete change-type → skill-selection table (mirror Matt Pocock's Step 3 table) instead of the vague "select the most relevant installed skills" instruction	Raise output density from ~0.9/run toward Matt Pocock's ~2.5/run
P1	Code Scanning Fixer	Add a self-assessment/partial-output checkpoint so a timeout still yields a diagnostic comment instead of 0 outputs	Convert 0% success runs into at least partial value
P2	AI Moderator	Add explicit success-vs-noop logging so "0 outputs, 92% success" is distinguishable from silent failure	Clarify whether 0-output pattern is expected behavior or regression
P2	Test Quality Sentinel	Add sharper act-vs-noop rubric to raise output density (currently $67/output)	Reduce cost per meaningful output
P3	Matt Pocock Skills Reviewer	Remove the duplicate inline fallback-triage table now that the pr-triage sub-agent is stable (89–90% success across siblings)	Lower prompt complexity/maintenance cost without losing coverage

Trends

• Overall run success rate: 84.4% today (partial data) vs. 95% (Aug 17) vs. 97% (Aug 16) — declining, but interpret cautiously since today's collection covers only ~50% of the 24h window and may over-sample failure-prone runs.
• Ecosystem scale: 66 active workflows today vs. 62 (Aug 17) vs. 51 (Aug 16) — increasing coverage, which naturally increases absolute failure counts even if per-agent rates hold steady.
• Cost data (total_cost_usd) newly available since Aug 17 collection; not comparable to Jan 2026 baseline (unavailable).

Skipped / Out of Scope This Run

• Full output-quality sampling (clarity/accuracy/completeness scoring of individual issues/PRs/comments) was skipped — gh issue list/gh pr list reads were blocked in this sandbox (gh subcommand restrictions returned a malformed-output error), so qualitative content review could not be performed this run. Recommend the next run verify gh read access before Phase 2.
• Campaign Manager (campaign-manager-latest.md) memory file was not found in shared storage this run — coordination notes from that orchestrator are unavailable; cross-checked only workflow-health-latest.md and shared-alerts.md.
• Full 24h ecosystem picture is a lower bound due to the partial log-pagination collection noted above.

Next Steps

1. File redesign-track issue for Design Decision Gate (P0).
2. File prompt-improvement issue for Impeccable Skills Reviewer skill-selection vagueness (P1).
3. Confirm gh CLI read access for the next run to enable qualitative output sampling.
4. Re-check AI Moderator's 0-output pattern against its intended behavior before flagging as a regression.
5. Monitor whether the 84.4% success dip on Aug 20 is a real regression or an artifact of partial log collection — re-run with full coverage next cycle.

---

Analysis period: Aug 20, 2026 (partial, ~12h window) with trend comparison to Aug 16–17, 2026
Next report: Aug 27, 2026

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · auto · 53.2 AIC · ⌖ 9.29 AIC · ⊞ 16.8K · ◷

• expires on Aug 21, 2026, 5:04 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-21T13:04:19.540Z.

Closed by Workflow