[prompt-clustering] Prompt Clustering Analysis - 2026-08-30

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-01 – 2026-08-30)
Total Tasks Analyzed: 1,000 copilot-authored PRs
Clusters Identified: 7
Overall Success Rate: 84.0% (840 merged / 155 closed / 5 open)

Full Analysis Report

General Insights

• Most common task type: Cluster 4 — refactoring/code-consolidation tasks (em/sub/aic/sous chef tokens, driven by the "PR Sous Chef" prompt template), 245 PRs (24.5% of total), also the highest success rate among large clusters at 91.8%.
• Highest success rate overall: Cluster 2 — "fix failing CI job" one-shot tasks, 20 PRs, 100% merged.
• Lowest success rate / outlier: Cluster 6 — gh-aw CLI / container-image tasks, 120 PRs, 65.8% merged — 18.2 points below the 84.0% baseline, and the only cluster meeting the outlier threshold (≥15 PRs, ≥10-point gap).
• Most complex tasks (by files touched / additions): Cluster 4 (avg 33.7 files, 725 additions) and Cluster 1 (avg 33.4 files, 431 additions) — both still merge at 84%+, so size alone isn't the failure driver here.
• Workflow run logs (turn counts, duration, cost) were not available for this run — gh-aw logs output wasn't pre-fetched, so this analysis relies on PR-level signals (comments, reviews, files changed) as complexity proxies instead of turn counts.

Cluster Analysis

Cluster 4: Refactoring & code consolidation ("Sous Chef" tasks)

• Size: 245 tasks (24.5%)
• Success Rate: 91.8% (225 merged / 18 closed / 2 open)
• Avg comments/reviews: 13.7 / 4.8 — highest engagement of any cluster
• Top Keywords: sous chef, em, sub, aic, run
• Characteristics: Templated refactor/consolidation prompts (interface extraction, struct dedup, spec-compliance tightening) issued via the "PR Sous Chef" tool. High comment volume reflects iterative back-and-forth, but converges to merge reliably.
• Example PRs: Give the four log-entry structs a shared LogEntry interface #52107, Split pkg/workflow/compiler_types.go by lifecycle: options, mutators, types #52109, Tighten SPDD compliance specs and fixtures #52111, Add preview Cloud Hypervisor runtime support for sandbox.agent.runtime with docker-sbx parity #52119, Consolidate JobStep/JobStepData identical structs in pkg/cli #52120

---

Cluster 3: Test coverage, lint rules & error-message compliance

• Size: 221 tasks (22.1%)
• Success Rate: 85.5% (189 merged / 30 closed / 2 open)
• Avg files/additions: 6.5 files, 133 additions — smallest, most surgical diffs
• Top Keywords: pkg, added, coverage, rule, error
• Characteristics: Narrow, mechanical fixes (lint-rule false positives, error-code compliance, coverage gates). Small blast radius correlates with a healthy merge rate.
• Example PRs: Allow repository_dispatch user rate limits #52101, Add formal RS-05a checkout gate coverage #52103, Consolidate duplicate ordered dedupe/merge helpers into pkg/sliceutil #52159, Raise error-message compliance in dispatch validation and CLI evaluation/logging paths #52173, Raise error-message compliance for sandbox and workflow run validation paths #52176

---

Cluster 1: Workflow features, safe-outputs & agent jobs

• Size: 198 tasks (19.8%)
• Success Rate: 83.8% (166 merged / 32 closed)
• Top Keywords: workflow, safe, agent, job, output
• Characteristics: Feature work on the gh-aw workflow/safe-outputs system itself — new job types, output schemas, preflight checks.
• Example PRs: Add Agent Job Health Monitor to own fleet-wide agent-job failure rate #52108, Trim unused Matt Pocock skills from skills-reviewer workflow #52122, Add safe-output noop fallback to Daily Go Test Stubs (Aider) #52123, Base full-mode patch/bundle on GITHUB_SHA when running from a non-default branch #52164, Add direct Ollama inference preflight and failure diagnostics #52197

---

Cluster 6: gh-aw CLI & container-image tasks — OUTLIER (lowest merge rate)

• Size: 120 tasks (12.0%)
• Success Rate: 65.8% (79 merged / 41 closed) — 18.2 points below the 84.0% overall rate
• Top Keywords: aw, gh, gh aw, image, github
• Characteristics: A mix of genuine gh-aw CLI/infra work and a large sub-population of container-image / CVE-remediation tasks (ghcr.io/github/gh-aw-firewall, api-proxy, cli-proxy, DIFC proxy) where the actual fix lives in an upstream repo the agent can't touch. See root-cause analysis below.
• Example PRs: Add regression tests preventing committed tool binaries #52540, Fix coverage.findProfile path matching so perf-linter coverage gating actually applies #52566, Track container CVE burn-down Critical-first with a consolidated parent issue #52580, [WIP] Update container image to resolve vulnerabilities #52630, Track upstream remediation for api-proxy 0.28.1 findings #53080

---

Cluster 0: Workflow docs & report-formatting normalization

• Size: 106 tasks (10.6%)
• Success Rate: 88.7% (94 merged / 12 closed)
• Top Keywords: md, daily, workflows, github, workflow
• Characteristics: Prompt/doc hygiene sweeps — normalizing report formatting, shared-import migration across many workflow .md files at once.
• Example PRs: Address Persona Explorer findings: trigger ambiguity, digest grouping, compliance example #52121, Normalize report formatting guidance for daily workflows #52413, Migrate inline report-formatting guidance to the shared/reporting.md import #52542, Extract shared smoke-test brevity prompt and import across 21 smoke workflows #52565, Normalize report formatting across 34 non-compliant workflows #52699

---

Cluster 5: Engine/model configuration (Copilot, Claude, MAI-Code)

• Size: 90 tasks (9.0%)
• Success Rate: 74.4% (67 merged / 22 closed / 1 open)
• Top Keywords: copilot, engine, model, workflow, claude
• Characteristics: Engine-adapter and model-pricing/inventory maintenance. Second-lowest success rate — likely sensitive to fast-moving upstream model/engine changes, worth a follow-up look if the gap persists.
• Example PRs: Allow jq in Breaking Change Checker #52128, Fix Linter Miner crash: unsupported model: inherited on inline sub-agents #52133, [model-inventory] Add missing MAI-Code model entry and correct Copilot cache-read pricing #52191, Retry Claude connection failures as fresh runs #52199, Route PR code-quality reviews through the Copilot gateway #52377

---

Cluster 2: One-shot "fix failing CI job" tasks

• Size: 20 tasks (2.0%)
• Success Rate: 100% (20/20 merged)
• Top Keywords: fix, job, actions, logs, check
• Characteristics: Narrowly scoped, single-job CI failure fixes — smallest diffs (avg 1.8 files, 27 additions) and a perfect merge record.
• Example PRs: [WIP] Fix failing GitHub Actions job 'Integration: CLI Completion & Other' #52927, [WIP] Fix failing GitHub Actions job Integration: CLI Compile & Poutine #53274, [WIP] Fix failing GitHub Actions job Integration: Workflow Features #53443, Add timing summaries to long-running validation workflows #53492, [WIP] Fix failing GitHub Actions job Integration: Workflow Compiler #54008

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files	Avg Additions	Top Keywords
2	Fix failing CI job	20	100.0%	1.8	27	fix, job, actions
4	Refactor / Sous Chef	245	91.8%	33.7	725	sous chef, em, sub
0	Docs / report formatting	106	88.7%	17.9	363	md, daily, workflows
3	Tests / lint / error-code	221	85.5%	6.5	133	pkg, coverage, rule
1	Workflow / safe-outputs features	198	83.8%	33.4	431	workflow, safe, agent
5	Engine / model config	90	74.4%	23.7	423	copilot, engine, model
6	gh-aw CLI / container images	120	65.8%	32.7	512	aw, gh, image

Key Findings

1. Templated refactor prompts ("Sous Chef") dominate volume and quality. Cluster 4 is both the largest cluster (24.5%) and the highest-engagement one (13.7 avg comments), yet still merges at 91.8% — heavy review iteration doesn't hurt outcomes when the task is well-scoped.
2. Small, mechanical diffs merge best. Clusters 2 and 3 have the smallest average diffs (1.8 and 6.5 files) and the two highest success rates (100%, 85.5%). Diff size is a stronger success signal here than task category.
3. Container/CVE-remediation tasks are a recurring, worsening failure mode. Cluster 6 is the only statistical outlier this run, and it has grown from 60 PRs / 66.7% merge rate (previous analysis, 2026-08-29) to 120 PRs / 65.8% merge rate now — double the volume at essentially the same depressed success rate. See root-cause section below.

Lowest-Merge Cluster Root Cause

• Cluster: 6 — gh-aw CLI & container-image tasks
• Merge Rate Gap: 65.8% vs 84.0% overall (gap: 18.2 points)
• Primary Blocker: Scope/complexity mismatch — task is blocked on a fix that must land in an upstream repository (github/gh-aw-firewall, container base images on ghcr.io, DIFC/api-proxy/cli-proxy images), so the agent cannot produce a real diff in gh-aw itself.
• Confidence: High
• Evidence PRs: [WIP] Update container image to resolve vulnerabilities #52630, No gh-aw change: Anthropic proxy fix requires gh-aw-firewall #52671, Document upstream blocker for cli-proxy vulnerability remediation #52705, Add regression guard for vulnerable gh-aw-firewall squid 0.27.44 container pin #52939, Track upstream remediation for api-proxy 0.28.1 findings #53080, [WIP] Update container scan findings for ghcr.io/github/gh-aw-firewall/agent:0.28.1 #53119, Verify CLI proxy container remediation availability #53131, [WIP] Update container findings for ghcr.io/github/gh-aw-firewall/squid:0.28.1 #53137, [WIP] Create /llms.txt for the documentation site #53158, [WIP] Remove probe5 item marked for deletion #52997

Within Cluster 6, PRs that failed to merge look structurally different from ones that did:

Signal	Not merged (n=41)	Merged (n=79)
Near-zero-diff (<20 additions, ≤2 files changed)	78% (32/41)	15% (12/79)
Zero comments/reviews (never triaged)	54% (22/41)	—

Representative closed PRs are titled things like "[WIP] Update container image to resolve vulnerabilities", "No gh-aw change: Anthropic proxy fix requires gh-aw-firewall", "Document upstream blocker for cli-proxy vulnerability remediation", and "Track upstream remediation for api-proxy 0.28.1 findings" — all opened against a CVE/vulnerability-scan trigger where the actual remediation isn't possible from this repo. A PR triage bot consistently scores these low priority / defer with notes like "DRAFT/WIP checklist PR ... 0 diff yet".

Notably, PR #53674 ("Add task-preflight skill to stop zero-diff retry PRs on container-vuln...") was itself a merged fix explicitly targeting this exact failure mode — confirming maintainers have already identified the pattern. Its effect hasn't yet shown up in the aggregate rate, likely because it was merged partway through this 30-day window and container-vuln-triggered tasks keep recurring on a schedule.

Blocker Category	Count	Share	Notes
Upstream-blocked / near-zero-diff	32	78%	Fix lives in gh-aw-firewall or a base container image, not this repo
CI/test failure	~4	10%	Standard failing-check retries, same as other clusters
Review friction (REQUEST_CHANGES)	~1	2%	e.g. #52986, inline-router size/risk pushback
Other/unclosed pattern	~4	10%	Miscellaneous (rate-limit investigation, footer link bug, etc.)

Recommendations

1. Add a preflight gate for CVE/vulnerability-triggered tasks that checks whether the fix is achievable within gh-aw before opening a PR — skip straight to filing/updating a tracking issue (or commenting on the existing upstream issue) when the remediation depends on gh-aw-firewall or a third-party base image. Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674 is a first step; consider extending its scope/coverage given the pattern persists at 2x volume.
2. Route "no gh-aw change possible" tasks to issues, not PRs. Several closed PRs in Cluster 6 are essentially status updates (docs of an upstream blocker) — an issue comment achieves the same communication without inflating PR-close counts or bot-triage overhead.
3. Keep leaning on templated, narrowly-scoped prompts (Clusters 2 and 4 patterns) for high-confidence task generation — small-diff, single-purpose prompts consistently outperform broader ones regardless of category.
4. Watch Cluster 5 (engine/model config, 74.4%) as a secondary, smaller-gap area — not a statistical outlier this run (below the 15-PR/10-point dual threshold isn't the issue; it's borderline at 90 PRs / 9.6-point gap) but worth a follow-up pass if the gap widens next cycle.

Methodology notes

• PR bodies were cleaned (code blocks, bot-footer boilerplate, URLs stripped) and vectorized with TF-IDF (unigrams–trigrams, 150 features, min_df=3).
• K-means with elbow-method k selection (k=7) grouped the 1,000 prompts; cluster themes are derived from top-weighted TF-IDF terms per centroid, not hand-labeled categories, so keyword lists can look sparse/oddly-tokenized (e.g. "em"/"sub"/"aic" in Cluster 4 comes from a recurring prompt-template abbreviation, not literal words).
• Outlier threshold: cluster size ≥ 15 PRs and merge rate ≥ 10 points below the overall 84.0% rate. Only Cluster 6 met both conditions this run.
• The PR full-data cache (comments/reviews/commits) was refreshed for 183 PRs that had entered the 30-day window since the last run, and 183 PRs that aged out were pruned. A separate staleness check caught 31 PRs whose cached state/mergedAt disagreed with the fresher search-API snapshot (all had transitioned OPEN→MERGED since the cache was written); those were corrected before clustering using the more recent snapshot.
• Workflow-run metrics (turn counts, duration, cost) were unavailable this run (no pre-fetched gh-aw logs output), so complexity is proxied via comments/reviews/files-changed/additions rather than agent turn counts.

---

Generated by Prompt Clustering Analysis (Run: 33304722135)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 211.2 AIC · ⌖ 11.9 AIC · ⊞ 15.7K · ◷

• expires on Aug 31, 2026, 2:14 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #57410.