[prompt-clustering] Prompt Clustering Analysis - 2026-09-05

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-11 to 2026-09-05)
Total Tasks Analyzed: 1,197
Clusters Identified: 4
Overall Success Rate: 77.1% (923 merged / 1,197)

⚠️ Merge rate dipped to a 7-run low today. It has held between 79.9% and 84.5% since 2026-08-29; today's 77.1% is driven almost entirely by one outlier cluster (see below) — excluding it, the rate would be 81.0%.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 0 — code fixes, test coverage, and CI/error-message work (443 tasks, 37% of total)
• Highest Success Rate: Cluster 3 — larger Go refactors/deduplication (88.6%)
• Most Complex Tasks: Cluster 3 also has the heaviest footprint (avg 733 additions, 35.6 files, 13.8 comments, 5.1 reviews per PR) — biggest diffs correlate with the most review engagement, not the lowest success
• Outliers: 1 outlier cluster flagged (Cluster 2, size ≥15 and merge rate ≥10 points below the 77.1% baseline)

Workflow-level turn/duration metrics were not included this run: correlating gh-aw's own --engine copilot run logs to individual PR numbers would have required a very large, slow log download with no reliable PR-to-run mapping, so this pass relies on PR data only (comments, reviews, file/line deltas, outcome).

Cluster Analysis

Cluster 1 (of 4): Code fixes, coverage & CI/error-message compliance

• Size: 443 tasks (37.0% of total)
• Success Rate: 80.8%
• Top Keywords: coverage, pkg, job, added, fixes, test, error, actions
• Characteristics: Bug fixes, added test coverage, CI job tweaks, and error-message compliance passes across pkg/*. Mid-size diffs (avg 248 additions, 15.7 files), moderate review load (avg 4.5 comments, 2.7 reviews).
• Example PRs: Allow repository_dispatch user rate limits #52101, Add formal RS-05a checkout gate coverage #52103, Consolidate duplicate ordered dedupe/merge helpers into pkg/sliceutil #52159, Base full-mode patch/bundle on GITHUB_SHA when running from a non-default branch #52164, Raise error-message compliance in dispatch validation and CLI evaluation/logging paths #52173

---

Cluster 2 (of 4): Agentic workflow (.md/.lock) authoring

• Size: 410 tasks (34.3% of total)
• Success Rate: 76.1%
• Top Keywords: workflow, workflows, github, copilot, agent, daily, md, lock
• Characteristics: Creating/editing gh-aw workflow markdown + compiled lock files — daily bots, monitors, skill trims, safe-output wiring. Largest average diffs of any cluster (avg 487 additions, 30.2 files) but the lowest review engagement of the non-outlier clusters (avg 3.0 comments, 1.7 reviews).
• Example PRs: Add Agent Job Health Monitor to own fleet-wide agent-job failure rate #52108, [WIP] Add AI-crawler robots.txt allow rules and llms.txt for the docs site #52115, Address Persona Explorer findings: trigger ambiguity, digest grouping, compliance example #52121, Trim unused Matt Pocock skills from skills-reviewer workflow #52122, Add safe-output noop fallback to Daily Go Test Stubs (Aider) #52123

---

Cluster 3 (of 4): Operational-value grader/study batch — ⚠️ OUTLIER

• Size: 64 tasks (5.3% of total)
• Success Rate: 7.8%
• Top Keywords: operational, operational value, value, evidence, study, github, null, metric
• Characteristics: A large, templated batch (mostly PR Add operational-value grading for Duplicate Code Detector #58452–Record blocked operational-value study for daily-choice-test #58559) adding "operational-value grader" configs and schema-compliant "study record" JSON for many different workflows/tools. 54 of 64 (84%) were closed with zero comments and zero reviews — a github-actions PR Triage Agent tagged several as Action: batch_review | Batch: operational-value-grading, but the batch was never actually individually reviewed before closure. Lowest diff footprint of any cluster (avg 2.4 files) and near-zero engagement (avg 0.3 comments, 0.09 reviews).
• Example PRs: Document the graders CLI command #55794, Rename aw-value skill to operational-value-designer and update grader guidance references #55829, Add YAML dashboard language specification #55841, Lint shell scripts declared in workflow frontmatter #56150, Document intent-driven workflow design #57005

---

Cluster 4 (of 4): Larger Go refactors & deduplication

• Size: 280 tasks (23.4% of total)
• Success Rate: 88.6% (highest)
• Top Keywords: (noisy — see note below) plus clearly Go-refactor titles: interface consolidation, struct dedup, error-message batches, sandbox runtime parity
• Characteristics: The heaviest tasks by far — avg 733 additions, 35.6 files changed, 13.8 comments, 5.1 reviews per PR — yet the most successful cluster. Bigger, well-scoped refactors get merged more reliably than small speculative additions (contrast with Cluster 3 below).
• Note on keywords: top TF-IDF terms for this cluster ("em", "sub", "aic", "sous chef", "chef", "pr sous") are an artifact of a recurring "PR Sous Chef" bot signature bleeding into PR body text, not the actual task content. This same artifact also appeared in the prior week's clustering runs. Titles are a more reliable signal: refactors, deduplication, error-message compliance, and CI/runtime fixes.
• Example PRs: Give the four log-entry structs a shared LogEntry interface #52107, Split pkg/workflow/compiler_types.go by lifecycle: options, mutators, types #52109, Tighten SPDD compliance specs and fixtures #52111, Add preview Cloud Hypervisor runtime support for sandbox.agent.runtime with docker-sbx parity #52119, Consolidate JobStep/JobStepData identical structs in pkg/cli #52120

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Additions	Avg Files	Avg Comments	Avg Reviews
1	Code fixes / coverage / CI compliance	443	80.8%	248	15.7	4.49	2.66
2	Workflow (.md/.lock) authoring	410	76.1%	487	30.2	3.02	1.67
3	Operational-value grader/study batch ⚠️	64	7.8%	225	2.4	0.31	0.09
4	Larger Go refactors / dedup	280	88.6%	733	35.6	13.76	5.05

Full Data Table (sample of 30 PRs per cluster available on request; showing top 5 examples per cluster)

PR #	Title	Cluster	Outcome
52101	Allow repository_dispatch user rate limits	1	Merged
52103	Add formal RS-05a checkout gate coverage	1	Merged
52159	Consolidate duplicate ordered dedupe/merge helpers into pkg/sliceutil	1	Merged
52108	Add Agent Job Health Monitor to own fleet-wide agent-job failure rate	2	(see PR)
52115	[WIP] Add AI-crawler robots.txt allow rules and llms.txt for the docs site	2	(see PR)
55794	Document the graders CLI command	3	Merged
56150	Lint shell scripts declared in workflow frontmatter	3	Merged
58452	Add operational-value grading for Duplicate Code Detector	3	Closed
58465	Add ACE Editor operational-value grader	3	Closed
58559	Record blocked operational-value study for daily-choice-test	3	Closed
52107	Give the four log-entry structs a shared LogEntry interface	4	Merged
52109	Split pkg/workflow/compiler_types.go by lifecycle: options, mutators, types	4	Merged

Key Findings

1. Templated batch work is the single biggest drag on merge rate. The operational-value grader/study batch (Cluster 3, 64 PRs) merges at 7.8% vs. the 77.1% baseline — a 69-point gap. It alone explains most of today's rate dip vs. the prior week (81.0% excluding it vs. 77.1% including it).
2. Bigger, well-scoped refactors outperform small speculative additions. Cluster 4 (large Go refactors, avg 733 additions) has the highest success rate (88.6%) despite the biggest diffs, while Cluster 3 (small, avg 225 additions, mostly config/schema files) has the worst. Size alone doesn't predict failure — lack of review engagement does.
3. Zero review/comment activity is a strong failure predictor. 84% of Cluster 3's PRs were closed with no comments and no reviews at all, versus single-digit percentages in the other three clusters. A github-actions PR Triage Agent tagged these for batch_review, but that batch review does not appear to have happened before closure.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 3 (of 4) — Operational-value grader/study batch
• Merge Rate Gap: 7.8% vs. 77.1% overall (gap: 69.3 points)
• Primary Blocker: Scope/complexity mismatch — a large templated batch of speculative "operational-value grader" / "study record" PRs generated across many unrelated workflows/tools, most never individually reviewed before being closed
• Confidence: High — 54 of 64 PRs (84%) in this cluster have zero comments and zero reviews, and the PRs that were reviewed and merged (e.g. Document the graders CLI command #55794, Lint shell scripts declared in workflow frontmatter #56150, Document intent-driven workflow design #57005) are documentation/lint-adjacent tasks, not schema/grader-scaffolding tasks like most of the closed ones
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Add ACE Editor operational-value grader #58465, Record operational value study for archivx-agentic-workflows-analyzer #58501, [WIP] Add operational-value grader for code scanning fixer #58521, Record blocked operational-value study for AWF spec surfacing #58553, Record blocked operational-value study for daily-choice-test #58559 (all closed, zero comments/reviews); Document the graders CLI command #55794, Lint shell scripts declared in workflow frontmatter #56150, Document intent-driven workflow design #57005 (merged, reviewed)

Blocker Category	Count	Share	Notes
CI/test failure	0	0%	No CI failure evidence found in comments/reviews for this cluster
Review friction	5	8%	A handful received actual review feedback (e.g. #57005's "changes recommended") and were then merged after follow-up
Scope/complexity mismatch (unreviewed batch closure)	54	84%	Zero comments, zero reviews; PR Triage Agent tagged batch_review/operational-value-grading but no visible follow-through before closing
Other/unknown	5	8%	Insufficient data (0 comments but non-zero commits; unclear closure reason)

Recommendations

1. Throttle or gate the operational-value grader/study batch generator. If this task template continues to run at this volume with an 8% merge rate, it is producing far more PRs than the review process can absorb — consider requiring an explicit maintainer opt-in per target workflow instead of generating one PR per candidate.
2. Make the "batch_review" triage action actually route to a reviewer, or auto-close with a documented reason instead of leaving PRs to age out silently — the current pattern (tag as batch_review, then close with zero engagement) gives no signal back to the task-generation process about why these are failing.
3. Favor batching several related grader/study additions into one PR rather than one PR per workflow — Cluster 4 shows that larger, well-scoped PRs merge more reliably than many small ones targeting the same kind of change.
4. Fix the "PR Sous Chef" text-extraction artifact in future clustering passes — the bot signature is bleeding into PR body text and polluting keyword extraction for the largest, most successful cluster (Cluster 4), which otherwise has a clean signal in its titles.

---

Generated by Prompt Clustering Analysis (Run: 33958724638)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 164.8 AIC · ⌖ 13.9 AIC · ⊞ 16.2K · ◷

• expires on Sep 6, 2026, 2:10 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #58967.