[prompt-analysis] Copilot PR Prompt Analysis - 2026-03-14

[github-actions[bot]]

Summary

Analysis Period: Last 30 days
Total PRs: 1,000 | Merged: 790 (79.0%) | Closed: 209 (20.9%) | Open: 1 (0.1%)

⬆️ Merge rate improved from 78.4% (yesterday) to 79.0% today — a gradual upward trend over the past 10 days.

---

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
🔧 Bug Fix	830	665	165	80.1%
✨ Feature Addition	36	28	8	77.8%
🧪 Tests	77	57	19	75.0%
♻️ Refactoring	7	5	2	71.4%
📚 Documentation	32	20	12	62.5%
🔀 Other	18	15	3	83.3%

Bug Fix is by far the dominant category (83% of all PRs), indicating the agent is primarily being used for corrective work. Documentation PRs have the lowest success rate at 62.5%.

---

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 975 words (longer, more detailed prompts)
• Top title keywords: safe, workflow, update, github, remove, docs, output, agent, copilot, error
• Prompts include specific root cause description and concrete reproduction steps

Example Merged PRs

1. PR #20901: fix: disable HTTP server timeouts in safe-outputs MCP server to prevent idle connection drops

   "The safe-outputs MCP server's HTTP connections were silently dropped after idle periods (~5+ min) during long agent runs, causing zero safe outputs to..." → Merged

2. PR #20899: fix: safe-output-items.jsonl always empty and SafeItemsCount always 0

   "safe-output-items.jsonl was always empty and SafeItemsCount always 0 for workflows executing modification-type safe outputs..." → Merged

3. PR #20898: Fix add-wizard/add --create-pull-request failing on GitHub Enterprise Server repos

   "gh pr create --repo owner/repo without --hostname always routes to github.com, so PR creation fails with a repository-not-found GraphQL error..." → Merged

4. PR #20897: fix: close_pull_request missing from config.json despite being in handler config

   "close_pull_request was correctly compiled into GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG but never written to /opt/gh-aw/safeoutputs/config.json..." → Merged

5. PR #20895: fix: correct stale with: version: when uses: comment and version drifted before upgrade

   "gh aw upgrade silently left with: version: stale when the uses: comment and with: version: were already out of sync..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 868 words (107 words shorter than merged PRs)
• Top title keywords: workflow, github, remove, update, copilot, feat, token, support, checkout, steps
• Notable: feat (15 occurrences) and token (13 occurrences) are disproportionately present in closed PRs
• [WIP] prefixed PRs are consistently closed

Example Closed PRs

1. PR #20896: fix: gh aw upgrade always corrects with: version: drift in copilot-setup-steps.yml

   "gh aw upgrade updated the uses: SHA/comment but silently left with: version: stale whenever pre-existing drift existed..." → Closed (superseded by fix: correct stale with: version: when uses: comment and version drifted before upgrade #20895)

2. PR #20894: fix: register close_pull_request in safe-outputs config.json

   "close_pull_request was present in GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG but absent from config.json..." → Closed (superseded by fix: close_pull_request missing from config.json despite being in handler config #20897)

3. PR #20874: [WIP] Add test for new feature

   "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date..." → Closed (incomplete/placeholder)

4. PR #20855: fix: bump TypeScript target/lib to es2022 to support Array.at()

   "Fixes the failing CI jobs js and js-integration-live-api..." → Closed (possibly superseded)

5. PR #20843: fix: restore 3 missing backward-compatible action pin entries

   "TestGetActionPinsSorting was failing because action_pins.json had 31 entries while the test expected 34..." → Closed (superseded)

---

Key Insights

• Longer, more specific prompts succeed more: Merged PR prompts average 975 words vs 868 words for closed (+12%). Prompts that include root cause analysis, file references, and reproduction steps consistently get merged.

• Superseded PRs are the primary driver of closures: Many closed PRs (like fix: register close_pull_request in safe-outputs config.json #20894→fix: close_pull_request missing from config.json despite being in handler config #20897, fix: gh aw upgrade always corrects with: version: drift in copilot-setup-steps.yml #20896→fix: correct stale with: version: when uses: comment and version drifted before upgrade #20895) are near-duplicates that were superseded by refined re-attempts. This is expected behavior in an iterative agent workflow.

• "feat:" and "token" correlate with closures: The feat keyword appearing disproportionately in closed PR titles suggests PRs using conventional commit format for new features tend to be more exploratory/iterative. token related PRs may indicate auth/credential challenges that require multiple attempts.

• Documentation PRs underperform (62.5%): Docs PRs have the lowest merge rate. This may indicate higher reviewer scrutiny or more frequent "not needed" determinations for doc changes.

---

Recommendations

1. DO: Include concrete root cause analysis and the specific code path causing the issue — prompts with clear problem statements have 80%+ merge rates.
2. DO: Reference specific files, functions, or test names in your prompt — this specificity correlates strongly with merged PRs.
3. AVOID: Using [WIP] prefix or vague placeholder descriptions ("I will get started on it...") — these always result in closure.
4. AVOID: Submitting near-duplicate PRs for the same issue — refine the prompt and create a single well-scoped PR instead.
5. DO: For documentation PRs, include clear justification for why the docs change is necessary to improve the 62.5% success rate.

---

Historical Trends (Last 7 Days)

Date	PRs	Merged	Closed	Success Rate
2026-03-14 (today)	1,000	790	209	79.0% ⬆️
2026-03-13	1,000	784	210	78.4%
2026-03-12	999	787	212	78.8%
2026-03-11	1,000	781	214	78.1%
2026-03-06	1,000	772	220	77.2%
2026-03-05	1,000	775	221	77.5%
2026-03-04	1,000	772	224	77.2%

Trend: 📈 Consistent improvement over the past 10 days — merge rate has risen from ~77.2% to 79.0% (+1.8 percentage points). The dataset covers the same rolling 30-day window so the improvement reflects both new successful PRs entering and old closed PRs aging out.

Data Quality Notes

• 5 PRs had null/empty bodies (0.5% of dataset) — categorized by title only
• Categories are assigned by keyword matching (title + body); "bug_fix" dominates because of the fix: conventional commit prefix
• Success rate for "test" category excludes the 1 open PR from the denominator
• The rolling 30-day window means daily snapshots capture slightly different PR populations

---

References: Workflow Run §23080879081

AI generated by Copilot PR Prompt Pattern Analysis · history

• expires on Mar 15, 2026, 5:05 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-15T05:05:20.309Z.

Closed by Workflow