[delight] User Experience Analysis Report - 2026-08-11

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (blog/2026-06-02-agent-of-the-day.md, guides/ai-issue-triage.md)
• 1 CLI command (gh aw add-wizard, help text sourced from pkg/cli/add_wizard_command.go since the compiled binary was unavailable offline)
• 2 workflow message configurations (daily-fact.md, smoke-agent-public-approved.md)
• 1 validation file (pkg/workflow/heredoc_validation.go)

Overall Quality: Mostly professional, with one concrete documentation gap that will cause a first-run failure for new users following the tutorial as written.

Key Finding: The AI issue triage guide's example workflow lists priority labels (priority/p0, priority/p1, priority/p2) in add-labels.allowed but never tells the reader these labels must already exist in the target repository — GitHub's label API returns a 404 if they don't, so a user who copies the example verbatim into a fresh repo will hit a confusing runtime failure with no guidance in the doc that caused it.

Quality Highlights ✅

Example 1: Clear security rationale in the issue triage guide

• File: docs/src/content/docs/guides/ai-issue-triage.md
• What works well: The guide explains why safe outputs matter for security, not just how to use them — tying add-labels/add-comment restrictions directly to prompt-injection risk.
• Quote: "add-labels and add-comment matter for security because the agent does not receive direct write access to issues. gh-aw validates label names and comment output before posting, which reduces the risk of prompt injection turning repository analysis into unrestricted writes."

Example 2: Well-documented auto_create pattern for milestones (contrast case)

• File: docs/src/content/docs/reference/safe-outputs.md (assign-milestone section)
• What works well: The milestone safe-output documents exactly what happens if the target doesn't exist and how to opt into auto-creation ("Without auto_create, the handler returns a clear error listing the available milestones and suggesting auto_create: true"). This is the standard the add-labels example in the issue-triage guide should meet, but currently doesn't.

Improvement Opportunities 💡

High Priority

Opportunity 1: Missing label pre-creation note in AI issue triage guide - Single File Improvement

• File: docs/src/content/docs/guides/ai-issue-triage.md
• Current State: The example frontmatter (lines ~20-30) lists allowed: [bug, feature, question, needs-info, priority/p0, priority/p1, priority/p2, duplicate] under add-labels, and the prose immediately below the code block (last paragraph) explains the security rationale for add-labels/add-comment but says nothing about label existence requirements.
• Issue: bug, feature, and question exist as GitHub's default repo labels, but priority/p0, priority/p1, priority/p2, and needs-info are not defaults. If a reader copies this example into a fresh repository (a very likely first action for anyone following a "getting started with issue triage" guide), the workflow will fail at runtime when it tries to apply a label that doesn't exist, with no signal in the doc explaining why or how to fix it.
• User Impact: New users following a copy-paste tutorial hit an opaque runtime error unrelated to anything they typed — this directly violates "Trust and Reliability" (predictable behavior) and "Documentation Quality" (completeness) principles for enterprise users evaluating the tool for the first time.
• Suggested Change: Add one sentence after the security-rationale paragraph noting that labels in allowed must already exist in the repository (unlike assign-milestone's auto_create option), with a one-line pointer to create them via gh label create or the repo Settings > Labels page before running the workflow.
• Design Principle: Documentation Quality (completeness) / Trust and Reliability (predictable behavior, clear error prevention)

Files Reviewed

Documentation

• docs/src/content/docs/blog/2026-06-02-agent-of-the-day.md - Rating: ✅
• docs/src/content/docs/guides/ai-issue-triage.md - Rating: ⚠️

CLI Commands

• gh aw add-wizard - Rating: ✅ (clear examples, precise flag documentation, good separation of interactive vs. CI usage)

Workflow Messages

• .github/workflows/daily-fact.md - Rating: ✅ (whimsical tone is intentional and consistent for this workflow's purpose; footer/status messages are clear)
• .github/workflows/smoke-agent-public-approved.md - Rating: ✅ (concise, professional status messages appropriate for a smoke test)

Validation Code

• pkg/workflow/heredoc_validation.go - Rating: ✅ (error messages are specific, name the exact invalid character/reason, and doc comments explain defense-in-depth rationale clearly)

Metrics

• Files Analyzed: 6
• Quality Distribution:
  • ✅ Professional: 5
  • ⚠️ Needs Minor Work: 1
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Task 1: Add label pre-creation note to AI issue triage guide

File to Modify: docs/src/content/docs/guides/ai-issue-triage.md

Current Experience

The guide's example workflow configures add-labels.allowed with priority/p0, priority/p1, priority/p2, and needs-info — none of which are GitHub default labels. The doc's closing paragraph explains only the security rationale for restricting labels, not that they must pre-exist.

Quality Issue

Design Principle: Documentation Quality (completeness) and Trust and Reliability (predictable behavior)

A reader who follows this guide in a brand-new repository will have the workflow run, classify an issue correctly, then fail silently or with an opaque GitHub API error when it tries to apply priority/p1 because the label doesn't exist yet. Nothing in the doc anticipates or explains this failure mode.

Proposed Improvement

Add one sentence to the final paragraph of the guide.

Before:

`add-labels` and `add-comment` matter for security because the agent does not receive direct write access to issues. gh-aw validates label names and comment output before posting, which reduces the risk of prompt injection turning repository analysis into unrestricted writes.

After:

`add-labels` and `add-comment` matter for security because the agent does not receive direct write access to issues. gh-aw validates label names and comment output before posting, which reduces the risk of prompt injection turning repository analysis into unrestricted writes.

Labels listed in `allowed` (such as `priority/p0` or `needs-info`) must already exist in the repository before the workflow runs — GitHub's label API returns an error for labels that don't exist yet. Create them first with `gh label create priority/p0 --color b60205` (repeat for each priority level and `needs-info`), or add them via the repository's Settings > Labels page.

Why This Matters

• User Impact: Prevents a confusing first-run failure for anyone following the tutorial in a fresh repository — a very common onboarding path.
• Quality Factor: Documentation completeness and predictable, trustworthy behavior.
• Frequency: High — this is a "getting started" style guide likely followed verbatim by new adopters.

Success Criteria

• Changes made to docs/src/content/docs/guides/ai-issue-triage.md only
• Added sentence clarifies label pre-creation requirement with an actionable gh label create example
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/guides/ai-issue-triage.md
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · auto · 49.2 AIC · ⌖ 2.51 AIC · ⊞ 12K · ◷

• expires on Aug 14, 2026, 7:07 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #52308.