[delight] User Experience Analysis Report - 2026-08-12

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (docs/src/content/docs/guides/open-telemetry.mdx, docs/src/content/docs/patterns/orchestration.md)
• 2 workflow message configurations (smoke-checkout-pr-dispatch.md, outcome-collector.md)
• 1 validation file (pkg/workflow/runtime_validation.go)

Overall Quality: Professional overall, with one clear inconsistency in error messaging that reduces actionability.

Key Finding: validateExpressionSizes in pkg/workflow/runtime_validation.go produces two different error messages for the same failure condition — one with an actionable remediation tip, one without — creating an inconsistent troubleshooting experience.

Quality Highlights ✅

Example 1: Well-structured OpenTelemetry guide

• File: docs/src/content/docs/guides/open-telemetry.mdx
• What works well: Clear separation between write-side (OTLP) and read-side (MCP) telemetry, a practical logSpan API reference table with types and defaults, and concrete debugging commands (gh aw logs, jq snippets) that reflect real troubleshooting workflows.
• Quote/Reference: "logSpan is non-fatal and never throws. Export failures are surfaced as console.warn." — this proactively sets expectations about failure behavior, which builds trust.

Example 2: Actionable validation errors (partial)

• File: pkg/workflow/runtime_validation.go
• What works well: The block-scalar size error (line 124) explains why the limit exists ("GitHub Actions has a 21KB limit...") and how to fix it ("Split the step into separate run: blocks..."). The container/package validation errors (lines 244, 304) follow a consistent "Fix the following issues:\n\n...\n\nEnsure: 1... 2... 3..." checklist format that is easy to scan.

Improvement Opportunities 💡

High Priority

Opportunity 1: Inconsistent expression-size error guidance

• File: pkg/workflow/runtime_validation.go
• Current State: Lines 84–91 build two different error strings for the same failure (a YAML line exceeding 21KB). When a key can be extracted (line 86), the message includes "Consider chunking the content or using artifacts instead." When no key is found (line 89), that same actionable suggestion is dropped entirely.
• Issue: Users hitting the unkeyed branch get a bare size-limit statement with no remediation path, while users hitting the keyed branch get concrete guidance. This is a predictability/trust gap — the same underlying problem should always produce a helpful hint.
• User Impact: A CI failure on this validation with no remediation text forces the user to search documentation or source code to learn what to do next, adding friction during a compile-time failure that already blocks their pipeline.
• Suggested Change: Append the same remediation clause ("Consider chunking the content or using artifacts instead.") to the key == "" branch so both messages give equivalent actionable guidance.
• Design Principle: Trust and Reliability / Consistent experience across touchpoints.

Medium Priority

Opportunity 2: Thin "Passing Correlation IDs" section

• File: docs/src/content/docs/patterns/orchestration.md
• Current State: The "Passing Correlation IDs" section (around line 360) is a single sentence: "If your workers need shared context, pass an explicit input such as tracker_id (string) and include it in worker outputs (e.g., writing it into a Project custom field)." No YAML/frontmatter example is given, unlike every other section in this document (which all include fenced code examples).
• Issue: This section breaks the established pattern of "explain, then show a code example" used throughout the rest of the document, making it feel incomplete relative to its neighbors.
• User Impact: Enterprise users adopting the orchestrator/worker pattern for multi-repo rollouts need a concrete example of how tracker_id flows from orchestrator to worker; without it they must infer the syntax from unrelated sections.
• Suggested Change: Add a short dispatch-workflow/call-workflow YAML snippet showing tracker_id passed as an input, consistent with the format used in the "Dispatch Workers" and "Call Workers" sections above it.
• Design Principle: Documentation Quality / Practical examples that reflect real use cases.

Files Reviewed

Documentation

• docs/src/content/docs/guides/open-telemetry.mdx - Rating: ✅
• docs/src/content/docs/patterns/orchestration.md - Rating: ⚠️

Workflow Messages

• .github/workflows/smoke-checkout-pr-dispatch.md - Rating: ✅
• .github/workflows/outcome-collector.md - Rating: ✅

Validation Code

• pkg/workflow/runtime_validation.go - Rating: ⚠️

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 3
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Add actionable remediation to unkeyed expression-size error

File to Modify: pkg/workflow/runtime_validation.go

Current Experience

In validateExpressionSizes (around lines 84–91), when a YAML line exceeds the 21KB expression size limit and a key can be extracted, the error includes a remediation suggestion. When no key can be extracted, the fallback error message omits that suggestion entirely:

var errorMsg string
if key != "" {
    errorMsg = fmt.Sprintf("expression value for %q (%s) exceeds maximum allowed size (%s) at line %d. GitHub Actions has a 21KB limit for expression values including environment variables. Consider chunking the content or using artifacts instead.",
        key, actualSize, maxSizeFormatted, lineNum+1)
} else {
    errorMsg = fmt.Sprintf("line %d (%s) exceeds maximum allowed expression size (%s). GitHub Actions has a 21KB limit for expression values.",
        lineNum+1, actualSize, maxSizeFormatted)
}

Quality Issue

Design Principle: Trust and Reliability — error messages should be consistently actionable regardless of which code path produces them.

Users who trigger the unkeyed branch receive a dead-end error with no suggested fix, while users on the keyed branch get concrete guidance for the identical underlying problem (an oversized expression value).

Proposed Improvement

Add the same remediation clause to the else branch so both messages end with actionable guidance.

Before:

errorMsg = fmt.Sprintf("line %d (%s) exceeds maximum allowed expression size (%s). GitHub Actions has a 21KB limit for expression values.",
    lineNum+1, actualSize, maxSizeFormatted)

After:

errorMsg = fmt.Sprintf("line %d (%s) exceeds maximum allowed expression size (%s). GitHub Actions has a 21KB limit for expression values. Consider chunking the content or using artifacts instead.",
    lineNum+1, actualSize, maxSizeFormatted)

Why This Matters

• User Impact: Every user hitting this compile-time validation error gets a clear next step instead of only sometimes.
• Quality Factor: Trust and Reliability — consistent, actionable error messages.
• Frequency: This validation runs on every gh aw compile, so any workflow with an oversized inline expression will hit one of these two paths.

Success Criteria

• Changes made to pkg/workflow/runtime_validation.go only
• Both the keyed and unkeyed error branches include the same remediation suggestion
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/runtime_validation.go
• No changes to other files required
• Can be completed independently

---

Task 2: Add a concrete example to the "Passing Correlation IDs" section

File to Modify: docs/src/content/docs/patterns/orchestration.md

Current Experience

The "Passing Correlation IDs" section (near line 360) is a single sentence with no code example, unlike the "Dispatch Workers" and "Call Workers" sections directly above it, which both include fenced YAML examples:

## Passing Correlation IDs

If your workers need shared context, pass an explicit input such as `tracker_id` (string) and include it in worker outputs (e.g., writing it into a Project custom field).

Quality Issue

Design Principle: Documentation Quality — practical examples that reflect real use cases; consistent visual hierarchy across sections.

Proposed Improvement

Add a short YAML snippet showing tracker_id passed via dispatch-workflow, matching the style of the preceding sections.

Before:

## Passing Correlation IDs

If your workers need shared context, pass an explicit input such as `tracker_id` (string) and include it in worker outputs (e.g., writing it into a Project custom field).

After:

## Passing Correlation IDs

If your workers need shared context, pass an explicit input such as `tracker_id` (string) and include it in worker outputs (e.g., writing it into a Project custom field).

\`\`\`yaml
safe-outputs:
  dispatch-workflow:
    workflows: [repo-triage-worker]
    inputs:
      tracker_id: "rollout-2026-08"
\`\`\`

The worker reads `tracker_id` from its `workflow_dispatch` inputs and includes it in its own outputs so the orchestrator (or a Project board) can correlate runs.

Why This Matters

• User Impact: Removes ambiguity for teams implementing correlation tracking across orchestrator/worker runs.
• Quality Factor: Documentation Quality — practical, consistent examples.
• Frequency: Every team adopting OrchestratorOps for multi-repo or phased rollouts will need correlation IDs.

Success Criteria

• Changes made to docs/src/content/docs/patterns/orchestration.md only
• "Passing Correlation IDs" section includes a fenced YAML example consistent with sibling sections
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/patterns/orchestration.md
• No changes to other files required
• Can be completed independently

📊 User experience analysis by Delight · auto · 32.1 AIC · ⌖ 3.29 AIC · ⊞ 12K · ◷

• expires on Aug 15, 2026, 7:06 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #52735.