[cli-tools-test] [audit] Comparison block shows "stable" / "No action needed" for failed workflow runs

[github-actions]

Problem Description

When auditing a failed workflow run, the comparison section classifies the run as stable and recommends No action needed — even though the run clearly failed. This creates a misleading and contradictory audit report where the critical key finding says the workflow failed but the comparison summary says no action is needed.

Command/Tool

• Tool: audit
• Command: agenticworkflows audit --run_id <failed-run-id>

Steps to Reproduce

1. Find a recently failed workflow run (e.g., run 28641847210 — Contribution Check, failed 2026-07-03)
2. Run: agenticworkflows audit --run_id 28641847210 --max_tokens 2000
3. Inspect the comparison and key_findings sections of the output

Expected Behavior

When the audited run has conclusion: failure, the comparison block should:

• Classify the run as something other than stable (e.g., regression or failed)
• Provide a recommendation that reflects the failure (e.g., "Investigate failure; run concluded with errors")
• Not say "No action needed" when there is a critical error

Actual Behavior

The audit output contains contradictory information:

// comparison block says:
{
  "classification": { "label": "stable" },
  "recommendation": { "action": "No action needed; this run matches the selected successful baseline closely." }
}

// key_findings block says:
[
  { "severity": "critical", "category": "error", "title": "Workflow Failed",
    "description": "Workflow 'Contribution Check' failed with 1 error(s)" }
]

The run conclusion is failure but the comparison block completely ignores that.

Root Cause Analysis

The comparison logic compares behavioral metrics only (turns, posture, blocked_requests — all 0) without factoring in the run conclusion field. Since both the current run and the baseline have the same behavioral shape (both read-only, 0 turns), it labels the run as stable. The conclusion: failure is not checked in the comparison classification.

Environment

• Repository: github/gh-aw
• Run ID tested: 28641847210 (Contribution Check, failed)
• Baseline run ID used by audit: 28632052731 (Contribution Check, success)
• Testing Run ID: 28642379750
• Date: 2026-07-03
• gh-aw version: 1.0.68

Impact

• Severity: High
• Frequency: Always (reproducible for any failed run with a matching behavioral baseline)
• Workaround: Check key_findings section separately; ignore comparison.recommendation when overview.conclusion is failure

Diagnostics

Full audit output showing the contradiction:

overview.conclusion: failure
comparison.classification.label: stable
comparison.recommendation.action: No action needed; this run matches the selected successful baseline closely.
key_findings[0].severity: critical
key_findings[0].title: Workflow Failed

References:

• §28641847210 — failed Contribution Check run (subject)
• §28632052731 — successful baseline used by audit
• §28642379750 — this testing run

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🧪 Daily Cli Tools Tester · 90.6 AIC · ⌖ 19.9 AIC · ⊞ 9.1K · ◷

• expires on Jul 9, 2026, 10:34 PM UTC-08:00