Skip to content

[aw-failures] [aw-fix] P0: Find and stop the workflow_dispatch storm hitting Approach Validator + Daily Syntax Error Quality Check #48813

Description

@github-actions

Stop whatever is bulk-dispatching these two workflows — that's what blew both daily AI-credit caps and cascaded into 57 straight failures.

Fixing the guardrail message alone won't help; the dispatcher is the actual bug.

Problem statement

In a single ~1.5h window (2026-07-29 ~04:35–06:05 UTC), Daily Syntax Error Quality Check (.github/workflows/daily-syntax-error-quality.lock.yml) and Approach Validator (.github/workflows/approach-validator.lock.yml) were each invoked roughly 29 times via workflow_dispatch. Neither workflow's triggers explain that volume:

  • daily-syntax-error-quality.md: schedule: daily + workflow_dispatch — should fire once/day from cron, not ~29x.
  • approach-validator.md: label_command (approach-proposal, needs-design labels) + slash_command — normally event-driven per issue/PR, not a tight burst.

Once each workflow's daily AI-credit budget was exhausted by the burst (GH_AW_MAX_DAILY_AI_CREDITS: 5000 for daily-syntax-error-quality, 10000 for approach-validator), every subsequent run that day failed at the Check daily workflow token guardrail step before ever calling the model — 57 of 60 unique failed runs in the 6h window.

Affected workflows and run IDs

  • Daily Syntax Error Quality Check — representative failed run §30427016013
  • Approach Validator — representative failed run §30427016566
  • Comparator/baseline (last known success): run 30341316460 (2026-07-28, Daily Syntax Error Quality Check)
  • Full list of 57 affected run IDs is in the parent issue's evidence section.

Probable root cause

One of:

  1. A bot/automation is re-dispatching these workflows repeatedly without dedup (e.g., re-applying approach-proposal/needs-design labels to many issues/PRs in a batch, or a retry loop with no backoff on failure).
  2. A manual or scripted bulk workflow_dispatch API call loop (e.g., a test or backfill script) targeting these two workflows specifically.

audit-diff on the representative run confirms the failure is a clean pre-flight abort, not a network/firewall issue: total_requests 1→0, token_usage_change -100%, firewall_diff shows 0 anomalies — so the model/API layer was never reached; only the guardrail check ran and failed.

Proposed remediation

  1. Pull triggering_actor and event payload for the 57 affected run IDs (or a representative sample) via the GitHub Actions API to identify the actual dispatcher.
  2. If it's a bot/automation: add dedup/throttling so the same workflow isn't re-dispatched for the same target within a short window, and add exponential backoff on retry.
  3. If it's a legitimate bulk operation: raise the daily AI-credit cap for that operation ahead of time, or stagger the dispatches so the guardrail doesn't cascade-fail the whole batch.
  4. Separately, consider making the guardrail failure mode softer — e.g., skip/queue instead of hard-fail with exit 1, so a capacity-exhaustion event doesn't read identically to a real workflow bug in monitoring.

Success criteria / verification

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 115.9 AIC · ⌖ 21.8 AIC · ⊞ 5.1K ·

  • expires on Aug 4, 2026, 11:53 PM UTC-08:00

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions