Add retry-with-backoff to AWF/threat-detect binary installers — one CDN blip failed 10+ PR-gate runs across 7 workflows
Problem: Between 21:33 and 21:58 UTC on 2026-08-12, every workflow that installs the AWF firewall binary (evals job) or the threat-detect binary (detection job) failed the same way:
curl: (56) Connection died, tried 5 times before giving up
##[error]Process completed with exit code 56.
For the AWF case this cascades into awf: command not found (exit 127), taking down the whole evals job. For the threat-detect case the job tolerates the missing binary (GH_AW_DETECTION_CONTINUE_ON_ERROR) but the step — and therefore the job — is still marked failure.
Affected workflows / runs (all within the same 25-minute window):
- Issue Monster — §31644531765 (AWF binary,
evals job)
- Matt Pocock Skills Reviewer — §31643436414, 31643152027, 31642850136 (threat-detect,
detection job)
- Impeccable Skills Reviewer — 31643310019, 31642800861
- PR Code Quality Reviewer — 31643310024
- Test Quality Sentinel — 31643185714 (+3 more in the same window)
- Ponytail Reviewer — 31643310020
Probable root cause: curl's built-in retry (--retry 5 or similar) exhausted in well under a second with no exponential backoff, so a few-second GitHub Releases CDN connectivity blip was enough to hard-fail the download instead of riding it out. No installer failures occurred before 21:33 or after 21:58 UTC — the blip was transient and self-resolved, but the installer has zero resilience to it.
Proposed remediation:
- In the install scripts for the AWF binary and the threat-detect binary (used by the
evals/detection job setup), replace the flat curl --retry 5 with exponential backoff (e.g. curl --retry 5 --retry-delay 2 --retry-max-time 60 or a manual backoff loop).
- Consider a secondary source (e.g. a mirrored/cached copy in the toolcache, matching the pattern already used for the Copilot CLI toolcache lookup in the same job) so a Releases CDN outage doesn't hard-fail every PR-gate workflow simultaneously.
- For the
detection job specifically: since the workflow already tolerates a missing threat-detect binary via GH_AW_DETECTION_CONTINUE_ON_ERROR, make the install step itself continue-on-error: true so a transient download failure doesn't mark the job (and the PR check) as failed when the workflow logic already treats it as non-fatal.
Success criteria:
- A simulated/injected transient connection failure (e.g. one dropped connection) no longer fails the install step — it retries with backoff and succeeds.
- The
detection job's overall conclusion is not failure when the threat-detect binary is unavailable and detection is configured to continue-on-error.
- No recurrence of correlated multi-workflow installer failures in the next Failure Investigator scan.
References:
Generated by 🔍 [aw] Failure Investigator (6h) · agent · 145.7 AIC · ⌖ 42 AIC · ⊞ 5.3K · ◷
Add retry-with-backoff to AWF/threat-detect binary installers — one CDN blip failed 10+ PR-gate runs across 7 workflows
Problem: Between 21:33 and 21:58 UTC on 2026-08-12, every workflow that installs the AWF firewall binary (
evalsjob) or the threat-detect binary (detectionjob) failed the same way:For the AWF case this cascades into
awf: command not found(exit 127), taking down the wholeevalsjob. For the threat-detect case the job tolerates the missing binary (GH_AW_DETECTION_CONTINUE_ON_ERROR) but the step — and therefore the job — is still markedfailure.Affected workflows / runs (all within the same 25-minute window):
evalsjob)detectionjob)Probable root cause:
curl's built-in retry (--retry 5or similar) exhausted in well under a second with no exponential backoff, so a few-second GitHub Releases CDN connectivity blip was enough to hard-fail the download instead of riding it out. No installer failures occurred before 21:33 or after 21:58 UTC — the blip was transient and self-resolved, but the installer has zero resilience to it.Proposed remediation:
evals/detectionjob setup), replace the flatcurl --retry 5with exponential backoff (e.g.curl --retry 5 --retry-delay 2 --retry-max-time 60or a manual backoff loop).detectionjob specifically: since the workflow already tolerates a missing threat-detect binary viaGH_AW_DETECTION_CONTINUE_ON_ERROR, make the install step itselfcontinue-on-error: trueso a transient download failure doesn't mark the job (and the PR check) as failed when the workflow logic already treats it as non-fatal.Success criteria:
detectionjob's overall conclusion is notfailurewhen the threat-detect binary is unavailable and detection is configured to continue-on-error.References:
Related to [aw-failures] [aw] Failure Investigator Report — 2026-08-13 (6h, updated 12:38 UTC) #52395