Fix the Cloud Hypervisor guest-network race that firewall v0.28.2 was supposed to close — it isn't closed. 11 of 16 failed runs in this window (69%) hit the identical fatal probe error, all after the fix release.
Fix the retry/readiness window in github/gh-aw-firewall v0.28.2 — the identical fault is recurring post-fix
Why: #53935 was closed 2026-08-20T21:15Z as fixed by gh-aw-firewall PRs #7555 and #7568, shipped in Release v0.28.2 (2026-08-20T17:57Z). Runs in this 6h window carry firewall_version: v0.28.2 and still fail with the exact same signature 3–7 hours after that release — the fix did not close the race, it only narrowed it.
Evidence
Confirmed via agenticworkflows audit on two representative runs, both on firewall v0.28.2:
[WARN] [cloud-hypervisor] stage=guest-connectivity status=failed: Cloud Hypervisor guest connectivity probe failed with exit code 4
[ERROR] Fatal error: Error: Cloud Hypervisor guest connectivity probe failed with exit code 4 (stderr: ...guest network state: 1: lo: <LOOPBACK,UP,LOWER_UP> ... eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> ... state DOWN ...)
Process exiting with code: 1
##[error]Agent execution exited with code 1
- §32431743603 — Ponytail Reviewer, 0 agent turns, immediate crash pre-engine-start.
- §32433611789 — Daily Go Test Parallelizer, same signature but after 2 successful API calls, meaning the race window can open mid-run too, not just at cold start.
Affected workflows and runs (11 of 16 failures this window)
Ponytail Reviewer — 9 consecutive failed runs, 21:04–00:12 UTC:
§32417537184, §32425901278, §32425988993, §32426039730, §32428025804, §32428076665, §32428127021, §32431448388, §32431743603
Daily Go Test Parallelizer — 2 failed runs, 22:36 & 00:41 UTC:
§32425052464, §32433611789
Not part of this cluster (5 singleton failures, likely unrelated — no repeat signature this window)
Each occurred once; not investigated further under this issue's P0 scope. Re-open if they recur.
Root cause
gh-aw-firewall PR #7568 brings loopback up and adds bounded retries before the guest-connectivity probe, but the retry bound is still too tight for the actual guest-network bring-up latency under load — eth0 is still state DOWN when the probe runs. This is a timing regression, not a reintroduction of the original bug.
Proposed remediation
- File a follow-up in
github/gh-aw-firewall against v0.28.2 referencing this issue and the two run IDs above with raw probe logs attached.
- Widen the retry/backoff window for
stage=guest-connectivity before declaring Fatal error, or add a distinct guest-network-not-ready retryable state instead of hard-failing on the first probe timeout.
- Add regression coverage that runs the probe under load (concurrent VM boot), since the current fix passed CI but not fleet conditions.
Success criteria
- Zero
Cloud Hypervisor guest connectivity probe failed occurrences across the next 6h Failure Investigator window.
audit-diff between a pre-fix and post-next-fix run shows turns no longer dropping to 0 on affected workflows.
References:
Generated by 🔍 [aw] Failure Investigator (6h) · agent · 107.8 AIC · ⌖ 7.62 AIC · ⊞ 5.9K · ◷
Fix the Cloud Hypervisor guest-network race that firewall v0.28.2 was supposed to close — it isn't closed. 11 of 16 failed runs in this window (69%) hit the identical fatal probe error, all after the fix release.
Fix the retry/readiness window in
github/gh-aw-firewallv0.28.2 — the identical fault is recurring post-fixWhy: #53935 was closed 2026-08-20T21:15Z as fixed by
gh-aw-firewallPRs #7555 and #7568, shipped in Release v0.28.2 (2026-08-20T17:57Z). Runs in this 6h window carryfirewall_version: v0.28.2and still fail with the exact same signature 3–7 hours after that release — the fix did not close the race, it only narrowed it.Evidence
Confirmed via
agenticworkflows auditon two representative runs, both on firewall v0.28.2:Affected workflows and runs (11 of 16 failures this window)
Ponytail Reviewer — 9 consecutive failed runs, 21:04–00:12 UTC:
§32417537184, §32425901278, §32425988993, §32426039730, §32428025804, §32428076665, §32428127021, §32431448388, §32431743603
Daily Go Test Parallelizer — 2 failed runs, 22:36 & 00:41 UTC:
§32425052464, §32433611789
Not part of this cluster (5 singleton failures, likely unrelated — no repeat signature this window)
Each occurred once; not investigated further under this issue's P0 scope. Re-open if they recur.
Root cause
gh-aw-firewallPR #7568 brings loopback up and adds bounded retries before the guest-connectivity probe, but the retry bound is still too tight for the actual guest-network bring-up latency under load —eth0is stillstate DOWNwhen the probe runs. This is a timing regression, not a reintroduction of the original bug.Proposed remediation
github/gh-aw-firewallagainst v0.28.2 referencing this issue and the two run IDs above with raw probe logs attached.stage=guest-connectivitybefore declaringFatal error, or add a distinctguest-network-not-readyretryable state instead of hard-failing on the first probe timeout.Success criteria
Cloud Hypervisor guest connectivity probe failedoccurrences across the next 6h Failure Investigator window.audit-diffbetween a pre-fix and post-next-fix run showsturnsno longer dropping to 0 on affected workflows.References:
Related to [aw-failures] [aw] Failure Investigator Report — 2026-08-20 18:41 UTC (6h) #54326