Repository navigation
fix(cd): a broken pipe mid-push no longer throws away the whole release (#2810) - #2828
Conversation
…se (#2810) Delivery was blocked for hours by an ACR blob upload that lost its connection, and because publication is all-or-nothing one leg took the whole release with it. Measured, run 33328213655 (target 84bf70b): 18:48:05 linux-arm64 pushed cleanly 18:52:58 [Registry] Encountered a HttpRequestException Unknown with message "Broken pipe". Pausing before retry. 18:53:07 CONTAINER1013 → CONTAINER1001: Failed to upload blob using PATCH https://meshweaver.azurecr.io/v2/memex-portal-ai/blobs/uploads/… Same pair on run 33322756606 (target 8cbf69a) at 16:49Z. The SDK's own retry did not recover either. Between them, ci.6846 promoted successfully at 18:14Z — so the fault is INTERMITTENT: not credentials, not quota, not a purge policy, and the registry is Premium SKU and healthy throughout. The portal-ai push now makes up to 3 attempts, with an `az acr login` between them (a token can expire during a multi-arch push whose legs take minutes each). 🚨 NOT a skip-trapdoor, and the distinction is the point: - a compile error, a missing project or a rejected credential fails identically on all three attempts and the step ends RED, naming the last attempt's output; - only a connection dying mid-upload gets a second chance; - three consecutive failures emit an ::error:: saying it is no longer a transient and wants investigating rather than another retry, and points at #2810. It mirrors the idiom already in node-repo-{compile-check,gate,publish-bake}.yml, which carry the same "Bounded retry, and deliberately NOT a skip-trapdoor" comment for their registry logins. Verified: - the YAML parses (yaml.safe_load); - the failure path was PROVED to exit 1 rather than assumed — the loop shape was simulated with an always-failing command and ends `::error::` + exit 1; - MeshWeaver.Documentation.Test 123/123 green, which is every guard that reads this workflow (ReleaseDeliveryChainGuard, MainRunsAreNeverCancelled- Guard, PlatformBakeLaneGuard, PlatformReleaseNotifyGuard, WorkflowJobIfContextGuard), so the step's rewrite does not break a parse. This is one of TWO blockers on delivery. The other is a CS0419 ambiguous cref in MeshWeaver.Plugins (core CD builds the portal hosts from plugins-repo), fixed by MeshWeaver.Plugins#993 — separate cause, separate commits: both ACR failures are on commits that PREDATE #2772's overload. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR improves resilience of the main continuous delivery pipeline so that intermittent Azure Container Registry connection failures (e.g., “Broken pipe” mid-blob upload) don’t discard an otherwise good multi-arch publish, while preserving the all-or-nothing promotion guarantee.
Changes:
- Adds a bounded (3-attempt) retry loop around the
memex-portal-aimulti-arch container publish inmain-cd.yml. - Adds a corresponding “What’s New” entry documenting the operational impact and rationale for the bounded retry.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-one-broken-pipe-no-longer-costs-a-whole-release.md |
New release note describing the intermittent ACR failure mode and the bounded-retry mitigation. |
.github/workflows/main-cd.yml |
Wraps the portal-ai publish in a 3-attempt retry with re-login between attempts, keeping the job red on persistent failures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| echo "attempt $attempt failed — re-authenticating and retrying in 30s" | ||
| az acr login --name meshweaver || true | ||
| sleep 30 | ||
| fi |
🚨 Correcting my own justification:
|
| run | commit | image | inner cause |
|---|---|---|---|
| 33322756606 | 8cbf69a 16:35 |
migration | Connection refused (meshweaver.azurecr.io) — no CONTAINER1001 at all |
| 33328213655 | 84bf70b 18:53 |
portal-ai | CONTAINER1001 … PATCH … 'RequestedRangeNotSatisfiable' (HTTP 416) |
CONTAINER1013 is the SDK's outer "failed to push" wrapper; it wraps different inner causes. I compared wrappers. That is the same same-symptom-⇒-same-cause shortcut this repo has paid for three times today — and the countermeasure took thirty seconds: open the error string per run.
What that means for this PR — it is narrower than I claimed
Still exactly right for the connection-refused class. Nothing was uploaded; the connection never opened. A bounded retry with a re-login is the correct answer, and it is the same fault that put the retry idiom into node-repo-*.yml in the first place.
Unproven for the 416, and I will not claim otherwise. The upload session's _state token decodes to {"Offset":0,"StartedAt":"…18:47:50Z"} and the PATCH was rejected at 18:53:07 — 5m17s in, still at offset 0. A 416 on blobs/uploads/ means the registry's offset no longer agrees with the client's. So:
- this retry does give it a fresh upload session each attempt, which the SDK's in-place resume does not — that is a real improvement over dying on an invalidated session;
- but if the mechanism is a single long-lived upload of a very large layer outliving the registry's session window, then attempts 2 and 3 take the same five minutes and hit the same wall, and this costs 3× the CI time to fail anyway.
The step's final message already says the right thing for that case — three consecutive failures emit "no longer a transient … wants investigating rather than another retry" — so the PR degrades honestly. But the body oversold one cause, and I have corrected it.
Keeping it, with the scope stated
I am not withdrawing this: one of the two blockers is squarely in scope, the other gets a materially better shot than the SDK's resume, and the failure mode is loud rather than silent. What it is not is a fix for the 416 mechanism — that wants its own investigation (layer size, chunked upload, or ACR's actual session window, none of which is measured), and belongs on #2827 rather than being quietly absorbed here.
(The wrapper-vs-inner-cause finding and the _state decode are the reviewing session's; the verification and this correction are mine.)
Delivery has been blocked for hours. Across the last 20 CD runs there is not one non-skipped
Promotejob, so every self-updating install is still on the previous image. There are two independent causes; this PR fixes the one nobody owns.The measurement
Run 33328213655 (target
84bf70b17):Same pair on run 33322756606 (target
8cbf69ac9) at 16:49Z. The SDK's own retry did not recover either.And
ci.6846promoted successfully at 18:14Z — between the two failures. So this is intermittent: not credentials, not quota, not a purge policy. The registry is Premium SKU and healthy throughout.Publication is all-or-nothing by design —
promoteapplies the real tags only once every leg succeeded, so an install can never pull a half-shipped set. The cost of that guarantee is that one broken pipe on one architecture throws away the entire release.The change
Up to three attempts on the portal-ai push, with an
az acr loginbetween them (a token can expire during a multi-arch push whose legs take minutes each).🚨 Not a skip-trapdoor, and the distinction is the whole point:
::error::saying it is no longer a transient and wants investigating rather than another retry, pointing at CD failed on main: run 33322756606 #2810.It mirrors the idiom already in
node-repo-{compile-check,gate,publish-bake}.yml, which carry the same "Bounded retry, and deliberately NOT a skip-trapdoor" comment for their registry logins — added after three ACR logins failed on 2026-08-28 with the registry healthy throughout.Verified
yaml.safe_load::error::+exit 1MeshWeaver.Documentation.Test123/123, which is every guard that reads this workflow:ReleaseDeliveryChainGuard,MainRunsAreNeverCancelledGuard,PlatformBakeLaneGuard,PlatformReleaseNotifyGuard,WorkflowJobIfContextGuardThe other blocker, so they are not conflated
A
CS0419ambiguous cref in MeshWeaver.Plugins (MeshStoreAccess.cs:72and two more) — core CD builds the portal hosts fromplugins-repo, and #2772's newObserveCompletionoverload made three bare<see cref>s ambiguous there. Fixed by MeshWeaver.Plugins#993.Different cause, different commits: both ACR failures are on commits that predate the overload. If
Promoteis still skipped after both land, read which error the portal-ai job carries —CS0419means the cref fix did not take;CONTAINER1001means the connection died three times running and this is no longer transient.🤖 Generated with Claude Code