A container leg that pushed nothing reported success by rbuergi · Pull Request #1031 · Systemorph/MeshWeaver

rbuergi · 2026-08-09T16:50:24Z

CD has since gone green on its own (31324251944, all five images for 86578a4), so this is an
intermittent failure — which is the worse kind, because when it fires it silently skips image
publication. This PR does not declare it transient and walk away: it establishes the exact mechanism,
rules out the repo-side hypotheses with evidence, and makes both the build failure and the missing
image impossible to mistake for anything else.

The exact mechanism

Timeline from the bake job log (linux-x64 had already pushed at 16:01:00.9):

time	event
16:01:03.9	`Memex.NodeType.Bake -> …/linux-arm64/publish/`
—	30.2 s of complete silence; `Building image …-linux-arm64` is never logged
16:01:34.1	`warning : The operation was canceled.`
16:01:34.2	`error : Cannot create image index …`

Every step of the chain is in the SDK:

CreateNewImage fetches the base image manifest (mcr.microsoft.com/dotnet/aspnet:10.0)
before it logs Building image …. That message is absent, so the 30 s was spent in that fetch,
not in the index.
Microsoft.NET.Build.Containers builds its registry client as
new SocketsHttpHandler { UseCookies = false, ConnectTimeout = TimeSpan.FromSeconds(30) }
(DefaultRegistryAPI). A stalled connect surfaces as
TaskCanceledException("The operation was canceled.") with an inner
TimeoutException("A connection could not be established within the configured ConnectTimeout.")
— dotnet/runtime #47484,
#78070. 30.2 s is that timeout, to the tenth.
CreateNewImage.Execute() catches it, calls Log.LogWarningFromException — which logs only
ex.Message, dropping the inner TimeoutException that names the host and the timeout — and
returns !Log.HasLoggedErrors, i.e. success, having produced no image.
_PublishSingleContainer emits its GeneratedContainer item unconditionally, every metadata
value empty.
CreateImageIndex.ParseImages rejects the empty metadata → the misleading error.

So the index never sees a race: it sees an item a leg was allowed to publish while failing. The
answer to "why intermittently" is simply "only when a registry connect stalls ≥ 30 s".

Hypotheses checked, with evidence

A multi-arch race (the ProduceReferenceAssembly prior art, PR #552 / CS0009). Ruled out, two ways:

That guard is active for Memex.NodeType.Bake — it lives in the root Directory.Build.props
under '$(ContainerRuntimeIdentifiers)' != '', not per-csproj, so it reaches the whole closure.
Queried directly for a library in that closure:

dotnet msbuild src/MeshWeaver.Graph -getProperty:ProduceReferenceAssembly -getProperty:BaseOutputPath \
  -p:ContainerRuntimeIdentifiers="linux-x64;linux-arm64" -p:ContainerRuntimeIdentifier=linux-arm64
→ { "ProduceReferenceAssembly": "false", "BaseOutputPath": "bin/container/linux-arm64/" }

Both the ref-assembly gate and the per-leg output isolation apply.

The signature is wrong anyway. That race breaks the compile (CS0009 / MSB3883). Here both legs
compiled and wrote their publish directories — …/linux-arm64/publish/ is right there in the log.
The failure is strictly after the build, in a registry call.

The #979 job split changing concurrency. Ruled out — it moved in the safe direction. GitHub gives
every job its own fresh VM, so after the split nodetype-bake shares no runner, no NuGet cache and no
output path with any other image job; before the split it was a second step in the migration job,
i.e. strictly more sharing. Within the job the two RID legs run in parallel either way (the SDK's
BuildInParallel default) — #979 did not touch that. And the job had been green in every run since it
was split at 00:08 that day, including 11 minutes before the failure.

Frequency. 1 failure in 27 image-building CD runs (last 100 completed runs: 26 success, 73 skipped,
1 failure). The runs immediately before (cb25f07) and after (d2db7dc, a superset tree containing
4f0c35cfc) both published memex-bake. On the same commit, the migration job did the identical
multi-arch publish against the identical base image and pushed both legs 12 s apart.

Conclusion: an upstream stall, amplified by an SDK defect that reports it as success. There is no
repo-side change that prevents a connect from stalling — and per the ground rules this PR adds no retry,
drops no architecture, disables no index and sets no continue-on-error, because none of those would
have produced an image.

What this PR changes — two detection layers

1. A leg that produced no image no longer reports success (root Directory.Build.props,
error MW1026). It fails where the failure actually happened, naming the RID and both registries and
pointing at the warning above it, instead of letting an empty item travel two tasks downstream to be
blamed on the image index.

It matters most on the single-RID path, where there is no CreateImageIndex at all: today a
swallowed cancellation there means dotnet publish exits 0 having pushed nothing.

2. verify-images — assert the image, never the tick (main-cd.yml). Runs if: always() after
every leg and asserts all five images carry this commit's short-SHA tag, and that the four .NET ones are
linux amd64+arm64 indexes rather than a surviving single arch. Every repository is reported, so one
miss cannot hide the next, and it is in alert-on-failure's needs, so a missing image opens the
ci-failure issue even when every publish leg reported success.

This closes the gap the issue actually left open. The alerting did fire for #1026 — but "CD failed"
never said which image was missing, and nothing was checking that a main commit ends up with the
full set:

memex-portal-ai   3.0.0-rc1.ci.2470, 4f0c35c   ← shipped
memex-bake        (neither tag)                ← missing

ci.2470 is a portal increment the NodeType bake gate never certified, and the self-updater picks the
newest version tag regardless.

Verification

verify-images logic run against the real registry, both directions: on 4f0c35c it names
memex-bake:4f0c35c is MISSING from ACR and exits 1; on 86578a4 it passes clean — reproducing
CD failed on main: run 31322066110 #1026 from ACR state alone.
MW1026 guard exercised against the real SDK targets: healthy legs → runs once per RID after
_PublishSingleContainer, sees all four Generated* properties populated, stays silent; forced to
fire → error MW1026, exit 1, CreateImageIndex never runs.
Full multi-arch Memex.NodeType.Bake publish with CD's flags (-c Release --no-self-contained -t:PublishContainer -p:ContainerRuntimeIdentifiers="linux-x64;linux-arm64" -p:CIRun=true) completes
green with the guard in place — both legs valid, index built, no MW1026.

Not decided here

Whether CD should refuse to ship a partial set (e.g. hold the portal tag until every leg lands) is a
delivery-policy call for the maintainer. verify-images makes the partial release loud and named; it
does not change what gets published.

No What's New entry

Build/CI-only change with no user-visible behaviour.

CD run 31322066110 failed the nodetype-bake job with error : Cannot create image index because provided images are invalid. Items must have 'Config', 'Manifest', 'ManifestMediaType' and 'ManifestDigest' metadata. which reads as a defect in the image index or in Memex.NodeType.Bake. It is neither. The linux-arm64 leg sat 30.2 s (16:01:03.9 -> 16:01:34.1) between its publish directory and a bare warning "The operation was canceled.", and that warning is the whole story: * Microsoft.NET.Build.Containers builds its registry client as `new SocketsHttpHandler { ConnectTimeout = TimeSpan.FromSeconds(30) }`, so a stalled TCP/TLS connect to the base registry (mcr.microsoft.com, fetched before "Building image" is ever logged) surfaces as TaskCanceledException. * CreateNewImage.Execute catches TaskCanceledException / OperationCanceledException, calls Log.LogWarningFromException — which drops the inner TimeoutException naming ConnectTimeout — and returns `!Log.HasLoggedErrors`, i.e. SUCCESS, having produced no image. * _PublishSingleContainer then emits its GeneratedContainer item unconditionally, with every metadata value empty, and the failure lands ~100 ms later in CreateImageIndex blaming the index. So a network stall is reported as an image-index defect, with neither host nor timeout anywhere in the log. Nothing in the bake project or in the job split (#979) is involved: the same tree published memex-bake fine in the run before and the run after, 26 of 27 image-building CD runs are green, and the identical publish in the migration job on that same commit succeeded. Refuse the success that produced no artifact. A guard in the root Directory.Build.props fails the leg where it actually failed (MW1026), naming the RID and both registries, so the next occurrence is one line to attribute instead of an hour of log archaeology. It matters most on the SINGLE-RID path, where there is no CreateImageIndex downstream at all and a swallowed cancellation means a publish that reports success while pushing nothing. Verified against the SDK targets: the guard runs once per RID leg after _PublishSingleContainer, sees the four Generated* properties populated on a healthy leg (inert), and when it does fire it fails the publish with exit 1 before CreateImageIndex runs. The stall itself is upstream infrastructure and is deliberately NOT papered over here: no retry around the publish, no dropped architecture, no continue-on-error. The workflow comment records the other consequence, which is a policy question rather than a bug — each image job pushes independently, so 4f0c35c shipped memex-portal-ai:3.0.0-rc1.ci.2470 with no matching memex-bake:ci.2470, a portal increment the NodeType bake gate never certified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…dowed targets file Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…772191f2e3e

Copilot

Pull request overview

This PR hardens the CI/CD container publishing pipeline by detecting when a per-RID container publish “succeeds” while producing no image artifacts (due to an SDK-swallowed cancellation), and by documenting the correct triage interpretation in the CD workflow.

Changes:

Add an MSBuild guard after _PublishSingleContainer to fail fast (MW1026) when the generated container manifest/config/digest metadata is missing.
Document in main-cd.yml that the downstream “invalid images”/image-index error is typically a symptom of an earlier cancelled/stalled registry connect, not an index defect.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
Directory.Build.props	Adds an MSBuild target that fails the build immediately when a container publish leg produced no image metadata, preventing misleading downstream failures.
.github/workflows/main-cd.yml	Adds workflow comments capturing the triage narrative so future incidents are diagnosed from the correct log signal (the 30s gap + cancellation warning).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-09T17:04:18Z