Repository navigation
CD failed on main: run 34190841613 #3680
Description
CD failed on main: run 34190841613 for commit 4fedf454975e013ef48eac6bd03a98f2027f92ed (entered via reconcile).
Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:
- an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
- preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.
The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. This issue CLOSES ITSELF when a CD run reports a successful heal (#3176) - a later failure files a fresh issue rather than appending here, so this one's title and age stay true to its own cause.
Activity
rbuergi commentedon Sep 8, 2026
Triage: the red job is
Plugins: bake + seal … / Bake + publish NodeType assemblies— the set is PROMOTED but NOT SEALEDImage jobs,
PromoteandVerify every image shippedwere green, so3.0.0-ci.8078exists in ACR (and the moving4fedf45short-sha tag now points at it) while no install can adopt it. Compile was clean (89/89 NodeType(s), 36 bundle(s), framework=s190dc0cae34d1602ddc92197d44461d4); the gate against the seeded mesh failed:Behind it, the gate mesh had two waves of stream updates whose owner stamped the patch but whose verdict never reached the cache hub (
… → PATCH_MERGE_STAMPED v=4 refused=0 → PATCH_ECHO_SEEN ⇒ a handler was entered and no reply, completion or fault has been recorded since):4fedf4549, SEALED)[UpdateQueue] FAILED … OwnerUnreachable … within 31s_Policy,_Access/*,_Activity/compile-stateat 06:24:42Z)OwnerUnreachablementions[PluginGating] reconcile failed … HubDisposedBeforeResponseException[PluginGating] Chess: reconcile is NOT CONVERGING — rewrote Chess/AlienGambit/_Access/…Same framework identity both runs. The only input that differs is the Plugins half (
2e1e430→7c640de: fourmanifest.lockregenerations, the i18n mirror37ab1b21, #1468, #1450 — nothing under Chess). So this is not a content change: the same race is present in the PASSING run at lower intensity, and in this run it crossed the 31 s update budget on the one node the idempotence check re-installs. A re-run may well seal, and would prove nothing about the race — the gate mesh loses patch verdicts under load whilePluginGatingdisposes plugin hubs mid-reconcile, and that is the defect to find.Discriminator in flight: run 8079 (34191519949, core
67cbbe0ee, the SAME Plugins7c640de) is in its bake since 06:08Z. Logs kept: both bake jobs (101955088724 failed, 101941978797 control).rbuergi commentedon Sep 8, 2026
Mechanism (measured in both bake logs, confirmed against the code)
Every one of the 15 lost verdicts sits inside a root RECYCLE. Both waves open on the same line from
PackageInstaller:and the control run's single loss has the identical shape (
05:00:41.521 recycling root Hosting→05:00:49.9 ADVANCE_WITHOUT_HANDOFF×7 underHosting/→05:01:15.9 FAILED Hosting/InstanceRequest/_Activity/compile-state).All 15 request trails end at
PATCH_ECHO_SEEN(+1…+45 ms) with noPATCH_FLUSH_SUBSCRIBED,PATCH_FLUSH_BOUND_ARMED/FIRED,PATCH_ACK,PATCH_FLUSH_FAULTED_SYNCornack=OwnerDisposing— and neither log carries a[LateWatch] VERDICT_EXPIREDor a "Patch verdict … reached no route" line. So the verdict was never minted, not lost in transit: the owner per-node hub stamped and echoed the patch, then its post-commit flush'sSubscribenever returned. That is the window the code itself names —DataExtensions.cs(at8b974323d, ~1560: "A trail that ends at PATCH_ECHO_SEEN without this stage names a flush whose Subscribe never returned (MeshWeaver#2543, arm 1)").StoragePostCommitFlush.Build→storage.WriteAndPublishUpdated→RoutingProxyAdapter.Write(.Take(1), no bound) routes the write to the PARTITION hub, whose prologue (_router.AddressFor,PersistenceService.TryWriteFrom's sequentialDefer) runs on the subscribing thread — the owner's echo thread — before the 10 s flush bound is armed. While the root is mid-recycle that write has no live target, so the prologue parks, the bound never arms, and the caller can only time out at 31 s. An owner disposal is excluded by the trail:RegisterOwnerDisposingNackwould have stampedPATCH_ACK nack=OwnerDisposing.Denominator. ~120 root recycles per gate run (two per root). Writes that missed the 5 s handoff (
ADVANCE_WITHOUT_HANDOFF) — this run: Chess 11, Hosting 6, Edu 5, RolePlay 4, Cornerstone 2, RemoteControl 2 (30); control run 8076: Hosting 7, Edu 5, Essentials 5, Chess 4, RolePlay 4, Cornerstone 3 (28). So the recycle strands in-flight writes under the recycled root in EVERY gate run at the same rate; whether that reds the gate depends only on whether one of the stranded writes is the idempotence re-install's own (Chess/_Policyhere, because the Chess recycle at 06:24:07 landed while Chess was being re-installed). A re-run rolls that dice again.Where to fix (root cause, not the bound):
PackageInstaller's root recycle must not dispose/re-activate a partition hub while owners under it are mid-flush — either drain the partition's in-flightPatchDataRequests before the recycle (the[QUIESCE-TIMEOUT] … forcibly cancellingat 2 s is the recycle refusing to wait), or make the flush's routed write observe the hub's disposal and fault (PATCH_FLUSH_FAULTED_SYNC) instead of parking in a prologue that runs before any bound exists. Widening 5 s / 10 s / 31 s changes nothing: the write has no target.rbuergi commentedon Sep 8, 2026
This is the seventh occurrence of #3510 (a package root is disposed while work under it is in flight) and a fourth arm of it — the UPDATE leg's post-commit flush parking under the root recycle;
4fedf4549already carries the three fixes landed for the other arms (#3512, #3550, #3603). The family and the fix are tracked on #3510; the measurement stays here. This issue closes itself on the next successful heal, and 8079 (same Plugins half) is in its bake now.github-actions commentedon Sep 8, 2026
🔁 CD reconcile attempt 1/3 for
67cbbe0(run).main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for
67cbbe0. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher-ci.<n>for it and roll every install backwards.rbuergi commentedon Sep 8, 2026
Triage: neither of the two — the SEAL broke. Images shipped, the event went out, nothing could adopt them.
Not the budget cap. The failing job executed 37 steps, not 0. September org spend is ≈ $2,482 net of $3,000 (Actions Linux 2,116 + storage 14 + Copilot 352), so the cap was not reached. Checked first, as the issue asks.
Which job went red
Assert the inputs exist(preflight)Resolve the target commit and decide whether to publishPromote: tag the full set (all-or-nothing)Verify every image shippedNotify platform-update registryPlugins: bake + seal … / Bake + publish NodeType assemblies to portal storagePlugins: bake + seal … / Register the publication with memexCD delivered, or had a good reason not toSo both of the issue's branches read green: a complete image set was published, and the release event did reach the control instance. What failed is the third thing — the seal — and
Register the publication with memexnever ran.What actually shipped vs what did not
Measured in ACR just now:
Shipped: the complete, promoted, multi-arch set
3.0.0-ci.8078(portal + migration + tester) for4fedf45 × plugins 7c640de. The "a FAILED main-cd run still promotes" trap is confirmed by measurement — the run's conclusion says nothing about what got tagged.Did not ship: the seal.
3.0.0-ci.8078existed and was unadoptable — every self-updating install holds such a build withheldReason: no sealed content bake …, andrelease.ymlwould refuse a tag on that commit.Already recovered, and I re-read it to be sure it is not a mid-transition reading. The next commit's run — 34191519949,
67cbbe0ee,3.0.0-ci.8079— hasPromote,Verify every image shipped,bake + sealandRegister the publication with memexallsuccess.mainnow points at 8079. The exposure was roughly 35 minutes in which the newest set could not be adopted, not a lost delivery.Why the bake failed
The bake was fine:
compile: 89/89 NodeType(s) compiled, 36 bundle(s). The--seedgate failed on package-install idempotence:Two components own a partition's
_Policyand its children's_Accessdenies and each undoes the other: core's pre-#902 access heal (reads the manifest) andPluginGate.SeedGating(MeshWeaver.Plugins, in-mesh source, reads the root node'sPluginContent). For a free non-pre-installed package they state opposite rules. The re-install retired 26 denies, the reconcile loggedNOT CONVERGING — rewrote 26 …, and the heal's ownChess/_Policywrite starved for 20 s and faulted.The control settles it: run 34186868747, 65 minutes earlier on the same commit, sealed — while doing the same thing at a smaller amplitude,
retiring 9instead ofretiring 26. Identical content, different count: the steady state was decided by a race, not by either rule.The heal's guard rested on "current code cannot produce that … only a pre-#902 installer could" — a survey of this repository's code, true of this repository's code, and blind to a producer that lives in the mesh.
Fix: #3681 — the heal is restricted to pre-installed partitions (the case #902 was about, and the one case the two components agree on). Analysis conserved as
Doc/Architecture/PartitionAccessOwnership.Bonus: this run answers #3510's open question
#3603's "the recycle names itself" line landed 2026-09-07 and this is the next occurrence. The disposer is
PackageInstaller.SettleRetypedRoot— the installer's own retyped-root recycle — which falsifies #3499's stated reading that it was "not the installer's retyped-root recycle". Detail on #3510. That race is real but is not what reddened this run; fixing it alone would not have saved the seal.github-actions commentedon Sep 8, 2026
✅ Healed
67cbbe0(run) — main's HEAD now has the complete image set (portal3.0.0-ci.8080), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494).Closing this issue, because the delivery hole it records is filled. A later failure files a FRESH
ci-failureissue rather than appending here (#3176), so an alert's title names its own cause and its age is the outage's age. Reopening is never needed — and closing also hands the reconciler a fresh heal budget.rbuergi commentedon Sep 8, 2026
Closed by #3681, merge commit
70874ac15, 2026-09-08T07:10:03Z. Verified onmainby content, not ancestry (a queue merge makesmerge-base --is-ancestora false negative): the guard line is present inPackageInstaller.csonorigin/mainandDoc/Architecture/PartitionAccessOwnershipis there.Still open, deliberately — two follow-ups this did NOT cover:
Systemorph/MeshWeaver.Plugins#1506.
DeclaredAccessInstallTest.LegacyScopedGate_OnAFullyPublicPackage_IsHealedpins the premise fix(#3680): the installer stops healing a partition the gating reconcile owns #3681 removes. That project builds against the pinnedMW_PLATFORM_REF(a57a6b52b…), so nothing is red today — it goes red the moment the pin moves past70874ac15, and the fix must ride that same pull request. This is cross-repo break shape 7 (behaviour behind an unchanged signature); no gate sees it. The issue carries the exact diff.The root-recycle race — A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510, now being fixed in fix(#3510): arm the patch flush bound before the flush is built, so a root recycle cannot strand the verdict #3683. Separate defect. This run's log answers A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510's open question: fix(#3510): the upsert's CREATE leg answers instead of hanging forever #3603's "the recycle names itself" instrumentation landed 2026-09-07 and its first occurrence names the disposer as
PackageInstaller.SettleRetypedRoot, which falsifies A package root is disposed mid-install under a reconcile — Chess, then Hosting, then Chess again; main-cd cannot seal #3499's stated reading that it was "not the installer's retyped-root recycle". Fixing it alone would not have saved this seal — the heal-vs-gating ping-pong is independent of it — which is why it was not bundled here.One correction to my own earlier comment, so the record is clean: I cited
Chess/_Policyat version 210,801 (2026-08-25) as evidence of this mechanism. It is not —PluginGate's own comments attribute that storm to the reconcile writing off blind subtree snapshots, since fixed by the targeted-readVerifiedWritechange. It belongs in the file only as evidence that these nodes are a contended surface with more than one way to loop. Corrected in the doc page and the PR body before merge.