CD: main 4aa76d3 has an incomplete image set #3037

Activity

1. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   🔁 CD reconcile attempt 1/3 for 4aa76d3 (run).

   main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 4aa76d3. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.
2. github-actions

   added

   ci-failureA main-branch CD run failed - no deployable image was publishedA main-branch CD run failed - no deployable image was published

   on Sep 2, 2026
3. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   CD failed on main: run 33577773075 for commit 429add6a752ac2c81e4b1a871c2ece5fa8d66b7d (entered via push).

   Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

   • an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
   • preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

   The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.
4. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   ✅ Healed 4aa76d3 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7516), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.
5. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   CD failed on main: run 33580429032 for commit 4aa76d3c2a145e3e1d4cbfce98c68c2fda3900e6 (entered via reconcile).

   Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

   • an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
   • preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

   The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.
6. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   CD failed on main: run 33580362222 for commit 4aa76d3c2a145e3e1d4cbfce98c68c2fda3900e6 (entered via push).

   Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

   • an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
   • preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

   The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.
7. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   🔁 CD reconcile attempt 1/3 for 02e9007 (run).

   main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 02e9007. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.
8. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   CD failed on main: run 33583373324 for commit 470cdd58e81ed5bbf0b82b3684aa8a5c50e842c6 (entered via push).

   Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

   • an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
   • preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

   The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.
9. rbuergi commented on Sep 2, 2026

   

   rbuergi

   on Sep 2, 2026

   Contributor

   More actions

   Closing per this collector's own criterion — CD has published past 4aa76d3:

   $ az acr repository show-tags --name meshweaver --repository memex-portal-ai --orderby time_desc
   3.0.0-rc9.ci.7525
   main
   470cdd5-p782fbe9
   470cdd5
   staging-470cdd5-33583373324
   

   470cdd5 (the #3040 merge, 04:24) promoted a full set, so no install is stuck on a missing image set. The image half of CD is healthy.

   🚨 Two things this collector's framing hides, both now diagnosed:

   1. The bake was failing on every publishing run — not the image jobs. Root-caused and fixed in #3041 (closes #3022): -p:Version=, added 2026-09-01 15:49 so the portal image reports its own build (#2555), is a global MSBuild property. It reached every referenced project and moved AssemblyInformationalVersion — one of exactly three attributes FrameworkBuildIdentity pins — so the portal image's identity forked from the bake host's. The gate refused correctly every time: bundles published under an address the shipped portal never asks for are inert (#1814's shape), and pods recompile the whole shipped mesh at boot.

   2. The portals are not self-updating at all, which no image-set collector can see. Measured just now:

   $ curl -s https://memex.meshweaver.cloud/api/version
   {"version":"3.0.0-rc9+0a1eabdc…","commit":"0a1eabdc028a6f6cf09b1f9927b1dbaf1dbb30c4"}
   
   0a1eabdc = 2026-09-01 05:41   — 226 commits behind main
   

   Both portals serve it. And the reason is a chicken-and-egg worth recording: the self-update fix #2935 (fcfcca231, "establishing the policy is not the same as re-establishing it") landed at 07:19 on 09-01 — 1h38m AFTER the build they are running. So they carry the defect that fix addresses: the first Admin/UpdatePolicy read faults, the retry is paced at the 6 h RetryInterval rather than 30 s, and the install is inert, not slow — the safety net cannot bound it because it sits behind the same gate. The fix for the stuck self-update is in a build the stuck self-update will not take.

   The recovery order matters and is not obvious. Rolling now would fix the self-update but land on a build whose bake is inert, so every pod recompiles the whole mesh at boot — the #2194 wave that took two silos to 20 GB. Correct sequence:

   1. let CD publish a build containing -p:Version= no longer forks the framework build identity (#3022) #3041, so the bake identity matches and bundles are adoptable;
   2. verify that build's framework-identity dirs exist in that instance's storage (/data/prebuilt-bundles/_releases/<version>);
   3. then roll the portals to it — which fixes the self-update in the same move.

   Closing this collector; the live work is the roll, and it is a maintainer decision rather than an automatic one.
10. rbuergi

    closed this as completedon Sep 2, 2026