Skip to content

Every satellite's publish-bake fails at preflight since #3760: MW_REGISTRY_PUBLISHER_PASSWORD is required and provisioned nowhere — nothing has baked since 2026-09-12T18:11Z #4142

Closed
Closed
Every satellite's publish-bake fails at preflight since #3760: MW_REGISTRY_PUBLISHER_PASSWORD is required and provisioned nowhere — nothing has baked since 2026-09-12T18:11Z#4142

Description

@rbuergi
rbuergi
opened on Sep 13, 2026

Every satellite's publish-bake is red at the same preflight step, and nothing has baked since 2026-09-12T18:11Z

Measured 2026-09-13T07:08Z, GitHub REST, counted by job, every CI run of the five satellites since 2026-09-12T12:00Z accounted for (Crm 32, Education 67, SocialMedia 32, Manufacturing 33, Reinsurance 8 CI runs).

The step and the line

Job publish-bake / Bake + publish NodeType assemblies to portal storage, step Preflight: the publish credentials must be provisioned, executed output (not the script echo):

##[error]The bake publication cannot reach any portal — required credentials are not provisioned. This job fails RED rather than skipping: a grey skip is indistinguishable from a pass and would silently leave every portal re-compiling this repo's content at boot.
##[error]missing: bundle-registry-publisher-password (secret, the caller's MW_REGISTRY_PUBLISHER_PASSWORD) — the fleet registry's publisher password (Key Vault 'Systemorph' → memexcloud-Registry-PublisherPassword). Provision it in BOTH stores of the calling repository: gh secret set MW_REGISTRY_PUBLISHER_PASSWORD --repo Systemorph/MeshWeaver.Reinsurance (Actions) AND gh secret set MW_REGISTRY_PUBLISHER_PASSWORD --app dependabot --repo Systemorph/MeshWeaver.Reinsurance (Dependabot), then pass it as bundle-registry-publisher-password on the uses: job

The denominator

publish-bake / Bake + publish … jobs across the five satellites, by start time relative to the #3760 merge (2026-09-12T18:11:47Z):

success failure
started 12:00Z → 18:11:47Z 22 2 (both the upstream gate, #3583's shape)
started after 18:11:47Z 4 (all four started 18:17Z, on lane refs resolved before the merge) 66 — every one at the preflight step above

Per satellite, preflight-red publish-bake jobs after the merge: Crm 18 · Manufacturing 18 · SocialMedia 17 · Education 11 · Reinsurance 2. Every publish-bake job started at or after 18:20:27Z has failed there — the first three were the meshweaver-upstream-published wakes of 18:20:27–28Z (Crm 34710888429, SocialMedia 34710889177, Manufacturing 34710889122), nine minutes after the merge.

The 2026-09-13 daily full runs — the run the fleet now relies on for validation and for the only satellite bake of the day — failed on all five, same step:

repo schedule run preflight job
MeshWeaver.SocialMedia 34735529110 03:27Z 103668498544
MeshWeaver.Manufacturing 34735877298 03:35Z 103669026500
MeshWeaver.Education 34736225609 03:44Z 103671855132
MeshWeaver.Reinsurance 34736500097 03:51Z 103671123258
MeshWeaver.Crm 34736758746 03:58Z 103670946105

So no satellite publication has been sealed for any framework identity since 18:11Z on 2026-09-12 — ~13 hours at the time of writing — and every portal that rolls onto a set from that window finds no crm/education/socialmedia/reinsurance/manufacturing bundle and compiles that content locally (#3583's condition, now total rather than partial).

The cause

#3760 (merged 2026-09-12T18:11:47Z, dcabce799) wired the fleet-registry push into node-repo-publish-bake.yml and gave the lane a new secret input, bundle-registry-publisher-password (.github/workflows/node-repo-publish-bake.yml:503-511), asserted by the lane's preflight (:628). The input is declared required: false on purpose — so that a caller which omits it fails in the preflight, RED, naming the secret and both stores, rather than as a workflow startup failure that names nothing. That is the right shape (Doc/Architecture/DependabotSecretStore, and the memory rule "a new REQUIRED input on a reusable workflow is a silent startup failure" — this one was made loud, correctly).

Every satellite calls the lane at @main, so all five picked the change up on their next run. None of the five provisions MW_REGISTRY_PUBLISHER_PASSWORD, and none passes bundle-registry-publisher-password: on its uses: job (checked on main of Reinsurance and Education: zero occurrences of either string in .github/workflows/ci.yml; the other three fail the same line). Core's own plugins-bake caller does pass it (main-cd.yml, bundle-registry-publisher-password: ${{ secrets.MW_REGISTRY_PUBLISHER_PASSWORD }}), which is why core's Plugins publication kept sealing (20 of 20 register jobs succeeded in the same window) while every satellite went dark.

What provisioning needs — per satellite, two acts, and NOT done here

  1. The secret, in BOTH stores of each of the five repositories: gh secret set MW_REGISTRY_PUBLISHER_PASSWORD --repo Systemorph/<repo> (Actions) and gh secret set MW_REGISTRY_PUBLISHER_PASSWORD --app dependabot --repo Systemorph/<repo> (Dependabot — a Dependabot-actor run resolves secrets.X against the second store, and there is no way to see that store from CI). The value is the fleet registry's publisher account password, authoritative source Key Vault Systemorph → memexcloud-Registry-PublisherPassword. Prove it against the live registry before writing it anywhere (a login as publisher to cr.meshweaver.cloud through registry-login.sh or oras login) — if that secret is a write-only slot rather than a KV-readable value (the ACR-token / mwi_ shape, where "mirror it" is impossible for everybody), the satellites need their own separately issued, separately revocable publisher credential, not a rotation of core's.
  2. The uses: job passes it: bundle-registry-publisher-password: ${{ secrets.MW_REGISTRY_PUBLISHER_PASSWORD }} under secrets: of the publish-bake call in each satellite's ci.yml — one PR per repository. check-pr-secret-preflight.py (via node-repo-validate) will want the lane's preflight to be the asserting preflight, which it already is.

Nothing was provisioned by this investigation: read-only, no secret written, no registry touched.

What behaved correctly, and what the gap is

The preflight did exactly what the no-skip-trapdoor rule asks: it went RED naming the secret and both stores, on the first run that could see the new input. The gap is that a red daily run on five repositories woke nobody for 13 hours. The main is red: file it, tell the control portal job ran beside every one of these reds, and its own step Did the inbox VERIFY the event? failed in at least Reinsurance 34699679792 / 34694999741 and Manufacturing (2 jobs) — so the red-main ledger route is itself not delivering. That is a second defect, worth its own reading; it is why this outage is being filed by a triage sweep rather than by the lane that exists to file it.

Refs

#3760 (the lane change), #3583 (the condition this makes total), Doc/Architecture/DependabotSecretStore, MeshWeaver.Plugins Hosting/BuildAndReleaseProcess (the daily run is the full run — and it is red).

🤖 Filed from Claude Code.

https://claude.ai/code/session_01BRLdhAiX5DjdZC2WUfPg9E

Activity

  1. rbuergi commented on Sep 13, 2026

    @rbuergi
    rbuergi
    on Sep 13, 2026
    ContributorAuthor

    Correction to "the gap is that a red daily run on five repositories woke nobody": the lane DID report, 63 of 63 times — the gap is the audience

    Counted over every satellite run since 2026-09-12T12:00Z (83 job lists). For the publish-bake jobs that failed at the preflight from 18:20Z on, the main is red: file it, tell the control portal job succeeded, and its Did the inbox VERIFY the event? step succeeded, in 63 of 63 report jobs (Crm 15, Education 11, Manufacturing 18, SocialMedia 17, Reinsurance 2). Executed lines from Crm job 103685250121:

    issue #95 open: https://github.com/Systemorph/MeshWeaver.Crm/issues/95
{"event":"ci-failure","repo":"Systemorph/MeshWeaver.Crm",…,"failedJobs":[{"name":"publish-bake / Bake + publish NodeType assemblies to portal storage",…,"step":"Preflight: the publish credentials must be provisioned"}],"platformSet":"3.0.0-ci.8457","issueUrl":"https://github.com/Systemorph/MeshWeaver.Crm/issues/95","issueNumber":95}
POST https://memex.systemorph.com/api/hooks/Hosting/PlatformBuilds -> 200
{"status":"accepted","signature":"verified"}

    Every satellite holds an OPEN ci-main-red: main is red ledger issue naming this step: Crm#95 (19:49Z), Education#324 (14:58Z), SocialMedia#182 (17:21Z), Manufacturing#84 (15:06Z), Reinsurance#201 (13:45Z) — the earlier ones opened on the pre-#3760 reds (the upstream gate, #3583's shape) and were updated, not duplicated. The four VERIFY failures visible in the window were all before 15:05Z on 2026-09-12 (Reinsurance 34694999741 / 34699679792, Manufacturing 34699422712 / 34699312600): the inbox then answered "signature":"not-required" — no SecretConfigKey on the control instance's target (#3312, #3338) — and by 18:20Z it answered verified, so that was fixed on the receiving instance during the afternoon.

    So the sentence above is withdrawn as written. The signal existed in three places for 13 hours — five ledger issues, 63 verified ci-failure events on the control instance, and the red runs themselves. What is missing is a reader: the ledger issues have no subscriber, and on the control instance the Hosting/Issue nodes those events become are, on 3 of 10 /health samples of memex.systemorph.com taken 07:19Z today, in content-types: Degraded (Hosting/Issue ×24, Hosting/LogEntry ×20784, Hosting/InstanceAction ×252, Hosting/Deployment ×8 — the Hosting module is not loaded on that replica, so those pages render empty; a viewer landing there sees nothing). Which of those two is the defect to fix is a scope call; neither is this issue's, and neither is the lane's.

  2. rbuergi
    added a commit that references this issue on Sep 13, 2026

    review(#4172): the compatibility branch has a stated end — warn durin…

    85ad180

  3. rbuergi commented on Sep 13, 2026

    @rbuergi
    rbuergi
    on Sep 13, 2026
    ContributorAuthor

    Closed 2026-09-13 20:15Z — the credential is provisioned on all five satellites and the bake runs green where it has been exercised

    Measured through the REST API, not inherited:

    repo MW_REGISTRY_PUBLISHER_PASSWORD in the repo's Actions secrets newest main run with a publish-bake job publish-bake
    MeshWeaver.Crm present 34768169172, 16:18Z success
    MeshWeaver.Manufacturing present 34768463741, 16:24Z success
    Education present (#325 merged 16:29Z hands it to the lane) 34773957223, 18:13Z skipped — the change set owed no bake
    MeshWeaver.Reinsurance present 34770922648, 17:13Z skipped — the change set owed no bake
    MeshWeaver.SocialMedia present (no main content push since) —

    So the title's condition — fails at preflight because the password is required and absent — no longer holds anywhere, and "nothing has baked since 2026-09-12T18:11Z" is false since 16:18Z today. Education, Reinsurance and SocialMedia will exercise the same preflight on their next content push; if any of them reds at that step again, reopen this with the run id rather than filing anew.

    Not covered here and still open on their own threads: #3583 (bundles baked once per framework identity) and #3461 (two writers on one prefix).

  4. rbuergi
    closed this as completedon Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions