Skip to content

No platform pins in module repos — build against the released platform, and be TOLERANT of drift (same major serves) #3842

Closed

Description

@rbuergi

Requirement (platform owner, 2026-09-09)

We must not pin anything. It must be tolerant.

Stated after the 2026-09-09 incident on memex.systemorph.com (#3583, comments of today): a one-line CSS merge in Plugins (#1555) left every Essentials/Email page dead for hours. Part of the chain was a pin:

12:41:44Z  Plugins main, push run for 4c4019b46: FAILURE — "Repo policy gates: 2 staleness problem(s)
           in .github/workflows/ci.yml" — the hand-maintained MW_PLATFORM_SET / MW_PLATFORM_REF pin
           went stale the moment core released a new framework; every later run on main failed the
           same way until Plugins#1560 re-pinned to 3.0.0-ci.8195 at 15:30Z.

So a core release turns every module repo's main red until a human bumps a number, and while it is red the registry still serves HEAD to live portals. Meanwhile the portal itself runs a THIRD version (3.0.0-ci.8057, s414bfb2…), which no pin anywhere names — see also #3768.

What "no pins, tolerant" means concretely

  1. Module repos do not carry a platform pin. The build lane resolves the platform to build against at run time — the latest released (sealed) set, and/or the identities the fleet actually runs (Hosting__Deployment / Hosting__ReportTo know them). No MW_PLATFORM_SET, no MW_PLATFORM_REF, no uses: …@<sha> lane refs that a policy gate has to police for staleness. The staleness gate itself then has nothing to check and goes away.
  2. Compatibility is a VERSION rule, not an exact match. A module built against platform X runs on platform Y unless one of them declares the break (a MAJOR bump). The same rule for module bundles vs. module source (Bundles are baked for ONE framework identity per run, so a portal's GitSynced sources stay days behind main (or its types go unadopted) — options 1-4 undecided #3583): same major ⇒ the adopted build keeps serving, with an honest "stale, vNEXT pending" status; major bump ⇒ "incompatible, awaiting bundle" on the page. Never a dead page, never an error caused by delivery alone.
  3. A red or cancelled main is invisible to portals. The registry serves the last green set, never HEAD.
  4. Readiness is announced. When a module version is actually adopted on a portal, the portal says so (notification + Updates page). Today the only signal between "merged" and "page dead" is the dead page.

Where the pins live today

  • MeshWeaver.Plugins/.github/workflows/ci.yml: MW_PLATFORM_REF, MW_PLATFORM_SET, MW_IMAGE_DIGEST, and the uses: …@<sha> lane refs; scripts/check-platform-pins.py and the "Repo policy gates / Neither frozen platform input is stale" step that failed today.
  • Every satellite repo (Crm, Reinsurance, Education, SocialMedia, Manufacturing) fetches the platform's compile-check.py at a pinned ref and carries the same policy inputs, per the "unified build process" doctrine in their AGENTS.md — that doctrine needs the same change.

Refs: #3583 (sources ahead of bundles; fallback requirement), #3768 (identity mismatch between registry and consumer), #2813 (the fingerprint gate).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PcriF2ByxHM2Dnn2eA7k3Y

Activity

  1. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    Retention adaptation is in #3843. The bundle sweep now uses at least 30 days rather than a count quota, preserves the latest sealed publication per source for every represented major plus reported/adopted consumers, and aborts cleanup on unresolved consumer references. 41 focused tests pass; removing age protection makes the two targeted regressions fail. The support policy and retention docs now explicitly supersede the old pin doctrine.

    Integration still required with the publication/CI work here: expose the advertised last-green released-set inventory and active build consumption records to protection before publication/consumption, ordered with cleanup. Runtime consumer reports must retain a serving fallback until adoption actually changes; readiness notifications alone cannot retire it. Removing module pins must not release legacy locks or turn an empty pin scan into proof of protection. The local sealed-publication rule is not a substitute for the registry's advertised last-green set.

    No live cloud purge settings or credentials were changed. #3843 deliberately does not close #3842 or #3438; the shared inventory and cleanup cutover remain outstanding.

  2. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    Retention follow-up now has a paired consumer-inventory fix: Systemorph/MeshWeaver.Plugins#1564 preserves optional adopted framework identities and explicit completeness in the existing report receiver; core #3849 reads local NodeType adoption stamps, sends them, and protects the remote identities in retention. Core #3849 stays draft until the receiver is merged and available.

    The real-mesh regression proves a portal reports and persists older adopted X alongside running Y; 17 focused tests pass. The receiver has 22 passing tests, including a negative control that fails when identity preservation is removed. Both halves build cleanly and the core documentation suite passes all 368 tests.

    #3843 now has green PR CI after correcting the documentation index; #3846 is merged with green post-merge checks. These changes do not arm live deletion. Fleet coverage/freshness and active publisher/consumer coordination are still outstanding under #3438; this does not claim the full cleanup protocol or runtime de-pinning is finished.

  3. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    Current evidence for the readiness/announcement part of this issue: core CD 8216 (34389781884, source 25a03d2) announced a platform whose Plugins publication was still being baked. In the shipped workflow, notify-platform-update needs promote but not plugins-bake, so an image announcement can precede the complete released artifact set (workflow).

    Reinsurance dispatch 34392133001 failed its upstream gate at 19:08:57 UTC: both plugins and crm were unavailable for identity s16349850bbe0df16ecefec772e9c8c30. The actual Plugins seal for that identity was written to both targets at 19:21:15 by core job 102606052000; core CD completed at 19:21:48. Reinsurance's next attempt 34392145204 then recognized Plugins and failed only on missing CRM at 19:23:35.

    These are correctly failing availability gates, not runtime-test regressions. The evidence distinguishes an image-published notification from readiness of the full required artifact set. The solution should preserve that distinction and dependency ordering; weakening the gates or manually rerunning against absent upstreams would conceal it. No notification ordering or publication policy was changed in this investigation.

  4. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    Maintenance coordination: the assessment task is taking the bounded Plugins publish-bake dependency repair (publisher currently starts despite failed portal-hosts aggregate), with regression coverage. No runtime policy, credential, permission or production configuration changes are included. A separate earlier module-pack upload path may expose registry bytes before final validation; changing that sequencing needs its own scope assessment. Please coordinate any overlapping workflow edits here before proceeding. This does not claim the full red-main invisibility rule is implemented.

  5. rbuergi commented on Sep 10, 2026

    @rbuergi
    ContributorAuthor

    Confirmed publication-gate violation, now covered by the bounded caller repair in Systemorph/MeshWeaver.Plugins#1583.

    Plugins push run 34414173231 had a failed required host aggregate, yet publisher job 102684899327 replaced both targets for framework identity se5d2c163b0d0c1469c5e7af69806b7a8. It verified 45/45 files as its own bytes and sealed at 23:54:20 / 23:54:37 UTC on September 9, from source b078f1bafb1e621845046d539a682a3e0564a76d; publication token Systemorph-MeshWeaver.Plugins-34414173231-1, targets-published=2. This replaced the earlier green core publication. By contrast, publisher 102687384576 in run 34412991750 was an idempotent skip: its success alone did not mean it wrote anything.

    PR #1583 adds portal-hosts-gate to the sealed publisher's prerequisites and preserves implicit success(). The existing repository guard has 23 passing mutation regressions; review corrections address valid five-space YAML continuations and remove a duplicate test invocation. The repair changes neither stale-execution policy nor credentials/configuration.

    Durable evidence: BuildReleaseSeparation.

    This does not close #3842: individual module-pack jobs can hand bundles to the registry before all repository gates finish, and core main-CD publishes Plugins through its own independent caller. Those paths need their own last-green enforcement. The separate missing-sentinel read race is tracked in #3876.

  6. rbuergi commented on Sep 10, 2026

    @rbuergi
    ContributorAuthor

    Follow-up split: #3878 covers the earlier module registry hand-over inside node-repo-module-pack.yml. Packing must stay ahead of compile/runtime validation because those jobs consume its artifacts; only live registry publication moves after the full validation verdict. This is separate from Plugins #1583 and does not claim to solve publication atomicity or the independent core-CD publisher.

  7. rbuergi commented on Sep 10, 2026

    @rbuergi
    ContributorAuthor

    Reopened — the 2026-09-10 sweep was not a triage decision

    This issue was closed not_planned at 2026-09-10T14:30:29Z by a bulk sweep that PATCHed every open issue in all 13 Systemorph repositories — 311 issues in twelve minutes, no comment on any of them — after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.

    All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: Platform-owner requirement, adoption in flight — MeshWeaver.Crm#81 is still open.

    Sixteen of the 82 were left closed — their fix had actually landed, or the author had withdrawn them; those should have read completed, not not planned.

  8. rbuergi commented on Sep 13, 2026

    @rbuergi
    ContributorAuthor

    Closing — the four requirements, verified on every repository's main on 2026-09-13; the two residuals have owners

    1. No platform pins in module repos — DELIVERED

    Audited .github/workflows/ci.yml on main of all six (Plugins, Crm, Reinsurance, Education, SocialMedia, Manufacturing), uncommented lines only:

    Plugins Crm Reinsurance Education SocialMedia Manufacturing
    MW_PLATFORM_REF / MW_PLATFORM_SET / MW_IMAGE_DIGEST / MW_PORTAL_IMAGE_DIGEST literals 0 0 0 0 0 0
    uses: Systemorph/MeshWeaver/…@<sha> 0 of 4 0 of 8 0 of 8 0 of 7 0 of 9 0 of 7
    check-platform-pins.py invoked no no no no no no
    resolves the newest sealed set at run time (resolve-platform.py) yes yes yes yes yes yes

    The one remaining reference is FREEZE: ${{ vars.MW_PLATFORM_REF }} in each — the incident-freeze repository VARIABLE the resolver honours, by design (Plugins#1565). PlatformPinUpdater found nothing to move and now sits behind the same switch as the wave (Plugins#1707). The staleness gate that reddened Plugins on 2026-09-09 is gone with the pins. Crm#81 (the last adoption) merged 2026-09-12T06:25Z.

    2. Compatibility is a version rule — DELIVERED for the platform surface, and the third-party hole is closed

    Same-MAJOR serves: BuildProvenance.StaleAdopted keeps the adopted build serving with an honest status (#3844); the resolver's contract and the process page state the rule (Plugins Hosting/BuildAndReleaseProcess rule 1). The measured hole in the assumption (#4083, the YamlDotNet crash loop) is closed by #4091 (merged 2026-09-12T13:08Z): the link probe compares assembly versions, roll-forward tolerated, roll-back a hard verdict, FileLoadException has a state.

    3. A red or cancelled main is invisible to portals — DELIVERED in the shape the maintainer chose

    The sealed NodeType publication of Plugins is gated on the whole validation set (publish-bake needs: […, test-repos, tests-ratchet, modules-floor, portal-hosts-gate], Plugins#1583). Module bundles are gated on their own suite — per-module deploy, the maintainer's 2026-09-11 directive (Plugins#1676, core #4044/#4062), which reversed the whole-run hand-over of #3878 nine hours after it landed; #3878 is closed on that basis with the production measurement. Core's plugins-bake bakes Plugins main HEAD (main-cd.yml:653-678, git ls-remote refs/heads/main) and seals only if its own bake gate (every NodeType compiled, Tests areas executed in the tester) is green — validation by the publication's own unit, which is the doctrine now.

    4. Readiness is announced — PARTIAL, and owned elsewhere

    BuildDeliveryHold.EventOf/Notify announces a NodeType's adoption on a hold transition into the Admin/_Notification bell (#3844). The package-level state ("X vN adopted for identity I on this replica") is unbuilt; the scope question is recorded on Doc/Architecture/FrameworkIdentityChurn and tracked as #3583's requirement 3 — not re-opened here.

    Two residuals, recorded, not blocking

    • The resolver is carried as a per-repo copy, and the copies drift: seven copies at six distinct sizes on 2026-09-13 (core 68,150 B; Crm 69,027; Reinsurance 69,035; SocialMedia 69,035; Manufacturing 69,037; Plugins 60,365; Education 46,786), no drift guard. The lanes fetch core's copy for their own resolution, so the copies decide only what a satellite's own jobs resolve. This is the exact concern raised on Plugins#1565's first comment, and Plugins#1565 stays open narrowed to it. Recorded on Doc/Architecture/TheReleaseWave (PR below).
    • The wave moved rather than stopped (core's per-build plugins registration wakes every satellite via upstream-published, ~23 runs per satellite per day) — Roland's decision, recorded on the same page and on Bundles are baked for ONE framework identity per run, so a portal's GitSynced sources stay days behind main (or its types go unadopted) — options 1-4 undecided #3583; not a pin.

    And one caveat on what "delivered" means today: since 2026-09-12T18:11Z every satellite's publish-bake is red at a preflight on the unprovisioned MW_REGISTRY_PUBLISHER_PASSWORD (#4142) — pins have nothing to do with it, but until it is provisioned the no-pin lanes bake nothing.

    Read-only: REST contents reads of six ci.ymls, code at cdb2878bb. No secret, registry or mesh touched.

    PR: #4152

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions