Skip to content

An image is a candidate until every node repo compiles against it - #1056

Closed
rbuergi wants to merge 4 commits into
mainfrom
feat/candidate-release-gate
Closed

rbuergi wants to merge 4 commits into
mainfrom
feat/candidate-release-gate

Conversation

@rbuergi

@rbuergi rbuergi commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Core deleted AddTracking on MessageHubConfiguration (eafd353). Three NodeTypes in MeshWeaver.SocialMedia still called it. Every gate we had was green, and production found it instead: CompileError → dependents UpstreamFailed → REFUSING READINESS → hubs never activate → every request burns the 60 s activation budget, on all three portals.

Gate Why it passed
dotnet build -c Release -warnaserror Node source is <None> content — the compiler never sees it
The full test suite Same reason; nothing compiles in-mesh source
The node repo's own Compile every NodeType (vs core) Compiles against a pinned core digest that still had the method
Its triggers (push/pull_request/workflow_dispatch) Fire when the plugin changes — never when core does

The pin is correct and does not move. A moving :latest makes two runs of identical code disagree (Plugins, 2026-08-04: main green on sha256:10462f9a, the same code red on sha256:d8895c8a an hour later). The defect is that nothing re-ran a dependent when the thing it is pinned to moved. This PR is that missing re-run.

What it does

Gate 1 of the Candidate Release Protocol, stacked directly on the staging→promote shape that #1039 just merged.

  • candidate-gate (new job) resolves the digest of the staged mw-plugin-test index — the image that is the framework, since the node repos unpack /app out of it for their compile ref set — dispatches each node repo's ci.yml with that digest as a one-run override of its committed pin, waits for all of them, and aggregates.
  • promote — the single manifest PUT that arms the self-updater — now additionally requires that closure to be clean.
  • preview (new job) is the other terminal state: same bytes tagged preview-<sha> (invisible to VersionSelect by construction — a leading letter fails ^\d+\.\d+\.\d+), every break in every repo posted to the ci-failure issue, and CD goes red. Nothing is rebuilt to recover; promotion is a manifest write.

Breadth-complete by construction: dispatch ALL, then poll ALL. Post, Profile and PostsHub carried the identical broken call and only Post was ever reported — the other two were UpstreamFailed: blocked by Post, so two of three bugs stayed invisible until the first was fixed. The same mistake at repo scale would hide whole repositories. blocked (not wired / no access / timed out) is never a pass.

Cost: candidate-gate needs only the plugin-test leg, so it runs concurrently with bake/portal/migration (~15 min) rather than after them — single-digit minutes on the CD critical path, not a whole node-repo CI.

🔧 It is INERT until a human arms it — deliberately

The gate dispatches workflows in other repositories, which the run's GITHUB_TOKEN cannot do. Until NODE_REPO_DISPATCH_TOKEN exists, candidate-gate passes the candidate through and annotates every run with a ::warning:: and a job-summary block naming the unverified repos. A loud pass-through, chosen over merging a gate that stops all delivery until someone mints a PAT.

Two things are needed, in this order (arming first would report every repo blocked and ship nothing) — both spelled out with exact YAML in Doc/Architecture/CandidateReleaseGate:

  1. Patch ci.yml in MeshWeaver.SocialMedia, MeshWeaver.Plugins, MeshWeaver.Reinsurance — three workflow_dispatch inputs (core_image_digest, core_ref, candidate_id), a run-name + concurrency group carrying the candidate id, and one step that makes the pin a default via $GITHUB_ENV. Validated by applying it to the real MeshWeaver.Plugins/ci.yml and parsing the result. (These live in other repos, so they are not in this PR.)
  2. Repository secret NODE_REPO_DISPATCH_TOKEN — a fine-grained PAT (or App installation token) scoped to exactly those three repos: Actions: read and write, Contents: read, Metadata: read. Nothing on Systemorph/MeshWeaver itself.

Found while wiring it

MeshWeaver.Plugins's compile-check job carries if: ${{ vars.MW_TEST_IMAGE != '' }}. An unset variable makes the compile gate skip — green, having compiled nothing, which is exactly how the 2026-07-21 outage shipped. MeshWeaver.Reinsurance already removed it. Called out in the doc as part of step 1.

Verification

  • DocumentationLinkIntegrityTest + WhatsNewEntryIntegrityTest green (Release), after merging current main.
  • main-cd.yml parses; job graph and every needs/if checked, including the skip paths (a gate that dies answers clean='' → neither promote nor preview runs, and alert-on-failure fires).
  • The node-repo patch applied to the real MeshWeaver.Plugins/ci.yml and parsed; the resolve step lands ahead of every consumer of MW_IMAGE_DIGEST in both jobs.

Spec: Doc/Architecture/CandidateReleaseProtocol (#1048) — link it from this page once that merges.

rbuergi and others added 3 commits August 9, 2026 23:34
Core deleted AddTracking on MessageHubConfiguration (eafd353). Three NodeTypes in
MeshWeaver.SocialMedia still called it, and every gate we had was green: -warnaserror
never sees node source (<None> content), no test compiles in-mesh source, and each node
repo's own "Compile every NodeType (vs core)" compiles against a PINNED core digest that
still had the method -- on triggers that fire when the PLUGIN changes and never when CORE
does. Production found it: CompileError -> UpstreamFailed -> REFUSING READINESS -> hubs
never activate -> every request burns the 60 s activation budget, on all three portals.

The pin is not the bug and does not move. A moving :latest makes two runs of identical
code disagree (Plugins, 2026-08-04). The bug is that nothing re-ran a dependent when the
thing it is pinned to moved.

So CD now treats its own image as a candidate. candidate-gate resolves the digest of the
staged mw-plugin-test index -- the image that IS the framework, since the node repos unpack
/app out of it -- dispatches each node repo's ci.yml with that digest as a one-run OVERRIDE
of its committed pin, waits for all of them, and aggregates. promote (the single manifest
PUT that arms the self-updater) now requires that closure to be clean; a broken one goes to
preview instead: the same bytes tagged preview-<sha>, invisible to VersionSelect by
construction, with every break in every repo named on the ci-failure issue.

Breadth-complete by construction: dispatch ALL, then poll ALL. Post, Profile and PostsHub
carried the identical broken call and only Post was ever reported, so two of three bugs
stayed invisible until the first was fixed -- the same mistake at repo scale would hide
whole repositories. blocked (not wired, no access, timed out) is never a pass.

Cost: candidate-gate needs only the plugin-test leg, so it runs concurrently with
bake/portal/migration rather than after them -- single-digit minutes on the critical path,
not a whole node-repo CI.

The gate is INERT until NODE_REPO_DISPATCH_TOKEN exists, and annotates every run with a
::warning:: saying so -- a loud pass-through, because merging a gate that stops all
delivery until a human mints a PAT would be worse than the hole it closes. The node-repo
patch and the token's least-privilege scope are in Doc/Architecture/CandidateReleaseGate;
the patch is validated against the real Plugins ci.yml.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
Re-searching for the run on every poll can hand back a DIFFERENT dispatch run mid-poll,
whose verdict we would then report as ours. Read it back by id once seen; a transient read
failure then leaves the previous snapshot in place instead of losing the run entirely.

Also record why a merge burst costs three node-repo runs per merge: neither concurrency
group may cancel a verification, because a cancelled verification is a candidate nobody
proved -- which is what the 45 min per-repo budget is absorbing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
Copilot AI lite review requested due to automatic review settings August 9, 2026 21:37
"Merged but no image" already has two documented causes; a held-back candidate is now a
third, and it is the one whose images DO exist (under preview-<sha>) while the version tag
does not. An agent reading only the all-or-nothing bullet would conclude a leg had failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds “Gate 1” of the Candidate Release Protocol to main-cd.yml, ensuring a core build is treated as a candidate until dependent node/plugin repositories have rebuilt against the exact staged framework image digest. If any dependent fails (or is not verifiably run), the build is published only as an inert preview-<sha> and CD is made red, preventing the self-updater arming tag from being written.

Changes:

  • Add a new candidate-gate job that resolves the staged mw-plugin-test image digest and dispatches/polls dependent node-repo ci.yml runs with a one-run pin override.
  • Add a preview terminal job and gate promote/verify-images on candidate-gate cleanliness.
  • Add documentation + a WhatsNew entry describing the new release safety gate and how to arm it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-09-an-update-is-checked-against-your-plugins-first.md Announces the new “candidate until dependents compile” safety guarantee.
src/MeshWeaver.Documentation/Data/Architecture/CandidateReleaseGate.md Documents the CI wiring, failure semantics, and required cross-repo setup/token.
src/MeshWeaver.Documentation/Data/Architecture.md Adds the new architecture page to the topic map.
.github/workflows/main-cd.yml Implements the candidate gate, preview path, and promotion gating in CD.
.github/scripts/candidate-gate.py New script to dispatch, correlate, poll, and summarize dependent repo CI runs.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +13 to +16
Exit code is ALWAYS 0 — this is a measurement, not the verdict. The verdict is `clean=true|false`
on $GITHUB_OUTPUT (plus a full markdown report on $GITHUB_STEP_SUMMARY and a JSON result file).
A script that exited non-zero on a broken dependent would stop the run before it could publish the
preview and name what broke, which is the one thing the protocol requires it to do.
@rbuergi

rbuergi commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by a design change from the user: core must release immediately — the release train does not wait on dependents, and the deploy gate moves to the instance.

Closing rather than reworking, because the trigger is the part that changes: an instance already knows its own installed plugin set, so it does not need core to know its dependents, and the cross-repo dispatch PAT this needed disappears with it.

What is retained on branch feat/candidate-release-gate: the promote-vs-preview framing, the preview-<sha> tag safety analysis (a leading letter is what keeps it invisible to VersionSelect.PlatformVersionTag), and the finding that MeshWeaver.Plugins's compile-check job still carries if: ${{ vars.MW_TEST_IMAGE != '' }} — an unset variable silently skips that repo's compile gate. That one is worth fixing on its own.

@rbuergi rbuergi closed this Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants