Repository navigation
Spec: a version is verified against its dependents before it is promoted - #1048
Merged
Merged
Conversation
Specifies the Candidate Release Protocol. A module builds itself, publishes the build as a CANDIDATE, and every dependent builds against that candidate; only a clean closure promotes it to a release. A broken closure publishes a preview carrying the full failure list. Grounded in the outage this came from. Every existing gate was green while the break reached all three portals: - dotnet build -c Release -warnaserror : node source is <None> content - the test suite : nothing compiles in-mesh source - "Compile every NodeType (vs core)" : compiles against a PINNED core digest - node-repo CI triggers : fire on plugin change, never on core The pin is a correct decision — a moving :latest makes two runs of identical code disagree. The defect is that nothing rebuilds a dependent when the thing it is pinned to moves. Failure semantics are breadth-complete, depth-stopped: every sibling is attempted so one break cannot hide another, but a dependent of something that produced no assembly is reported as blocked rather than given invented diagnostics. The incident is the proof — Post, Profile and PostsHub all carried the same call, and two were masked as UpstreamFailed behind the first. Also covers cycles (compiled as one unit; Store/Coupon-Order-Plugin is a real one), concurrent requests (release trains), cross-partition reporting (complete gate, filtered report), cost, and the two enforcement points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds architecture documentation that defines a “Candidate Release Protocol” where a version is only promoted after every dependent is rebuilt against the candidate and the full dependent-closure compiles cleanly, preventing in-mesh compile failures from reaching production unnoticed.
Changes:
- Adds a new Architecture doc page specifying the Candidate Release Protocol (states, failure semantics, cycles, concurrent trains, and enforcement points).
- Adds a corresponding “What’s New” entry announcing the protocol documentation and summarizing the intent.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-09-a-new-version-is-checked-against-what-depends-on-it.md | Adds a What’s New entry summarizing the dependents-verified promotion concept. |
| src/MeshWeaver.Documentation/Data/Architecture/CandidateReleaseProtocol.md | Introduces the detailed spec for candidate → dependent rebuild → promote/preview semantics and enforcement points. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Test Results (shard 0)701 tests 701 ✅ 13m 48s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Contributor
Test Results (shard 4)1 284 tests 1 279 ✅ 6m 19s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Contributor
Test Results (shard 5)1 756 tests 1 564 ✅ 5m 14s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Contributor
Test Results (shard 1)1 198 tests 1 198 ✅ 6m 33s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Contributor
Test Results (shard 3) 11 files 11 suites 6m 1s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Contributor
Test Results (shard 2)1 967 tests 1 965 ✅ 7m 7s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Contributor
Test Results 65 files 65 suites 45m 3s ⏱️ Results for commit b1545f7. ♻️ This comment has been updated with latest results. |
Releases are fast; deploys are gated. Core no longer waits on its dependents:
no cross-repo fan-out, no dispatch credentials, no held release train.
The reason is not cost, it is correctness — NO central set of dependents is the
right set. A NodeType broken on a plugin nobody deployed is not an outage; one
broken on a plugin a single portal installed is an outage for exactly that
portal. Only the instance knows its own set.
So the gate moves to the deploy boundary: before a portal ADOPTS a candidate
image, it verifies that every plugin it has installed still compiles against that
candidate, and refuses to roll if any regresses.
Most of this already exists. DynamicTypePreWarmer captures a WasHealthy baseline
and refuses readiness on regression, saying "the rollout will stall with the
previous image still serving". Two gaps make that untrue in practice, and they
are what implementation has to close:
- the check runs INSIDE the new pod, after adoption — on a single-replica
deployment there is no previous pod still serving, so the portal is simply
down; replicas/maxUnavailable/maxSurge are part of this gate
- verification happens after adoption rather than before it; the natural home
is the self-update poll — verify the candidate, THEN adopt
The node repos' own pinned digest stays: a plugin PR must not go red because core
moved underneath it. The instance gate is what makes an unsafe pin bump harmless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
Corrects this spec's own account of why the pre-warmer did not protect prod. It is not that the check runs too late, and it is not single-replica. The health check that consumes the regression state is registered only when PreWarm:GateReadiness is true, and that is FALSE on all three portals. The sweep runs, records every regression into NodeTypeBakeGateState, and nothing reads it. The gate STATE is registered unconditionally, so the pod still logs REFUSING READINESS — ... The rollout will stall with the previous image still serving while going Ready and taking traffic. That line is why the outage looked inexplicable from the logs: it claims a protection that is not wired up. It was switched off on 2026-08-03 (563019e) because the first gated roll stalled on seven FALSE regressions — cross-silo SubscribeRequest timeouts, zero compiler diagnostics. 974016b fixed exactly that the next day ("a timeout is not a regression"). The config was never turned back on and the OFF rationale in values.aks.yaml still argues from the pre-fix code. Rollout shape, measured live: memex-cloud and atioz are maxSurge:1/maxUnavailable:0 — surge-first, so a 1-replica portal IS protected once the check exists, and the surge pod itself IS the pre-adoption verification. But memex runs maxUnavailable:1 at replicas:1, where k8s may delete the only serving pod before the replacement is ready. Arming the gate there without fixing the strategy first would create false confidence. Also records: progressDeadlineSeconds unset (600s default against a 3h startup budget), and that a blocked upgrade is invisible to the admin tab. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
Sharpens the gate to what is actually required: check EACH INSTANCE, for THAT
instance's exact combination of candidate image x installed module VERSIONS, that
every module builds and every test is green — before anything ships to prod.
Two things earlier drafts of this page got subtly wrong:
- It is a COMBO, not a repo. A node repo's CI verifies that repo at HEAD against
a PINNED image; neither factor matches production, which runs a specific image
and a specific cross-repo set of module versions no single repo can see.
PackageManifest already pins what is needed per instance (Id, Version,
ModuleVersion, Source/SourceFolder, Requires), so the combo is fully
identifiable from the instance's own install records.
- It is build AND TESTS, not compilation. A NodeType that compiles can still be
broken — a signature survives while behaviour changes.
This needs no new harness. mw-plugin-test already boots a fresh in-process mesh,
installs each package, waits for every NodeType to reach a terminal compile
status, renders each default area, and EXECUTES each type's Tests area, where a
red test fails the run. What is missing is only the combo assembler: point that
same executable at an instance's set — each module materialised at its recorded
ModuleVersion (the repos already tag per module) — running inside the candidate
image.
Also demotes the surge pod from "the pre-adoption verification" to the LAST LINE.
It has the right scope but it is compile-only and it discovers the problem by
failing in production instead of before shipping. It catches what the combo check
missed; it is not the check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
… into docs/candidate-release-protocol
…ownstream
Measured on memex-cloud, not inferred:
- The module SET is knowable: 40+ top-level nodeType:Store/Plugin nodes.
- The module VERSIONS are not. Plugins/* — the partition PackageInstaller
writes Package records into (InstalledPartition="Plugins",
PackageNodeType="Package") — holds only _Policy. No install records exist,
so nothing pins ModuleVersion per module, and the plugin root node's
PluginContent carries no version either.
Cause: those modules arrived via GitSync / repo import, not PackageInstaller.
Only the installer path writes a PackageManifest record; the sync path does not.
So "which version of each module is this portal running" currently has no answer
on the portal.
This reorders the work. Writing the combo assembler first would yield a gate that
verifies THE LATEST OF EACH REPO — not the instance's combo — which is precisely
the false confidence this design exists to remove. First every path that lands a
module on an instance must record what it landed (module id + ModuleVersion +
source ref), in one place, installer and sync alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
The previous commit claimed an instance cannot state its combo. Wrong, and worth
correcting before anything is built on it.
Modules reach an instance two ways and each records its coordinate differently:
GitSync / repo import -> {Space}/_GitSync (nodeType:GitHubSyncConfig)
repositoryUrl + branch + subdirectory + lastSyncCommitSha
PackageInstaller -> Plugins/{id} (nodeType:Package)
PackageManifest.ModuleVersion
Verified live on memex-cloud: SocialMedia/_GitSync carries
repositoryUrl=.../MeshWeaver.SocialMedia, branch=main, subdirectory=SocialMedia,
lastSyncCommitSha=d19534d6 — a complete, exact combo coordinate.
Why the mistake was easy: Plugins/* holds only _Policy on that portal, and "zero
install records" reads as "no version recorded". But ModuleDiscoveryService
already documents the true state — "memex carries 37 sync configs and zero
install records". The information was never missing, only in the other shape.
So the first task is a READER, not a writer: one query returning an instance's
full combo, folding both shapes into one list. A reader that handles only Package
records would report almost nothing on a real portal and look like a healthy empty
set — the same false-confidence failure in a new place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Specifies the Candidate Release Protocol — the durable fix for the outage that took all three portals down.
The gap it closes
A module builds itself, publishes the build as a candidate, and every dependent builds against that candidate. Only a clean closure promotes it to a release; a broken closure publishes a preview carrying the full failure list.
Every gate that exists today was green while the break reached prod:
dotnet build -c Release -warnaserror<None>contentCompile every NodeType (vs core)The pin is a correct decision — a moving
:latestmakes two runs of identical code disagree, and the workflow comment records exactly that. The defect is that nothing rebuilds a dependent when the thing it is pinned to moves.Failure semantics
Breadth-complete, depth-stopped. Every sibling is attempted so one break cannot hide another; a dependent of something that produced no assembly is reported as
Blockednaming its blocker, rather than given invented diagnostics.The incident proves why this matters:
SocialMedia/Post,ProfileandPostsHubeach carried the identical.AddTracking()call, but onlyPostwas ever reported — the other two were masked asUpstreamFailed: blocked by SocialMedia/Post. Two of three bugs were invisible.Also specified
Cycles compiled as one atomic unit (
Store/Coupon → Store/Order → Store/Pluginis a real cycle today); release trains for concurrent version requests; cross-partition walks that stay complete while the report is filtered to what the caller may read; closure-only rebuilds with level parallelism and an explicit log for anything bounded.Scope
Documentation only — this is the spec, to be reviewed before implementation. It states plainly what the gate does not do: it verifies dependents compile, not that they still behave correctly.
Verified:
DocumentationLinkIntegrityTest+WhatsNewEntryIntegrityTest2/2 passed.🤖 Generated with Claude Code
https://claude.ai/code/session_01CPQKMhQDpitUzsW1MqTVhH