Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/MeshWeaver.Documentation/Data/Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ The catalog below lists every page; these are the load-bearing reads per theme.
| **Plugins & content delivery** | [Plugins](Plugins) — node repos from git, no NuGet | [Plugin Manual](PluginAuthoring) — author · publish · install · own registry · [Plugin Registry](PluginRegistry) — memex re-serves plugins over REST · [Webhook Inbox](WebhookInbox) — external services deliver into {target}/_Inbox · [Plugin Packaging](PluginPackaging) — bundles, the framework identity, and the `Release` node that links a release to its assemblies per architecture · [Install Readability](InstallReadability) — the two doors an install can open, and the cover-grant deadlock detector · [Static Repo Import](StaticRepoImport) · [Node Type Compilation](NodeTypeCompilation) |
| **Reliability & wedges** | [Error Propagation & Wedges](ErrorPropagationAndWedges) — drive wedges to 0 | [Action-Block Wedge Prevention](ActionBlockWedgePrevention) · [Riding Out a ShuttingDown Address](RidingOutAShuttingDownAddress) — the one transient NACK, and the two axes a ride-out must bound separately · [Hub Initialization Failure](HubInitializationFailure) · [Orleans Stream Pub-Sub Durability](OrleansStreamPubSubDurability) — a publish with no subscriber succeeds, so a cross-silo reply can vanish with nothing logged · [Durable Streams Are Mesh Nodes](DurableStreamsViaMeshNodes) — the design that retires the memory stream without a provider · [The Pod-Hub Claim Must Be Re-Asserted](PodHubClaimReassertion) — a claim asserted once into a directory that is re-partitioned on every membership change is lost silently, and forever · [Oversized Delivery Refusal](OversizedDeliveryRefusal) — a message too large for its transport destroys the connection carrying it; refuse at the producer, never raise the limit · [An Unreachable Store Is Not a Refusal](StoreUnreachableIsNotARefusal) — one classification, three consumers; reporting an availability failure as a verdict is how a retried create becomes a duplicate · [Reading a Silo Eviction](ReadingASiloEviction) — a heartbeat newer than the suspect votes is not proof the silo was healthy; the control arm that tells a correct eviction from a false positive |
| **Testing & debugging** | [Writing Tests](WritingTests) | [Negative Controls](NegativeControls) — a pin is only a pin if it fails against the defect · [Reactive Test Assertions](ReactiveTestAssertions) · [Test State Isolation](TestStateIsolation) · [Disposable-mesh e2e](DisposableMeshE2E) · [Debugging Message Flow](DebuggingMessageFlow) · [Debugging Disposal & Leaks](DebuggingDisposalAndLeaks) · [Ambient Test-Host Hangs](AmbientTestHostHangs) — what decides whether a killed test host can be diagnosed at all, and the readings of it already falsified · [In-Mesh Tests and the Seal](InMeshTestsAndTheSeal) — a Tests area no required context executes is a latent trunk red the seal detonates fleet-wide; how to measure a gate before requiring it |
| **Deployment & ops** | [Deployment](Deployment) (the router) | [AKS](DeploymentAKS) · [Container Apps](DeploymentContainerApps) · [Local Dev Workflow](LocalDevWorkflow) · [Onboarding a New Environment](OnboardingNewEnvironment) · [Release & Self-Update Strategy](ReleaseStrategy) · [The Continuous Delivery Contract](ContinuousDeliveryContract) — all-or-nothing publication; verify the image, never the tick · [Bake Identity Mismatch](BakeIdentityMismatch) — why a green CD can publish a bake no portal adopts, and the one rule that keeps two images of one commit on one address · [Release Availability Gates](ReleaseGates) — one predicate; never roll or build into a release a package cannot survive · [Combo Gate Wiring](ComboGateWiring) — the roll consults the combo verdict; Red refuses, and "could not find out" is neither · [Release Process](ReleaseProcess) · [Repository Dependency Direction](RepositoryDependencyDirection) — the platform never depends on a plugin repo; the inventory of every edge that still does · [The Cross-Repo Pair Gate](CrossRepoPairGate) — a removal here reds a plugin repo's trunk hours later; the deleting half lands LAST · [The Merge Queue](MergeQueue) — one entry built at a time so nothing churns, and a steward that re-queues an ejected PR on evidence and never re-runs · [Carving Projects Out Of Core](CarvingProjectsOutOfCore) — what a SOURCE move costs and what it does not · [Red-Log Watching & Ticketing](LogWatchTriage) — every `fail:`/`crit:` becomes exactly one triaged issue · [Measuring a Live Portal Read-Only](MeasuringALivePortalReadOnly) — the four read-only instruments, and why an absence needs a coverage fact before it counts as evidence · [Chart Ownership and the Runner Pool](ChartOwnershipAndRunnerPool) — why the chart's gate is here, what a relocation must carry, and why path-filtering it is unsafe |
| **Deployment & ops** | [Deployment](Deployment) (the router) | [AKS](DeploymentAKS) · [Container Apps](DeploymentContainerApps) · [Local Dev Workflow](LocalDevWorkflow) · [Onboarding a New Environment](OnboardingNewEnvironment) · [Release & Self-Update Strategy](ReleaseStrategy) · [The Continuous Delivery Contract](ContinuousDeliveryContract) — all-or-nothing publication; verify the image, never the tick · [The Self-Update Schema Wall](SelfUpdateSchemaWall) — every schema-bumping release is un-takeable by self-update, the stall is invisible, and a promoted tag is not a deployable tag · [Bake Identity Mismatch](BakeIdentityMismatch) — why a green CD can publish a bake no portal adopts, and the one rule that keeps two images of one commit on one address · [Release Availability Gates](ReleaseGates) — one predicate; never roll or build into a release a package cannot survive · [Combo Gate Wiring](ComboGateWiring) — the roll consults the combo verdict; Red refuses, and "could not find out" is neither · [Release Process](ReleaseProcess) · [Repository Dependency Direction](RepositoryDependencyDirection) — the platform never depends on a plugin repo; the inventory of every edge that still does · [The Cross-Repo Pair Gate](CrossRepoPairGate) — a removal here reds a plugin repo's trunk hours later; the deleting half lands LAST · [The Merge Queue](MergeQueue) — one entry built at a time so nothing churns, and a steward that re-queues an ejected PR on evidence and never re-runs · [Carving Projects Out Of Core](CarvingProjectsOutOfCore) — what a SOURCE move costs and what it does not · [Red-Log Watching & Ticketing](LogWatchTriage) — every `fail:`/`crit:` becomes exactly one triaged issue · [Measuring a Live Portal Read-Only](MeasuringALivePortalReadOnly) — the four read-only instruments, and why an absence needs a coverage fact before it counts as evidence · [Chart Ownership and the Runner Pool](ChartOwnershipAndRunnerPool) — why the chart's gate is here, what a relocation must carry, and why path-filtering it is unsafe |
| **Contributing docs** | [Authoring Documentation](AuthoringDocumentation) | [Docs Follow The Functionality](DocsFollowTheFunctionality) — which repo a page belongs in, and what pins the rest here · [Specifying Software](SpecifyingSoftware) · [Glossary](/Doc/Glossary) |
| **Licensing** | [Dependency Licensing](DependencyLicensing) — Apache-2.0/MIT compatible only; the CI gate that enforces it | |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,17 @@ moving pointer like `latest`, untagging *destroys* it rather than reverting it.
| **C** | **`memex-portal-ai:<version>`** — one manifest PUT, the last thing the pipeline does | a single PUT either happened or did not; there is no half-armed state |

Phase C is the **arming write** and nothing else is, because `SelfUpdateHostedService` lists tags for
`memex-portal-ai` **only**, picks the newest `^\d+\.\d+\.\d+` one, and patches the portal *and*
migration Deployments to it. Everything the roll will need — the matching
`memex-portal-ai` **only**, picks the newest `^\d+\.\d+\.\d+` one, and patches the portal Deployment
to it. Everything the roll will need — the matching
`memex-migration:<version>`, the bake image that certified these node types — is already tagged by
phase A. **Do not move that step, and do not add anything after it.**

🚨 **The roll patches the portal and NOTHING else** — since #2797 there is no second PATCH, and a
guard in MeshWeaver.Plugins fails the build if one comes back. The matching `memex-migration` image
that phase A tagged is therefore *available* to the roll but never *run* by it: only `helm upgrade`
mints the migration Job. That is why a release which bumps the schema is un-takeable by self-update
at all — [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall).

One deliberate coupling: the GHCR mirror sits in **phase B**, so a GHCR outage *blocks* the release
rather than silently leaving the plugins repo's `latest` stale. `check-image-set.sh` only observes
ACR, so a GHCR miss placed after the arming write would never be reconciled and nothing would notice.
Expand Down Expand Up @@ -310,6 +316,38 @@ what sets that tail.
the rolls, so each published set restarts every install that selects it. That is the reason the tick
is hourly and not faster, and it is tracked as #1778 rather than papered over with a slower CD.

### A promoted tag is not a deployable tag — the seal is the discriminator

The rule above stops at the image. **A tag whose images all exist, and are all pullable, can still be
unusable**, and the two checks that sound like they say otherwise are green in exactly that state:

- **`Promote: tag the full set (all-or-nothing)`** green means the three platform images
(`memex-portal-ai`, `memex-migration`, `mw-plugin-test`) carry the tag.
- **`Verify every image shipped`** green means they are pullable.
- **Neither says anything about the plugin modules.** The binding job is
**`Plugins: bake + seal the publication for this identity`**, which seals the module set *for that
platform identity*. It runs after the promote — see "After the promote" below — and, as recorded
there, the reconciler does not heal it.

Measured on two consecutive runs, 2026-09-03 — `33746020109` (`ci.7669`) and `33749847612`
(`ci.7674`), both seals FATALing on the one-producer guard (#3175):

```text
Promote: tag the full set (all-or-nothing) success
Verify every image shipped success
Plugins: bake + seal the publication for this identity FAILURE
└─ Register the publication with memex skipped
```

Both tags are **platform-present, plugin-modules-absent** for their framework identity. Rolling to
one produces the half-broken state the standing availability rule forbids — *all plugins available
for the correct platform version, else nothing goes*
([Release Availability Gates](/Doc/Architecture/ReleaseGates)). And note the skipped step: a failed
seal never registers the publication with memex, so the availability predicate has no record to
read. **The number of images promoted was never the question.** The operator-facing checklist for
choosing a target is [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall) →
"What makes a tag a safe target".

## Property 1a — how the set already spans repos, and what the GUI move actually breaks

🚨 **Correction, 2026-08-26.** An earlier draft of this section claimed the shipped artefacts have
Expand Down Expand Up @@ -604,3 +642,4 @@ green lane red.
- [Release Process & Versioning](/Doc/Architecture/ReleaseProcess) — where the version number comes from.
- [Deployment](/Doc/Architecture/Deployment) — the route router (AKS vs Container Apps).
- [Deploying Plugin Changes](/Doc/Architecture/DeployingPluginChanges) — what the `mw-plugin-test` leg is for.
- [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall) — which releases an install can take by itself, and the three conditions a tag must clear before it is a safe helm target.
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ MeshWeaver has **two distinct deploy routes**. They target different infrastruct
| **Database backups** & disaster recovery — managed PITR, geo-redundancy, restore | [DatabaseBackups.md](/Doc/Architecture/DatabaseBackups) |
| Understand the release model, merge gates, version channels, and **policy-driven self-update** | [ReleaseStrategy.md](/Doc/Architecture/ReleaseStrategy) |
| Know what CD **guarantees** about a published image set — all-or-nothing publication, the promote ordering, the self-healing reconciler — and why you verify the IMAGE and never the green tick | [ContinuousDeliveryContract.md](/Doc/Architecture/ContinuousDeliveryContract) |
| Work out whether an install can actually **take** the newest release — the schema boundary self-update cannot cross, why the resulting stall is invisible, and the three conditions a tag must clear before it is a safe `helm upgrade` target | [SelfUpdateSchemaWall.md](/Doc/Architecture/SelfUpdateSchemaWall) |
| Ship a code update to the `memex` portal on the shared AKS cluster | [DeploymentAKS.md](/Doc/Architecture/DeploymentAKS) |
| Deploy an Aspire-orchestrated `test`/`prod` Container Apps environment | [DeploymentContainerApps.md](/Doc/Architecture/DeploymentContainerApps) |
| Understand the private-AKS-cluster architecture & operations behind the shared portal | [MemexCloudDeployment.md](/Doc/Architecture/MemexCloudDeployment) |
Expand Down
17 changes: 17 additions & 0 deletions src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,14 @@ construction, un-takeable by self-update. CD does build and push a correctly-tag
path ever runs it. `main-cd.yml` says as much: *"a migration that can never run surfaces only at the
deploy that needs it."* Only `helm upgrade` mints the Job.

> **This is a property of the fleet, not of one namespace, and the remedy is an OPEN decision.** An
> install rolls itself forward until it meets its first schema-bumping release and stops there; an
> install that has not stopped has not arrived yet, not been configured differently (`memex-cloud`
> served `ci.7621` healthily the same day for exactly that reason). Clearing one instance at one tag
> clears today and re-arms for the next bump. Why that matters for the control instance, what the
> three candidate remedies trade, and how to pick a target when an operator does carry an install
> across, are on [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall).

**What that looks like when it fires** (memex, 2026-09-03 — MeshWeaver#3207): self-update rolled the
portal to three successive builds needing `db_version` 55 against a database at 54. Each new pod hit
`DbVersionGate`, logged `Critical`, and exited; the ReplicaSet never went Ready and the rollout
Expand All @@ -334,6 +342,15 @@ must hold a roll, the decision belongs BEFORE the portal is patched — that is
that guard prescribes, and re-attempting a migration PATCH after the portal has already rolled is
the defect #2797 removed.

🚨 **And where a `helm upgrade` genuinely is the sanctioned path, choosing its target is a separate
problem with its own trap.** A tag whose three platform images are all promoted and all pullable can
still have no plugin modules for its framework identity — `Promote: tag the full set` and `Verify
every image shipped` both go green in that state, and the job that discriminates is `Plugins: bake +
seal the publication for this identity`. Check the seal, the ancestry of the fix you need, and a
`memex-migration` image at the SAME tag, before naming any build a target:
[The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall) → "What makes a tag a safe
target".

**Startup ordering — what actually happens if the portal outruns the schema.** Two mechanisms exist, and
only the first is a wait:

Expand Down
Loading
Loading