fix(retention): prebuilt-bundle sweep is report-only, and its registration is now asserted (#3963) by rbuergi · Pull Request #3972 · Systemorph/MeshWeaver

rbuergi · 2026-09-10T19:42:05Z

Fixes #3963 — but not as filed. The premise is falsified, and the real defect points the other way.

The zero-callers finding does NOT hold

AddPrebuiltBundleRetention has a caller, on main, since the commit that created the service:

memex/Memex.Portal.Shared/MemexConfiguration.cs:259   (e2fb5f03b, 2026-09-08)
    builder.ConfigureServices(services => services.AddPrebuiltBundleRetention(configuration));

It sits inside ConfigureMemexMesh — unconditionally, two lines below its sibling AddModuleGenerationsGc — and both portal hosts call it (Memex.Portal.Distributed/Program.cs:392, Memex.Portal.Monolith/Program.cs:79), both referencing memex/Memex.Portal.Shared by project. MeshHostApplicationBuilder is constructed as base(x => x.Invoke(Host.Services), address), so MeshBuilder.ConfigureServices runs against the host IServiceCollection and AddHostedService<…> really does reach the generic host's roster.

The issue's measurement was grep -rn AddPrebuiltBundleRetention src/ test/. This repo has three roots that ship code — src/, test/ and memex/ — and the registration is in the third. Using AddAssemblyCacheRetention as the positive control confirmed the error rather than catching it: that one reads as registered on the same restricted search only because it happens to be duplicated per host in MeshWeaver.Plugins.

The real defect: deletion was armed, and could not be turned off

PrebuiltBundleRetention.Delete defaults to true, and PreWarm__PrebuiltBundleRetention__Delete was templated nowhere. The portal ConfigMap enumerates every key explicitly — no catch-all range over .Values.config — so a values file, a deployment overlay or a Hosting/Deployment record's extraPortalConfig setting it reached no container, silently. Same silent-drop shape as #1660 WS3 and #2203, except this omission deletes.

Measured read-only on the live control instance, 2026-09-10:

instance	`PreWarm__PrebuiltBundleRoot`	`…Retention__Delete`
`memex` (record v75)	`/data/prebuilt-bundles` (extraPortalConfig and inlineEnv)	absent everywhere ⇒ true
`memex-cloud`	`/data/prebuilt-bundles`	absent everywhere ⇒ true

deploy/aks/values.aks.yaml:276 and deploy/homebrew/share/values.local.defaults.yaml:60 mount the same root, so every AKS instance and every memex-local install was in the same state.

What this changes

This change deletes nothing. It removes strictly less than before: previously armed, now report-only.

deploy/helm/templates/memex-portal/config.yaml — bind PreWarm__PrebuiltBundleRetention__Delete with an explicit | default "false", mirroring AssemblyCache__Retention__Delete. This both disarms the fleet and makes the knob reachable for the first time.
deploy/helm/values.yaml — declare it, so the reviewable surface says so and the existing EveryPreWarmKeyInValues_IsTemplatedInTheConfigMap guard keeps the halves tied.

The sweep still runs on every boot and daily after, still plans, still writes its ledger line. Arming becomes a per-instance decision taken against a real report — the rule the sibling's own comment already states.

The controls, in both directions

1. The registration is asserted — PrebuiltBundleRetentionIsRegisteredTest (Memex.Portal.Shared.Test). Drives the real ConfigureMemexMesh and asserts an IHostedService descriptor whose implementation type is the sweep — not that the type exists, not that it resolves: registering it as a plain AddSingleton compiles, resolves and never runs. The sibling AddModuleGenerationsGc registration is asserted first as the positive control, so a harness that reached no registration at all fails naming itself instead of letting the subject's absence read as a pass.

Verified by removing line 259 and rebuilding:

FAILED: ConfigureMemexMesh must register the prebuilt-bundle retention sweep as an
IHostedService (services.AddPrebuiltBundleRetention(configuration), beside
AddModuleGenerationsGc). It is registered NOWHERE else: ...
Failed!  - Failed: 1, Passed: 0

…with the positive control still passing. Restored, Passed! - Failed: 0, Passed: 1.

2. Deletion is never armed by omission — PrebuiltBundleRetentionArmingGuard (MeshWeaver.Hosting.Test). Reads the shipped default off PrebuiltBundleRetention.Default.Delete rather than assuming it, and only then requires the chart binding, following PrebuiltBundleRetentionExtensions.DeleteConfigKey rather than a copied string. Proven non-vacuous against two mutations: binding line removed ⇒ no match; default "false" flipped to "true" ⇒ match but assertion fails.

3. The report names its denominator. TheReportNamesItsDenominator_SoAnUnexaminedStoreCannotReadAsACleanOne asserts a populated store reads 3 identity directory(ies) … collectable=0 while an empty one reads 0 identity directory(ies) … collectable=0, and that the two strings differ — so a sweep aimed at an unmounted share can never read as a clean one.

The gap this does NOT close, named

DeploymentPinnedReferences builds the protected pin set from exactly two things: Hosting/Deployment records' pinnedImageTag, and Hosting/ModuleInventory reports filed by registered instances. A satellite CI gate pinned to an older platform build is neither — an MW_PLATFORM_REF in Plugins/Reinsurance/SocialMedia/Manufacturing/Crm/Education files no report, holds no record and writes no adoption stamp into this mesh. Such an identity is protected only by rule 3 (a clean, non-pre-release _releases marker) or rule 7 (the 30-day window) — and satellite pins are X.Y.Z-ci.<n> pre-releases, so rule 3 never applies. Past 30 days, a satellite's pinned platform build is collectable. That is the #3876 / Memex#219 shape exactly, and it is why arming on the registry (memex-cloud) is the dangerous half. Written up in the doc page with what closing it needs.

Also noted rather than fixed: PrebuiltBundleRetentionStatus is written and has no reader, so "read one report" is break-glass today (a file on the data volume, or Loki).

Work product

Doc/Architecture/PrebuiltBundleRetention — registration and how to prove it, the two-way default, the report format, the eight keep rules and the gap, and the four-step rollout. Listed in Data/Architecture.md; ArchitectureTopicMapTest + DocumentationLinkIntegrityTest green.

Verification

dotnet build -c Release -warnaserror, 0 Error(s) / 0 Warning(s): MeshWeaver.Hosting.Test, Memex.Portal.Shared.Test, MeshWeaver.Documentation.Test.
Tests: Hosting.Test ~PrebuiltBundleRetention 53/53; Shared.Test registration control 1/1; Documentation.Test topic-map + link-integrity + PlatformBakeLaneGuard + TestTimeoutLiteralRatchetGuard + What's New 21/21.

No public type or member removed; no public interface member added; no i18n catalog key touched. No TimeSpan timeout literal added.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S25PJZbVeCbkWhCfMub9P7

…, and its registration is asserted #3963 reported `AddPrebuiltBundleRetention` as having zero callers — never run on any portal. Re-measured, that is FALSE, and the error is a denominator one: the issue's own sweep was `grep -rn ... src/ test/`, and the registration lives in the repo's THIRD code root, `memex/Memex.Portal.Shared/MemexConfiguration.cs:259`, inside `ConfigureMemexMesh` — the boot path BOTH portal hosts call. `MeshBuilder.ConfigureServices` runs against the HOST service collection, so the hosted service does start. The real defect is the opposite direction, and it is live. `PrebuiltBundleRetention.Delete` defaults to `true`, and the disarming key was templated NOWHERE: the portal ConfigMap enumerates keys explicitly, so a values file, an overlay or a Deployment record's extraPortalConfig setting `PreWarm__PrebuiltBundleRetention__Delete` reached no container, silently. Measured on the live control instance 2026-09-10, `memex` and `memex-cloud` both mount `PreWarm__PrebuiltBundleRoot=/data/prebuilt-bundles` and carry no Delete key anywhere; values.aks.yaml and the homebrew defaults mount the same root. Every one of them swept ARMED, against defaults nobody had read a report for. - deploy/helm: bind `PreWarm__PrebuiltBundleRetention__Delete` with `default "false"` and declare it in values.yaml — the fleet becomes report-only and the knob becomes reachable. THIS CHANGE DELETES NOTHING; it removes strictly less than before. - PrebuiltBundleRetentionIsRegisteredTest: drives the real ConfigureMemexMesh and asserts an IHostedService descriptor for the sweep, with the sibling AddModuleGenerationsGc asserted first as the positive control. Verified in both directions — removing the registration fails it on the named assertion while the control still passes. - PrebuiltBundleRetentionArmingGuard: deletion is never armed by omission. Reads the code default off the type, then requires the chart binding with `default "false"`. Verified non-vacuous against a removed line and a flipped default. - TheReportNamesItsDenominator...: "examined 3, collectable 0" and "examined 0" must not produce the same report. - Doc/Architecture/PrebuiltBundleRetention: the whole finding, including the gap the protected set cannot see — a satellite CI gate's MW_PLATFORM_REF is neither a Deployment record nor an instance report, so past 30 days a satellite's pinned build is collectable (the #3876 / Memex#219 shape). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S25PJZbVeCbkWhCfMub9P7

Copilot

🟢 Approval recommended

The Helm wiring, tests, and documentation consistently align to make deletion opt-in while asserting both boot-path registration and safe configuration defaults.

Pull request overview

This PR makes the prebuilt-bundle retention sweep report-only by default and adds guardrails to ensure (1) the deletion knob is actually templated into the portal ConfigMap and (2) the sweep’s registration on the portal boot path is asserted by tests. It also documents the operational intent and the known “satellite CI pin” protection gap.

Changes:

Helm: template PreWarm__PrebuiltBundleRetention__Delete into the portal ConfigMap with a default "false" and declare the key in values.yaml to keep the surface reviewable.
Tests: add a chart-arming guard test, a boot-path registration assertion, and a report-denominator assertion to prevent “examined nothing” from reading as “clean”.
Docs: add an architecture page plus a What’s New entry describing the new safe-by-default behavior and how to roll out deletion.

File summaries

File	Description
test/MeshWeaver.Hosting.Test/PrebuiltBundleRetentionTest.cs	Adds a test that the sweep summary names its denominator so an unexamined/empty store can’t read as “nothing collectable”.
test/MeshWeaver.Hosting.Test/PrebuiltBundleRetentionArmingGuard.cs	New guard test that enforces the Helm template binds the delete key with `default "false"` and that `values.yaml` declares it.
test/Memex.Portal.Shared.Test/PrebuiltBundleRetentionIsRegisteredTest.cs	New test asserting `ConfigureMemexMesh` registers the retention sweep as an `IHostedService`, with `AddModuleGenerationsGc` as a positive control.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-10-the-bundle-store-cleanup-now-reports-before-it-removes.md	User-facing release note describing report-only-by-default and safer rollout.
src/MeshWeaver.Documentation/Data/Architecture/PrebuiltBundleRetention.md	Architecture documentation for where it’s registered, arming semantics, report format, keep rules, and known gap.
src/MeshWeaver.Documentation/Data/Architecture.md	Adds the new architecture page to the topic map.
deploy/helm/values.yaml	Declares `PreWarm__PrebuiltBundleRetention__Delete: "false"` under `config.memex_portal`.
deploy/helm/templates/memex-portal/config.yaml	Renders `PreWarm__PrebuiltBundleRetention__Delete` into the ConfigMap with `

Review details

Files reviewed: 8/8 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-10T19:47:08Z