Per-environment content composition: what each environment pre-installs, expressed as feature flags, with required parameters routed through service discovery #1735

Activity

1. rbuergi commented on Aug 17, 2026

   

   rbuergi

   on Aug 17, 2026

   ContributorAuthor

   More actions

   Ask 3 context, from actually standing a portal up on Cosmos — the case this issue cites. Full findings in Systemorph/MeshWeaver.Plugins#1140; the parts that bear on parameter routing:

   One premise here is now stale. "The Cosmos and Snowflake storage backends … ship in no publish lane" was fixed by 447c1b8: both joined the CLOSURE lane in MeshModulesPublish.targets, so the DLLs reach the image. What remains true, and is the better framing for this issue, is that no deployment lists them in Modules:Assemblies — the bits ship, activation is unexercised everywhere.

   The parameter surface, measured. Cosmos needs four things, of which exactly one is reachable from configuration:

   Parameter	Reachable today
   connection string	✅ Graph:Storage:ConnectionString
   database name	❌ Graph:Storage:DatabaseName exists in GraphStorageConfig and is silently ignored — the factory reads CosmosStorageOptions.DatabaseName, pinned to "MeshWeaver"
   container names	❌ no binding
   connection mode	❌ no knob at all

   CosmosStorageOptions is bound to no configuration section — there is no .BindConfiguration(...) anywhere, and its only registration site passes configure: null, so IOptions<CosmosStorageOptions>.Value is always all-defaults. Its doc comment names ConnectionStrings:memexcosmos as the intended source; nothing implements that. Snowflake documents the same intent and likewise never performs it.

   This is the failure shape the issue says to avoid. An operator setting Graph:Storage:DatabaseName gets no error — they get the default database, silently. That is worse than the "skipped install that reads as success" case, because there is no install step to inspect. It argues for the declare-and-fail-loudly requirement being enforced at the options-binding level, not only at install/provision time: a backend should declare its parameters, and an unbound-but-supplied key should be a hard error rather than a default.

   Service-graph routing has nothing to bind to yet. The Aspire AppHost models no Cosmos resource (grep -i cosmos memex/aspire/ → nothing). PostgreSQL is wired as AddPostgres(...).AddDatabase("memex") → ConnectionStrings__memex. So "route a package's declared parameter from the environment's service graph" would, for the first real case, require adding the resource to the AppHost as well as the declaration mechanism — worth scoping in, since Cosmos is the example this issue leads with.

   One nuance for the design: the Cosmos factory already prefers Aspire-injected keyed Container services over any connection string, and falls back only if they are absent. That is closer to "resolve from the service graph" than a config key is, and it is the branch I used to get a portal booting. A parameter-routing design that can express "this package wants a resource handle, not a string" would fit what the code already does.

   I have not implemented anything from this issue — Systemorph/MeshWeaver.Plugins#1140 carries only the fixes proven by a test (a startup-throwing cast, and a frozen change feed).
2. rbuergi commented on Aug 17, 2026

   

   rbuergi

   on Aug 17, 2026

   ContributorAuthor

   More actions

   Design note, part 2 — the acceptance criterion, and the one shape change it forced

   memex (memex.meshweaver.cloud, namespace memex-cloud) → all of Plugins.
   systemorph (memex.systemorph.com, namespace memex) → the same, without the games and fun stuff.

   The allow-list I sketched in part 1 cannot say that without re-typing 17 package names, so the flag
   gained a direction, decided by Enabled:

   • an ENABLED flag INCLUDES the packages it names;
   • a declared-but-DISABLED flag EXCLUDES them, and the exclusion wins over every other selection
     signal — the platform's own preInstalled baseline included.

   That makes the maintainer's sentence literal, and — the part that matters operationally — it makes
   the per-environment delta one line on top of one shared declaration:

   # shared
   features:
     plugins:
       description: "The whole platform plugin repo."
       packages: ["Plugins/*"]
     games:
       description: "Games and demos."
       packages: ["Plugins/Chess", "Plugins/DoublePendulum", "Plugins/FractalStars", "Plugins/ThreeBody"]
   
   # deployments/aks/memex/values.memex.yaml   (namespace memex = memex.systemorph.com = "systemorph")
   features:
     games:
       enabled: false

   memex-cloud (= "memex") adds nothing — both flags are declared, so both are on. RolePlay is a
   values decision: put it in games, in its own flag, or leave it alone. No package name appears
   anywhere in platform code, and the exclusion is source-scoped through the same PluginGrantEntry
   matcher as everything else, so Plugins/Chess and Education/Chess are different packages.

   Allow-list vs exclusion — the tradeoff, stated

   They fail differently and neither is universally right:

   	silent failure
   allow-list	omits a package newly added to a repo — it never reaches the portal, and nobody notices
   exclusion	includes one

   I did not pick one; the flag expresses whichever the operator means, and I removed the silence
   instead:

   • the Composition admin tab names which flag decided each package;
   • the boot log names the composition it installed, and warns when a flag's patterns matched nothing;
   • a pattern naming a source this installation does not have is reported at Error — with the
     exclusion direction called out specifically, because that one fails open (the packages it
     meant to keep out get installed).

   For this case the wildcard-plus-exclusion reading is also the safer one: Plugins/* is already what
   pluginCatalog.installByDefault says today, so a new plugin appearing is the status quo, whereas a
   17-entry allow-list would silently withhold every future plugin from both portals.

   Why this is a new lane and not the old knob

   Confirmed as stated: InstallByDefault is ledger-gated and re-asserts never, so on either live
   portal — both long-populated — setting it is a no-op. Its seed-once meaning is deliberate (an admin
   who uninstalls something must not be fought by the next restart) and is left exactly as is,
   comments included. The flag lane is separate and reconciled on every boot; a reconciled candidate
   is exempt from the seed ledger, which is precisely the difference between a policy and a seed. There
   is a test that populates an instance, then re-runs the production decision and requires the same
   composition to be re-asserted with zero writes.

   Delivery path

   Everything is settable through the values path the private repo already uses. The chart's ConfigMap
   enumerates keys by hand — an un-templated key is silently dropped — so this PR adds the range block
   that renders Features__Flags__{name}__{Enabled,Description,Packages__N} from a features: map, and
   features: {} plus a worked example to deploy/helm/values.yaml. Verified with helm template.

   While there: PluginCatalog__InstallPreInstalledPackages was itself never templated, so the
   documented opt-out could not be exercised per environment at all. It is now rendered when set.

   I have not touched Systemorph/Memex. Landing the maintainer's case needs the features: block
   above added to deployments/aks/{memex,memex-cloud}/values.*.yaml (and the secret-free public
   mirror the chart-drift gate renders against) — happy to hand over the exact snippet, but that repo
   is yours to change.

   One correction to part 1

   The Cosmos/Snowflake example is stale: 447c1b879 put both on the MeshModuleClosure publish lane,
   so they do ship in the image now. The point survives in a sharper form — their only actual
   connection-string channel is Graph:Storage:ConnectionString, while CosmosStorageOptions documents
   a ConnectionStrings:memexcosmos convention that nothing reads. That is exactly the
   package-invents-its-own-lookup shape ask 3 replaces, with one of the two inventions dead on arrival.
3. rbuergi commented on Aug 17, 2026

   

   rbuergi

   on Aug 17, 2026

   ContributorAuthor

   More actions

   Naming hazard for this work, found while auditing all 43 Store packages (#1749).

   If per-environment composition is going to name packages in config, two collisions need a decision first:

   Path	Repo	Display name	What it actually is	Price
   DataModelling	MeshWeaver.Plugins	"Data Modelling"	free, pre-installed dev tool (data-model browser)	free
   DataModeling	education	"Data Modelling"	paid course	900 CHF
   SST	MeshWeaver.Reinsurance	"Swiss Solvency Test"	free live model (FINMA StandRe)	free
   SwissSolvencyTest	education	"The Swiss Solvency Test"	paid course	900 CHF

   DataModeling / DataModelling is the dangerous one: byte-identical display name, and paths differing by a single l. A config typo silently selects a 900 CHF course instead of a free pre-installed tool, or vice versa — and both are legitimate live packages, so nothing would error.

   They are not duplicates and nothing should be deleted. Suggested resolution is display-name disambiguation only (e.g. DataModelling → "Data Model Browser", DataModeling → "Data Modelling (Course)"), keeping both paths — renaming paths would break installs and cross-links.

   Flagging rather than acting; it needs a maintainer call.
4. rbuergi

   closed this as completedin #1744on Aug 17, 2026
5. rbuergi commented on Oct 11, 2026

   

   rbuergi

   on Oct 11, 2026

   ContributorAuthor

   More actions

   (re-posted 2026-10-11; originally by @rbuergi on 2026-08-17)

   Design note — per-environment composition as feature flags, with service-referenced parameters

   Survey first, then shapes. Two premises in the issue turned out to be wrong; I say so rather than
   building on them, because both change the design.

   0. Two falsified premises

   (a) "There is NO server-side feature-flag concept today." There is — and it is already the
   deploy-time one. memex/Memex.Portal.Shared/MemexFeatureOptions.cs binds the Features
   section and is documented as Feature Flags:
   Features:Ai:Providers:OpenAI, Features:Onboarding:InvitationOnly,
   Features:StaticRepoSync:Partitions, Features:Orleans:Clustering, Features:SignalR. It binds in
   MemexConfiguration.ConfigureMemexServices and is read through IOptions<T>; the Helm chart
   already renders Features__StaticRepoSync__Partitions__{i} per environment.

   So the gap is narrower and more precise than "no flags":

   1. every flag is a fixed field on one C# record — a deployment cannot declare a flag the
      platform does not already know about, which is exactly what "pre-install from any repo,
      per environment" needs;
   2. it is not observable — IOptions<T> is a startup snapshot, and PluginCatalogOptions is an
      even flatter one (configuration.GetSection(…).Get<PluginCatalogOptions>() registered as a plain
      singleton), so nothing reacts to a reload even though MemexConfiguration opens its JSON with
      reloadOnChange: true;
   3. nothing hangs content off a flag, and nothing declares a package's parameters.

   (b) "Cosmos and Snowflake … ship in no publish lane." Not any more — 447c1b879 put both on the
   MeshModuleClosure lane in memex/MeshModulesPublish.targets, so the DLLs are in the image today.
   The case still supports ask 3, but for a sharper reason: their only connection-string channel is
   Graph:Storage:ConnectionString. CosmosStorageOptions documents a ConnectionStrings:memexcosmos
   convention that nothing reads, and neither CosmosStorageOptions nor SnowflakeStorageOptions
   is ever bound from configuration. That is precisely "each package inventing its own config lookup",
   and one of the two inventions is dead on arrival.

   Two more survey facts that shape the design:

   • There is no environment NAME to key off. Nothing in deploy/ sets ASPNETCORE_ENVIRONMENT, so
     memex, memex-cloud and client-i all run as Production and load byte-identical appsettings.
     Per-environment differentiation is env vars only, from a values file that is gitignored and
     lives in the private Systemorph/Memex repo (deploy/aks/envs/.gitignore).
   • The chart enumerates every key by hand (deploy/helm/templates/memex-portal/config.yaml) — an
     un-templated key is silently dropped, a trap the chart's own comments record three times. Today
     PluginCatalog__InstallPreInstalledPackages is one of the dropped ones: the existing opt-out
     cannot be set per environment at all.

   The MAUI FeatureFlags is a third vocabulary (device-local Preferences, compile-time defaults,
   bool IsEnabled) and is deliberately left alone — it has no IConfiguration host and its concerns
   are per-device, not per-environment. The server side extends Features; it does not mint a fourth
   vocabulary.

   ---

   1. Asks 1 + 2 — the flag IS the composition unit: Features:Flags:{name}

   "Features": {
     "Flags": {
       "store":     { "Packages": ["Plugins/Store", "Plugins/Essentials"] },  // declared ⇒ ON here
       "education": { "Enabled": false, "Packages": ["Education/*"] },        // declared, off in THIS env
       "betaChat":  true                                                      // bare bool: a pure flag
     }
   }

   Environment-variable form — which is what actually reaches AKS and ACA:

   Features__Flags__store__Packages__0=Plugins/Store
   Features__Flags__education__Enabled=false
   Features__Flags__betaChat=true
   

   • Packages reuses the existing Source/Package notation and the existing PluginGrantEntry
     matcher against PackageManifest.Source — so it is already "from any repo" and already fails
     closed when a registry does not stamp the source. No second matching rule, no second parser.
   • Declared ⇒ enabled; Enabled: false turns a flag off without deleting the declaration, which
     is what makes "shared base values file + per-environment override" work under helm's last-wins
     layering. Undeclared ⇒ disabled.
   • A flag with no Packages is a plain named boolean — the extension point the fixed record cannot
     offer.
   • No Features:Environments:{name} map, deliberately: the portal cannot tell which environment
     it is (see above), so a name-keyed map would silently select the wrong branch. Each environment
     sets its own flags. That is what "per environment" can mean here.

   Reconciled — and NOT the same knob as InstallByDefault. Three lanes, three different questions,
   all feeding the one ordered install in InstanceAutoRegistrationService.InstallDefaults:

   lane	who declares it	when	why
   preInstalled on the manifest	the package author	every boot	the platform's own baseline; the only thing that heals a lost partition (#902)
   Features:Flags:{name}:Packages (new)	the environment	every boot	"this environment always has X" — it must re-assert, or it is not a policy
   PluginCatalog:InstallByDefault	the operator	once, ledger-gated	seeds a fresh deployment; must never fight an admin who later uninstalls

   InstallByDefault's semantics, ledger and doc comments are untouched; its XML doc gains one
   paragraph pointing at the reconciled sibling so the contrast is stated where someone reads it. The
   new lane joins the selection beside the preInstalled predicate in Candidates(…), inheriting
   the whole existing machinery for free: source-scoped matching, the dependency closure, the
   commercial-package refusal, the tolerant dependency order, sequential install, and the
   content-identity gate that keeps a reconciled boot at one catalog listing and zero writes.

   Disabling a flag does not uninstall. Turning store off stops the environment asserting it; it
   does not delete a partition full of user content, and an unattended uninstall racing per-node hubs on
   boot is not a thing to do by default. The admin surface shows "declared, not asserted", so the state
   is visible rather than silent. A later explicit Remove: true can add the destructive half when
   somebody actually wants it.

   2. How a flag is observed — IObservable<T>, not bool IsEnabled

   src/MeshWeaver.Hosting/FeatureFlags/:

   public interface IFeatureFlags
   {
       IObservable<ImmutableSortedDictionary<string, FeatureFlag>> All { get; }
       IObservable<bool> IsEnabled(string name);          // DistinctUntilChanged
       IObservable<FeatureFlag?> Get(string name);
   }
   
   public sealed record FeatureFlag(string Name, bool Enabled,
       ImmutableList<string> Packages, string? Description);

   ConfigurationFeatureFlags is a mesh-scoped singleton (dies with the mesh — no static cache, no
   Clear()), holding a BehaviorSubject on an instance field, re-read by
   ChangeToken.OnChange(config.GetReloadToken, …). That is a push from the configuration provider —
   no timer, no poller, no watchdog. IsEnabled is All.Select(…).DistinctUntilChanged(), so a live
   view binds it directly. Nothing async, no Task<T>, no SemaphoreSlim.

   3. Ask 3 — declared parameters, resolved through Aspire-style service references

   Declare, on the manifest (PackageManifest.Parameters), authored exactly the way preInstalled
   / publicSegments / module already are — in the node-repo root's content, or in a package.json:

   "parameters": [
     { "name": "warehouse", "kind": "ConnectionString",
       "description": "The Snowflake warehouse this pack reads." },
     { "name": "crm",    "kind": "Endpoint" },
     { "name": "apiKey", "kind": "Value", "optional": true }
   ]

   Optional defaults to false — required is the default, and the CLR-default false round-trips
   loss-free under the default-suppressing serializer (the declared-true bool trap already diagnosed
   on this codebase).

   Resolve from the environment's SERVICE GRAPH, not a package-specific key. The resolver reads
   exactly the keys the two routes already inject:

   kind	config key	who injects it
   ConnectionString	ConnectionStrings:{name}	Aspire WithReference(db) → ConnectionStrings__memex; AKS: chart secret / Key Vault CSI
   Endpoint	Services:{name}:{https|http|default}:0	Aspire WithReference(project) → services__{name}__https__0 — the Microsoft.Extensions.ServiceDiscovery shape, and that package is already referenced and AddServiceDiscovery() already called in Memex.Portal.ServiceDefaults
   Value	Parameters:{name}	Aspire AddParameter / plain env

   So a package says "I need the warehouse connection" and the environment decides whether that
   is an Aspire resource reference, a Helm value, or a Key Vault secret. That is the "route to other
   services" the issue asks for, expressed once instead of per package.

   It fails CLOSED, at the one funnel, naming exactly what to provision.
   CatalogLayoutAreas.InstallOrUpdate is the single orchestrator every lane goes through (boot default
   install, the Store's Provision click, the auto-update reconciler) and already carries one fail-closed
   gate — PackageEntitlement.Authorize, which faults the observable with a speaking exception. The
   parameter gate sits immediately beside it and faults with PackageParameterException:

   Package 'finance-pack' requires 1 parameter this environment does not supply:
     warehouse (ConnectionString) — The Snowflake warehouse this pack reads.
       provision: ConnectionStrings__warehouse
   Nothing was installed.
   

   Never a half-configured install, never a silent skip: the unattended lane counts it Failed in
   the DefaultInstallSummary (logged at Error, surfaced), the same discipline as the CI preflight — a
   gate that reports green on no evidence is worse than a red one. It also closes the asymmetry the
   survey found: today ConnectionStrings:orleans missing gets a purpose-written throw naming what to
   set, while ConnectionStrings:memex missing surfaces only as an Aspire-internal exception at first
   resolution.

   4. Exposure — audience: the platform ADMIN, in the portal

   A global-admin settings tab (Administration → Composition), the same shape as the existing
   InstanceGrantAdminSettingsTab: Controls.DataGrid over a plain row record with
   PropertyColumnControls — flag, enabled, packages, and the outcome (installed / up to date /
   declared-not-asserted / refused: ConnectionStrings__warehouse missing). No hand-built HTML, no
   RenderHtml-shaped helper. Every string is a key in strings.{en,de}.json, mirrored into
   clients/react/src/i18n/.

   A machine-readable endpoint is deliberately not added: "why did this package install here" is a
   human question, the tab answers it, and the registry API already exists for anything a machine needs.

   5. Slicing

   One PR: flag surface + reconciled composition lane + parameter declaration/resolution/gate + admin
   tab + chart templating + docs + tests. The parameter half is small precisely because it rides the
   existing funnel and copies the existing entitlement-gate shape; splitting it would land a manifest
   field nobody reads — the dead-metadata defect class this repo has already hit three times
   (preInstalled, publicSegments, contactEmail).

   The chart change is mandatory, not cosmetic: an un-templated key is silently dropped, so
   Features__Flags__* gets a range block — and while there, PluginCatalog__InstallPreInstalledPackages
   gets templated too, since today the documented opt-out cannot be set per environment at all.

   6. Decisions worth confirming

   1. Features:Flags rather than a new top-level section — one operator vocabulary, at the cost of
      the Features section having two binders (the typed MemexFeatureOptions record, which ignores
      unknown keys, and the dynamic flag reader).
   2. Declared ⇒ enabled. Alternative (Enabled required) is more explicit but doubles the length
      of the common per-environment line.
   3. Disabling never uninstalls.
   4. Additive to InstallByDefault, which is untouched. Migrating the AKS values from
      pluginCatalog.installByDefault: ["Plugins/*"] to a plugins flag is an operator decision in the
      private repo, not part of this change.