Hosting: RotateRegistryKey — instance keys rotated through the Hosting plugin, never by hand (first verb of the record-driven deployment process)

Tracking issue for the maintainer directive of 2026-08-30: a clean deployment process, administered through the Hosting plugin, with instance records in Systemorph/Memex — starting with the verb the day's incident demands, registry-key rotation.

Why now

Two live mwi_ instance keys (memex, memex-cloud) need rotating. Today that cannot be done cleanly: MeshWeaverInstanceService.ReissueKey exists and has no caller (no endpoint, no UI, no action), the key is stored as a literal in memex's helm values and as a cluster-only patch on memex-cloud (helm get values / helm get manifest: 0, live deployment: 1 — a helm upgrade would drop it), and the operator's hosting-kv-ensure marks PluginCatalog-RegistryToken as REQUIRED = "supplied out-of-band". Rotation by hand would be a workaround; the verb is the fix.

Measured state (2026-08-30)

fact	value
Deployment records	committed in `Systemorph/Memex` `mesh/Deployments/{memex,memex-cloud,client-b,client-i}.json` (Memex #146, core #2686, Plugins #918 — all merged)
…in the registry mesh	zero — `nodeType:Hosting/Deployment` returns 0, no GitSync node references `Systemorph/Memex`; the Hosting plugin cannot act on records it cannot see
record secret model	both records still use the LEGACY `secretMounts` hand-made SPC; `keyVaultSecrets` (typed) is `null`/empty, so the registry token is declared nowhere
vault `Systemorph`	access-policy model (`enableRbacAuthorization=false`); the operator identity's Key Vault Secrets Officer role assignment was inert; policy `get/set/list` granted today to `hosting-operator` (oid `bd994331…`) and read back
operator ↔ Azure	Jobs run as SA `hosting-operator` with workload identity (`azure.workload.identity/use`), image `azure-cli:2.77`
operator ↔ mesh	one-way: the mesh reads `::hosting::` marker lines out of the pod log (`OperatorOutput`, the audit line); the Job holds NO mesh credential

The verb: `InstanceActionKind.RotateRegistryKey`

Designed so the raw key never leaves the operator job, and only its hash crosses back — the same channel the backup verifier and the audit already use.

Operator script hosting-kv-rotate (core deploy/aks/operator/bin/, the hosting-kv-ensure idiom): generate mwi_ + base64url(32 random bytes) (InstanceKeys scheme), az keyvault secret set --vault <record.keyVault> --name <prefix>PluginCatalog-RegistryToken from a variable that is never echoed, unset, then hosting::say key_hash <sha256 hex>. Wait for the synced Secret to carry the new value (compare hashes, never values), then the existing Restart the portal + Wait for rollout steps, then hosting-verify.
Control plane (Plugins InstanceActionControlPlane.Track): when the streamed output carries ::hosting:: key_hash=, adopt it before the job reaches the restart step — write KeyHash/KeyIssuedAt on the instance node and the MeshWeaverInstance/{hashPrefix} index — via a new core method MeshWeaverInstanceService.AdoptKeyHash(instancePath, hash) (a ReissueKey that takes the hash instead of minting the key; reuses WriteIndex). The old index entry is deleted in the same write.
Plan (InstanceActionPlan.Steps): RotateRegistryKey => RotateSteps(vault, prefix, ns); request validation joins Roll/Restart (name-the-instance confirmation, no backup question); Describe gains the verb.
Records: both records declare the token under typed keyVaultSecrets (key: PluginCatalog__RegistryToken → object <prefix>PluginCatalog-RegistryToken), so HelmValues renders the SPC and the env is secretKeyRef, retiring the literal and the cluster-only patch (Memex#148's most serious row). DeploymentContent gains RegistryInstanceId (defaults to the deployment id).
Records reach the mesh: Systemorph/Memex mesh/ GitSynced into the registry's operational space, two-way (the control plane already assumes "the record space is two-way GitSynced").

Order, and what each step unblocks

vault policy — done (this issue's date)
core: AdoptKeyHash + hosting-kv-rotate + RegistryInstanceId plumbing
Plugins: the verb, plan, Track adoption, in-mesh tests
Memex: records declare keyVaultSecrets; GitSync so they exist in the mesh
run RotateRegistryKey for memex and memex-cloud; verify by hash and by the portal authenticating — never by printing a value

Not in scope here

Everything else the directive covers (the standard baked chart, RAM/CPU knobs, BYO helm, ACA) is the wider program in memory; this issue is the first verb, chosen because it is forced.

Progress

step	state
1. vault policy for `hosting-operator` (get/set/list on `Systemorph`)	done, read back
2. core: `IInstanceKeyRegistry` + `MeshWeaverInstanceService.AdoptKeyHash` + `hosting-kv-rotate`	#2803 open — `InstanceKeyAdoptionTest` 2/2 on the real mesh; script shellcheck-clean
3. Plugins: `RotateRegistryKey` verb, plan, mid-run adoption, tests, DI	branch `feat/hosting-rotate-registry-key` pushed; PR opens once #2803 is on main (it compiles against core, which does not yet carry the contract)
4. records declare `keyVaultSecrets`; GitSync into the mesh	next
5. run the verb for `memex` and `memex-cloud`	after 4

Two facts learned on the way that the design now rests on:

The registry-side adoption happens MID-RUN, not after the job. The operator restarts the portal later in the same job, and the new pods must find a registry that already knows the new key — so the control plane adopts the hash the moment the ::hosting:: key_hash= line streams (Track), before the job's CSI-sync wait and restart. Ordering is pinned by a plan test.
The operator's Key Vault role assignment was inert. Systemorph uses access policies, not RBAC, so Secrets Officer granted nothing; hosting-kv-ensure could never have satisfied its own REQUIRED token on this vault. Worth a line in Memex's envs.json or docs so the next environment does not rediscover it.

Merge pull request #3015 from Systemorph/docs/program-state-of-record

Closed by #3015 (merged as 8a31eb40), which converts this program to a committed state-of-record page.

The PR body said "Closes #1707, #2193, #2550 and #2802" — GitHub only parses the keyword before the first number, so only #1707 auto-closed. Closing the rest by hand; the work is landed either way.

Where this now lives:

The Compile Program — State of Record — every directive measured against the code, with what is implemented, what is partial (and which path bypasses it), and what is not started.
Instance Lifecycle — State of Record — the same for setup and key rotation, including which repo owns each half, which is the fact neither issue could answer.

Both ship with the platform under Doc/Architecture/, so the next session finds them without reading an issue thread.

🚨 The remaining work is named in those pages, not lost. The sharpest items:

Modules:RequirePrebuilt is real, load-bearing, and set in ZERO config files — so "never fall back to compiling" is provided for and enforced nowhere.
A Store update does not rebuild its dependents. The mesh-wide closure exists and has two callers; the package installer is not one of them. That is the mechanism behind the 2026-08-25 Store outage, and it is a live bug rather than a program — written up with the fix shape.
The awaiting-setup flag is set and read by nothing, so an empty image now boots successfully into a portal that can do nothing and says nothing about why — strictly worse than the exception it replaced.
One ordering hazard called out so it is not discovered the hard way: verification compares a bundle's fingerprint against the live mesh source nodes, so removing source from the mesh demotes every adoption from verified to unverified, which the execution gate permits. Doing that before the adopt-only gate is enforced loses verification silently, with nothing red.

Carried out in the same PR: deleted the dead CompilationLock (no caller, and async/Task.Delay-based, which the house rules forbid); taught the operator gate the direction that rots (it could only fail when a script was deleted, never when one shipped unclaimed); and gave hosting-kv-rotate behaviour tests including the one property whose failure is unrecoverable — it must never print the minted key, mutation-proven.

test(#2802): the stand-in registry runs without awk, inside the opera…

fix(#2802): commit and revoke refuse while the Deployment sets the ke…

fix(#2802): adoption retires a staged key; tokens bind to the key tha…

docs(#2802): list the rotation page in the topic map; the new tests t…

fix(#2802): a key transition reads the owner, and an identical index …

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-08-30)

Step 4b done — the fleet Space exists in the registry mesh and is GitSynced from `Systemorph/Memex`

Before: nodeType:Hosting/Deployment → 0 on the registry; no Space named Deployments; no _GitSync referencing the config repo. The Hosting plugin could not act on records it could not see.

Created on the registry (memex.meshweaver.cloud), via MCP:

+ Deployments                 nodeType: Space         (category Operations; body says it is synced from Systemorph/Memex)
+ Deployments/_GitSync        nodeType: GitHubSyncConfig
+   repositoryUrl: https://github.com/Systemorph/Memex   branch: main   subdirectory: mesh/Deployments
+   twoWay: true   createBranchIfMissing: true   createRepoIfMissing: false

then git_hub_sync op=update space=Deployments (activity Deployments/_Activity/d44ca55a). After:

nodeType:Hosting/Deployment → 4
  Deployments/memex-cloud   Deployments/memex   Deployments/client-b   Deployments/client-i   (imported 16:38:42Z)

Private-repo update was already proven on this registry — every Systemorph/Education course Space syncs from a private repo the same way — so no new credential was needed.

Two things to note:

The records that landed are main's — the keyVaultSecrets declarations are in Memex fix(blazor): cure the render-storm circuit wedge (re-bind guard) + surface bind errors #151, still open, and reach the mesh on the next update after it merges. That is the intended order: nothing in the mesh may reference vault objects the vault does not hold yet.
Point reads on the registry (get) repeatedly answered "reached no verdict within 10s" during this work while search answered fine — the live face of OwnerDisposing NACK is dropped exactly when the parent is disposing hosted hubs — the caller hangs for its full budget #2778. It did not block the writes or the sync; it did block reading back the diff.

Step 5 (running the verb) waits on core #2803 → Plugins verb PR → Memex #151.

Navigation Menu

Repository navigation

Hosting: RotateRegistryKey — instance keys rotated through the Hosting plugin, never by hand (first verb of the record-driven deployment process) #2802

Why now

Measured state (2026-08-30)

The verb: `InstanceActionKind.RotateRegistryKey`

Order, and what each step unblocks

Not in scope here

Progress

Step 4b done — the fleet Space exists in the registry mesh and is GitSynced from `Systemorph/Memex`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Hosting: RotateRegistryKey — instance keys rotated through the Hosting plugin, never by hand (first verb of the record-driven deployment process) #2802

Description

Why now

Measured state (2026-08-30)

The verb: InstanceActionKind.RotateRegistryKey

Order, and what each step unblocks

Not in scope here

Activity

rbuergi commented on Aug 30, 2026

Progress

rbuergi commented on Sep 1, 2026

rbuergi commented on Oct 11, 2026

Step 4b done — the fleet Space exists in the registry mesh and is GitSynced from Systemorph/Memex

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The verb: `InstanceActionKind.RotateRegistryKey`

Step 4b done — the fleet Space exists in the registry mesh and is GitSynced from `Systemorph/Memex`