feat(hosting): instance keys rotate through the operator — AdoptKeyHash + hosting-kv-rotate (#2802) by rbuergi · Pull Request #2803 · Systemorph/MeshWeaver

rbuergi · 2026-08-30T16:27:10Z

The platform half of #2802. The Hosting plugin's RotateRegistryKey verb follows in MeshWeaver.Plugins once this is on main (cross-repo halves land platform first).

Why

Two live mwi_ instance keys need rotating. MeshWeaverInstanceService.ReissueKey exists with no caller, and the only way to land a new value in a running portal was a terminal that would have to see the key — which is how today's exposure happened in the first place (an audit selected .value).

The design: the raw key exists in exactly two places, ever

where	what it holds
the operator job's memory	the raw key, until `unset`
Key Vault	the raw key
the job log / the mesh / this repo	the hash only

hosting-kv-rotate — the one operator script allowed to overwrite a vault secret (hosting-kv-ensure never will: the master key it guards seals stored credentials, which is why this is a separate verb and kv-ensure must never grow a --force). Mints mwi_ + base64url(32 random bytes) — the InstanceKeys scheme exactly — sets it in the vault from a variable that is never echoed, discards it, reports ::hosting:: key_hash=<sha256>, then waits for the synced Kubernetes Secret to carry the new value, compared by hash (DeploymentAKS → "KeyVault CSI env timing": a restart that reads the old synced Secret puts the old key into the new pods) and refuses with a message naming the SPC object if it never does. Dry-run aware; shellcheck clean.

IInstanceKeyRegistry (contract assembly) + MeshWeaverInstanceService.AdoptKeyHash — the registry-side half: the instance takes the new hash and issued-at, a fresh index entry points at it, and the previous index entry is deleted so the old key stops authenticating the moment adoption lands (ReissueKey left that "to be cleaned up separately"; a lifecycle verb must not). Refuses anything that is not 64-char lowercase hex — a raw key arriving here by mistake is precisely what must never happen. On the contract assembly so the Hosting plugin's in-mesh control plane can resolve it from the hub without referencing the portal host.

Verified

InstanceKeyAdoptionTest — 2/2 on the real Monolith mesh: register → adopt by instance id through the contract → new hash + later KeyIssuedAt on the node, new index entry resolves to the instance, old index entry gone; plus the shape refusal. Index nodes are read as System through GetMeshNode, the way the registry's own tests read fresh nodes (the first draft point-read them and hit the absent-node NotFound — fixed by reading the way the house rules say, not by widening a wait).

Already done outside the repo

The hosting-operator identity had a Key Vault Secrets Officer role assignment that was inert — the Systemorph vault uses access policies (enableRbacAuthorization=false). It now holds get/set/list as a policy, verified by read-back. Recording it here because it is the precondition the script depends on and it lives nowhere else yet.

🤖 Generated with Claude Code

…sh + hosting-kv-rotate (#2802) The platform half of MeshWeaver#2802. Two live mwi_ instance keys need rotating, and there was no clean path: MeshWeaverInstanceService.ReissueKey existed with NO caller, and the only way to land a new value in a running portal was a terminal that would have to see the key. - deploy/aks/operator/bin/hosting-kv-rotate: the ONE operator script allowed to overwrite a vault secret (kv-ensure never will — the master key it guards seals stored credentials, which is why this is a separate verb and kv-ensure must never grow a --force). Mints mwi_ + base64url(32 random bytes) — the InstanceKeys scheme exactly — sets it in Key Vault from a variable that is never echoed, unsets it, and reports ONLY `::hosting:: key_hash=<sha256>`. Then waits for the synced Kubernetes Secret to carry the new value, compared by HASH of the decoded value (DeploymentAKS → "KeyVault CSI env timing": a restart that reads the old synced Secret puts the OLD key into the new pods), and refuses with a message naming the SecretProviderClass object if it never does. Dry-run aware. shellcheck clean. - IInstanceKeyRegistry (MeshWeaver.Mesh.Contract/Security): the registry-side contract, on the contract assembly so the Hosting plugin's in-mesh control plane can resolve it from the hub's ServiceProvider without referencing the portal host. Hashes only, by instance id. - MeshWeaverInstanceService.AdoptKeyHash: a ReissueKey that takes the hash instead of minting the key. The instance node takes KeyHash/KeyIssuedAt, a fresh index entry points at it, and the PREVIOUS index entry is DELETED — ReissueKey left that "to be cleaned up separately"; a rotation driven by a lifecycle verb must not leave the old key authenticating. Refuses anything that is not a 64-char lowercase hex (a raw key sent here by mistake is exactly the thing that must never arrive). Idempotent for an adopted hash. Tests (Memex.Portal.Shared.Test/InstanceKeyAdoptionTest, 2/2 on the real Monolith mesh): register → adopt by instance id through the contract → instance carries the new hash and a later KeyIssuedAt, the new index entry resolves to the instance, the old index entry is GONE; and the shape refusal. Index nodes are read as System through GetMeshNode, the way the registry's own tests read fresh nodes. The Hosting plugin's RotateRegistryKey verb, which drives both halves, is the Plugins half and follows once this is on main (cross-repo halves land platform first). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Implements the platform-side support for rotating MeshWeaver instance registry keys via the Hosting operator, ensuring the raw mwi_ key only exists in the operator job’s memory and in Key Vault while the registry adopts only the SHA-256 hash and retires the previous index entry.

Changes:

Added IInstanceKeyRegistry (contract) for adopting a rotated key by hash.
Implemented MeshWeaverInstanceService.AdoptKeyHash and an instance-id based adapter for in-mesh callers.
Added the hosting-kv-rotate operator script plus a Monolith integration test and a What’s New entry.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/Memex.Portal.Shared.Test/InstanceKeyAdoptionTest.cs	Adds an end-to-end Monolith test covering hash adoption + old-index retirement behavior.
src/MeshWeaver.Mesh.Contract/Security/IInstanceKeyRegistry.cs	Introduces a contract surface for adopting a rotated instance key by hash.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-instance-keys-rotate-through-the-hosting-operator.md	Documents the new rotation split (operator mints key, registry adopts hash).
memex/Memex.Portal.Shared/Authentication/MeshWeaverInstanceService.cs	Implements hash adoption and contract adapter logic in the registry host.
deploy/aks/operator/bin/hosting-kv-rotate	Adds the operator-side rotation script that writes the new secret to Key Vault and reports only the hash.

Suppressed comments (1)

memex/Memex.Portal.Shared/Authentication/MeshWeaverInstanceService.cs:503

DeleteIndex currently switches access context manually and then swallows all delete failures. That can leave the old index entry in place (so the old key may keep authenticating) while AdoptKeyHash still reports success. Use the sealed impersonation boundary (RunAsSystem) and only treat the specific "Node not found" case as ignorable; propagate any other failure.

            var disposable = accessService.SwitchAccessContext(
                new AccessContext { ObjectId = WellKnownUsers.System, Name = "system-security" });
            return nodeFactory.DeleteNode(indexPath)
                .Select(_ => Unit.Default)
                .Catch<Unit, Exception>(ex =>
                {
                    // Absent already: not a failure. Anything else IS one — a stale entry that
                    // still authenticates is the hole this method exists to close.
                    logger.LogDebug(ex, "No index entry to remove at {Path}", indexPath);
                    return Observable.Return(Unit.Default);
                })

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        return Observable.Using(
+                () => accessService.ImpersonateAsSystem(),
+                _ => meshService.Query(request))
+            .Take(1)
+            .Timeout(TimeSpan.FromSeconds(10))
+            .SelectMany(results =>
+            {
+                // The index nodes share the NodeType and use the hash prefix as id, so an instance
+                // record is the hit whose id IS the instance id and whose namespace is not the index.
+                var node = results.FirstOrDefault(n =>
+                    string.Equals(n.Id, instanceId, StringComparison.Ordinal)
+                    && !string.Equals(n.Namespace, MeshWeaverInstanceNodeType.IndexNamespace, StringComparison.Ordinal));
+                return node?.Path is null
+                    ? Observable.Throw<Unit>(new InvalidOperationException(
+                        $"no MeshWeaverInstance is registered as '{instanceId}' — nothing to rotate"))
+                    : AdoptKeyHash(node.Path, keyHash);
+            });


github-actions · 2026-08-30T16:34:37Z