Hosting operator: hosting-audit names what lives only on the cluster by rbuergi · Pull Request #2734 · Systemorph/MeshWeaver

rbuergi · 2026-08-30T07:30:19Z

What

The operator image gains hosting-audit — one read-only command that diffs an instance's namespace against its helm release's manifest (helm get manifest + helm get hooks, turned into objects by kubectl apply --dry-run=client) and reports everything that lives ONLY on the cluster, by name and never by value:

the portal Deployment's live-only (and manifest-only) env names, envFrom sources, volumes, mounts, sidecar containers, pod annotations, and the pod-spec fields a kubectl patch reaches (replicas — unless an autoscaler owns them — nodeSelector, tolerations, affinity, SA, grace period, the portal container's resources/lifecycle/startup-probe budget);
the chart ConfigMaps' live-only, manifest-only and value-differing keys (values compared on the cluster, only key names leave it);
live objects the chart does not own, grouped by kind (owned objects, helm release records, SA tokens, CSI-synced secrets and Kubernetes' own furniture excluded);
Secrets the live pod reads that are neither chart-managed nor CSI-synced;
entries whose name matches Token|Secret|Key|Password carried as a plain value (inline env on any container, or a chart ConfigMap key).

The verdict (clean/drift) plus the full typed report ride ::hosting:: audit=<base64 JSON> for the Hosting plugin's new Audit instance action (MeshWeaver.Plugins PR, follows this one) to record on the mesh. Drift exits 0 — a finding, not a failed run. Could not measure refuses: no release, no live portal Deployment, or a kind the ClusterRole may not list each fail the run, because a kind the audit could not read must not be reported as clean.

Why

The memex portal crashed at boot because its Email section existed twice — chart ConfigMap defaults vs a cluster-only secret patched as env — and .NET picks the winner at random per pod start. helm upgrade preserves live-edited ConfigMap keys and never touches objects it does not own, so this class of drift never fails a deploy; it fails months later as an outage. Systemorph/Memex#148 is the hand-measured inventory (25 env patches, 10 live-edited keys, sidecars, a plain-text registry token in the pod spec); this is the machinery that keeps it measured. The rule enforced: nothing lives only on the cluster — every value is a typed field on the Hosting/Deployment record, rendered by the chart; secrets are Key Vault names.

How it is proven

test/run-tests.sh gains 26 audit cases run against helm/kubectl stubs over fixture manifests: a drifting namespace (every category found by name), a clean one (verdict clean, an unserved kind recorded as skipped — never silently absent), a missing release (refused), a Forbidden kind (refused naming the ClusterRole), injection-shaped names (refused before anything runs), and the one that matters most: no fixture secret value ever appears in the output. 65/65 green locally.
The CI gate updates its tracked-file counts (18), the plan-command list (hosting-audit), asserts _audit.jq ships and parses (jq exit 3 = red), and the image check now requires jq (present in the Azure Linux base; the Dockerfile pins it via tdnf anyway).
The operator ClusterRole gains read-only list/get on the audited kinds (daemonsets, PDBs, HPAs, ScaledObjects, network policies, roles/rolebindings) — an audit measures, it never reconciles. Re-applying the RBAC on the cluster is a separate, deliberate ops step.

🤖 Generated with Claude Code

…luster The memex portal crashed at boot because its Email config existed twice: the chart ConfigMap defaults and a cluster-only secret patched onto the Deployment as env — .NET picks the winner at random per pod start, and a helm upgrade preserves live edits, so nothing ever went red (Systemorph/Memex#148 is the hand-measured inventory: 25 env patches, 10 live-edited ConfigMap keys, sidecars, unmanaged objects, a plain-text registry token in the pod spec). The operator image gains hosting-audit + _audit.jq: diff a namespace against 'helm get manifest' + 'helm get hooks' (the truth) and report, by NAME and never by value, the portal Deployment's live-only env/envFrom/volumes/mounts/ containers/pod-spec patches, the chart ConfigMaps' live-only and value-differing keys, objects the chart does not own, secrets the pod reads that nothing renders, and secret-shaped names carried as plain values. Verdict clean/drift rides '::hosting:: audit=<base64 JSON>' for the Hosting plugin's new Audit instance action to record; drift exits 0 (a finding, not a failed run), while 'could not measure' — no release, no live portal, a kind the ClusterRole may not list — refuses rather than reporting clean. Fixture-tested via helm/kubectl stubs (drift, clean, forbidden, missing-release, injection, value-leak guard) in test/run-tests.sh; the operator ClusterRole gains read-only list/get on the audited kinds; the image installs jq and the CI gate pins the file counts, the plan-command list and the jq program's parse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…joins the plan-command gate The tracked-file floor said 16 while 17 files were tracked, and the every-command-the-plan-can-emit list never carried hosting-pv-purge although the teardown plan emits it — a missing pv-purge would have failed INSIDE a live teardown, after the namespace delete, exactly what the gate exists to refuse. Now: 19 files (16 commands + run.sh + _common.sh + _audit.jq), 16-entry plan list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds a new read-only hosting-audit operator command that detects “live-only” Kubernetes configuration drift by diffing a namespace’s live objects against the Helm release manifest/hooks, and wires it into documentation, RBAC, CI, and operator test fixtures.

Changes:

Add deploy/aks/operator/bin/hosting-audit plus its jq-based detection program (_audit.jq) to compute a typed drift report.
Extend operator tests with helm/kubectl stubs and fixtures to assert drift/clean/refusal behaviors and “no secret value leakage” guarantees.
Update operator image dependencies (jq), RBAC read permissions for audited kinds, docs, and the hosting-operator CI gate.

Reviewed changes

Copilot reviewed 35 out of 40 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-the-cluster-can-no-longer-hide-configuration.md	What’s New entry describing `hosting-audit` and its drift/refusal semantics.
deploy/aks/operator/test/stubs/kubectl	Kubectl stub to serve fixture JSON for audit tests.
deploy/aks/operator/test/stubs/helm	Helm stub to serve fixture status/manifest/hooks for audit tests.
deploy/aks/operator/test/run-tests.sh	Adds `hosting-audit` behavior tests and report assertions.
deploy/aks/operator/test/fixtures/audit/missing/.keep	Placeholder for “missing release” fixture directory.
deploy/aks/operator/test/fixtures/audit/forbidden/status.json	Fixture Helm status JSON for “forbidden kind” scenario.
deploy/aks/operator/test/fixtures/audit/forbidden/manifest.json	Fixture manifest objects for “forbidden kind” scenario.
deploy/aks/operator/test/fixtures/audit/forbidden/live/role.forbidden	Marker file to force a Forbidden response for `role` listing.
deploy/aks/operator/test/fixtures/audit/forbidden/live/deployment.json	Fixture live Deployment list for “forbidden kind” scenario.
deploy/aks/operator/test/fixtures/audit/forbidden/live/configmap.json	Fixture live ConfigMap list for “forbidden kind” scenario.
deploy/aks/operator/test/fixtures/audit/drift/status.json	Fixture Helm status JSON for “drift” scenario.
deploy/aks/operator/test/fixtures/audit/drift/manifest.json	Fixture manifest objects for “drift” scenario.
deploy/aks/operator/test/fixtures/audit/drift/live/serviceaccount.json	Fixture live ServiceAccounts for drift detection/exclusions.
deploy/aks/operator/test/fixtures/audit/drift/live/service.json	Fixture live Services including unmanaged ones.
deploy/aks/operator/test/fixtures/audit/drift/live/secretproviderclass.json	Fixture live SecretProviderClass list for CSI context.
deploy/aks/operator/test/fixtures/audit/drift/live/secret.json	Fixture live Secrets list (chart-managed, CSI-managed, unmanaged, furniture).
deploy/aks/operator/test/fixtures/audit/drift/live/persistentvolumeclaim.json	Fixture live PVCs including unmanaged ones.
deploy/aks/operator/test/fixtures/audit/drift/live/job.json	Fixture live Jobs including owned and hook job cases.
deploy/aks/operator/test/fixtures/audit/drift/live/ingress.json	Fixture live Ingresses including unmanaged ones.
deploy/aks/operator/test/fixtures/audit/drift/live/deployment.json	Fixture live Deployments including drifted portal and unmanaged deployment.
deploy/aks/operator/test/fixtures/audit/drift/live/cronjob.json	Fixture live CronJobs including unmanaged ones.
deploy/aks/operator/test/fixtures/audit/drift/live/configmap.json	Fixture live ConfigMaps with edited/differing keys and unmanaged configmap.
deploy/aks/operator/test/fixtures/audit/drift/hooks.json	Fixture hook Job object to ensure hooks are treated as managed.
deploy/aks/operator/test/fixtures/audit/clean/status.json	Fixture Helm status JSON for “clean” scenario.
deploy/aks/operator/test/fixtures/audit/clean/manifest.json	Fixture manifest objects for “clean” scenario.
deploy/aks/operator/test/fixtures/audit/clean/live/serviceaccount.json	Fixture live ServiceAccounts for clean scenario.
deploy/aks/operator/test/fixtures/audit/clean/live/service.json	Fixture live Services for clean scenario.
deploy/aks/operator/test/fixtures/audit/clean/live/secret.json	Fixture live Secrets for clean scenario (including helm record, CSI secret).
deploy/aks/operator/test/fixtures/audit/clean/live/scaledobject.notype	Marker file to simulate an unserved CRD kind (skippedKinds).
deploy/aks/operator/test/fixtures/audit/clean/live/job.json	Fixture live Jobs for clean scenario.
deploy/aks/operator/test/fixtures/audit/clean/live/ingress.json	Fixture live Ingresses for clean scenario.
deploy/aks/operator/test/fixtures/audit/clean/live/deployment.json	Fixture live Deployment for clean scenario.
deploy/aks/operator/test/fixtures/audit/clean/live/configmap.json	Fixture live ConfigMaps for clean scenario.
deploy/aks/operator/test/fixtures/audit/clean/hooks.json	Fixture hook Job object for clean scenario.
deploy/aks/operator/Dockerfile	Installs `jq` to support the audit’s jq-based detection.
deploy/aks/operator/bin/hosting-audit	New operator command: reads helm/kubectl, computes report, emits `::hosting::` lines.
deploy/aks/operator/bin/_audit.jq	New jq program implementing the drift detection and report shape.
deploy/aks/manifests/hosting-operator/operator-rbac.yaml	Adds read-only list/get permissions for additional audited kinds.
deploy/aks/INSTANCE-LIFECYCLE.md	Documents `hosting-audit` as part of the instance lifecycle flow.
.github/workflows/hosting-operator.yml	Updates gate expectations (tracked files, plan commands), parse-checks `_audit.jq`, requires jq in image checks.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…e, never a command line Every other compared pod-spec field carries placement and sizing (replicas, selectors, quantities, probe numbers), but a lifecycle hook holds COMMAND LINES, and a command line can embed anything — tokens included. The comparison keeps using the full value; the report now shows the hook's structure with every string scalar replaced by '…', which is enough to see THAT a hook was patched and WHERE without quoting what it runs. Pinned by a fixture case whose command string joins the must-never-print leak guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

github-actions · 2026-08-30T07:57:01Z

Test Results (shard 3)

701 tests ±0 509 ✅ ±0 4m 48s ⏱️ -2s
5 suites ±0 192 💤 ±0
5 files ±0 0 ❌ ±0

Results for commit 9f52772. ± Comparison against base commit d3272b5.

github-actions · 2026-08-30T07:58:07Z

Test Results (shard 4)

10 files 10 suites 5m 48s ⏱️
3 466 tests 3 466 ✅ 0 💤 0 ❌
4 110 runs 4 110 ✅ 0 💤 0 ❌

Results for commit 9f52772.

github-actions · 2026-08-30T07:58:16Z

Test Results (shard 2)

968 tests ±0 968 ✅ ±0 5m 51s ⏱️ -7s
6 suites ±0 0 💤 ±0
6 files ±0 0 ❌ ±0

Results for commit 9f52772. ± Comparison against base commit d3272b5.

♻️ This comment has been updated with latest results.

github-actions · 2026-08-30T07:58:44Z

Test Results (shard 0)

913 tests 913 ✅ 5m 30s ⏱️
6 suites 0 💤
6 files 0 ❌

Results for commit 9f52772.

github-actions · 2026-08-30T08:00:46Z

Test Results (shard 5)

1 533 tests ±0 1 533 ✅ ±0 6m 38s ⏱️ + 1m 7s
8 suites ±0 0 💤 ±0
8 files ±0 0 ❌ ±0

Results for commit 9f52772. ± Comparison against base commit d3272b5.

github-actions · 2026-08-30T08:00:52Z

Test Results (shard 1)

792 tests ±0 788 ✅ ±0 6m 49s ⏱️ -5s
6 suites ±0 4 💤 ±0
6 files ±0 0 ❌ ±0

Results for commit 9f52772. ± Comparison against base commit d3272b5.

github-actions · 2026-08-30T08:01:23Z

Test Results

41 files ±0 41 suites ±0 35m 27s ⏱️ +34s
8 373 tests ±0 8 177 ✅ ±0 196 💤 ±0 0 ❌ ±0
9 017 runs +1 8 821 ✅ +1 196 💤 ±0 0 ❌ ±0

Results for commit 9f52772. ± Comparison against base commit d3272b5.

♻️ This comment has been updated with latest results.

rbuergi · 2026-08-30T08:02:27Z

CI red on shard 2 was SilentReadNackTest.OwnerDisposedWithReadOutstanding_IsNacked_NotLeftHanging — the two-disposal-arms race tracked as #1599 ("35% flaky on main") / #2673. This PR's diff carries no C# at all (git diff --name-only origin/main...HEAD | grep -c '\.cs$' → 0: bash, jq, YAML, an RBAC manifest and a doc), so it cannot reach Monolith hub disposal. Re-running the failed jobs with that attribution; the root cause of the race stays with #2673/#2700.

rbuergi · 2026-08-30T08:12:46Z

Second red on the re-run, a DIFFERENT test in the same shard: UserHomeLegacyPathResolutionTest.LegacyUserPrefix_RendersHomeArea_NotAreaNotFoundPlaceholder — a 30 s methodTimeout inside a Monolith render test (1 of 462), not an assertion. Two unrelated single failures in MeshWeaver.Hosting.Monolith.Test across two runs of a diff that carries zero C# is the shard-2 wedge cluster, not this change; #2727 tracks the disposal race first seen after #2438 landed this morning. Noting both occurrences on #2727 and re-running the failed jobs once more; a third red stops here and gets reported rather than looped.

Copilot AI lite review requested due to automatic review settings August 30, 2026 07:30

Copilot started reviewing on behalf of rbuergi August 30, 2026 07:34

View session

Copilot AI reviewed Aug 30, 2026

View reviewed changes

rbuergi mentioned this pull request Aug 30, 2026

Race: RecycleSurvivesItsOwnDisposeTest.TheHubThatConfirmsTheRecycle_OutlivesTheHubItRecycles — the recycled address's GetDataRequest is access-checked on the DISPOSED hub's LifetimeScope (shard 1, first seen after #2438) #2727

Closed

rbuergi merged commit f9c3eb6 into main Aug 30, 2026
48 of 52 checks passed

rbuergi deleted the feat/hosting-audit-operator branch August 30, 2026 08:22

rbuergi mentioned this pull request Oct 3, 2026

fix(ci): project scope reads glob inputs/links; a shipped static asset reaches no referencing project #6021

Merged

Navigation Menu

Repository navigation

Hosting operator: hosting-audit names what lives only on the cluster - #2734

Hosting operator: hosting-audit names what lives only on the cluster#2734
rbuergi merged 3 commits into
mainfrom
feat/hosting-audit-operator

rbuergi commented Aug 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026 •

edited

Loading

Uh oh!

rbuergi commented Aug 30, 2026

Uh oh!

rbuergi commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rbuergi commented Aug 30, 2026

What

Why

How it is proven

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Aug 30, 2026

Test Results (shard 3)

Uh oh!

github-actions Bot commented Aug 30, 2026

Test Results (shard 4)

Uh oh!

github-actions Bot commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (shard 2)

Uh oh!

github-actions Bot commented Aug 30, 2026

Test Results (shard 0)

Uh oh!

github-actions Bot commented Aug 30, 2026

Test Results (shard 5)

Uh oh!

github-actions Bot commented Aug 30, 2026

Test Results (shard 1)

Uh oh!

github-actions Bot commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

rbuergi commented Aug 30, 2026

Uh oh!

rbuergi commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Aug 30, 2026 •

edited

Loading

github-actions Bot commented Aug 30, 2026 •

edited

Loading