Repository navigation
fix(operator): the role writes every chart kind; refuse before helm what helm refuses inside; the Azure grants a Provision needs - #3778
Merged
meshweaver-cloud[bot] merged 1 commit intoSep 9, 2026
Conversation
…fuses before helm what helm would refuse inside; the Azure grants a Provision needs Measured 2026-09-09 01:59Z, Deployments/memex-reconcile-20260909-adopt — adoption worked (adopted=1), then helm: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: failed to create resource: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:memex-ops:hosting-operator" cannot create resource "poddisruptionbudgets" core #3774 made the chart render a PodDisruptionBudget for every two-replica instance; the ClusterRole granted poddisruptionbudgets get+list (the audit block). Nothing paired the chart change with its grant. And Deployments/pearl-provision-20260909-b (the first Provision through the lane, az login now working): step 3/14 hosting-federate → AuthorizationFailed on Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/write over memexaks-portal-mi — the operator identity had no role on the portal identity. - operator-rbac.yaml: poddisruptionbudgets, scaledobjects, roles/rolebindings, endpoints create/update/patch/delete; jobs patch/update. ClusterRole/ClusterRoleBinding stay excluded. - test/check-chart-kinds-granted.sh: every `kind:` under deploy/helm/templates must be create/patch/delete-able by the role (16 missing on main's role; 0 now). Falls back to /opt/hosting/chart/templates in the image. Wired into run-tests.sh. - hosting-deploy: after adoption, `kubectl auth can-i <verb> <kind>` per rendered kind, ONE refusal naming every denial before helm; `helm status` pending-* refused by name (the rollback is a person's call). Stubs: auth can-i from denied.txt, helm status from status.json. - backups.bicep: Managed Identity Contributor on the portal identity (param portalIdentityId); dns-zone-operator-role.bicep (DNS Zone Contributor on the zone, its own resource group; param dnsZoneId on backups.bicep deploys it as a module). Both compile (az bicep build). No lane deploys bicep — README names the break-glass `az deployment group create`. - What's New (Fix). Suite: 157 passed, 0 failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Several new/updated comments and messages describe checking “kinds” but the runtime/test preflight actually checks kubectl resource strings (e.g., deployment.apps), which will be confusing when diagnosing denials.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens the AKS hosting operator’s deployment path by ensuring its Kubernetes RBAC covers every chart-rendered resource, adding a pre-Helm authorization preflight to fail fast with clear diagnostics, and extending Azure infrastructure grants required for first-time Provision runs.
Changes:
- Expand the operator ClusterRole to include write verbs for newly chart-rendered resources (e.g., PDBs, KEDA ScaledObjects, Roles/RoleBindings, Endpoints) and Job patch/update.
- Add a static “chart kinds granted” test plus runtime
kubectl auth can-ipreflight (and releasepending-*refusal) to prevent Helm upgrades from failing mid-run. - Extend infra docs and Bicep modules to grant the operator identity permissions needed for federated credentials and DNS record management.
File summaries
| File | Description |
|---|---|
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-09-the-operator-refuses-before-helm-what-helm-would-refuse-inside.md | Adds a What’s New entry describing the incident and the new preflight/ratchet behavior. |
| deploy/aks/operator/test/stubs/deploy/kubectl | Extends the kubectl stub to support auth can-i for new preflight tests. |
| deploy/aks/operator/test/stubs/deploy/helm | Extends the helm stub to support helm status used for pending-* refusal logic. |
| deploy/aks/operator/test/run-tests.sh | Adds tests for “deny before helm” preflight and release pending-state refusal; wires in the new chart-kind grant ratchet. |
| deploy/aks/operator/test/fixtures/deploy/status.json | Adds fixture data for helm status responses. |
| deploy/aks/operator/test/check-chart-kinds-granted.sh | New ratchet test ensuring every Helm template kind: is writable by the operator ClusterRole. |
| deploy/aks/operator/bin/hosting-deploy | Adds runtime RBAC preflight via kubectl auth can-i and release pending-state refusal before helm upgrade. |
| deploy/aks/manifests/hosting-operator/README.md | Documents the Azure role assignments required for Provision and how they’re applied (break-glass). |
| deploy/aks/manifests/hosting-operator/operator-rbac.yaml | Expands ClusterRole permissions to match chart-rendered resource set (including PDB/KEDA/RBAC/Endpoints and Job patch/update). |
| deploy/aks/infra/modules/dns-zone-operator-role.bicep | New module to assign DNS Zone Contributor at the DNS zone scope. |
| deploy/aks/infra/modules/backups.bicep | Adds optional grants for portal identity federation writes and DNS zone contributor wiring. |
Review details
- Files reviewed: 10/11 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+635
to
+651
| # ── hosting-deploy refuses BEFORE helm when the identity cannot write a rendered kind ─────────── | ||
| # Measured 2026-09-09 01:59Z on memex: helm died on `poddisruptionbudgets.policy is forbidden` | ||
| # and its --atomic rollback erred too. The preflight asks `kubectl auth can-i` per rendered kind | ||
| # and names every denial in ONE refusal, with helm never reached. | ||
| echo | ||
| echo "── hosting-deploy: writable-kinds preflight and release state ───" | ||
| _pf_dir="$(mktemp -d)"; cp -R "$DP_FIXTURES/." "$_pf_dir/"; _pf_log="$_pf_dir/calls.log"; : > "$_pf_log" | ||
| _pf_vals="$_pf_dir/values.yaml"; printf '# GENERATED from the Hosting/Deployment record by HelmValues\nreplicas:\n portal: 1\n' > "$_pf_vals" | ||
| printf 'create secretproviderclass.secrets-store.csi.x-k8s.io\npatch deployment.apps\n' > "$_pf_dir/denied.txt" | ||
| _pf_out="$(env PATH="$DP_STUBS:$PATH" HOSTING_CHART=/tmp HOSTING_DEPLOY_FIXTURE="$_pf_dir" HOSTING_DEPLOY_STUB_LOG="$_pf_log" \ | ||
| hosting-deploy --namespace memex --release memex --database memex --values "$_pf_vals" --image cr.example.test/memex-portal-ai:1 2>&1)"; _pf_rc=$? | ||
| if [ "$_pf_rc" -ne 0 ] && printf '%s' "$_pf_out" | grep -q 'denied:.*create:secretproviderclass.secrets-store.csi.x-k8s.io' \ | ||
| && printf '%s' "$_pf_out" | grep -q 'denied:.*patch:deployment.apps' && ! grep -q '^helm upgrade' "$_pf_log"; then | ||
| ok "every denied verb:kind is named in ONE refusal, and helm never runs" | ||
| else | ||
| bad "denied kinds are named before helm" "rc=${_pf_rc} out: ${_pf_out} log: $(cat "$_pf_log")" | ||
| fi |
Comment on lines
+33
to
+37
| - Before `helm upgrade`, the operator asks the cluster (`kubectl auth can-i`) about every kind the | ||
| record renders and refuses with the whole list in one line, so a missing grant is named once | ||
| instead of discovered one failed upgrade at a time. A release helm cannot upgrade — | ||
| `pending-upgrade` after a cut-off run — is refused by name too, with the rollback that resolves | ||
| it left to a person. |
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The role can write every chart kind; hosting-deploy refuses before helm what helm would refuse inside; the Azure grants a Provision needs
Measured 2026-09-09 01:59Z (
Deployments/memex-reconcile-20260909-adopt, operator 921bcdb, through the API): adoption worked (adopted=1, 15 owned, 2 to create), then helm:#3774 made the chart render a PodDisruptionBudget for every two-replica instance; the ClusterRole knew that kind from its read-only audit block only. Nothing paired a chart change with its grant — every Provision of a two-replica instance would have hit it. The
--atomicrollback landed as revision 42 (audit read-only: release readable, memex 200).And
Deployments/client-b-provision-20260909-b(the first Provision ever run through the lane, after #3776's sign-in): database created, vault secrets created/found, then step 3/14hosting-federate→AuthorizationFailed … federatedIdentityCredentials/write over …/memexaks-portal-mi/…— the operator identity had never been granted anything on the portal identity.What changes
operator-rbac.yaml:poddisruptionbudgets,scaledobjects,roles/rolebindings,endpointscreate/update/patch/delete;jobspatch/update (helm patches the migration Job on every upgrade). ClusterRole/ClusterRoleBinding stay excluded by design — a Job must never widen cluster-scoped RBAC.check-chart-kinds-granted.sh(bash 3.2 + grep/sed): everykind:underdeploy/helm/templatesmust be create/patch/delete-able by the role. Against main's role: 16 missing; now 0. Falls back to/opt/hosting/chart/templatesin the image. Wired intorun-tests.sh.hosting-deploy: after adoption,kubectl auth can-i <verb> <kind>per rendered kind — ONE refusal naming every denial, helm never reached (test: two denials named,helm upgradeabsent from the stub log).helm statuspending-*refused by name (the rollback revision is a person's call).backups.bicepgains Managed Identity Contributor on the portal identity (paramportalIdentityId, scoped to that one resource) and deploysdns-zone-operator-role.bicep(DNS Zone Contributor on the zone, in the zone's resource group) whendnsZoneIdis set. Both compile. No lane deploys bicep —manifests/hosting-operator/README.mdnames the break-glassaz deployment group createfor a person with Owner.Also found tonight, recorded for the next reader
memex's record carried no
portalNextblock, so the record-driven upgrade removed/nextbefore it failed on the PDB — that is the rollback's "no Service portal-next-service" line. The live record now carriesportalNext(v55); the repo record follows in Systemorph/Memex.