Repository navigation
fix(chart): let the runner read CRDs and common custom resources - #728
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new configuration option, runner.customResourceRead (defaulting to true), and adds a new Helm template runner-custom-resource-read.yaml. This template defines a ClusterRole and ClusterRoleBindings to grant the runner service account read-only access to custom resource definitions (CRDs), VictoriaMetrics resources, and Prometheus monitoring resources, alongside binding it to the built-in view ClusterRole. No review comments were provided, and I have no feedback to offer on these changes.
The CRDs tab lists the cluster's CRDs and, per CRD, its objects through get_resource. The runner had no read on customresourcedefinitions and almost no CR groups, so both lists failed with "forbidden" on every install. runner.customResourceRead (opt-in, default false; both modes) renders one file: - get/list/watch on customresourcedefinitions - a ClusterRoleBinding to the built-in `view` role (no Secrets by default), which covers built-ins and every operator that aggregates to view - named VictoriaMetrics and prometheus-operator kinds, which do not aggregate to view; vmusers and vmalertmanagers are left out because they can carry credentials inline The Secrets RBAC CI gate now also renders with the opt-in on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3a5fa29 to
367b55a
Compare
|
Follow-up suggestion: the VictoriaMetrics and prometheus-operator kinds are hardcoded in
Changing |
…-only Follow-up to #728. The VictoriaMetrics and prometheus-operator kinds were hardcoded in the template, so a CRD outside `view` (Traefik, for example) needed a chart release, and the only other route, runner.customClusterRoleRules, accepts any verb. runner.customResourceRead becomes a map (unreleased, so no shim): - enabled (default false): the opt-in; CRD list read - viewRole (default true): the built-in `view` binding - groups: API group -> resources, always get/list/watch. A map so an install adds a group without repeating the defaults (Helm merges maps, replaces lists) and drops one with `<group>: null`. The core group, "*" and "secrets" fail the render, since the CI Secrets gate only sees the chart's own values. The enabled render's rules are unchanged from #728. Suggested-by: mayankpande88 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks. #728 merged before this came in, so it's in a follow-up: #729. It does what you suggested (a map with |
…-only Follow-up to #728. The VictoriaMetrics and prometheus-operator kinds were hardcoded in the template, so a CRD outside `view` (Traefik, for example) needed a chart release, and the only other route, runner.customClusterRoleRules, accepts any verb. runner.customResourceRead becomes a map (unreleased, so no shim): - enabled (default false): the opt-in; CRD list read - viewRole (default true): the built-in `view` binding - groups: API group -> resources, always get/list/watch. A map so an install adds a group without repeating the defaults (Helm merges maps, replaces lists) and drops one with `<group>: null`. The core group, "*" and "secrets" fail the render, since the CI Secrets gate only sees the chart's own values. The enabled render's rules are unchanged from #728. Suggested-by: mayankpande88 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
The Nudgebee CRDs tab lists a cluster's CRDs and, for each one, its objects, read through the runner. The runner's service account could do neither, so on every install the tab showed a "forbidden" error. This adds an opt-in,
runner.customResourceRead, that lets the runner read them without any Secrets access.Decision: the runner gets the built-in
viewrole, plus named VictoriaMetrics and prometheus-operator kinds. A wildcard read was the alternative; it would cover every CRD, but RBAC cannot exclude Secrets from a wildcard, which would undo #714. Cost: CRDs whose operator doesn't add itself toview(Traefik, for example) still show "forbidden" until they are added withrunner.customClusterRoleRules.Off by default: an upgrade changes no install's permissions. The tab stays "not allowed" until the value is set to
true.Look at:
runner-custom-resource-read.yaml, where the exclusions are listed.Type of change
Chart version
versionis 0.1.33 on main, and 0.1.33 is not released yet (only an rc)Test plan
--set runner.customResourceRead=true, then open the cluster in Nudgebee → Apps & Infra → CRDs. The list loads. Before, it said "Could not read CRDs … forbidden".helm lint charts/nudgebee-agentpassesct lint(left to CI)helm templateoutput reviewed: nothing by default; three objects withrunner.customResourceRead=true, in standard and read-only modenudgebee-agent-dev)Risks
viewis an aggregated role: any operator installed later that adds itself toviewwidens what the runner can read, with no chart change. Kubernetes keeps Secrets out ofview. A customer who needs a fixed, auditable grant can leave the opt-in off and grant throughrunner.customClusterRoleRules.viewgrants.Engineering detail
Root cause. Neither runner ClusterRole has ever granted
customresourcedefinitions(git log -S customresourcedefinitions -- chartsshows only the forwarder role). The CRDs tab (nudgebee-enterprise #42932) callsget_resourcewithapiextensions.k8s.io/v1/customresourcedefinitions, all_namespaces: true, then the same call per CRD with that CRD's group and plural.Why named kinds and not
*for the two operators.vmuserscan carrypassword/bearerTokeninline, andvmalertmanagerscan embed receiver credentials inconfigRawYaml. The runner's Secret policy (pkg/kube/secretpolicy.go) refuses only the Secret-type kinds, so these two have to stay out at the RBAC level. The named lists match the kinds served on GKE dev today; kinds an operator adds later need a chart update.Verification on GKE dev. I applied the rendered file to
nudgebee-agent-devwith Helm ownership metadata, so the next upgrade adopts the objects. Results ofkubectl auth can-i list … --as=<runner SA>:view)In the UI, the CRDs tab loaded 256 CRDs, and VMAgent → Resources listed
victoria/victoria-victoria-metrics-k8s-stackwith its printer columns. On that cluster,viewcovers 20 of the 46 CRD groups.Checks run locally.
helm lintpasses. The CI Secrets gate script, run against the default,enableWritePermissions,readOnlyandhelmReleaseDiscoveryrenders, gives the same output as on main.kubectl apply --dry-run=serveron GKE dev passes.Not in this PR. Each install needs a
helm upgradeto pick this up. The tab's error message, which points atrunner.customClusterRoleRules, is nudgebee/nudgebee-enterprise#44236.🤖 Generated with Claude Code