Skip to content

fix(chart): let the runner read CRDs and common custom resources - #728

Merged
blue4209211 merged 1 commit into
mainfrom
fix/runner-read-crds
Nov 13, 2026
Merged

blue4209211 merged 1 commit into
mainfrom
fix/runner-read-crds

Conversation

@RamanKharchee

@RamanKharchee RamanKharchee commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Nudgebee CRDs tab lists a cluster's CRDs and, for each one, its objects, read through the runner. The runner's service account could do neither, so on every install the tab showed a "forbidden" error. This adds an opt-in, runner.customResourceRead, that lets the runner read them without any Secrets access.

Decision: the runner gets the built-in view role, plus named VictoriaMetrics and prometheus-operator kinds. A wildcard read was the alternative; it would cover every CRD, but RBAC cannot exclude Secrets from a wildcard, which would undo #714. Cost: CRDs whose operator doesn't add itself to view (Traefik, for example) still show "forbidden" until they are added with runner.customClusterRoleRules.

Off by default: an upgrade changes no install's permissions. The tab stays "not allowed" until the value is set to true.

Look at: runner-custom-resource-read.yaml, where the exclusions are listed.

Type of change

  • Bug fix (non-breaking)

Chart version

  • No version bump needed: version is 0.1.33 on main, and 0.1.33 is not released yet (only an rc)

Test plan

  1. Upgrade an install to this chart with --set runner.customResourceRead=true, then open the cluster in Nudgebee → Apps & Infra → CRDs. The list loads. Before, it said "Could not read CRDs … forbidden".
  2. Expand a VictoriaMetrics row such as VMAgent → Resources. Its objects are listed. Before, it showed the same forbidden banner.
  3. Expand a VMUser row → Resources. It still says forbidden; that kind is deliberately excluded.
  • helm lint charts/nudgebee-agent passes
  • ct lint (left to CI)
  • helm template output reviewed: nothing by default; three objects with runner.customResourceRead=true, in standard and read-only mode
  • Verified on a real cluster (GKE dev, release nudgebee-agent-dev)

Risks

  • view is an aggregated role: any operator installed later that adds itself to view widens what the runner can read, with no chart change. Kubernetes keeps Secrets out of view. A customer who needs a fixed, auditable grant can leave the opt-in off and grant through runner.customClusterRoleRules.
  • The CI Secrets gate now also renders with the opt-in on, but it checks only rules in roles the chart renders, not bindings, so it does not see what view grants.
Engineering detail

Root cause. Neither runner ClusterRole has ever granted customresourcedefinitions (git log -S customresourcedefinitions -- charts shows only the forwarder role). The CRDs tab (nudgebee-enterprise #42932) calls get_resource with apiextensions.k8s.io/v1/customresourcedefinitions, all_namespaces: true, then the same call per CRD with that CRD's group and plural.

Why named kinds and not * for the two operators. vmusers can carry password / bearerToken inline, and vmalertmanagers can embed receiver credentials in configRawYaml. The runner's Secret policy (pkg/kube/secretpolicy.go) refuses only the Secret-type kinds, so these two have to stay out at the RBAC level. The named lists match the kinds served on GKE dev today; kinds an operator adds later need a chart update.

Verification on GKE dev. I applied the rendered file to nudgebee-agent-dev with Helm ownership metadata, so the next upgrade adopts the objects. Results of kubectl auth can-i list … --as=<runner SA>:

resource before after
customresourcedefinitions no yes
vmanomalies.operator.victoriametrics.com no yes
alertmanagers.monitoring.coreos.com no yes
certificates.cert-manager.io (via view) no yes
vmusers / vmalertmanagers no no
secrets no no
ingressroutes.traefik.io no no

In the UI, the CRDs tab loaded 256 CRDs, and VMAgent → Resources listed victoria/victoria-victoria-metrics-k8s-stack with its printer columns. On that cluster, view covers 20 of the 46 CRD groups.

Checks run locally. helm lint passes. The CI Secrets gate script, run against the default, enableWritePermissions, readOnly and helmReleaseDiscovery renders, gives the same output as on main. kubectl apply --dry-run=server on GKE dev passes.

Not in this PR. Each install needs a helm upgrade to pick this up. The tab's error message, which points at runner.customClusterRoleRules, is nudgebee/nudgebee-enterprise#44236.

🤖 Generated with Claude Code

@RamanKharchee
RamanKharchee requested a review from a team as a code owner November 13, 2026 15:06

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new configuration option, runner.customResourceRead (defaulting to true), and adds a new Helm template runner-custom-resource-read.yaml. This template defines a ClusterRole and ClusterRoleBindings to grant the runner service account read-only access to custom resource definitions (CRDs), VictoriaMetrics resources, and Prometheus monitoring resources, alongside binding it to the built-in view ClusterRole. No review comments were provided, and I have no feedback to offer on these changes.

The CRDs tab lists the cluster's CRDs and, per CRD, its objects through
get_resource. The runner had no read on customresourcedefinitions and
almost no CR groups, so both lists failed with "forbidden" on every
install.

runner.customResourceRead (opt-in, default false; both modes) renders one
file:
- get/list/watch on customresourcedefinitions
- a ClusterRoleBinding to the built-in `view` role (no Secrets by
  default), which covers built-ins and every operator that aggregates
  to view
- named VictoriaMetrics and prometheus-operator kinds, which do not
  aggregate to view; vmusers and vmalertmanagers are left out because
  they can carry credentials inline

The Secrets RBAC CI gate now also renders with the opt-in on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blue4209211
blue4209211 added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 5567067 Nov 13, 2026
5 checks passed
@blue4209211
blue4209211 deleted the fix/runner-read-crds branch November 13, 2026 15:22
@mayankpande88

Copy link
Copy Markdown
Contributor

Follow-up suggestion: the VictoriaMetrics and prometheus-operator kinds are hardcoded in runner-custom-resource-read.yaml, so adding a CRD that isn't in view (Traefik, for example) needs a chart release. The other option, runner.customClusterRoleRules, accepts any verb. We could move the hardcoded kinds into values and keep them read-only:

runner:
  customResourceRead:
    enabled: false
    viewRole: true          # bind the built-in `view` ClusterRole
    groups:
      operator.victoriametrics.com: [vmagents, vmalerts, ...]   # vmusers / vmalertmanagers stay out
      monitoring.coreos.com: [prometheuses, servicemonitors, ...]
  • The template renders customresourcedefinitions plus each group's resources with get/list/watch only, so values can't widen the verbs.
  • A map, not a list: Helm replaces lists, so adding one group to a list would mean repeating every default. Helm merges maps, so --set-json 'runner.customResourceRead.groups={"traefik.io":["ingressroutes"]}' keeps the defaults, and groupname: null drops one.
  • Guard in the template: fail on the core group "", on secrets, and on "*". The CI Secrets gate renders only the chart's own values, so it won't see what an install puts there.

Changing customResourceRead from a bool to a map is a breaking value change. Because 0.1.33 isn't released yet, it can go in before the release without a compatibility shim.

RamanKharchee pushed a commit that referenced this pull request Nov 13, 2026
…-only

Follow-up to #728. The VictoriaMetrics and prometheus-operator kinds
were hardcoded in the template, so a CRD outside `view` (Traefik, for
example) needed a chart release, and the only other route,
runner.customClusterRoleRules, accepts any verb.

runner.customResourceRead becomes a map (unreleased, so no shim):
- enabled (default false): the opt-in; CRD list read
- viewRole (default true): the built-in `view` binding
- groups: API group -> resources, always get/list/watch. A map so an
  install adds a group without repeating the defaults (Helm merges maps,
  replaces lists) and drops one with `<group>: null`. The core group,
  "*" and "secrets" fail the render, since the CI Secrets gate only
  sees the chart's own values.

The enabled render's rules are unchanged from #728.

Suggested-by: mayankpande88
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RamanKharchee

RamanKharchee commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks. #728 merged before this came in, so it's in a follow-up: #729. It does what you suggested (a map with enabled / viewRole / groups, always get/list/watch, and "", "*" and secrets fail the render). I checked the merge behaviour with real renders: --set-json adds a group and keeps the defaults, and null drops one. One small correction: the bool was added in #728 and hasn't shipped in any release or rc, so this doesn't break anything for users. The old runner.customResourceRead=true now fails with a message naming .enabled. Without that guard Helm failed with only a template position. The UI banner text is updated to match in nudgebee/nudgebee-enterprise#44237.

RamanKharchee pushed a commit that referenced this pull request Nov 13, 2026
…-only

Follow-up to #728. The VictoriaMetrics and prometheus-operator kinds
were hardcoded in the template, so a CRD outside `view` (Traefik, for
example) needed a chart release, and the only other route,
runner.customClusterRoleRules, accepts any verb.

runner.customResourceRead becomes a map (unreleased, so no shim):
- enabled (default false): the opt-in; CRD list read
- viewRole (default true): the built-in `view` binding
- groups: API group -> resources, always get/list/watch. A map so an
  install adds a group without repeating the defaults (Helm merges maps,
  replaces lists) and drops one with `<group>: null`. The core group,
  "*" and "secrets" fail the render, since the CI Secrets gate only
  sees the chart's own values.

The enabled render's rules are unchanged from #728.

Suggested-by: mayankpande88
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants