Skip to content

fix(operator): scope owned namespaced resource caches to KAI namespace - v0.18 - #2707

Open
NVGreg wants to merge 1 commit into
kai-scheduler:v0.18from
NVGreg:backport-2676-to-v0.18
Open

NVGreg wants to merge 1 commit into
kai-scheduler:v0.18from
NVGreg:backport-2676-to-v0.18

Conversation

@NVGreg

@NVGreg NVGreg commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Backport of #2676 to v0.18.

Problems

  • On v0.18, cmd/operator/app/app.go scopes only Pod, Lease, and EndpointSlice informers to opts.Namespace, leaving ConfigMap, Secret, Deployment, DaemonSet, Service, and PodDisruptionBudget cached across all namespaces, and cmd/operator/config/options.go defaults --namespace to "runai-engine".

Changes

Things to note

  • Scopes DefaultNamespaces to opts.Namespace ("kai-scheduler" by default) with a single ByObject exemption for ServiceAccount (cache.AllNamespaces).

The operator indexes and watches namespaced resources (including
Secret, ConfigMap, Deployment, DaemonSet, Service, PodDisruptionBudget,
Pod, Lease, EndpointSlice, Prometheus, ServiceMonitor, and
VerticalPodAutoscaler) through controller-runtime's shared cache to
reconcile KAIConfig and SchedulingShard operands. Previously only Pod,
Lease, and EndpointSlice were restricted to the KAI namespace in
cache.Options.ByObject, causing the operator to list and watch
cluster-wide Secrets (including Helm release secrets) and other
namespaced objects across all namespaces.

Configure DefaultNamespaces in NewCacheOptions so every namespaced
resource is scoped to the KAI namespace by default (falling back to
constants.DefaultKAINamespace when empty), while cluster-scoped
resources continue to use the cluster cache. Exempt only ServiceAccount
in ByObject with cache.AllNamespaces because the binder and
node-scale-adjuster operands also reconcile ServiceAccounts in their
dedicated resource-reservation and scale-adjust namespaces.

Also update the operator --namespace flag default from "runai-engine"
to constants.DefaultKAINamespace ("kai-scheduler") to match
kaiv1.ConfigSpec.SetDefaultsWhereNeeded() and the scheduler options.

(cherry picked from commit 57db87d)
Signed-off-by: Gregory Giecold <ggiecold@nvidia.com>
@davidLif davidLif changed the title fix(operator): scope owned namespaced resource caches to KAI namespace fix(operator): scope owned namespaced resource caches to KAI namespace - v0.18 Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants