feat(k8s-reporter): pod scheduling controls to avoid blocking Karpenter consolidation

charts/k8s-reporter/Chart.yaml

@@ -15,7 +15,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 2.2.1
+version: 2.3.0
 
 # This is the version number of the (CLI) application being deployed. This version number should be
 # incremented each time you make changes to the application. They should reflect the version the

charts/k8s-reporter/MINTLIFY.md.gotmpl

@@ -20,6 +20,8 @@ description: A Helm chart for installing the Kosli K8S reporter as a cronjob.
 
 {{ template "mintlify.customCA" . }}
 
+{{ template "mintlify.karpenter" . }}
+
 {{ template "mintlify.configurations" . }}
 
 ---

charts/k8s-reporter/README.md

@@ -4,7 +4,7 @@ title: Kubernetes Reporter Helm Chart
 
 # k8s-reporter
 
-![Version: 2.2.1](https://img.shields.io/badge/Version-2.2.1-informational?style=flat-square)
+![Version: 2.3.0](https://img.shields.io/badge/Version-2.3.0-informational?style=flat-square)
 
 A Helm chart for installing the Kosli K8S reporter as a CronJob.
 The chart allows you to create a Kubernetes cronjob and all its necessary RBAC to report running images to Kosli at a given cron schedule.
@@ -149,9 +149,59 @@ Both options use `secret`-backed volumes, which are permitted under the Pod Secu
 
 If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute a corporate CA bundle into a well-known ConfigMap in every namespace, point `extraVolumes` / `extraVolumeMounts` at that ConfigMap instead of creating a per-namespace Secret.
 
+## Running on EKS with Karpenter (or another node autoscaler)
+
+By default the reporter runs as a CronJob every 5 minutes. On clusters that use [Karpenter](https://karpenter.sh) for node autoscaling, this frequent scheduling can prevent nodes from being **consolidated** (scaled down).
+
+The cause is Karpenter's `consolidateAfter` timer: Karpenter only consolidates a node once it has seen no pod scheduling activity on it for the configured window. A reporter pod arriving every 5 minutes keeps resetting that timer, so any node whose `consolidateAfter` is longer than the reporter interval never becomes eligible for consolidation (see [karpenter#1921](https://github.com/kubernetes-sigs/karpenter/issues/1921)). This is Karpenter working as designed, not a reporter bug.
+
+There are three good ways to avoid it, in order of preference.
+
+### 1. Widen the report interval
+
+The simplest fix. Set `cronSchedule` longer than your NodePool's `consolidateAfter` so nodes get quiet windows long enough to consolidate. Environment snapshots rarely need 5-minute freshness.
+
+```yaml
+cronSchedule: "*/15 * * * *"
+```
+
+### 2. Pin the reporter to a stable node group
+
+If you run a stable managed node group that Karpenter does not manage, schedule the reporter there so it never disturbs Karpenter-managed nodes. Use `nodeSelector`, and `tolerations` if that node group is tainted:
+
+```yaml
+nodeSelector:
+  eks.amazonaws.com/nodegroup: system   # your managed node group
+
+tolerations:
+  - key: dedicated
+    operator: Equal
+    value: system
+    effect: NoSchedule
+```
+
+To steer the reporter *away* from Karpenter-managed nodes instead, use `affinity` (a plain `nodeSelector` cannot express "not on these nodes"):
+
+```yaml
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: karpenter.sh/nodepool
+              operator: DoesNotExist
+```
+
+### 3. Run the reporter out of the cluster
+
+For zero footprint on cluster nodes, run `kosli snapshot k8s` on a schedule outside the cluster (for example a CI cron job) with kubeconfig access. See the [Kubernetes environment reporting tutorial](https://docs.kosli.com/tutorials/report_k8s_envs/).
+
+> Note: `karpenter.sh/do-not-disrupt: "true"` is **not** a fix here. It prevents Karpenter from disrupting the pod, which protects a mid-run report from interruption but makes consolidation of that node *less* likely, not more. Likewise `cluster-autoscaler.kubernetes.io/safe-to-evict` only affects the Kubernetes Cluster Autoscaler and is ignored by Karpenter.
+
 ## Configurations
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
+| affinity | object | `{}` | affinity rules for scheduling the reporter pod. Supports nodeAffinity, podAffinity and podAntiAffinity. |
 | concurrencyPolicy | string | `"Replace"` | specifies how to treat concurrent executions of a Job that is created by this CronJob |
 | cronSchedule | string | `"*/5 * * * *"` | the cron schedule at which the reporter is triggered to report to Kosli |
 | customCA | object | `{"enabled":false,"key":"ca.crt","secretName":""}` | convenience wrapper for mounting a corporate / custom CA bundle. See the "Running behind a TLS-inspecting proxy" section of the README for usage. |
@@ -170,8 +220,10 @@ If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/t
 | kosliApiToken.secretKey | string | `"key"` | the name of the key in the secret data which contains the Kosli API token |
 | kosliApiToken.secretName | string | `"kosli-api-token"` | the name of the secret containing the kosli API token |
 | nameOverride | string | `""` | overrides the name used for the created k8s resources. If `fullnameOverride` is provided, it has higher precedence than this one |
-| podAnnotations | object | `{}` | any custom annotations to be added to the cronjob |
+| nodeSelector | object | `{}` | node labels for scheduling the reporter pod. On EKS with Karpenter, use this to pin the reporter to a stable managed node group (e.g. `eks.amazonaws.com/nodegroup: <name>`) so it does not interfere with node consolidation. See the "Running on EKS with Karpenter" section of the README. |
+| podAnnotations | object | `{}` | annotations to add to the CronJob object itself. For pod-level annotations (added to each reporter pod), use `podTemplateAnnotations` instead. |
 | podLabels | object | `{}` | custom labels to add to pods |
+| podTemplateAnnotations | object | `{}` | annotations to add to the reporter pod template (applied to each Job pod that the CronJob creates) |
 | reporterConfig.dryRun | bool | `false` | whether the dry run mode is enabled or not. In dry run mode, the reporter logs the reports to stdout and does not send them to kosli. |
 | reporterConfig.environments | list | `[]` | List of Kosli environments to report to. Each entry has required 'name' and optional namespace selectors. Use one entry to report a single environment; use multiple entries to report to multiple environments with different selectors. Per entry: name (required), namespaces, namespacesRegex, excludeNamespaces, excludeNamespacesRegex (optional). Leave namespace fields unset for an entry to report the entire cluster to that environment. |
 | reporterConfig.httpProxy | string | `""` | the http proxy url |
@@ -188,6 +240,7 @@ If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/t
 | serviceAccount.name | string | `""` | the name of the service account to use. If not set and create is true, a name is generated using the fullname template |
 | serviceAccount.permissionScope | string | `"cluster"` | specifies whether to create a cluster-wide permissions for the service account or namespace-scoped permissions. allowed values are: [cluster, namespace] |
 | successfulJobsHistoryLimit | int | `3` | specifies the number of successful finished jobs to keep |
+| tolerations | list | `[]` | tolerations for scheduling the reporter pod, e.g. to run on a dedicated or tainted node group. |
 
 ----------------------------------------------
 Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)

charts/k8s-reporter/README.md.gotmpl

@@ -19,6 +19,8 @@ title: Kubernetes Reporter Helm Chart
 
 {{ template "extra.customCA" . }}
 
+{{ template "extra.karpenter" . }}
+
 {{ template "extra.valuesHeader" . }}
 {{ template "chart.valuesTable" . }}
 

charts/k8s-reporter/_mintlify_templates.gotmpl

@@ -123,6 +123,59 @@ Both options use `secret`-backed volumes, which are permitted under the Pod Secu
 If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute a corporate CA bundle into a well-known ConfigMap in every namespace, point `extraVolumes` / `extraVolumeMounts` at that ConfigMap instead of creating a per-namespace Secret.
 {{- end }}
 
+{{ define "mintlify.karpenter" -}}
+## Running on EKS with Karpenter (or another node autoscaler)
+
+By default the reporter runs as a CronJob every 5 minutes. On clusters that use [Karpenter](https://karpenter.sh) for node autoscaling, this frequent scheduling can prevent nodes from being **consolidated** (scaled down).
+
+The cause is Karpenter's `consolidateAfter` timer: Karpenter only consolidates a node once it has seen no pod scheduling activity on it for the configured window. A reporter pod arriving every 5 minutes keeps resetting that timer, so any node whose `consolidateAfter` is longer than the reporter interval never becomes eligible for consolidation (see [karpenter#1921](https://github.com/kubernetes-sigs/karpenter/issues/1921)). This is Karpenter working as designed, not a reporter bug.
+
+There are three good ways to avoid it, in order of preference.
+
+### 1. Widen the report interval
+
+The simplest fix. Set `cronSchedule` longer than your NodePool's `consolidateAfter` so nodes get quiet windows long enough to consolidate. Environment snapshots rarely need 5-minute freshness.
+
+```yaml
+cronSchedule: "*/15 * * * *"
+```
+
+### 2. Pin the reporter to a stable node group
+
+If you run a stable managed node group that Karpenter does not manage, schedule the reporter there so it never disturbs Karpenter-managed nodes. Use `nodeSelector`, and `tolerations` if that node group is tainted:
+
+```yaml
+nodeSelector:
+  eks.amazonaws.com/nodegroup: system   # your managed node group
+
+tolerations:
+  - key: dedicated
+    operator: Equal
+    value: system
+    effect: NoSchedule
+```
+
+To steer the reporter away from Karpenter-managed nodes instead, use `affinity` (a plain `nodeSelector` cannot express "not on these nodes"):
+
+```yaml
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: karpenter.sh/nodepool
+              operator: DoesNotExist
+```
+
+### 3. Run the reporter out of the cluster
+
+For zero footprint on cluster nodes, run `kosli snapshot k8s` on a schedule outside the cluster (for example a CI cron job) with kubeconfig access. See the [Kubernetes environment reporting tutorial](/tutorials/report_k8s_envs).
+
+<Warning>
+`karpenter.sh/do-not-disrupt: "true"` is **not** a fix here. It prevents Karpenter from disrupting the pod, which protects a mid-run report from interruption but makes consolidation of that node *less* likely, not more. Likewise `cluster-autoscaler.kubernetes.io/safe-to-evict` only affects the Kubernetes Cluster Autoscaler and is ignored by Karpenter.
+</Warning>
+{{- end }}
+
 {{ define "mintlify.versionInfo" -}}
 <Info>
 This reference applies to **chart version {{ .Version }}**, which defaults to CLI **{{ .AppVersion }}** via `appVersion`. Override with `image.tag`.

charts/k8s-reporter/_templates.gotmpl

@@ -153,6 +153,57 @@ Both options use `secret`-backed volumes, which are permitted under the Pod Secu
 If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute a corporate CA bundle into a well-known ConfigMap in every namespace, point `extraVolumes` / `extraVolumeMounts` at that ConfigMap instead of creating a per-namespace Secret.
 {{- end }}
 
+{{ define "extra.karpenter" -}}
+## Running on EKS with Karpenter (or another node autoscaler)
+
+By default the reporter runs as a CronJob every 5 minutes. On clusters that use [Karpenter](https://karpenter.sh) for node autoscaling, this frequent scheduling can prevent nodes from being **consolidated** (scaled down).
+
+The cause is Karpenter's `consolidateAfter` timer: Karpenter only consolidates a node once it has seen no pod scheduling activity on it for the configured window. A reporter pod arriving every 5 minutes keeps resetting that timer, so any node whose `consolidateAfter` is longer than the reporter interval never becomes eligible for consolidation (see [karpenter#1921](https://github.com/kubernetes-sigs/karpenter/issues/1921)). This is Karpenter working as designed, not a reporter bug.
+
+There are three good ways to avoid it, in order of preference.
+
+### 1. Widen the report interval
+
+The simplest fix. Set `cronSchedule` longer than your NodePool's `consolidateAfter` so nodes get quiet windows long enough to consolidate. Environment snapshots rarely need 5-minute freshness.
+
+```yaml
+cronSchedule: "*/15 * * * *"
+```
+
+### 2. Pin the reporter to a stable node group
+
+If you run a stable managed node group that Karpenter does not manage, schedule the reporter there so it never disturbs Karpenter-managed nodes. Use `nodeSelector`, and `tolerations` if that node group is tainted:
+
+```yaml
+nodeSelector:
+  eks.amazonaws.com/nodegroup: system   # your managed node group
+
+tolerations:
+  - key: dedicated
+    operator: Equal
+    value: system
+    effect: NoSchedule
+```
+
+To steer the reporter *away* from Karpenter-managed nodes instead, use `affinity` (a plain `nodeSelector` cannot express "not on these nodes"):
+
+```yaml
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: karpenter.sh/nodepool
+              operator: DoesNotExist
+```
+
+### 3. Run the reporter out of the cluster
+
+For zero footprint on cluster nodes, run `kosli snapshot k8s` on a schedule outside the cluster (for example a CI cron job) with kubeconfig access. See the [Kubernetes environment reporting tutorial](https://docs.kosli.com/tutorials/report_k8s_envs/).
+
+> Note: `karpenter.sh/do-not-disrupt: "true"` is **not** a fix here. It prevents Karpenter from disrupting the pod, which protects a mid-run report from interruption but makes consolidation of that node *less* likely, not more. Likewise `cluster-autoscaler.kubernetes.io/safe-to-evict` only affects the Kubernetes Cluster Autoscaler and is ignored by Karpenter.
+{{- end }}
+
 {{ define "extra.valuesHeader" -}}
 ## Configurations
 {{- end }}

charts/k8s-reporter/templates/cronjob.yaml

@@ -3,7 +3,10 @@ apiVersion: batch/v1
 kind: CronJob
 metadata:
   name: {{ include "reporter.fullname" . }}
-  annotations: {{ toYaml .Values.podAnnotations }}
+  {{- with .Values.podAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   labels:
     {{- include "reporter.labels" . | nindent 4 }}
 
@@ -17,12 +20,28 @@ spec:
     spec:
       template:
         metadata:
+          {{- with .Values.podTemplateAnnotations }}
+          annotations:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
           labels:
             {{- range $key, $value := .Values.podLabels }}
               {{ $key }}: {{ $value | quote }}
             {{- end }}
         spec:
           serviceAccountName: {{ include "reporter.serviceAccountName" . }}
+          {{- with .Values.nodeSelector }}
+          nodeSelector:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          {{- with .Values.affinity }}
+          affinity:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          {{- with .Values.tolerations }}
+          tolerations:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
           volumes:
           - name: environments-config
             configMap:

charts/k8s-reporter/values.yaml

@@ -133,9 +133,21 @@ customCA:
   # -- key within the Secret that holds the PEM-formatted CA certificate (single cert or multi-cert PEM bundle)
   key: "ca.crt"
 
-# -- any custom annotations to be added to the cronjob
+# -- annotations to add to the CronJob object itself. For pod-level annotations (added to each reporter pod), use `podTemplateAnnotations` instead.
 podAnnotations: {}
 
+# -- annotations to add to the reporter pod template (applied to each Job pod that the CronJob creates)
+podTemplateAnnotations: {}
+
+# -- node labels for scheduling the reporter pod. On EKS with Karpenter, use this to pin the reporter to a stable managed node group (e.g. `eks.amazonaws.com/nodegroup: <name>`) so it does not interfere with node consolidation. See the "Running on EKS with Karpenter" section of the README.
+nodeSelector: {}
+
+# -- tolerations for scheduling the reporter pod, e.g. to run on a dedicated or tainted node group.
+tolerations: []
+
+# -- affinity rules for scheduling the reporter pod. Supports nodeAffinity, podAffinity and podAntiAffinity.
+affinity: {}
+
 resources:
   requests:
     # -- the memory request