priority queue metrics cause unbounded memory growth and scrape timeouts

[chenghu2]

Summary

When using controller-runtime's priority queue (UsePriorityQueue: true), the workqueue_depth metric can cause unbounded memory growth and extremely slow metrics serialization times, leading to Prometheus scrape timeouts.

Problem

The workqueue_depth metric includes a priority label that creates a new metric time series for each unique priority value:

workqueue_depth{name="my-controller", controller="my-controller", priority="0"} 0
workqueue_depth{name="my-controller", controller="my-controller", priority="1"} 0
workqueue_depth{name="my-controller", controller="my-controller", priority="12345"} 0

The Prometheus client library never automatically cleans up these metric entries, even after items are dequeued. If an application uses incrementing priority values (e.g., for LIFO ordering), each enqueue creates a persistent metric entry.

Impact

In a real-world scenario with ~70 controllers using incrementing priorities:

• 810K+ unique metric entries accumulated over time
• 15+ second metrics serialization time (exceeds typical 10s scrape timeout)
• Prometheus scrape failures with "broken pipe" errors

Root Cause

The metric is defined in pkg/internal/metrics/workqueue.go:

var (
    depth = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Subsystem: WorkQueueSubsystem,
        Name:      DepthKey,
        Help:      "Current depth of workqueue by workqueue and priority",
    }, []string{"name", "controller", "priority"})  // <-- priority label

And used in depthWithPriorityMetric:

func (g *depthWithPriorityMetric) Inc(priority int) {
    depth.WithLabelValues(append(g.lvs, strconv.Itoa(priority))...).Inc()
}

Each unique priority integer becomes a distinct label value, creating a new persistent metric entry.

Workaround

Applications using custom priorities should bound their priority values to a small range (e.g., 0-100) to limit metric cardinality:

const maxPriority = 100
var priorityCounter atomic.Int32

func enqueue(item T) {
    priority := priorityCounter.Add(1)
    if priority >= maxPriority {
        priorityCounter.Store(0)
    }
    queue.AddWithOpts(priorityqueue.AddOpts{Priority: ptr.To(int(priority))}, item)
}

Potential Solutions

1. Document the cardinality risk - Add warnings to priority queue documentation about metric cardinality when using custom priorities

2. Make priority label optional - Add configuration to disable the priority label for users who don't need per-priority observability

Environment

• controller-runtime version: v0.22.2
• Go version: 1.24.x
• Kubernetes version: 1.31.x

[alvaroaleman]

As an idea, an alternate to making the priority label optional could be to make the implementation smarter and only emit the priority label if the cardinality for that doesn't exceed some threshold. The main drawbacks of that is that in the beginning of the process, the numbers would always be emitted until the threshold is hit which may cause confusion and that it might generally appear a bit "magic".

[alvaroaleman]

As a stopgap workaround, you could override the default metric provider with a wrapper that does not implement the MetricsProviderWithPriority interface

[chenghu2]

As a stopgap workaround, you could override the default metric provider with a wrapper that does not implement the MetricsProviderWithPriority interface

I tried implementing this fix of calling workqueue.SetProvider() before controller-runtime's init() runs, but the approach is more complicated for the priority queue.

The priority queue does not use the global workqueue.SetProvider() but

https://github.com/kubernetes-sigs/controller-runtime/blob/v0.22.2/pkg/controller/priorityqueue/priorityqueue.go#L63-L65

if opts.MetricProvider == nil {
    opts.MetricProvider = metrics.WorkqueueMetricsProvider{}
}

The priority queue has its own MetricProvider option that defaults directly to metrics.WorkqueueMetricsProvider{}, bypassing the global provider set via workqueue.SetProvider().

To workaround that, we would need to

1. Create a custom NewQueue function that wraps priorityqueue.New() with a custom MetricProvider
2. Pass this function to every controller via controller.Options.NewQueue

at this point it gets too complicated so I would probably just stick with the simpler workaround for now