Expose Prometheus histograms in both classic and native form #3817

Summary

Our Prometheus histograms are almost all classic histograms, most of them with prometheus.DefBuckets, and two services still use Summaries.

Prometheus native histograms have been stable since Prometheus v3.8. They give much better resolution with far fewer time series, and need no bucket tuning.

Because self-hosters scrape OpenCloud with whatever Prometheus setup they already run, we should not switch over hard.

Instead, expose both the classic and the native form of every histogram (dual exposition), which client_golang supports out of the box.

Nothing changes for setups that don't have native histograms enabled: they keep ingesting the classic histograms. Setups that have native histograms enabled (scrape_native_histograms: true) ingest the native form instead, and stop getting the classic _bucket, _sum and _count series unless they also set always_scrape_classic_histograms: true, so their dashboards and alerts need to be migrated or that setting enabled.

Low priority: nothing is broken.

Current state

Notes:

• DefBuckets stops at 10s. Anything slower, for example a slow LDAP call or backend request, lands in +Inf.
• Summaries compute quantiles on each instance, and those quantiles can't be combined across replicas.

Native vs. classic histograms

Native histograms

Pros:

• No bucket tuning: sparse exponential buckets are created only where values actually land. A single growth factor (NativeHistogramBucketFactor: 1.1, about 10% relative error) replaces hand-picked boundaries.
• Much better resolution for about the same storage. histogram_quantile is much more accurate than interpolating between around 11 coarse buckets.
• Far fewer time series: 1 series per label set instead of one per bucket plus _sum and _count (14 with DefBuckets). That matters for HistogramVecs with several labels (method × service × result, …).
• Can be combined across instances, even when they use different resolutions.
• Better PromQL: histogram_quantile(0.99, rate(x[5m])) without le/sum by (le), plus histogram_count, histogram_sum, histogram_avg, histogram_fraction and histogram_stddev.
• Maps onto OpenTelemetry exponential histograms, should we move to OTel instrumentation later.

Cons:

• The server and everything downstream must support them. They are only transferred over the protobuf scrape format and must be enabled explicitly (see the next section). Remote-write targets and long-term storage (Thanos, Mimir, Cortex, VictoriaMetrics, …) also need to support them, and support there varies by product and version.
• Dashboards and alerts break on a hard switch. _bucket{le=…} series disappear, so existing rate(x_bucket[…]) queries, recording rules and Grafana panels stop working.
• No exact thresholds: bucket boundaries are fixed by the schema, so an SLO like "share of requests under 300ms" is interpolated by histogram_fraction instead of counted exactly.
• Slightly more memory on the client while values are spread widely. This can be capped with NativeHistogramMaxBucketNumber and NativeHistogramMinResetDuration.
• Less familiar: fewer ready-made dashboards and less community knowledge.

Classic histograms

Pros: supported everywhere (any Prometheus version, any remote-write target, any dashboard), exact bucket boundaries for SLOs, simple and well-understood.

Cons: buckets must be chosen in advance and are often wrong; quantiles are coarse; every bucket multiplies cardinality; changing buckets breaks continuity.

Prometheus server support

• v2.40.0 (2022-11-08): native histograms introduced as an experimental feature, enabled with --enable-feature=native-histograms. The flag also makes protobuf the preferred scrape format.
• v3.0.0 (2024-11-14): the scrape setting scrape_classic_histograms was renamed to always_scrape_classic_histograms.
• v3.8.0 (2025-11-28): native histograms are stable, but optional. They are controlled by the new scrape_native_histograms setting (global or per scrape config, default false). The native-histograms feature flag is being phased out; in v3.8 it only sets the default of scrape_native_histograms to true.
• v3.9.0 (2026-01-06): the native-histograms feature flag is a no-op. scrape_native_histograms must be set to true explicitly. The default stays false, and there is no announced plan to change it.
• v3.13.0 (2026-07-01): per-target override through the __scrape_native_histograms__ and __always_scrape_classic_histograms__ relabeling labels.

So no Prometheus version enables native histograms by default. Admins have to opt in, either with --enable-feature=native-histograms (v2.40 to v3.8) or with scrape_native_histograms: true (v3.8 and later; required from v3.9 on). With scrape_native_histograms: true, the default scrape_protocols puts PrometheusProto first.

For a histogram exposed in both forms, Prometheus behaves like this:

• scrape_native_histograms: false: the native parts are ignored and the classic histogram is stored, exactly as today.
• scrape_native_histograms: true: the native histogram is stored. The classic series are only kept as well if always_scrape_classic_histograms: true, which is useful while migrating dashboards.

References:

• Prometheus CHANGELOG (entries for 2.40.0, 3.0.0, 3.8.0, 3.9.0, 3.13.0)
• scrape_native_histograms configuration
• Feature flag docs for v3.8.0

Dual exposition:

• Native histograms spec, Instrumentation libraries: libraries should be able to expose a classic and a native version of the same histogram in parallel
• Native histograms spec, Scraping both classic and native histograms: how Prometheus ingests a histogram exposed in both forms (scrape_native_histograms, always_scrape_classic_histograms)
• Native histograms spec, Migration considerations: the recommended migration path, starting with exposing both forms in the instrumentation
• client_golang HistogramOpts.Buckets: to expose both forms, Buckets must be set explicitly, since it defaults to no classic buckets once native buckets are used

Proposal

1. Dual exposition for all histograms: keep Buckets and add the native settings to the same HistogramOpts:

   prometheus.HistogramOpts{
       // ...
       Buckets:                         prometheus.DefBuckets, // classic, unchanged
       NativeHistogramBucketFactor:     1.1,
       NativeHistogramMaxBucketNumber:  100,
       NativeHistogramMinResetDuration: time.Hour,
   }

   Ideally do this through a small helper in pkg/metrics that applies these defaults, so all services stay consistent.

2. Review the classic bucket boundaries. DefBuckets doesn't fit LDAP and Graph backend latencies well.

3. Deprecate and later remove the Summaries (*_latency_microseconds in invitations and webfinger), since the *_duration_seconds histograms already exist next to them. Announce it, because people may have dashboards on those names.

4. Document it for admins: how to enable scrape_native_histograms (plus always_scrape_classic_histograms while migrating), and example PromQL for both forms.

Costs and risks: slightly higher client-side memory per histogram. No change for scrapers that use the text format or don't enable native histograms.

Cardinality trade-off: dual exposition only saves time series for setups that ingest the native form (scrape_native_histograms: true without always_scrape_classic_histograms). Setups without native histograms keep paying one series per bucket for every label set, as they do today. For histogram vectors with several labels, adding classic buckets can therefore be expensive. In those cases an alternative is a native-only histogram with all labels, plus a separate classic histogram with fewer labels as a fallback for setups without native histograms. Which approach fits should be decided per metric.