feat(metrics): use native request latency histogram

[ReubenBond]

Problem

Application request latency is currently preaggregated into three observable counters (-bucket, -count, and -sum). This was introduced in #7814 in 2022 because native OpenTelemetry histogram aggregation was prohibitively expensive at the time, and the concern was reiterated in #9609.

The custom representation has non-standard semantics: bucket series are disjoint ranges labeled with duration, collection is process-lifetime cumulative, and the sum and bucket selection truncate elapsed time to integer milliseconds. It also prevents exporters from treating request latency as a normal histogram distribution.

Change

Replace the custom aggregation with one System.Diagnostics.Metrics.Histogram<double> named orleans-app-requests-latency, with unit ms, no metric attributes, and untruncated TimeSpan.TotalMilliseconds observations.

Orleans does not prescribe bucket boundaries. Latency objectives and useful long-tail coverage are application-specific, so deployments should configure this instrument using an exact-name OpenTelemetry View. Instances whose distributions are aggregated together should use the same boundaries. A coarse general-purpose millisecond profile could be:

1, 5, 10, 25, 50, 100, 250,
500, 1000, 2500, 5000, 10000, 30000, 60000

This is an example, not an Orleans default. Applications should add thresholds corresponding to their own SLOs and timeout policy.

The custom aggregator and its tests are removed. Correctness coverage verifies fractional timing through CallbackData, disabled instrumentation, application-configured boundaries, exact-boundary and overflow behavior, floating-point count/sum/MinMax, cumulative and delta readers, and concurrent recording.

This also updates the repository's OpenTelemetry packages, exporters, instrumentation, documentation snippets, and affected samples from 1.16 to 1.18. OpenTelemetry 1.18 contains the histogram synchronization optimization from open-telemetry/opentelemetry-dotnet#7601 and is the version used for the sustained-load comparison below. On .NET 10, System.Diagnostics.DiagnosticSource is supplied by the shared runtime (10.0.12 in the test environment), and OpenTelemetry 1.18 does not add a separate DiagnosticSource package dependency for net10, so this PR does not add a redundant framework-package reference.

Compatibility

This is an intentional breaking metrics change with no dual-publication period:

• Removes orleans-app-requests-latency-bucket, orleans-app-requests-latency-count, and orleans-app-requests-latency-sum as separate Orleans instruments.
• Removes the duration attribute and its mutually exclusive range semantics.
• Replaces them with one standard histogram distribution. Exporters determine the transport shape; Prometheus classic export uses cumulative le buckets plus count and sum series.
• Changes the sum from truncated integer milliseconds to fractional milliseconds.
• Delegates bucket policy and cumulative or delta temporality to the application's metrics pipeline.
• Updates the repository's OpenTelemetry dependency family to 1.18.0, including the Prometheus exporter to 1.18.0-beta.1.

Exporter Views, dashboards, alerts, recording rules, and queries must be updated together with the Orleans upgrade. Use the same explicit boundaries across instances or regions which must be aggregated exactly. Differing explicit histograms can be coarsened only where their boundaries are compatible; finer resolution cannot be reconstructed after collection.

Performance evidence

Benchmarks were run as a bounded prototype and are intentionally not included in this commit. Evidence and validation are focused on .NET 10.

Environment: BenchmarkDotNet 0.15.6, macOS 26.6.2, Apple M4 Max (14 physical/logical cores), .NET SDK 10.0.401, .NET 10.0.12, Server GC, and OpenTelemetry 1.18. Each sustained-load run used one client and one silo in the same process, 100 workers, a 150,000-request warmup, ten 500,000-request rounds, one reader, cumulative temporality, one-second export cadence, MinMax enabled, and no request tags. Three alternating runs were collected per implementation.

The native benchmark used 22 explicit boundaries, more than the coarse example above, so it is conservative with respect to explicit-bucket lookup and retained bucket state.

Implementation	Mean of run medians	Mean throughput	CPU/run	Peak RSS
Custom 17-bound aggregator	198,414/s	196,017/s	337.9s	329.0 MiB
Native 22-bound histogram	199,836/s	197,869/s	336.8s	327.4 MiB

The native histogram is within 1% of custom and slightly ahead under sustained Orleans request load. This is the relevant comparison for the production migration.

The focused .NET 10 ShortRun microbenchmark shows the local tradeoff:

Operation	Custom	Native
Disabled branch	below measurement resolution	below measurement resolution
Untagged record	3.53 ns	14.02 ns
Concurrent record	148.53 ns	255.84 ns
Collection	1,321.78 ns / 1,232 B	97.34 ns / 256 B
Concurrent record and collection	178.13 ns	259.81 ns
Record with two tags	n/a	76.81 ns

Native recording remains slower in isolation, while collection is about 14x faster and allocates about one fifth as much. Under the end-to-end request workload, the recording difference was not measurable. This is why the migration does not add grain type or local/remote tags; those dimensions can be evaluated separately.

OpenTelemetry 1.16 already contained the explicit-bound lookup improvements from open-telemetry/opentelemetry-dotnet#7165. Moving the repository to 1.18 also brings in open-telemetry/opentelemetry-dotnet#7601, which replaced an interlocked lock-release operation with Volatile.Write and reported 2-5.6% hot-path histogram improvements upstream.

Bucket and percentile policy

A native histogram does not improve percentile approximation if boundaries are unchanged. Its benefits here are standard distribution semantics, fractional observations, MinMax, native temporality, and standard exporter handling.

Bucket selection belongs to the application or deployment because the useful thresholds depend on its SLOs, timeout, expected latency range, and backend capabilities. Submillisecond buckets are not enabled by Orleans. A 1 ms lower boundary is sufficient for the example profile, while 30-second and 60-second boundaries retain useful long-tail visibility. Deployments which need finer low-latency resolution can opt into it using their View.

The local benchmarks are directional rather than publication-grade. Dedicated Linux/CI hardware, multiple readers, delta-temporality load, precise retained memory per series, and remote exporter serialization remain unmeasured.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Copilot review overview

🔵 Needs a closer look

This breaking metrics migration has broad exporter and dashboard compatibility implications requiring human review.

Review tier: Lite
Findings: None

What changed in this PR

Replaces custom request-latency counters with a native Histogram<double> using fractional millisecond observations and explicit bucket boundaries.

Changes:

• Adds native histogram instrumentation and comprehensive behavior tests.
• Removes obsolete aggregators and related tests.
• Updates monitoring documentation and migration guidance.

File	Description
test/​Orleans.Runtime.Tests/​Orleans.Runtime.Tests.csproj	Adds histogram test dependencies.
test/​Orleans.Runtime.Tests/​Diagnostics/​ApplicationRequestInstrumentsTests.cs	Tests histogram behavior and boundaries.
test/​Orleans.Runtime.Tests/​CallbackDataTests.cs	Verifies fractional callback latency.
test/​Orleans.Core.Tests/​General/​HistogramAggregatorTests.cs	Removes obsolete aggregator tests.
src/​Orleans.Core/​Runtime/​CallbackData.cs	Preserves fractional elapsed milliseconds.
src/​Orleans.Core/​Diagnostics/​Metrics/​ApplicationRequestInstruments.cs	Implements native latency histogram recording.
src/​Orleans.Core/​Diagnostics/​Metrics/​Aggregators/​HistogramBucketAggregator.cs	Removes obsolete bucket aggregation.
src/​Orleans.Core/​Diagnostics/​Metrics/​Aggregators/​HistogramAggregator.cs	Removes obsolete aggregation.
docs/​site/​src/​content/​docs/​host/​monitoring/​troubleshooting-symptom-catalog.md	Updates latency troubleshooting guidance.
docs/​site/​src/​content/​docs/​host/​monitoring/​metrics.md	Documents the new metric and migration.
docs/​site/​src/​content/​docs/​host/​monitoring/​metrics-catalog.md	Updates the metrics catalog.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Resolve the net8 build issue and update the stale GPSTracker latency dashboard query.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 1

Open findings (1)

• Guard the bucket array for net10 builds · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Update the GPSTracker Grafana dashboard to query the native histogram and remove the obsolete duration grouping.

Review tier: Lite
Findings: None

Resolved findings (1)

• Guard the bucket array for net10 builds

[Copilot]

Copilot review overview

🔵 Needs a closer look

The broad, intentionally breaking metrics and dependency migration warrants final human validation.

Review tier: Lite
Findings: None

[Copilot]

Copilot review overview

🟡 Changes recommended

A critical test issue and documentation corrections remain outstanding.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 1

Open findings (1)

• Tests require MinMax without enabling it for default views · New

[github-actions]

Code coverage

Metric	Pull request	Current main	Variance
Lines	81.60% (109,802 / 134,555)	81.59% (109,822 / 134,600)	+0.0124 pp
Branches	70.49% (31,197 / 44,260)	70.49% (31,209 / 44,272)	-0.0080 pp

Report-only conclusion: mixed.

The current-main baseline is commit e69d956b23 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

• Pull request commit: 4ac5043f0a
• View the pull request test run and coverage artifacts
• View the trusted coverage reporting run