feat(otel-v2): emit the 6 gen_ai.client.* metrics at parity with v1

[yassin-berriai]

Relevant issues

Extends #30257 (LIT-3600) to the OpenTelemetry v2 integration.

Dependency

This is stacked on #30257 and is based on that branch, because it reuses the metric attribute filter helpers introduced there. The base is litellm_fix_otel_metrics_cardinality for a clean v2-only diff; retarget to litellm_internal_staging once #30257 merges.

Changes

The v2 OTEL integration (litellm/integrations/otel/) was a span engine; it declared two metric histograms but never created a meter or recorded anything, so a v2-default deployment emitted no gen_ai.client.* metrics at all. This brings it to parity with v1.

It adds the four missing metric names, all six histograms, a meter-provider builder that mirrors v1's exporter selection, and a GenAIMetricRecorder that records token usage (split input/output), cost, operation duration, time to first token (streaming only), time per output token, and response duration on the success hook. Everything is gated on config.enable_metrics, so the default is unchanged.

The attribute cardinality filter is reused from v1 by import (no duplication of the valid-name set or validation) and resolved lazily from callback_settings.otel.attributes, the same as v1. A misconfigured filter raises out of the recorder; the logger surfaces it once at ERROR and records nothing, rather than silently disabling metrics, and a corrected config recovers without a restart.

Type

New Feature

Screenshots / Proof of Fix

Live proxy on the v2 path (LITELLM_OTEL_V2=true) with metrics on, the console metric exporter, and an exclude_list configured. Run with the repo root on PYTHONPATH so the branch code loads (otherwise python litellm/proxy/proxy_cli.py resolves litellm from an installed copy).

PYTHONPATH="$(pwd)" LITELLM_OTEL_V2=true LITELLM_OTEL_INTEGRATION_ENABLE_METRICS=true \
  OTEL_EXPORTER=console OTEL_ENDPOINT="" \
  python litellm/proxy/proxy_cli.py --config config.yaml --detailed_debug 2>&1 | tee litellm.log

curl -s -N http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"gpt","messages":[{"role":"user","content":"count to three"}],"stream":true}'

A streaming call emits all six instruments:

gen_ai.client.operation.duration
gen_ai.client.token.usage
gen_ai.client.token.cost
gen_ai.client.response.time_to_first_token
gen_ai.client.response.time_per_output_token
gen_ai.client.response.duration

With the exclude_list set, the recorded gen_ai.client.token.usage data point carries the identity attributes and gen_ai.token.type but none of the excluded keys (hidden_params, metadata.requester_metadata, metadata.requester_ip_address, ...), matching v1.

Docs: BerriAI/litellm-docs#344

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 44 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/otel/plumbing/providers.py	12.12%	29 Missing ⚠️
litellm/integrations/otel/plumbing/metrics.py	88.88%	13 Missing ⚠️
litellm/integrations/otel/logger.py	90.90%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR brings the OTel v2 integration to metric parity with v1 by adding the four missing histogram instruments (token.cost, response.time_to_first_token, response.time_per_output_token, response.duration), wiring a MeterProvider into OpenTelemetryV2.__init__, and introducing a GenAIMetricRecorder that records all six gen_ai.client.* histograms on the async success hook. The cardinality filter is reused from v1 with lazy resolution, and everything is gated on the existing enable_metrics flag so default behaviour is unchanged.

• metrics.py (new) — GenAIMetricRecorder handles attribute building, cardinality filtering, and per-metric recording; mirrors v1 timing math for TTFT, TPOT, and response duration.
• providers.py — adds build_metric_reader / build_meter_provider factory functions mirroring the span-exporter selection, plus _otlp_metrics_endpoint for OTLP/HTTP signal-path rewriting.
• Tests — test_otel_v2_metrics.py covers the six-histogram path, streaming vs non-streaming split, include/exclude cardinality filtering, and the gen_ai.token.type discriminator guard; test_otel_v2_logger.py gains three integration tests for the logger layer.

Confidence Score: 4/5

The change is purely additive and gated on enable_metrics=False by default, so existing deployments are unaffected; both findings are non-blocking edge-case issues.

The core recording logic correctly mirrors v1 behaviour, the cardinality filter is reused rather than duplicated, and all six histograms are well-tested with an InMemoryMetricReader. The two findings — a silent console-exporter fallback for in_memory in build_metric_reader, and a broad ValueError catch that can mis-attribute SDK errors as filter misconfiguration — are both in low-traffic or edge-case paths and do not affect the default configuration.

litellm/integrations/otel/plumbing/providers.py (missing in_memory handler in build_metric_reader) and litellm/integrations/otel/logger.py (broad ValueError catch scope in _record_metrics)

Important Files Changed

Filename	Overview
litellm/integrations/otel/plumbing/metrics.py	Core new file: adds GenAIMetrics dataclass, create_genai_metrics factory, and GenAIMetricRecorder with lazy cardinality filter resolution; generally well-structured at parity with v1
litellm/integrations/otel/plumbing/providers.py	Adds build_metric_reader and build_meter_provider factories; _otlp_metrics_endpoint mirrors the traces counterpart but build_metric_reader silently falls back to ConsoleMetricExporter for unrecognized kinds including "in_memory"
litellm/integrations/otel/logger.py	Wires metric recorder into async_log_success_event; _record_metrics is best-effort and won't break request path; ValueError catch scope is slightly broad
litellm/integrations/otel/model/semconv.py	Adds four new Metric constants (TOKEN_COST, TIME_TO_FIRST_TOKEN, TIME_PER_OUTPUT_TOKEN, RESPONSE_DURATION); clean additive change
tests/test_litellm/integrations/otel/test_otel_v2_logger.py	Adds three integration tests for metrics (invalid filter, six-metric happy path, disabled-by-default); all use InMemoryMetricReader with no real network calls
tests/test_litellm/integrations/otel/test_otel_v2_metrics.py	Comprehensive unit tests for the recorder layer; _drive_success mutates litellm.callback_settings globally without monkeypatch, inconsistent with other tests in the same file

_{Reviews (1): Last reviewed commit: "test(otel-v2): drop duplicate misconfig ..." | Re-trigger Greptile}

[greptile-apps]

build_metric_reader doesn't recognise "in_memory" / "inmemory" / "memory" as exporter kinds, while the span-exporter path (_exporter_from_spec) does. Any config that reaches this function with exporter="in_memory" silently falls through to ConsoleMetricExporter, which is hard to diagnose because metrics appear to "work" in the console but aren't available for programmatic inspection. Adding the same guard that _exporter_from_spec already has keeps the two paths consistent.

Suggested change

	kind = (config.exporter or "console").lower()
	if kind in ("otlp_http", "http", "http/protobuf", "http/json"):
	kind = (config.exporter or "console").lower()
	if kind in ("in_memory", "inmemory", "memory"):
	from opentelemetry.sdk.metrics.export import InMemoryMetricReader
	return InMemoryMetricReader()
	if kind in ("otlp_http", "http", "http/protobuf", "http/json"):

[greptile-apps]

Broad ValueError catch mis-attributes SDK errors as filter misconfiguration

_ensure_filter is the intended source of ValueError here, but any ValueError raised deeper in record() — e.g. if the OTEL SDK or an attribute coercion rejects a value — will be caught by the same branch and logged as "invalid otel.attributes metric filter". On the first occurrence it also permanently sets _metric_filter_error_logged = True, silencing all future ValueErrors for the lifetime of the logger. Narrowing the catch to ValueError raised specifically by _ensure_filter (or using a dedicated sentinel exception) would prevent the misleading message and the premature silencing of unrelated errors.

[veria-ai]

Medium: Unbounded metric cardinality from request metadata

With metrics enabled, requester_metadata is copied from the client's request metadata and this loop records it as an OTEL histogram attribute by default. An authenticated caller can send a unique metadata object on each completion request and force the process/exporter to allocate a distinct metric series for every call; default to excluding requester-controlled metadata from metrics unless an operator explicitly includes it, or bound/normalize the values before recording.

[veria-ai]

PR overview

This PR adds OpenTelemetry v2 support for emitting the six gen_ai.client.* metrics in parity with the existing v1 instrumentation. The touched metrics plumbing records completion-related metric attributes for LiteLLM requests.

There is one open security concern: requester-supplied metadata is included as metric attributes by default, which can let an authenticated caller create unbounded metric cardinality and pressure the process or metrics exporter. No issues have been fixed yet, so the PR still needs a guardrail such as excluding, bounding, or normalizing caller-controlled metadata before it is ready from a security perspective.

Open issues (1)

• Medium: Unbounded metric cardinality from request metadata — litellm/integrations/otel/plumbing/metrics.py:132

Fixed/addressed: 0 · PR risk: 6/10