perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths

[yassin-berriai]

What this PR does

Reduces per-request and per-chunk overhead on the Anthropic /v1/messages streaming path in the LiteLLM proxy. Same wire output, parity-tested.

• Skip work that's a no-op in the default config. Don't run the per-chunk Datadog span when tracing is off, don't await the per-chunk streaming hook when no callback / guardrail / cost-injection is active, and skip the agentic post-processing wrapper when no callback overrides its hook (it otherwise buffers every chunk and rebuilds the response from SSE only to call hooks that all return (False, {})).
• Stop doing the same work twice per request. Serialize the request body once (reuse for pre-call log + wire), memoize the optional-params type-hints resolution (~80µs/request), skip the second strip_empty_text_blocks scan when the async wrapper already sanitized.
• Cheaper end-of-stream response reconstruction. Collapse the homogeneous run of content_block_delta text events into a single equivalent SSE event before stream_chunk_builder, removing O(num_output_tokens) ModelResponseStream Pydantic constructions. Tool-use / thinking / citations streams fall back to the unchanged legacy path.
• Cheaper logging on the hot path. Gate debug f-string evaluation behind isEnabledFor(DEBUG) (no more serializing full message payloads at non-debug levels), hoist cost_injection_active out of the per-chunk loop, drop one async-generator layer per chunk in async_sse_data_generator.
• Parity-guaranteed. Every fast path falls back to the legacy path for anything it doesn't recognize. New parity tests assert byte-identical logged/billed payloads between fast and legacy paths, plus unit tests for agentic-hook detection, pre-serialized body reuse, and memoized key resolution.
• Reproducible benchmark. scripts/benchmark_anthropic_messages_perf.py boots a local mock Anthropic SSE provider + the proxy under test for apples-to-apples TTFT / TPM measurements across commits.

Benchmark

Local scripts/benchmark_anthropic_messages_perf.py against the mock Anthropic SSE provider on the same host. 500 requests/run, concurrency 20, 80-request warmup, median of 5 runs, back-to-back on the same machine. Baseline = HEAD^ (this PR reverted), Optimized = HEAD (this PR).

Metric	Baseline	Optimized	Δ
TTFT p50 (ms)	241.88	90.89	−62.4% (2.66× faster)
TTFT p95 (ms)	463.86	148.23	−68.0% (3.13× faster)
TTFT p99 (ms)	1313.46	155.77	−88.1% (8.43× faster)
Full-request p50 (ms)	242.26	91.32	−62.3%
Full-request p95 (ms)	464.24	148.88	−67.9%
Output tokens / s	4,394.5	12,504.4	+184.6% (2.85×)
Requests / s	68.66	195.38	+184.6% (2.85×)

Reproduce:

uv run python scripts/benchmark_anthropic_messages_perf.py \
    --label optimized --proxy-port 4099 --provider-port 8098 \
    --requests 500 --concurrency 20 --warmup 80 --repeats 5

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🚄 Infrastructure
🧹 Refactoring
✅ Test

Changes

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ yassinkortam
❌ claude
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 84.71338% with 24 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../proxy/pass_through_endpoints/streaming_handler.py	56.25%	14 Missing ⚠️
..._handlers/anthropic_passthrough_logging_handler.py	93.05%	5 Missing ⚠️
litellm/llms/custom_httpx/llm_http_handler.py	85.71%	3 Missing ⚠️
litellm/proxy/common_request_processing.py	90.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR reduces per-request and per-chunk overhead on the Anthropic /v1/messages streaming path by removing several layers of redundant work in the default (no-callbacks, no-tracing) configuration.

• Fast-path chunk collapsing (_collapse_pure_text_chunks): N per-token Pydantic ModelResponseStream constructions are replaced by a single merged event before stream_chunk_builder; parity tests assert byte-identical logged/billed payloads between paths, with automatic fallback for tool-use, thinking, and citations streams.
• Per-request deduplication: request-body serialization reused across pre-call log and wire, get_type_hints memoized with @lru_cache, and redundant strip_empty_text_blocks scan skipped via _litellm_messages_presanitized sentinel.
• Per-chunk hot-path gates: DD tracer context-manager overhead skipped via an import-time constant, per-chunk async_post_call_streaming_hook coroutine bypassed when no callbacks/guardrails are registered, cost_injection_active hoisted out of the chunk loop, and one async for: yield trampoline removed from async_sse_data_generator.

Confidence Score: 5/5

Safe to merge. All fast paths fall back to the unchanged legacy path for anything they don't recognise, and parity tests assert byte-identical logged/billed payloads between the two paths.

The changes are well-scoped and well-tested. Every new fast path has a concrete fallback, and the parity test suite directly guards the most complex optimization (chunk collapsing). The two notes left are minor: one is about a hypothetical hook mutation pattern that is unsupported by convention, the other is an import-time constant that the tests already patch correctly.

No files require special attention; the fast-path logic in anthropic_passthrough_logging_handler.py and llm_http_handler.py is the most complex but is covered by dedicated parity and retry tests.

Important Files Changed

Filename	Overview
litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py	Adds fast-path chunk collapsing via _collapse_pure_text_chunks that merges N text-delta events into a single event before legacy reconstruction, plus _build_complete_streaming_response_legacy refactor. Parity tests confirm byte-identical outputs.
litellm/llms/custom_httpx/llm_http_handler.py	Pre-serializes request body once (reused for pre-call log and wire), adds _has_agentic_completion_hook to skip agentic wrapper when no override present, changes signed_json_body type to Union[str, bytes].
litellm/proxy/common_request_processing.py	Adds _DD_STREAMING_TRACE_ENABLED module-level constant to skip DD span overhead per-chunk when tracing is off; adds fast-path in async_streaming_data_generator bypassing per-chunk hook; removes one generator layer from async_sse_data_generator.
litellm/proxy/pass_through_endpoints/streaming_handler.py	Hoists cost_injection_active out of the per-chunk loop; extracts _build_passthrough_logging_result for testability; refactors the hot path to avoid branching per-chunk.
litellm/proxy/utils.py	Changes async_post_call_streaming_hook detection from leaf-class dict check to MRO walk via function-identity comparison, ensuring inherited overrides are detected.
litellm/llms/anthropic/experimental_pass_through/messages/handler.py	Passes _litellm_messages_presanitized=True sentinel from async wrapper to handler to skip redundant strip_empty_text_blocks scan; handler pops the flag to prevent provider leakage.
litellm/llms/anthropic/experimental_pass_through/messages/utils.py	Memoizes get_type_hints(AnthropicMessagesRequestOptionalParams) via @lru_cache(maxsize=1), removing ~80us per-request cost.
scripts/benchmark_anthropic_messages_perf.py	New benchmark script booting a mock Anthropic SSE provider and LiteLLM proxy for apples-to-apples TTFT/TPM measurements across commits.
tests/test_litellm/proxy/pass_through_endpoints/llm_provider_handlers/test_anthropic_passthrough_logging_handler.py	Adds TestPureTextFastPathParity with 10 scenarios asserting byte-identical usage/logged payload between fast and legacy paths; adds fallback tests for tool-use and thinking blocks.
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py	Adds tests for _has_agentic_completion_hook detection and three tests for the pre-serialized body reuse/retry path.
tests/test_litellm/proxy/test_common_request_processing.py	Adds DD-trace skip test and TestAsyncStreamingDataGeneratorFastPath covering both fast and slow paths of the per-chunk hook gate.

_{Reviews (4): Last reviewed commit: "fix(mypy): narrow model_name to str in c..." | Re-trigger Greptile}

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai