Litellm benchmark for LLM response Streaming

[sreekarkamireddy]

We are seeing a significant increase in the cpu utilization when streaming the LLM model response, We also observe that increasing the completion tokens on the model also increases the cpu utilization considerably. We are observing around 75% cpu utilization even at 1/6th the rate of traffic (5 calls per second) of the non-streaming model calls.
We couldn't find any benchmarks for streaming response on litellm github. Could you please share benchmarks that you run for your releases so that we can compare and see the performance?

[ishaan-jaff]

• what model / provider is this ?
• are you using any logging integrations
• please share a curl to repro

[sreekarkamireddy]

@ishaan-jaff Thanks for the quick response
We are using OpenAI provider and model is gpt-41-nano
No logging integrations
Are you looking for the payload used for the perf test?
{ "messages": [ { "role": "user", "content": "Please tell me about benefits of using a curved monitor at work. Your answer must be at least 1500 characters long." } ], "max_tokens": 150, "temperature": 0.4, "stream": true }

[tanvithakur94]

@ishaan-jaff Did you get a chance to look at this? Were you able to reproduce the issue?

[del-zhenwu]

Introduce A performance benchmark platform For LLM services: https://github.com/MigoXLab/LMeterX

[mjozwiak-pwell]

Any idea when is it going to be resolved?

[berri-teddy]

@sreekarkamireddy @mjozwiak-pwell @tanvithakur94 Please update to our newest stable and let me know if the issue has been resolved, or please provide me with:
CPU:
RAM:
Prompt:
Model:
Config.yaml(PII Redacted)

Streaming benchmark is a good idea that we're currently lacking. Will add to our roadmap

[sreekarkamireddy]

@berri-teddy
We tested the streaming with litellm version - 1.79.0 but we saw the same performance
Model : gpt-41-nano
RPS (requests per second) : 5
CPU: 90% utilization (1 cpu)
Memory: 10% utilization (6 GB)
Sample Prompt:
{ "messages": [ { "role": "user", "content": "Please tell me about benefits of using a curved monitor at work. Your answer must be at least 1500 characters long." } ], "max_tokens": 150, "stream": true }

[gt-libo28]

We encountered the same issue — with the same machine configuration and QPS, the CPU utilization of the streaming mode is approximately 4 to 5 times that of the non-streaming mode.

[adi-yolabs]

We are using streaming extensively -I assume it is added to roadmap right? but still as for now it's like @gt-libo28 wrote above 4-5 times CPU utilization of the non-streaming mode?

[ashgold]

I am experiancing same issue too

[kuldeepjain]

@ishaan-jaff Any update here? Do you plan to triage this issue anytime soon? It looks like multiple people have reported this and are encountering the same problem.

I also noticed the recent release (v1.81.3 — 25% CPU Usage Reduction). Does that address this streaming CPU usage issue, or is it from unrelated perf changes?

[ashgold]

I am analyzing this issue to improve it.
There is a logic for processing each received event (chunk), so it is naturally slower than non-streaming, but after profiling it, there is room for improvement, so I am revising it.
I will upload a PR soon.

[kuldeepjain]

I also tried updating our service implementation to call Bedrock Sonnet 4 model directly skipping the LiteLLM which showed up to 50% reduction in the CPU %.

The direct Bedrock implementation bypassing LiteLLM's Pydantic parsing has achieved a ~45-50% CPU reduction during load tests (3 mins duration with 3 users). This indicates that LiteLLM's Pydantic model creation for each streaming chunk was a significant CPU bottleneck.

[sreekarkamireddy]

@ishaan-jaff @krrishdholakia Any updates on this regarding the bottleneck on CPU from litellm?

Working on reproducing + fixing it. We’re hoping to have someone on this in the next 2 weeks. Can you bump me on this end of March?

…

On Mon, Mar 16, 2026 at 11:37 AM sreekar kamireddy ***@***.***> wrote: *sreekarkamireddy* left a comment (BerriAI/litellm#13505) <#13505 (comment)> @ishaan-jaff <https://github.com/ishaan-jaff> @krrishdholakia <https://github.com/krrishdholakia> Any updates on this regarding the bottleneck on CPU from litellm? — Reply to this email directly, view it on GitHub <#13505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEF7LK7TSG42SJRGAZ5MRWT4RBCW3AVCNFSM6AAAAACDT4JPICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANRZG44DQMRZGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>