[Feature]: customLLM return only object for GenericStreamingChunk

[dragonTalon]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

We have a self-hosted, closed-source model in our company, and we would like to integrate it into LiteLLM through customLLM.

However, we found that the custom streaming interface only supports returning GenericStreamingChunk. The fields in GenericStreamingChunk are quite limited, and using provider_specific_fields would break the original interface compatibility.

What we would like to do is to actively convert our model's streaming response into the common OpenAI-compatible format.

From my understanding, this should be a very common use case. I noticed that acompletion seems to support this behavior, but astreaming does not.

Could you please consider supporting OpenAI-compatible streaming responses in customLLM, or provide a recommended way to achieve this?

Thanks!

Motivation, pitch

custom llm add litellm

What part of LiteLLM is this about?

No response

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response

[tanmay958]

Hey @Sameerlite and @krrish-berri-2 I've dug into the code and would like to contribute a fix for this. Wanted to check with you first before opening a PR.

Root cause:
In chunk_creator (streaming_handler.py:1137-1153), when custom_llm_provider in litellm._custom_providers, every chunk gets force-cast to GChunk regardless of its actual type. So even if a custom provider yields an already-formatted ModelResponseStream, it gets mangled trying to read GChunk fields like "text" and "is_finished".

Proposed fix:
Add a short-circuit at the top of chunk_creator before the GChunk block:

if isinstance(chunk, ModelResponseStream):
    if chunk.choices and chunk.choices[0].finish_reason:
        self.received_finish_reason = chunk.choices[0].finish_reason
    return chunk

The finish_reason line is important without it, finish_reason_handler() silently overwrites tool_calls / length with "stop". The rest of the downstream pipeline (logging, cost tracking, stream_chunk_builder) already operates on ModelResponseStream so nothing else is affected.

Since ModelResponseStream inherits directly from openai._models.BaseModel, users with self-hosted OpenAI-compatible models can yield chunks as-is with zero translation.

Would you accept a PR for this? Happy to add tests covering the finish_reason passthrough too.

[krrish-berri-2]

sure

[tanmay958]

hey @krrish-berri-2, could you take a look at this when you get a chance?