fix(streaming): guard raise_on_model_repetition against empty choices

[shivamrawat1]

Resolves LIT-3776

Relevant issues
Fixes mid-stream IndexError: list index out of range for Vertex Gemini Flash / Flash Lite streaming with web_search_options (related: #28884)

Type
Bug Fix

Cause
Vertex Gemini Flash and Flash Lite with web_search_options={} + stream=True emit metadata-only and usage-only SSE chunks that have no streamable delta. With stream_options={"include_usage": True}, those chunks are stored in CustomStreamWrapper.chunks with choices=[].

Later, when a real content chunk arrives, return_processed_chunk_logic() calls raise_on_model_repetition(), which unconditionally reads self.chunks[-1].choices[0] and self.chunks[-2].choices[0]. If either historical chunk has empty choices, LiteLLM raises IndexError, which surfaces to callers as APIConnectionError / MidStreamFallbackError.

Gemini Pro is less affected because its streaming shape rarely inserts these empty-choice chunks before content arrives.

Fix
Add an early return in raise_on_model_repetition() when either of the last two stored chunks has empty choices. Repetition detection only needs comparable text content; metadata/usage chunks should be skipped. This matches the existing empty-choices guard already used in _has_special_delta_content().

Changes
litellm/litellm_core_utils/streaming_handler.py: guard before accessing choices[0] on historical chunks
tests/test_litellm/litellm_core_utils/test_streaming_handler.py: regression test test_raise_on_model_repetition_tolerates_empty_choices (parametrized for empty chunk at -1 and -2)

Run this e2e test:

import asyncio
import os
from litellm import acompletion, completion
MODELS = [
    "vertex_ai/gemini-3.1-flash-lite-preview",
    "vertex_ai/gemini-3.1-flash-lite",
    "vertex_ai/gemini-3-flash-preview",
]
PROMPT = "Search the web and summarize the latest news about OpenAI."
COMMON = dict(
    messages=[{"role": "user", "content": PROMPT}],
    stream=True,
    web_search_options={},
    stream_options={"include_usage": True},
    vertex_project=os.environ.get("VERTEXAI_PROJECT"),
    vertex_location=os.environ.get("VERTEXAI_LOCATION", "global"),
    vertex_credentials=os.environ.get("GOOGLE_APPLICATION_CREDENTIALS"),
)
async def repro_async() -> None:
    for model in MODELS:
        print(f"\n=== {model} (async) ===")
        try:
            response = await acompletion(model=model, **COMMON)
            async for chunk in response:
                print(chunk)
            print("OK")
        except Exception as e:
            print(f"FAILED: {type(e).__name__}: {e}")
def repro_sync() -> None:
    model = MODELS[0]
    print(f"\n=== {model} (sync) ===")
    response = completion(model=model, **COMMON)
    for chunk in response:
        print(chunk)
if __name__ == "__main__":
    asyncio.run(repro_async())

Before: list index out of range

After: Streaming finishes

[greptile-apps]

Greptile Summary

Guards raise_on_model_repetition() against IndexError when Vertex Gemini Flash / Flash Lite emits metadata-only or usage-only SSE chunks with choices=[] during streaming — a pattern triggered by web_search_options combined with stream_options={"include_usage": True}. The fix adds a single early-return check consistent with the identical guard already present in _has_special_delta_content().

• streaming_handler.py: Inserts if not self.chunks[-1].choices or not self.chunks[-2].choices: return before the two choices[0] accesses in raise_on_model_repetition(), preventing IndexError (which previously surfaced to callers as APIConnectionError / MidStreamFallbackError).
• test_streaming_handler.py: Adds a parametrized regression test covering both empty-chunk positions (-1 and -2) using mock data only, with no real network calls.

Confidence Score: 5/5

Safe to merge — the change is a minimal, targeted early-return that only fires when choices is empty, and is consistent with an identical guard already used elsewhere in the same class.

The fix is a single, well-scoped guard that mirrors the existing empty-choices check in _has_special_delta_content(). Both accesses to choices[0] in raise_on_model_repetition() are protected by the new guard, repetition-detection logic for non-empty chunks is completely unchanged, and the regression test covers both empty-chunk positions (-1 and -2) using mock-only data.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_handler.py	Adds a 6-line early-return guard in raise_on_model_repetition() to skip repetition detection when either of the last two stored chunks has empty choices, preventing IndexError on usage/metadata-only Vertex Gemini streaming chunks.
tests/test_litellm/litellm_core_utils/test_streaming_handler.py	Adds parametrized regression test test_raise_on_model_repetition_tolerates_empty_choices covering both empty-chunk positions (-1 and -2); uses only mock data, no real network calls.

_{Reviews (1): Last reviewed commit: "fix(streaming): guard raise_on_model_rep..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

LGTM; thanks!