Skip to content

test(gemini): de-flake test_gemini_image_size_limit_exceeded#28039

Merged
yuneng-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_/nostalgic-johnson-eeb7c3
May 16, 2026
Merged

test(gemini): de-flake test_gemini_image_size_limit_exceeded#28039
yuneng-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_/nostalgic-johnson-eeb7c3

Conversation

@yuneng-berri

@yuneng-berri yuneng-berri commented May 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

tests/llm_translation/test_gemini.py::test_gemini_image_size_limit_exceeded is flaky in CI. It downloaded a 50MB+ image from upload.wikimedia.org to assert the image exceeds MAX_IMAGE_URL_DOWNLOAD_SIZE_MB. Wikimedia intermittently rate-limited the runner (HTTP 429), so the code raised ImageFetchError("Unable to fetch image ... Status code: 429"). pytest.raises(litellm.ImageFetchError) still matched, but assert "Image size" in error_message failed.

Fix

Remove the external-network dependency by mirroring the established LargeImageClient pattern in tests/test_litellm/litellm_core_utils/test_image_handling.py:

  • Stub litellm.module_level_client with a client returning a response whose Content-Length reports 100MB (> 50MB default limit).
  • Bypass SSRF validation (image_handling.safe_get) so no DNS resolution / network call occurs.
  • Keep the completion(model="gemini/...") call so the gemini path's size-limit rejection is still exercised end-to-end.

The test now deterministically hits the size-limit path and is fully offline.

Test plan

uv run pytest tests/llm_translation/test_gemini.py::test_gemini_image_size_limit_exceeded -v

Passes in 0.24s with no network access.


Note

Low Risk
Low risk: test-only change that removes an external network dependency by mocking HTTP image download behavior; no production logic is modified.

Overview
Makes test_gemini_image_size_limit_exceeded deterministic by removing the real Wikimedia download and instead monkeypatching image fetching to return a mocked httpx.Response with an oversized Content-Length.

The test now stubs image_handling.safe_get and litellm.module_level_client (via a local LargeImageClient) to force the size-limit rejection path offline, and updates the URL/assertions to validate the expected "exceeds maximum allowed size" error text.

Reviewed by Cursor Bugbot for commit 8c537f7. Bugbot is set up for automated code reviews on this repo. Configure here.

Mock the image fetch instead of downloading a 50MB+ image from
upload.wikimedia.org. The runner was intermittently rate-limited
(HTTP 429), so the code raised "Unable to fetch image ... Status
code: 429" and the size-limit assertions failed even though
pytest.raises(litellm.ImageFetchError) still matched.

Mirror the established LargeImageClient pattern in
tests/test_litellm/litellm_core_utils/test_image_handling.py: stub
litellm.module_level_client with a response whose Content-Length
exceeds the 50MB limit and bypass SSRF validation, so the
size-limit rejection path is exercised deterministically with no
external network dependency.
@greptile-apps

greptile-apps Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR de-flakes test_gemini_image_size_limit_exceeded by replacing a real network request to upload.wikimedia.org with a fully mocked HTTP client, mirroring the LargeImageClient pattern already established in tests/test_litellm/litellm_core_utils/test_image_handling.py.

  • Mock setup: litellm.module_level_client is patched with a LargeImageClient that returns a 200 response with Content-Length: 104857600 (100 MB), and image_handling.safe_get is replaced with a passthrough lambda to bypass SSRF validation — both are restored automatically by monkeypatch.
  • Coverage preserved: The completion(model=\"gemini/...\") call still traverses the full Gemini translation path; _process_image_response raises ImageFetchError on the Content-Length header check exactly as before, and both original assertions (\"Image size\" and \"exceeds maximum allowed size\") are kept intact.

Confidence Score: 5/5

Safe to merge — a single test file is changed, with no modifications to production code.

The change removes an external network dependency causing intermittent CI failures and replaces it with a deterministic mock. The tested logic path and assertions are identical to the original. The only minor observation is that the mock allocates 100 MB of memory unnecessarily, but this is a pre-existing pattern mirrored from elsewhere in the test suite and does not affect correctness.

No files require special attention.

Important Files Changed

Filename Overview
tests/llm_translation/test_gemini.py De-flaked test_gemini_image_size_limit_exceeded by replacing an external Wikimedia URL with a fully mocked HTTP client; assertions and the tested code path are unchanged.

Reviews (1): Last reviewed commit: "test(gemini): de-flake test_gemini_image..." | Re-trigger Greptile

@codecov

codecov Bot commented May 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Comment on lines +1616 to +1626
def get(self, url, follow_redirects=True):
size_bytes = int(100 * 1024 * 1024) # 100MB > 50MB default limit
return Response(
status_code=200,
headers={
"Content-Type": "image/jpeg",
"Content-Length": str(size_bytes),
},
content=b"x" * size_bytes,
request=Request("GET", url),
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The content=b"x" * size_bytes expression eagerly allocates 100 MB of memory, even though _process_image_response raises ImageFetchError on the Content-Length header check before ever calling iter_bytes. Since the body is never read, b"" is sufficient and avoids the allocation entirely.

Suggested change
def get(self, url, follow_redirects=True):
size_bytes = int(100 * 1024 * 1024) # 100MB > 50MB default limit
return Response(
status_code=200,
headers={
"Content-Type": "image/jpeg",
"Content-Length": str(size_bytes),
},
content=b"x" * size_bytes,
request=Request("GET", url),
)
def get(self, url, follow_redirects=True):
size_bytes = int(100 * 1024 * 1024) # 100MB > 50MB default limit
return Response(
status_code=200,
headers={
"Content-Type": "image/jpeg",
"Content-Length": str(size_bytes),
},
content=b"", # Content-Length triggers rejection before body is read
request=Request("GET", url),
)

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high mode and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2130bdc. Configure here.

Comment thread tests/llm_translation/test_gemini.py Outdated
"safe_get",
lambda client, url, **kw: client.get(url, follow_redirects=True),
)
monkeypatch.setattr(litellm, "module_level_client", LargeImageClient())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mocked image download code is never invoked by Gemini path

High Severity

The monkeypatches of image_handling.safe_get and litellm.module_level_client are never exercised because the Gemini transformation path in _process_gemini_media passes https:// URLs with recognized extensions (like .jpg) directly as file_data references without downloading them. The URL https://example.com/large-image.jpg matches the third branch in _process_gemini_media (via _get_image_mime_type_from_url returning "image/jpeg"), so convert_url_to_base64 is never called. The test does not actually exercise the size-limit rejection path it claims to test.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2130bdc. Configure here.

The Content-Length header check in _process_image_response rejects the
image before the body is streamed, so the mock body never needs to be
materialized. Use an empty body instead of b"x" * 100MB (addresses
greptile/cursor review feedback).
@yuneng-berri yuneng-berri merged commit ec2f3aa into litellm_internal_staging May 16, 2026
107 of 108 checks passed
@yuneng-berri yuneng-berri deleted the litellm_/nostalgic-johnson-eeb7c3 branch May 16, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants