test: make custom_tokenizer proxy tests hermetic

[yuneng-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Type

🧹 Refactoring
✅ Test

Changes

tests/proxy_unit_tests/test_custom_tokenizer_bug.py loaded Xenova/llama-3-tokenizer from HuggingFace Hub at test time. On shared CI runners HF periodically returns 429 Too Many Requests, after which huggingface_hub falls back to an empty local cache and re-raises a LocalEntryNotFoundError ("check your connection"); the surfaced error makes it look like a connectivity problem when the real cause is rate limiting. The failures show up on unrelated PRs because the file lands in the test_[a-j]*.py matrix group that every PR runs, so the flake is not tied to any one change. These were integration tests sitting in a unit-test path.

Rather than delete the coverage, this rewrites the suite to mock the single network boundary (litellm.utils.Tokenizer.from_pretrained) while running the proxy's real extraction-and-selection path. The regression test now asserts the identifier configured in model_info.custom_tokenizer actually reaches from_pretrained and that the response reports the HuggingFace tokenizer. That is stronger than the old test_custom_tokenizer_from_model_info, which named its model llama-3.1-8b-instant; the built-in default tokenizer path already maps any llama-3 model to Xenova/llama-3-tokenizer, so that test reported huggingface_tokenizer with or without the fix and gave no real signal. A control test pins the no-custom-tokenizer case to the OpenAI tokenizer and asserts from_pretrained is never called.

I verified the tests are real by reintroducing the original bug (leaving model_info unpopulated from the matched deployment): the regression test fails with from_pretrained called 0 times, while the control stays green. The full suite runs in ~0.05s with HF_HUB_OFFLINE=1, confirming no network access.

Screenshots / Proof of Fix

Tests pass offline (no HuggingFace access):

$ HF_HUB_OFFLINE=1 uv run --no-sync pytest tests/proxy_unit_tests/test_custom_tokenizer_bug.py -q
..                                                                       [100%]
2 passed in 0.04s

Regression check, with the original bug reintroduced (model_info not read from the deployment):

$ HF_HUB_OFFLINE=1 uv run --no-sync pytest tests/proxy_unit_tests/test_custom_tokenizer_bug.py -q
E       AssertionError: Expected 'from_pretrained' to be called once. Called 0 times.
1 failed, 1 passed

[greptile-apps]

Greptile Summary

This PR rewrites tests/proxy_unit_tests/test_custom_tokenizer_bug.py to eliminate network calls to the HuggingFace Hub, which were causing intermittent CI failures due to rate limiting (429). The suite shrinks from four tests to two, trading breadth for hermeticity and signal quality.

• Mocked boundary: litellm.utils.Tokenizer is patched via patch.object, intercepting all from_pretrained calls inside litellm/utils.py's create_pretrained_tokenizer function — the correct patch site since Tokenizer is a module-level name in utils.py.
• Stronger regression signal: The new regression test uses a model name (self-hosted-embedder) that matches no built-in HuggingFace tokenizer heuristic, so the test fails with 0 calls if the proxy doesn't read custom_tokenizer from the deployment's model_info — unlike the old llama-3.1-8b-instant test whose built-in llama-3 mapping produced a false pass.
• Improved teardown: monkeypatch.setattr replaces bare setattr so the global llm_router is restored between tests.

Confidence Score: 5/5

Safe to merge — only test-file changes, no production code touched, and the new tests are strictly stronger than the ones they replace.

The change is confined to a single test file. The mock target (litellm.utils.Tokenizer) is the module-level name used by create_pretrained_tokenizer in utils.py, so the patch intercepts the real call site. The regression test is demonstrably tighter than the original: the old test used a model name whose built-in llama-3 route would satisfy the assertion even without the bug fix, while the new test uses an unrecognized model name that forces the proxy to actually read custom_tokenizer from model_info. No production logic is modified.

No files require special attention.

Important Files Changed

Filename	Overview
tests/proxy_unit_tests/test_custom_tokenizer_bug.py	Rewrites four network-dependent HuggingFace tests into two hermetic unit tests by patching litellm.utils.Tokenizer.from_pretrained; switches setattr to monkeypatch for proper teardown; regression test correctly verifies the custom tokenizer identifier reaches from_pretrained.

_{Reviews (2): Last reviewed commit: "test: make custom_tokenizer proxy tests ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yuneng-berri]

@greptile