Add granian as a ASGI compliant web server. Provider better throughput stability,

[harish-berri]

Add granian as a ASGI compliant web server. Provider better throughput stability, failure rate is nil and 10-20 RPS improvements over uvicorn. We achieve a peak throughput increase from 337 RPS to 365 RPS on a GCP VM with 4vCPU and 16GB RAM. Redis Cache Enabled

Check below for some numbers

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

Raw Throughput of the Granian (4 workers (subprocesses) + 1 runtime worker)

Raw Throughput of the Uvicorn Server(4 workers (subporcesses))

Raw Throughput of the LLM endpoint

[greptile-apps]

Greptile Summary

This PR adds Granian as an optional ASGI server backend for the LiteLLM proxy, exposed via --run_granian and --granian_threads CLI flags. Most issues from the previous review round have been resolved: granian==2.5.7 is now correctly placed in the proxy optional-dependencies, the license_cache.json version matches the pin, requires-python is updated to >=3.10, and SSL test coverage has been added.

• All five granian unit tests (test_init_granian_server*) use @patch(\"granian.Granian\") as a decorator with pytest.importorskip(\"granian\") in the function body. The decorator resolves and imports granian before the body runs, so the tests error with ModuleNotFoundError rather than skipping on environments where granian is not installed. The fix (module-level pytest.importorskip) was outlined in a prior review comment but has not been applied to any of the five tests.

Confidence Score: 4/5

Safe to merge for production use; the test defect only affects CI environments where granian is absent and doesn't impact runtime behavior.

Core implementation is correct and well-structured. All major issues from the previous review round (dependency placement, license version, Python floor, missing SSL tests) are resolved. The remaining P1 is confined to test infrastructure: the @patch/importorskip ordering affects all five granian tests and will cause errors (not skips) on environments without granian installed, which can break CI.

tests/test_litellm/proxy/test_proxy_cli.py — all five test_init_granian_server* tests need the module-level pytest.importorskip guard applied.

Important Files Changed

Filename	Overview
litellm/proxy/proxy_cli.py	Adds _init_granian_server static method and --run_granian / --granian_threads CLI flags; implementation is clean with SSL validation, workers clamping, and informational warnings for unsupported options.
tests/test_litellm/proxy/test_proxy_cli.py	Adds SSL and partial-SSL test cases for granian (addressing previously missing coverage), but all five granian tests still share the @patch("granian.Granian") / pytest.importorskip ordering defect that causes errors instead of skips when granian is absent.
pyproject.toml	granian==2.5.7 correctly placed in proxy optional-dependencies (not core); requires-python updated to >=3.10; no remaining placement or marker concerns.
license_cache.json	Records granian:2.5.7 with BSD-3-Clause license — version now matches the pyproject.toml pin; prior mismatch resolved.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[litellm proxy run_server CLI] --> B{--run_granian?}
    B -- Yes --> C[Import granian or raise ImportError]
    C --> D[_init_granian_server]
    D --> E{SSL flags?}
    E -- both cert+key --> F[Add ssl_cert / ssl_key to kwargs]
    E -- only one --> G[raise ClickException]
    E -- neither --> H[No SSL kwargs]
    F --> I{granian_runtime_threads?}
    H --> I
    I -- set --> J[kwargs runtime_threads = N]
    I -- None --> K[omit runtime_threads]
    J --> L[Granian kwargs .serve blocking]
    K --> L
    B -- No --> M{--run_gunicorn?}
    M -- Yes --> N[_run_gunicorn_server]
    M -- No --> O{--run_hypercorn?}
    O -- Yes --> P[_init_hypercorn_server]
    O -- No --> Q[uvicorn.run default path]

Loading

_{Reviews (5): Last reviewed commit: "update uv lock to fix granian import err..." | Re-trigger Greptile}

[greptile-apps]

@patch resolves before pytest.importorskip in new SSL tests

The three newly added SSL tests (test_init_granian_server_ssl, test_init_granian_server_ssl_requires_cert_and_key, and test_init_granian_server_runtime_threads) all carry @patch("granian.Granian") decorators with pytest.importorskip("granian") in their bodies. unittest.mock.patch resolves the target attribute (importing granian and looking up Granian) when the decorated function is invoked, before the function body executes. On an environment where granian is absent, this raises ModuleNotFoundError rather than skipping the test cleanly. The two original granian tests already had this pattern flagged; the same defect is present in all newly-added tests. Moving the skip guard to module level (as suggested in the earlier thread) would fix all five tests at once.

[codecov]

Codecov Report

❌ Patch coverage is 70.58824% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/proxy_cli.py	70.58%	10 Missing ⚠️

📢 Thoughts on this report? Let us know!