Skip to content

Commit 4b48b29

Browse files
anthonyshewanthonyshew
authored
Add LiteLLM Gateway example (#1432)
1 parent a7488dd commit 4b48b29

14 files changed

Lines changed: 624 additions & 0 deletions

File tree

‎python/litellm-gateway/.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
node_modules/
2+
.next/
3+
__pycache__/
4+
*.pyc
5+
.env
6+
.env.local
7+
.vercel

‎python/litellm-gateway/README.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# LiteLLM Gateway on Vercel
2+
3+
Self-hosted AI gateway using [LiteLLM](https://www.litellm.ai/) deployed on Vercel with [Services](https://vercel.com/docs/services). A Next.js chat frontend talks to a LiteLLM proxy backend — both deployed as a single Vercel project.
4+
5+
[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fvercel%2Fexamples%2Ftree%2Fmain%2Fpython%2Flitellm-gateway&env=VERCEL_AI_GATEWAY_API_KEY&envDescription=Vercel%20AI%20Gateway%20API%20key%20from%20your%20dashboard&envLink=https%3A%2F%2Fvercel.com%2Fdocs%2Fai-gateway)
6+
7+
## Architecture
8+
9+
Two services deployed together via `experimentalServices` in `vercel.json`:
10+
11+
- **frontend** (Next.js) at `/` — Chat UI using the [AI SDK](https://sdk.vercel.ai)
12+
- **gateway** (LiteLLM/FastAPI) at `/gateway` — OpenAI-compatible proxy that routes to any LLM provider
13+
14+
```
15+
Browser → /api/chat (Next.js) → GATEWAY_URL/v1/chat/completions (LiteLLM) → Vercel AI Gateway → Provider
16+
```
17+
18+
Vercel Services automatically generates a `GATEWAY_URL` environment variable so the frontend can reach the gateway without hardcoded URLs.
19+
20+
## Project structure
21+
22+
```
23+
litellm-gateway/
24+
├── gateway/
25+
│ ├── app.py # LiteLLM proxy entrypoint
26+
│ ├── litellm_config.yaml # Model + provider config
27+
│ └── pyproject.toml # Python dependencies
28+
├── frontend/
29+
│ ├── app/
30+
│ │ ├── api/chat/route.ts # Proxies to LiteLLM via GATEWAY_URL
31+
│ │ ├── layout.tsx
32+
│ │ ├── page.tsx # Chat UI
33+
│ │ └── globals.css
34+
│ ├── package.json
35+
│ └── next.config.js
36+
└── vercel.json # Services configuration
37+
```
38+
39+
## Setup
40+
41+
### 1. Configure models
42+
43+
Edit `gateway/litellm_config.yaml` to define your model routing. The default config routes through the [Vercel AI Gateway](https://vercel.com/docs/ai-gateway):
44+
45+
```yaml
46+
model_list:
47+
- model_name: gpt-4o-mini
48+
litellm_params:
49+
model: vercel_ai_gateway/openai/gpt-4o-mini
50+
api_key: os.environ/VERCEL_AI_GATEWAY_API_KEY
51+
```
52+
53+
You can also route directly to providers (OpenAI, Anthropic, etc.) — see the [LiteLLM provider docs](https://docs.litellm.ai/docs/providers/) and the [config reference](https://docs.litellm.ai/docs/proxy/configs).
54+
55+
### 2. Set environment variables
56+
57+
| Variable | Required | Description |
58+
| --------------------------- | ------------------------ | --------------------------------------------------------------- |
59+
| `VERCEL_AI_GATEWAY_API_KEY` | Yes (for default config) | [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) API key |
60+
| `LITELLM_MASTER_KEY` | No | Require auth for LiteLLM proxy endpoints |
61+
62+
If routing directly to providers instead, set their API keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.) and update `litellm_config.yaml`.
63+
64+
### 3. Update the frontend model list
65+
66+
The chat UI has a hardcoded model list in `frontend/app/page.tsx`. Update the `MODELS` array to match your `litellm_config.yaml`.
67+
68+
### 4. Deploy
69+
70+
Set the project framework to **Services** in your Vercel project settings, then:
71+
72+
```bash
73+
vercel deploy
74+
```
75+
76+
## Local development
77+
78+
Install frontend dependencies:
79+
80+
```bash
81+
cd frontend
82+
npm install
83+
```
84+
85+
Run all services together:
86+
87+
```bash
88+
cd ..
89+
vercel dev -L
90+
```
91+
92+
Open http://localhost:3000 to use the chat UI. The LiteLLM gateway runs at `/gateway` — try `/gateway/health/liveliness` to verify it's up.
93+
94+
## How it works
95+
96+
The Next.js API route at `/api/chat` creates an OpenAI-compatible client pointed at the LiteLLM gateway:
97+
98+
```ts
99+
const litellm = createOpenAI({
100+
baseURL: `${process.env.GATEWAY_URL}/v1`,
101+
apiKey: process.env.LITELLM_MASTER_KEY || 'not-needed',
102+
})
103+
```
104+
105+
`GATEWAY_URL` is auto-generated by Vercel Services — no hardcoded URLs needed. The AI SDK's `streamText` function handles streaming the LLM response back to the browser.
106+
107+
## Tuning
108+
109+
The gateway service is configured with `maxDuration: 120` (seconds) in `vercel.json`. You can also set `memory` (128–10240 MB) if your config requires more resources. See the [Services docs](https://vercel.com/docs/services) for all options.

‎python/litellm-gateway/frontend/app/api/chat/route.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { createOpenAI } from '@ai-sdk/openai'
2+
import { streamText, convertToModelMessages, type UIMessage } from 'ai'
3+
import { headers } from 'next/headers'
4+
5+
export const maxDuration = 120
6+
7+
export async function POST(req: Request) {
8+
const {
9+
messages,
10+
model = 'gpt-4o-mini',
11+
}: { messages: UIMessage[]; model: string } = await req.json()
12+
13+
// Forward auth cookies so the request passes Vercel's deployment protection
14+
const requestHeaders = await headers()
15+
const cookie = requestHeaders.get('cookie') || ''
16+
17+
const litellm = createOpenAI({
18+
baseURL: `${process.env.GATEWAY_URL}/v1`,
19+
apiKey: process.env.LITELLM_MASTER_KEY || 'not-needed',
20+
headers: { cookie },
21+
})
22+
23+
const result = streamText({
24+
model: litellm(model),
25+
messages: await convertToModelMessages(messages),
26+
})
27+
28+
return result.toUIMessageStreamResponse()
29+
}

‎python/litellm-gateway/frontend/app/globals.css‎

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
* {
2+
margin: 0;
3+
padding: 0;
4+
box-sizing: border-box;
5+
}
6+
7+
body {
8+
font-family:
9+
-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu,
10+
Cantarell, sans-serif;
11+
background: #000;
12+
color: #fff;
13+
line-height: 1.6;
14+
height: 100vh;
15+
overflow: hidden;
16+
}
17+
18+
.container {
19+
display: flex;
20+
flex-direction: column;
21+
height: 100vh;
22+
}
23+
24+
/* Nav */
25+
26+
header {
27+
border-bottom: 1px solid #222;
28+
flex-shrink: 0;
29+
}
30+
31+
nav {
32+
display: flex;
33+
align-items: center;
34+
padding: 0.75rem 1rem;
35+
gap: 0.75rem;
36+
max-width: 800px;
37+
margin: 0 auto;
38+
width: 100%;
39+
}
40+
41+
.logo {
42+
font-weight: 600;
43+
font-size: 0.9rem;
44+
}
45+
46+
.new-chat {
47+
background: #111;
48+
border: 1px solid #333;
49+
color: #fff;
50+
width: 32px;
51+
height: 32px;
52+
border-radius: 6px;
53+
cursor: pointer;
54+
font-size: 1.1rem;
55+
display: flex;
56+
align-items: center;
57+
justify-content: center;
58+
transition: background 0.15s;
59+
}
60+
61+
.new-chat:hover {
62+
background: #222;
63+
}
64+
65+
.model-select {
66+
margin-left: auto;
67+
background: #111;
68+
border: 1px solid #333;
69+
color: #fff;
70+
padding: 0.4rem 0.6rem;
71+
border-radius: 6px;
72+
font-size: 0.8rem;
73+
cursor: pointer;
74+
}
75+
76+
/* Main */
77+
78+
main {
79+
flex: 1;
80+
overflow-y: auto;
81+
padding: 1rem;
82+
}
83+
84+
.empty-state {
85+
display: flex;
86+
flex-direction: column;
87+
align-items: center;
88+
justify-content: center;
89+
height: 100%;
90+
text-align: center;
91+
gap: 1rem;
92+
}
93+
94+
.empty-state h1 {
95+
font-size: 2.5rem;
96+
font-weight: 700;
97+
background: linear-gradient(to right, #fff, #888);
98+
-webkit-background-clip: text;
99+
-webkit-text-fill-color: transparent;
100+
background-clip: text;
101+
}
102+
103+
.empty-state p {
104+
color: #888;
105+
font-size: 0.95rem;
106+
max-width: 500px;
107+
}
108+
109+
.empty-state a {
110+
color: #999;
111+
text-decoration: underline;
112+
text-underline-offset: 2px;
113+
}
114+
115+
.empty-state a:hover {
116+
color: #fff;
117+
}
118+
119+
/* Messages */
120+
121+
.messages {
122+
max-width: 800px;
123+
margin: 0 auto;
124+
display: flex;
125+
flex-direction: column;
126+
gap: 1.5rem;
127+
padding-bottom: 1rem;
128+
}
129+
130+
.message {
131+
display: flex;
132+
flex-direction: column;
133+
gap: 0.25rem;
134+
}
135+
136+
.message-role {
137+
font-size: 0.7rem;
138+
font-weight: 600;
139+
color: #555;
140+
text-transform: uppercase;
141+
letter-spacing: 0.05em;
142+
}
143+
144+
.message.user .message-content {
145+
background: #111;
146+
border: 1px solid #222;
147+
border-radius: 8px;
148+
padding: 0.75rem 1rem;
149+
}
150+
151+
.message.assistant .message-content {
152+
color: #ccc;
153+
line-height: 1.7;
154+
white-space: pre-wrap;
155+
}
156+
157+
/* Error */
158+
159+
.error {
160+
max-width: 800px;
161+
margin: 0 auto 0.5rem;
162+
padding: 0.75rem 1rem;
163+
background: #1a0000;
164+
border: 1px solid #4a0000;
165+
border-radius: 8px;
166+
color: #ff6b6b;
167+
font-size: 0.85rem;
168+
width: calc(100% - 2rem);
169+
}
170+
171+
/* Input */
172+
173+
footer {
174+
border-top: 1px solid #222;
175+
padding: 1rem;
176+
flex-shrink: 0;
177+
}
178+
179+
form {
180+
max-width: 800px;
181+
margin: 0 auto;
182+
}
183+
184+
.input-row {
185+
display: flex;
186+
gap: 0.5rem;
187+
background: #111;
188+
border: 1px solid #333;
189+
border-radius: 10px;
190+
padding: 0.5rem;
191+
}
192+
193+
.input-row input {
194+
flex: 1;
195+
background: transparent;
196+
border: none;
197+
color: #fff;
198+
font-size: 0.9rem;
199+
padding: 0.4rem 0.5rem;
200+
outline: none;
201+
}
202+
203+
.input-row input::placeholder {
204+
color: #555;
205+
}
206+
207+
.send {
208+
background: #fff;
209+
color: #000;
210+
border: none;
211+
padding: 0.4rem 1rem;
212+
border-radius: 6px;
213+
font-size: 0.85rem;
214+
font-weight: 500;
215+
cursor: pointer;
216+
transition: opacity 0.15s;
217+
}
218+
219+
.send:disabled {
220+
opacity: 0.3;
221+
cursor: not-allowed;
222+
}
223+
224+
.send:not(:disabled):hover {
225+
opacity: 0.9;
226+
}
227+
228+
@media (max-width: 640px) {
229+
.empty-state h1 {
230+
font-size: 1.8rem;
231+
}
232+
233+
nav {
234+
padding: 0.5rem 0.75rem;
235+
}
236+
}

0 commit comments

Comments
 (0)