Skip to content

Commit c6aa3ea

Browse files
ishaan-berrimichelligabrieleSameerliteredhelixSynergy
authored
Litellm ishaan april1 try2 (#25110)
* Litellm ishaan april1 (#25103) * fix(proxy): enforce upperbound key params on key/update and add custom_key_update hook The /key/update endpoint did not enforce upperbound_key_generate_params, allowing users to bypass configured limits (tpm_limit, rpm_limit, max_budget, duration, budget_duration) by updating an existing key instead of generating a new one. Extract the upperbound enforcement logic from _common_key_generation_helper() into a standalone _enforce_upperbound_key_params() function and call it from both the generate and update paths. For updates, None values are skipped (not filled with defaults) since they mean "don't change this field". Also adds a custom_key_update config option and user_custom_key_update global, mirroring the existing custom_key_generate pattern, so custom key validation logic can fire during key updates as well. * fix(proxy): invoke custom_key_update hook in bulk update path The user_custom_key_update hook was only called in update_key_fn (single key update) but not in _process_single_key_update (bulk update path), allowing custom validation to be bypassed via the /key/update/bulk endpoint. Mirror the hook invocation in both paths. * fix(proxy): pass UpdateKeyRequest to hook in bulk path, not BulkUpdateKeyRequestItem Move the custom_key_update hook invocation to after UpdateKeyRequest is constructed so the hook receives the same type in both single and bulk update paths. Previously the bulk path passed BulkUpdateKeyRequestItem (5 fields only), which would cause AttributeError for hooks accessing fields like tpm_limit or models. * fix(bedrock): promote cache usage to message_delta for Claude Code (#24850) Ensure Bedrock/Anthropic-compatible streaming exposes cache usage where Claude Code reads it by promoting message_stop usage onto message_delta and preserving usage fields in fake-streamed message_delta events. Made-with: Cursor * fix(search): Support self-hosted Firecrawl response format in search transform (#24866) The `transform_search_response` method only handled Firecrawl Cloud (v2) response format where `data` is a dict with `web`/`news` keys. Self-hosted Firecrawl (v1) returns `data` as a flat list of result objects, causing an `AttributeError: 'list' object has no attribute 'get'`. Detect the response format by checking if `data` is a list (self-hosted) or dict (cloud) and handle both cases. Cloud format: {"data": {"web": [...], "news": [...]}} Self-hosted: {"success": true, "data": [{"url": "...", "title": "...", ...}]} Co-authored-by: Synergy <synergyoclaw@gmail.com> * feat: add environment and user tracking to prompt management (#24855) * feat: add environment and user tracking to prompt management - Add environment (development/staging/production) and created_by columns to LiteLLM_PromptTable - Update unique constraint to [prompt_id, version, environment] - All CRUD endpoints support environment filtering and user tracking - Redesigned prompt detail page with environment tabs and version history - UI: environment filter on list page, environment selector in editor - 8 new tests for environment and user tracking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Black formatting and add environments to PromptInfoResponse TypeScript type Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address Greptile review findings - P1: delete_prompt scopes in-memory cleanup to environment when provided - P2: dotprompt_content parsed directly regardless of environment flag - P2: use distinct for environments query - P2: fix double-fetch on initial mount in prompt_info.tsx - fix: remove unsupported select kwarg from find_many Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address remaining Greptile review comments - Remove unused useCallback import (index.tsx) - Remove unused ENV_COLORS variable (prompt_info.tsx) - P1: in-memory fallback in get_prompt_versions now respects environment filter - P1: reset selectedEnv when promptId changes to avoid stale state - Cyclic imports are pre-existing pattern, not introduced by this PR Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: scope patch_prompt to environment using primary key - Add environment query param to patch_prompt endpoint - Look up target row by composite key (prompt_id + version + environment) - Update by primary key (id) to target exactly one row - Fixes Greptile finding: patch with multiple environments no longer ambiguous Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use actual start_time for failed request spend logs (#24906) async_post_call_failure_hook set both start_time and end_time to datetime.now(), making all failed requests show duration=0. Use the actual start_time from litellm_logging_obj instead, so spend logs reflect the real request duration on timeout and other failures. Fixes #24888 * feat(bedrock): add nova canvas image edit support (#24869) * feat(bedrock): add nova canvas image edit support * fix(bedrock): support PathLike inputs for nova image edit * chore: sync schema.prisma copies from root * fix(mypy): correct type-ignore code for delta_usage arg-type * fix(mypy): cast status_code to str, suppress intentional str yield * fix(lint): extract _create_content_block_chunks to fix PLR0915 * fix(lint): extract helpers to fix PLR0915 in prompt endpoints --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: redhelix <amin.lalji@gmail.com> Co-authored-by: Synergy <synergyoclaw@gmail.com> Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: madhu19991 <madhu@thunkai.com> Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(test): update model armor streaming test to handle string or int error code --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: redhelix <amin.lalji@gmail.com> Co-authored-by: Synergy <synergyoclaw@gmail.com> Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: madhu19991 <madhu@thunkai.com> Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent fc885af commit c6aa3ea

34 files changed

Lines changed: 2819 additions & 503 deletions

File tree

‎docs/my-website/docs/providers/bedrock_image_gen.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,29 @@ curl -L -X POST 'http://0.0.0.0:4000/v1/images/generations' \
111111
</TabItem>
112112
</Tabs>
113113

114+
## Amazon Nova Canvas - Image Edit
115+
116+
Use OpenAI-compatible `image_edit()` with Bedrock Nova Canvas (`amazon.nova-canvas-v1:0`). Requests use the same `InvokeModel` API as generation; LiteLLM maps inputs to [Nova Canvas task types](https://docs.aws.amazon.com/nova/latest/userguide/image-gen-access.html):
117+
118+
| Scenario | `taskType` sent to Bedrock |
119+
|----------|----------------------------|
120+
| Image + prompt (no mask) | `IMAGE_VARIATION` |
121+
| Image + prompt + mask | `INPAINTING` (`inPaintingParams.image`, `maskImage` or `maskPrompt`) |
122+
| `taskType: OUTPAINTING` + `mask` or `maskPrompt` | `OUTPAINTING` (Bedrock requires one; LiteLLM raises a clear error if both are missing) |
123+
| `taskType: BACKGROUND_REMOVAL` | `BACKGROUND_REMOVAL` |
124+
125+
```python
126+
from litellm import image_edit
127+
128+
response = image_edit(
129+
image=open("photo.png", "rb"),
130+
prompt="Add soft sunset lighting",
131+
model="bedrock/amazon.nova-canvas-v1:0",
132+
)
133+
```
134+
135+
For **`BACKGROUND_REMOVAL`**, the AWS request must not include `imageGenerationConfig`; LiteLLM omits it for that task even if you pass `size`, `n`, `seed`, etc. Additional Nova Canvas inference IDs for image edit should set **`supports_nova_canvas_image_edit`: true** in `model_prices_and_context_window.json` (see `amazon.nova-canvas-v1:0`).
136+
114137
## Using Inference Profiles with Image Generation
115138

116139
For AWS Bedrock Application Inference Profiles with image generation, use the `model_id` parameter to specify the inference profile ARN:
@@ -147,4 +170,3 @@ model_list:
147170
## Authentication
148171
149172
All standard Bedrock authentication methods are supported for image generation. See [Bedrock Authentication](./bedrock#boto3---authentication) for details.
150-
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
-- AlterTable
2+
ALTER TABLE "LiteLLM_PromptTable" ADD COLUMN "environment" TEXT NOT NULL DEFAULT 'development';
3+
ALTER TABLE "LiteLLM_PromptTable" ADD COLUMN "created_by" TEXT;
4+
5+
-- DropIndex (old unique constraint)
6+
DROP INDEX IF EXISTS "LiteLLM_PromptTable_prompt_id_version_key";
7+
8+
-- CreateIndex (new unique constraint)
9+
CREATE UNIQUE INDEX "LiteLLM_PromptTable_prompt_id_version_environment_key" ON "LiteLLM_PromptTable"("prompt_id", "version", "environment");
10+
11+
-- CreateIndex (new composite index)
12+
CREATE INDEX "LiteLLM_PromptTable_prompt_id_environment_idx" ON "LiteLLM_PromptTable"("prompt_id", "environment");

‎litellm-proxy-extras/litellm_proxy_extras/schema.prisma‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -321,11 +321,12 @@ model LiteLLM_MCPServerTable {
321321
byok_description String[] @default([])
322322
byok_api_key_help_url String?
323323
source_url String?
324-
approval_status String? @default("active")
325-
submitted_by String?
326-
submitted_at DateTime?
327-
reviewed_at DateTime?
328-
review_notes String?
324+
// BYOM submission lifecycle
325+
approval_status String? @default("active")
326+
submitted_by String?
327+
submitted_at DateTime?
328+
reviewed_at DateTime?
329+
review_notes String?
329330
330331
@@index([approval_status])
331332
}
@@ -1001,12 +1002,15 @@ model LiteLLM_PromptTable {
10011002
id String @id @default(uuid())
10021003
prompt_id String
10031004
version Int @default(1)
1005+
environment String @default("development")
1006+
created_by String?
10041007
litellm_params Json
10051008
prompt_info Json?
10061009
created_at DateTime @default(now())
10071010
updated_at DateTime @updatedAt
10081011
1009-
@@unique([prompt_id, version])
1012+
@@unique([prompt_id, version, environment])
1013+
@@index([prompt_id, environment])
10101014
@@index([prompt_id])
10111015
}
10121016

‎litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py‎

Lines changed: 116 additions & 138 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,102 @@ def __init__(self, response: AnthropicMessagesResponse):
3838
self.chunks = self._create_streaming_chunks()
3939
self.current_index = 0
4040

41+
def _create_content_block_chunks(
42+
self, block_dict: Dict[str, Any], index: int
43+
) -> List[bytes]:
44+
"""Build SSE chunks for a single content block."""
45+
chunks = []
46+
block_type = block_dict.get("type")
47+
48+
if block_type == "text":
49+
content_block_start = {
50+
"type": "content_block_start",
51+
"index": index,
52+
"content_block": {"type": "text", "text": ""},
53+
}
54+
chunks.append(
55+
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
56+
)
57+
text = block_dict.get("text", "")
58+
content_block_delta = {
59+
"type": "content_block_delta",
60+
"index": index,
61+
"delta": {"type": "text_delta", "text": text},
62+
}
63+
chunks.append(
64+
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
65+
)
66+
67+
elif block_type == "thinking":
68+
content_block_start = {
69+
"type": "content_block_start",
70+
"index": index,
71+
"content_block": {"type": "thinking", "thinking": "", "signature": ""},
72+
}
73+
chunks.append(
74+
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
75+
)
76+
thinking_text = block_dict.get("thinking", "")
77+
if thinking_text:
78+
content_block_delta = {
79+
"type": "content_block_delta",
80+
"index": index,
81+
"delta": {"type": "thinking_delta", "thinking": thinking_text},
82+
}
83+
chunks.append(
84+
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
85+
)
86+
signature = block_dict.get("signature", "")
87+
if signature:
88+
signature_delta = {
89+
"type": "content_block_delta",
90+
"index": index,
91+
"delta": {"type": "signature_delta", "signature": signature},
92+
}
93+
chunks.append(
94+
f"event: content_block_delta\ndata: {json.dumps(signature_delta)}\n\n".encode()
95+
)
96+
97+
elif block_type == "redacted_thinking":
98+
content_block_start = {
99+
"type": "content_block_start",
100+
"index": index,
101+
"content_block": {"type": "redacted_thinking"},
102+
}
103+
chunks.append(
104+
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
105+
)
106+
107+
elif block_type == "tool_use":
108+
content_block_start = {
109+
"type": "content_block_start",
110+
"index": index,
111+
"content_block": {
112+
"type": "tool_use",
113+
"id": block_dict.get("id"),
114+
"name": block_dict.get("name"),
115+
"input": {},
116+
},
117+
}
118+
chunks.append(
119+
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
120+
)
121+
input_data = block_dict.get("input", {})
122+
content_block_delta = {
123+
"type": "content_block_delta",
124+
"index": index,
125+
"delta": {"type": "input_json_delta", "partial_json": json.dumps(input_data)},
126+
}
127+
chunks.append(
128+
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
129+
)
130+
131+
content_block_stop = {"type": "content_block_stop", "index": index}
132+
chunks.append(
133+
f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()
134+
)
135+
return chunks
136+
41137
def _create_streaming_chunks(self) -> List[bytes]:
42138
"""Convert the non-streaming response to streaming chunks"""
43139
chunks = []
@@ -69,152 +165,34 @@ def _create_streaming_chunks(self) -> List[bytes]:
69165

70166
# 2-4. For each content block, send start/delta/stop events
71167
content_blocks = response_dict.get("content", [])
72-
if content_blocks:
73-
for index, block in enumerate(content_blocks):
74-
# Cast block to dict for easier access
75-
block_dict = cast(Dict[str, Any], block)
76-
block_type = block_dict.get("type")
77-
78-
if block_type == "text":
79-
# content_block_start
80-
content_block_start = {
81-
"type": "content_block_start",
82-
"index": index,
83-
"content_block": {"type": "text", "text": ""},
84-
}
85-
chunks.append(
86-
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
87-
)
88-
89-
# content_block_delta (send full text as one delta for simplicity)
90-
text = block_dict.get("text", "")
91-
content_block_delta = {
92-
"type": "content_block_delta",
93-
"index": index,
94-
"delta": {"type": "text_delta", "text": text},
95-
}
96-
chunks.append(
97-
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
98-
)
99-
100-
# content_block_stop
101-
content_block_stop = {"type": "content_block_stop", "index": index}
102-
chunks.append(
103-
f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()
104-
)
105-
106-
elif block_type == "thinking":
107-
# content_block_start for thinking
108-
content_block_start = {
109-
"type": "content_block_start",
110-
"index": index,
111-
"content_block": {
112-
"type": "thinking",
113-
"thinking": "",
114-
"signature": "",
115-
},
116-
}
117-
chunks.append(
118-
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
119-
)
120-
121-
# content_block_delta for thinking text
122-
thinking_text = block_dict.get("thinking", "")
123-
if thinking_text:
124-
content_block_delta = {
125-
"type": "content_block_delta",
126-
"index": index,
127-
"delta": {
128-
"type": "thinking_delta",
129-
"thinking": thinking_text,
130-
},
131-
}
132-
chunks.append(
133-
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
134-
)
135-
136-
# content_block_delta for signature (if present)
137-
signature = block_dict.get("signature", "")
138-
if signature:
139-
signature_delta = {
140-
"type": "content_block_delta",
141-
"index": index,
142-
"delta": {
143-
"type": "signature_delta",
144-
"signature": signature,
145-
},
146-
}
147-
chunks.append(
148-
f"event: content_block_delta\ndata: {json.dumps(signature_delta)}\n\n".encode()
149-
)
150-
151-
# content_block_stop
152-
content_block_stop = {"type": "content_block_stop", "index": index}
153-
chunks.append(
154-
f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()
155-
)
156-
157-
elif block_type == "redacted_thinking":
158-
# content_block_start for redacted_thinking
159-
content_block_start = {
160-
"type": "content_block_start",
161-
"index": index,
162-
"content_block": {"type": "redacted_thinking"},
163-
}
164-
chunks.append(
165-
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
166-
)
167-
168-
# content_block_stop (no delta for redacted thinking)
169-
content_block_stop = {"type": "content_block_stop", "index": index}
170-
chunks.append(
171-
f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()
172-
)
173-
174-
elif block_type == "tool_use":
175-
# content_block_start
176-
content_block_start = {
177-
"type": "content_block_start",
178-
"index": index,
179-
"content_block": {
180-
"type": "tool_use",
181-
"id": block_dict.get("id"),
182-
"name": block_dict.get("name"),
183-
"input": {},
184-
},
185-
}
186-
chunks.append(
187-
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
188-
)
189-
190-
# content_block_delta (send input as JSON delta)
191-
input_data = block_dict.get("input", {})
192-
content_block_delta = {
193-
"type": "content_block_delta",
194-
"index": index,
195-
"delta": {
196-
"type": "input_json_delta",
197-
"partial_json": json.dumps(input_data),
198-
},
199-
}
200-
chunks.append(
201-
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
202-
)
203-
204-
# content_block_stop
205-
content_block_stop = {"type": "content_block_stop", "index": index}
206-
chunks.append(
207-
f"event: content_block_stop\ndata: {json.dumps(content_block_stop)}\n\n".encode()
208-
)
168+
for index, block in enumerate(content_blocks):
169+
block_dict = cast(Dict[str, Any], block)
170+
chunks.extend(self._create_content_block_chunks(block_dict, index))
209171

210172
# 5. message_delta event (with final usage and stop_reason)
173+
# Include cache usage fields so clients that only read message_delta
174+
# (like Claude Code's SDK) see the full input token breakdown.
175+
delta_usage: Dict[str, Any] = {
176+
"output_tokens": usage.get("output_tokens", 0) if usage else 0,
177+
}
178+
if usage:
179+
if usage.get("input_tokens") is not None:
180+
delta_usage["input_tokens"] = usage["input_tokens"]
181+
if usage.get("cache_creation_input_tokens") is not None:
182+
delta_usage["cache_creation_input_tokens"] = usage[
183+
"cache_creation_input_tokens"
184+
]
185+
if usage.get("cache_read_input_tokens") is not None:
186+
delta_usage["cache_read_input_tokens"] = usage[
187+
"cache_read_input_tokens"
188+
]
211189
message_delta = {
212190
"type": "message_delta",
213191
"delta": {
214192
"stop_reason": response_dict.get("stop_reason"),
215193
"stop_sequence": response_dict.get("stop_sequence"),
216194
},
217-
"usage": {"output_tokens": usage.get("output_tokens", 0) if usage else 0},
195+
"usage": delta_usage,
218196
}
219197
chunks.append(
220198
f"event: message_delta\ndata: {json.dumps(message_delta)}\n\n".encode()

0 commit comments

Comments
 (0)