Repository navigation
fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary - #44800
Conversation
|
| # and keep the client-facing Anthropic error frame. | ||
| if self.litellm_logging_obj is not None: | ||
| try: | ||
| await self.litellm_logging_obj.async_failure_handler( |
There was a problem hiding this comment.
Sync failure callbacks are skipped When a standalone async Messages stream fails, this calls
async_failure_handler directly. That handler runs only async callbacks; unlike dispatch_failure_handlers, it does not schedule sync failure callbacks. SDK users who configure only litellm.failure_callback receive the error frame but never receive their failure callback.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…failure boundary AnthropicStreamWrapper.async_anthropic_sse_wrapper swallowed every upstream exception and emitted its own Anthropic error frame, so a provider drop on a streamed /v1/messages call never reached the proxy's streaming boundary — no failure spend row, no failure callbacks. Re-raise when the stream is proxy-managed (detached failure hook armed) so async_streaming_data_generator runs post_call_failure_hook once and serializes the error frame; when consumed standalone (SDK litellm.messages path), run the logging object's async failure handler and keep the client-facing error frame. Fixes BerriAI#44742
74088c3 to
125a25e
Compare
|
Addressed the review, thanks for catching the real one. Sync failure callbacks skipped (P1, line 1065): confirmed, fixed. Transformed failures lose error events (P1, line 1059): not a regression from this PR. That Proxy outcome untested (P2, test_sse_wrapper.py): addressed, tests moved. The regression cases now live in the mapped file Verification: 10 passed in that file, 375 passed across |
…elope so the router keeps its pre-content retry
…1/messages streams Nine integration cells for the chat bridge: raw httpx and the Anthropic SDK (sync and async) streams that fail after the first content delta now land exactly one failure SpendLogs row carrying the provider's error class, the non-streaming 500, the response-cache twin and a client that leaves mid-stream keep their bookkeeping, a 24-stream burst lands every call id once, and the standalone SDK stream reports the provider error to failure callbacks once.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8ce7974. Configure here.
mateo-berri
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the contribution!
…essages error frames (#45465) #44800 re-raises the provider's own error from the chat adapter, so a bridged /v1/messages stream's error frame now reads litellm.RateLimitError without the litellm.MidStreamFallbackError: prefix that #44989's tests pinned shortly before #44800 merged. The three tests now expect the provider error text once and no sentinel, matching the chat_limited case and every other assertion in the two files Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
TLDR
Problem this solves:
/v1/messagescall that fails mid-stream writes no spend log row/v1/chat/completionserrorevent, so nothing looks wrongHow it solves it:
erroreventerroreventUser Flow
Before: a developer streaming
/v1/messagesonto a non-Anthropic deployment (an Azure or Vertex model served through the chat translation) sees the provider error arrive in the stream, but the gateway never records the call"stream": trueand their model, and the provider rate-limits or drops the stream after it startedevent: message_startfollowed byevent: errorcarrying the provider's error (rate_limit_errorfor a 429,api_errorfor a dropped connection)status = failure, so the two endpoints disagreeAfter: the same failure lands as one failure row, like every other endpoint
"stream": trueand their model, and the provider rate-limits or drops the stream after it startedevent: message_startfollowed byevent: errorcarrying the provider's error, unchanged on the wirestatus = failureandspend = 0, and the failure callbacks ranRelevant issues
Fixes #44742
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Last updated: 8ce7974, which adds the /audit integration cells and no product code. QA, /live-pr-risk and /audit ran on 2709675, the last commit with a product diff
Two proxies with a Postgres database, 2 uvicorn workers each, the same config: model group
gpt-6.1-sol->azure/gpt-6.1-sol-lit9298, a real Azure Foundry deployment in eastus2 with a 1K TPM limit. Before: merge base 127278f on port 50989. After: 2709675 on port 40410. Every request carries one 19K-character user message, aget_weathertool,max_tokens: 32and"stream": true, so Azure accepts the request and rate-limits the stream aftermessage_start. The clients are the Anthropic Python SDK and curl on/v1/messages, and the OpenAI Python SDK on/v1/chat/completionsas the control that already logged its failureBefore (127278f, port 50989)
Anthropic SDK
curl
OpenAI SDK control on
/v1/chat/completionsSpend rows,
GET /spend/logs?request_id=<x-litellm-call-id>polled every 5 s for up to 120 sBoth
/v1/messagescalls answered 200 with theerrorevent and never got a spend row; the chat control got its failure rowAfter (2709675, port 40410)
The same three calls against port 40410
Every call now has exactly one
failurerow within 6 s, and the wire still carriesmessage_startthen therate_limit_errorframe on a 200. The frame text now names the provider error without thelitellm.MidStreamFallbackError:prefixSDK without a proxy, both commits
The chat stream wrapper already logs this Azure failure once at the SDK level, and the adapter's own dispatch does not log it a second time
Observations from the run, none caused by this PR unless said so
call_typeresponses, the chat control too: the Azure model's chat call runs through the chat-to-responses bridge/v1/messageserror text names the provider error and drops thelitellm.MidStreamFallbackError:prefix (this PR); the chat control keeps itmessage_startnames the group, the row names the deploymentLive A/B on the dependents (/live-pr-risk)
Verdict: at 2709675 no dependent path answers differently from the merge base except where this PR says so (the failure row, the failure callbacks, and the error text). The previous tip, c208fae, broke the chat bridge's pre-content retry; 2709675 fixes it
Rig: the two proxies above (real Azure, a wire recorder in front of it, a
CustomLoggerrecording every hook) plus the integration harness undertests/integration(a scripted upstream,num_retries: 0unless the test sets its own), both booted from the merge base and from the PR tree, every scenario run on both sidesBreaking
/v1/messagesstream that dropped after the role-only chat chunk, before any content, was no longer retried:tests/integration/messages_endpoint/chat_bridge/test_chat_bridge_pre_content_retry_wire.pypassed 12/12 at the merge base, 11/12 at c208fae ([drop_after_pre_content_frame-chat]saw one upstream POST and anerrorevent instead of a second POST), 12/12 at 2709675. The chat stream wrapper raises aMidStreamFallbackErrorwhoseis_pre_first_chunkis false once the role-only chunk went out, and the router declines to retry a raised envelope that says so even though the Anthropic client had only seenmessage_start. Fixed in 2709675: the adapter re-raises the provider's own exception, which the router judges by its own content flagBackward incompatible
errorframe text on a bridged/v1/messagesfailure no longer starts withlitellm.MidStreamFallbackError:(before and after texts are in the proof above).type(rate_limit_error) and the 200 status are unchanged. Kept on purpose in 2709675: the text now names the same error the failure row and the callbacks recordexception_classand the OTEL error for a bridged/v1/messagesstream name the provider's exception class (RateLimitErrorhere), where/v1/chat/completionsnamesMidStreamFallbackErrorfor the same Azure failure (the chat control above). Chosen in 2709675 over the two alternatives: raising the envelope breaks the pre-content retry (c208fae), and teaching the router to judge a raised envelope by its own content flag changes a rule every source iterator shareslitellm.anthropic.messages.acreatewithout a proxy) now get their failure callbacks when the provider iterator itself raises mid-stream, with the provider's exception object; before, nothing fired on that shape. A failure the chat stream wrapper already logs (the Azure rate limit above) is still logged once on both commitsRegression risk
/v1/messages: the exception now travels through the guardrail's iterator the way it does on/v1/chat/completionstoday, so text the guardrail still holds when the provider fails is not delivered before theerrorframe; at the merge base the frame was an ordinary chunk and the guardrail flushed first. Not driven live, no streaming guardrail vendor was in the rigerrorframe, the same as every other streaming route today. Not driven liveDependency graph
AnthropicStreamWrapper.async_anthropic_sse_wrapperexcept branch (the diff): verified live, proxy and SDK, both commits_aanthropic_messages_recover_stream_errorand_anthropic_stream_fallback_error_for_raised(raised-error retry and fallback): verified live, the 12 bridge cases, both commitspost_call_failure_hookconsumers (spend row,async_post_call_failure_hook, alerts, Prometheus, OTEL): verified live for the row and the hooks, both commits; Prometheus and OTEL traced from the hook's exception, not scrapedanthropic_error_sse_frame(frame type from the raised status, pre-message_startJSON error): verified live forrate_limit_erroron a 200 aftermessage_start; the pre-message_startpath is the[http_500-*]and[drop_after_headers-*]bridge casesLogging.dispatch_failure_handlersdedupe against the chat stream wrapper's owncreate_taskdispatch: verified live, callbacks fire once on both commitstests/integration/providers/test_bedrock_converse_stream_event_frames_wire.py,test_bedrock_stream_timeout_wire.py): 78 passed on both commits;test_c02_worker_sigkill_mid_burst_leaves_the_sibling_rejecting_the_held_streamsfailed on both with "Owned proxy readiness deadline exceeded" (it boots its own proxy on a saturated machine), no difference between the legstests/unit/llms/anthropic/pass_through/...,tests/unit/proxy/test_common_request_processing.py): 800 passed at the tipmessages/streaming_iterator.py), the responses adapter (responses_adapters/streaming_iterator.py), Vertex and Bedrock native routes: not on the diff's path, untouchedPre-existing on both commits, not caused by this PR
/v1/messagesleaves no spend row, while the same disconnect on/v1/chat/completionswrites a success row; tracked in LIT-8603[200, 401, 200, 401, 200, 401]right after the delete, all 401 after 72 s); filed separatelyNot verified
post_call_failure_hookgot, which the hook recorder captured/audit: integration cells, merge base vs head
Rig: three legs on one box through
tests/integration/run.py, each with its own Postgres database, redis, scripted upstream (tests/integration/_support) and two-worker proxy ontests/integration/proxy_config.yaml. base = merge base 4417bf0, head = 2709675 (run twice with the same seeds), ctl = 2709675 for the controls, every lane--seed 697 --order-seed 697, lanes side by side. Run ids:base-providers,base-sdk,head-providers-run1,head-sdk-run1,head-providers-run2,head-sdk-run2,ctl-providers,ctl-extensions(roundslp697, 352 s wall). The cells are committed in 8ce7974 byte-identical to the files the lanes ran. Every call id below is the client-suppliedx-litellm-call-idthe cell generated, echoed in the response header and read back asmetadata->>'litellm_call_id'on theLiteLLM_SpendLogsrowtests/integration/messages_endpoint/chat_bridge/test_chat_bridge_mid_stream_failure_bookkeeping_wire.py::unless named)test_bridged_stream_failing_after_content_writes_one_failure_row[drop_after_content]error_codeagreetest_bridged_stream_failing_after_content_writes_one_failure_row[error_frame_after_content]messages.streamtest_anthropic_sdk_sync_stream_failing_after_content_raises_and_writes_one_failure_rowAPIStatusErrorafter "Hello", one failure row)test_anthropic_sdk_async_stream_failing_after_content_raises_and_writes_one_failure_rowtest_bridged_request_rejected_before_any_body_keeps_its_error_and_failure_rowtest_bridged_stream_served_from_the_response_cache_keeps_its_success_bookkeepingcache_hitTrue row)test_client_leaving_a_bridged_stream_before_its_failure_leaves_the_proxy_servingtest_concurrent_bridged_streams_failing_after_content_each_land_one_rowlitellm.anthropic.messages.acreate(stream=True)tests/integration/sdk/test_anthropic_messages_bridge_mid_stream_failure_sdk.py::test_sdk_bridged_stream_failing_after_content_reports_the_provider_error_oncetest_chat_bridge_pre_content_retry_wire.py(pre-content retry per deployment and request budget, SDK retry, drop after content not retried, chat and responses bridges),test_anthropic_thinking_signature_logging_wire.py(anthropic and bedrock invoke pass-through streams),test_bedrock_passthrough_stream_wire.py::test_bedrock_passthrough_converse_stream_response_carries_event_stream_content_type,routing/test_deployment_num_retries_generic_routes_wire.py::test_vllm_passthrough_rejected_before_it_opens_is_retried_per_the_deployment_budget[stream],observability/test_passthrough_upstream_error_visibility.py::test_gemini_passthrough_streaming_200_relays_every_chunkPass-through routes never run the changed wrapper (it is built only by the chat adapter at
adapters/transformation.py), so their streaming rows are controls. No inventory row is unverifiedType
🐛 Bug Fix
Caveats (if any)
Medium
errorframe;/v1/chat/completionsbehaves the same today. Not driven liveLow
/v1/messageserror frame text drops thelitellm.MidStreamFallbackError:prefix and names the provider error;typeand status are unchanged/v1/messagesstream name the provider's exception class, where the chat route namesMidStreamFallbackErrorerrorevent, the same as every other streaming route todayosv-scanis red on sixnext16.3.6 advisories published 2026-10-07 20:30Z, after main's last green scan; the dashboard lockfile is identical on main, so main's next scan goes red the same way until the dependency bump landsFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
2709675 passes /live-pr-risk
8ce7974 passes /audit (ran at 2709675, the same product code)
Note
Medium Risk
Changes streaming error handling on a heavily used proxy route (observability and re-raise vs swallow); wire error message text and logged exception class shift slightly for bridged failures, with intentional behavior for router retry.
Overview
Fixes bridged
/v1/messagesstreams that fail after content already started: the Anthropic chat adapter no longer treats mid-stream errors as “emit an SSEerrorframe and stop.” It now routes failures through LiteLLM’s logging boundary so spend logs and callbacks match/v1/chat/completions.In
async_anthropic_sse_wrapper, when the upstream iterator raises,_provider_errorunwrapsMidStreamFallbackErrorto the underlying provider exception (so router retry/fallback and error text stay correct). Ifon_detached_stream_failureis set (proxy-managed stream), that exception is re-raised for the proxy to record failure and still send the client-facing error. Otherwise the adapter callsdispatch_failure_handlerswith the provider error, then yields the same AnthropicerrorSSE event; a broken failure handler cannot suppress the frame.AnthropicStreamWrapperkeeps a reference tolitellm_logging_objfor this path. New integration, SDK, and unit tests cover failure rows, single callback dispatch, concurrent streams, and proxy vs standalone behavior.Reviewed by Cursor Bugbot for commit 8ce7974. Bugbot is set up for automated code reviews on this repo. Configure here.