Skip to content

fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary - #44800

Merged
mateo-berri merged 11 commits into
BerriAI:mainfrom
Winoren:fix/anthropic-messages-mid-stream-failure-bookkeeping
Oct 8, 2026
Merged

mateo-berri merged 11 commits into
BerriAI:mainfrom
Winoren:fix/anthropic-messages-mid-stream-failure-bookkeeping

Conversation

@Winoren

@Winoren Winoren commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • A streamed /v1/messages call that fails mid-stream writes no spend log row
  • Its failure callbacks and alerts never run, unlike /v1/chat/completions
  • The client still gets the right Anthropic error event, so nothing looks wrong

How it solves it:

  • The adapter re-raises a mid-stream error instead of swallowing it
  • The proxy's streaming boundary then logs the failure and sends the same error event
  • Outside the proxy (SDK use), the adapter runs failure logging itself and keeps the error event
  • What it re-raises is the provider's own error, not the chat stream wrapper's fallback envelope, so the router's pre-content retry and fallback keep working

User Flow

Before: a developer streaming /v1/messages onto a non-Anthropic deployment (an Azure or Vertex model served through the chat translation) sees the provider error arrive in the stream, but the gateway never records the call

  1. They send POST https://litellm-domain/v1/messages with "stream": true and their model, and the provider rate-limits or drops the stream after it started
  2. The response is 200 with event: message_start followed by event: error carrying the provider's error (rate_limit_error for a 429, api_error for a dropped connection)
  3. GET https://litellm-domain/spend/logs?request_id= returns nothing a minute later, and no failure callback or alert fired
  4. The same failure sent to POST https://litellm-domain/v1/chat/completions is listed with status = failure, so the two endpoints disagree

After: the same failure lands as one failure row, like every other endpoint

  1. They send POST https://litellm-domain/v1/messages with "stream": true and their model, and the provider rate-limits or drops the stream after it started
  2. The response is 200 with event: message_start followed by event: error carrying the provider's error, unchanged on the wire
  3. GET https://litellm-domain/spend/logs?request_id= lists one row with status = failure and spend = 0, and the failure callbacks ran
  4. The same failure sent to POST https://litellm-domain/v1/chat/completions is listed the same way

Relevant issues

Fixes #44742

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Last updated: 8ce7974, which adds the /audit integration cells and no product code. QA, /live-pr-risk and /audit ran on 2709675, the last commit with a product diff

Two proxies with a Postgres database, 2 uvicorn workers each, the same config: model group gpt-6.1-sol -> azure/gpt-6.1-sol-lit9298, a real Azure Foundry deployment in eastus2 with a 1K TPM limit. Before: merge base 127278f on port 50989. After: 2709675 on port 40410. Every request carries one 19K-character user message, a get_weather tool, max_tokens: 32 and "stream": true, so Azure accepts the request and rate-limits the stream after message_start. The clients are the Anthropic Python SDK and curl on /v1/messages, and the OpenAI Python SDK on /v1/chat/completions as the control that already logged its failure

Before (127278f, port 50989)

Anthropic SDK

client = anthropic.Anthropic(base_url="http://127.0.0.1:50989", api_key=KEY, max_retries=0)
raw = client.messages.with_raw_response.create(model="gpt-6.1-sol", max_tokens=32, messages=MESSAGES, tools=TOOLS, stream=True)
call_id = raw.headers["x-litellm-call-id"]
events = [event.type for event in raw.parse()]
http 200 call_id bbaa6605-ba56-460f-a34d-bc87117f1ab5 events ['message_start'] raised APIStatusError: {'type': 'error', 'error': {'type': 'rate_limit_error', 'message': 'litellm.MidStreamFallbackError: litellm.MidStreamFallbackError: litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have exceeded token rate limit. ...'}}

curl

curl -sN -D headers.txt http://127.0.0.1:50989/v1/messages -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" -d @messages.json
HTTP/1.1 200 OK
x-litellm-call-id: 499c5e93-bf9e-489d-a074-ad0fc3236af8

event: message_start
data: {"type":"message_start","message":{"id":"msg_d6ca9c35-b39e-409b-b869-c6958a3d2f6e","type":"message","role":"assistant","content":[],"model":"gpt-6.1-sol","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"output_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}}}

event: error
data: {"type": "error", "error": {"type": "rate_limit_error", "message": "litellm.MidStreamFallbackError: litellm.MidStreamFallbackError: litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have exceeded token rate limit. Original exception: RateLimitError: litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have exceeded token rate limit."}}

OpenAI SDK control on /v1/chat/completions

client = openai.OpenAI(base_url="http://127.0.0.1:50989/v1", api_key=KEY, max_retries=0)
client.chat.completions.with_raw_response.create(model="gpt-6.1-sol", max_tokens=32, messages=MESSAGES, tools=TOOLS, stream=True)
http 429 call_id 34cf0789-adaf-45b5-b63e-1b44386fe5a8 RateLimitError: Error code: 429 - {'error': {'message': 'litellm.MidStreamFallbackError: litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have ex...

Spend rows, GET /spend/logs?request_id=<x-litellm-call-id> polled every 5 s for up to 120 s

GET /spend/logs?request_id=bbaa6605-ba56-460f-a34d-bc87117f1ab5 -> [] (after 120s)
GET /spend/logs?request_id=499c5e93-bf9e-489d-a074-ad0fc3236af8 -> [] (after 120s)
GET /spend/logs?request_id=34cf0789-adaf-45b5-b63e-1b44386fe5a8 -> [{"request_id": "34cf0789-adaf-45b5-b63e-1b44386fe5a8", "status": "failure", "spend": 0.0, "model": "azure/gpt-6.1-sol-lit9298", "call_type": "responses"}]

Both /v1/messages calls answered 200 with the error event and never got a spend row; the chat control got its failure row

After (2709675, port 40410)

The same three calls against port 40410

http 200 call_id f55e118f-efde-4667-95e0-4881bbeeccf4 events ['message_start'] raised APIStatusError: {'type': 'error', 'error': {'type': 'rate_limit_error', 'message': 'litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have exceeded token rate limit.'}}
HTTP/1.1 200 OK
x-litellm-call-id: 72b03119-9e9b-4a0c-842b-e86312ad1884

event: message_start
data: {"type":"message_start","message":{"id":"msg_...","type":"message","role":"assistant","content":[],"model":"gpt-6.1-sol","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"output_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}}}

event: error
data: {"type": "error", "error": {"type": "rate_limit_error", "message": "litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have exceeded token rate limit."}}
http 429 call_id d12ab2b3-a3db-4adb-aa8a-8d2279eea983 RateLimitError: Error code: 429 - {'error': {'message': 'litellm.MidStreamFallbackError: litellm.RateLimitError: AzureException RateLimitError - Your requests to gpt-6.1-sol for gpt-6.1-sol-lit9298 in eastus2 have ex...
GET /spend/logs?request_id=f55e118f-efde-4667-95e0-4881bbeeccf4 -> [{"request_id": "f55e118f-efde-4667-95e0-4881bbeeccf4", "status": "failure", "spend": 0.0, "model": "azure/gpt-6.1-sol-lit9298", "call_type": "responses"}] (after 6s)
GET /spend/logs?request_id=72b03119-9e9b-4a0c-842b-e86312ad1884 -> [{"request_id": "72b03119-9e9b-4a0c-842b-e86312ad1884", "status": "failure", "spend": 0.0, "model": "azure/gpt-6.1-sol-lit9298", "call_type": "responses"}] (after 6s)
GET /spend/logs?request_id=d12ab2b3-a3db-4adb-aa8a-8d2279eea983 -> [{"request_id": "d12ab2b3-a3db-4adb-aa8a-8d2279eea983", "status": "failure", "spend": 0.0, "model": "azure/gpt-6.1-sol-lit9298", "call_type": "responses"}] (after 6s)

Every call now has exactly one failure row within 6 s, and the wire still carries message_start then the rate_limit_error frame on a 200. The frame text now names the provider error without the litellm.MidStreamFallbackError: prefix

SDK without a proxy, both commits

litellm.callbacks = [Recorder()]  # a CustomLogger counting async_log_failure_event
litellm.failure_callback = [sync_failure]  # a plain function counting calls
stream = await litellm.anthropic.messages.acreate(model="azure/gpt-6.1-sol-lit9298", api_base=AZURE_BASE, api_key=AZURE_KEY, max_tokens=32, messages=MESSAGES, tools=TOOLS, stream=True)
frames = [event type of each SSE frame async for chunk in stream]
before (127278f951): frames ['message_start', 'error'] failure callbacks {'async_failure': 1, 'sync_failure': 1}
after  (2709675ca3): frames ['message_start', 'error'] failure callbacks {'async_failure': 1, 'sync_failure': 1}

The chat stream wrapper already logs this Azure failure once at the SDK level, and the adapter's own dispatch does not log it a second time

Observations from the run, none caused by this PR unless said so

  • Every spend row says call_type responses, the chat control too: the Azure model's chat call runs through the chat-to-responses bridge
  • After, the /v1/messages error text names the provider error and drops the litellm.MidStreamFallbackError: prefix (this PR); the chat control keeps it
  • message_start names the group, the row names the deployment
  • Every cost header on the 200 reads 0.0
  • SDK failure callbacks fire once on both commits, no duplicate

Live A/B on the dependents (/live-pr-risk)

Verdict: at 2709675 no dependent path answers differently from the merge base except where this PR says so (the failure row, the failure callbacks, and the error text). The previous tip, c208fae, broke the chat bridge's pre-content retry; 2709675 fixes it

Rig: the two proxies above (real Azure, a wire recorder in front of it, a CustomLogger recording every hook) plus the integration harness under tests/integration (a scripted upstream, num_retries: 0 unless the test sets its own), both booted from the merge base and from the PR tree, every scenario run on both sides

Breaking

  • At c208fae a bridged /v1/messages stream that dropped after the role-only chat chunk, before any content, was no longer retried: tests/integration/messages_endpoint/chat_bridge/test_chat_bridge_pre_content_retry_wire.py passed 12/12 at the merge base, 11/12 at c208fae ([drop_after_pre_content_frame-chat] saw one upstream POST and an error event instead of a second POST), 12/12 at 2709675. The chat stream wrapper raises a MidStreamFallbackError whose is_pre_first_chunk is false once the role-only chunk went out, and the router declines to retry a raised envelope that says so even though the Anthropic client had only seen message_start. Fixed in 2709675: the adapter re-raises the provider's own exception, which the router judges by its own content flag

Backward incompatible

  • The error frame text on a bridged /v1/messages failure no longer starts with litellm.MidStreamFallbackError: (before and after texts are in the proof above). type (rate_limit_error) and the 200 status are unchanged. Kept on purpose in 2709675: the text now names the same error the failure row and the callbacks record
  • The failure row, alerts, Prometheus exception_class and the OTEL error for a bridged /v1/messages stream name the provider's exception class (RateLimitError here), where /v1/chat/completions names MidStreamFallbackError for the same Azure failure (the chat control above). Chosen in 2709675 over the two alternatives: raising the envelope breaks the pre-content retry (c208fae), and teaching the router to judge a raised envelope by its own content flag changes a rule every source iterator shares
  • SDK users (litellm.anthropic.messages.acreate without a proxy) now get their failure callbacks when the provider iterator itself raises mid-stream, with the provider's exception object; before, nothing fired on that shape. A failure the chat stream wrapper already logs (the Azure rate limit above) is still logged once on both commits

Regression risk

  • A buffering streaming guardrail on /v1/messages: the exception now travels through the guardrail's iterator the way it does on /v1/chat/completions today, so text the guardrail still holds when the provider fails is not delivered before the error frame; at the merge base the frame was an ordinary chunk and the guardrail flushed first. Not driven live, no streaming guardrail vendor was in the rig
  • A failure callback that replaces the exception with an HTTPException ends the stream with no error frame, the same as every other streaming route today. Not driven live

Dependency graph

  • AnthropicStreamWrapper.async_anthropic_sse_wrapper except branch (the diff): verified live, proxy and SDK, both commits
  • Router _aanthropic_messages_recover_stream_error and _anthropic_stream_fallback_error_for_raised (raised-error retry and fallback): verified live, the 12 bridge cases, both commits
  • Proxy post_call_failure_hook consumers (spend row, async_post_call_failure_hook, alerts, Prometheus, OTEL): verified live for the row and the hooks, both commits; Prometheus and OTEL traced from the hook's exception, not scraped
  • anthropic_error_sse_frame (frame type from the raised status, pre-message_start JSON error): verified live for rate_limit_error on a 200 after message_start; the pre-message_start path is the [http_500-*] and [drop_after_headers-*] bridge cases
  • Logging.dispatch_failure_handlers dedupe against the chat stream wrapper's own create_task dispatch: verified live, callbacks fire once on both commits
  • Bedrock Converse and stream-timeout wire tests (tests/integration/providers/test_bedrock_converse_stream_event_frames_wire.py, test_bedrock_stream_timeout_wire.py): 78 passed on both commits; test_c02_worker_sigkill_mid_burst_leaves_the_sibling_rejecting_the_held_streams failed on both with "Owned proxy readiness deadline exceeded" (it boots its own proxy on a saturated machine), no difference between the legs
  • Unit tests naming the class or the detached-failure hook (tests/unit/llms/anthropic/pass_through/..., tests/unit/proxy/test_common_request_processing.py): 800 passed at the tip
  • Native Anthropic passthrough (messages/streaming_iterator.py), the responses adapter (responses_adapters/streaming_iterator.py), Vertex and Bedrock native routes: not on the diff's path, untouched

Pre-existing on both commits, not caused by this PR

  • A client that disconnects mid-stream on /v1/messages leaves no spend row, while the same disconnect on /v1/chat/completions writes a success row; tracked in LIT-8603
  • A deleted key keeps answering 200 on sibling workers until the in-memory key cache TTL expires (two workers: [200, 401, 200, 401, 200, 401] right after the delete, all 401 after 72 s); filed separately

Not verified

  • A real Bedrock cross-region "ConverseStream ended without messageStop" drop (the issue's shape): nothing induces it on demand, so it is covered by the unit tests and the scripted-upstream wire tests only
  • The streaming guardrail and HTTPException-from-hook paths above
  • Prometheus and OTEL were not scraped; what they receive is the exception post_call_failure_hook got, which the hook recorder captured

/audit: integration cells, merge base vs head

Rig: three legs on one box through tests/integration/run.py, each with its own Postgres database, redis, scripted upstream (tests/integration/_support) and two-worker proxy on tests/integration/proxy_config.yaml. base = merge base 4417bf0, head = 2709675 (run twice with the same seeds), ctl = 2709675 for the controls, every lane --seed 697 --order-seed 697, lanes side by side. Run ids: base-providers, base-sdk, head-providers-run1, head-sdk-run1, head-providers-run2, head-sdk-run2, ctl-providers, ctl-extensions (round slp697, 352 s wall). The cells are committed in 8ce7974 byte-identical to the files the lanes ran. Every call id below is the client-supplied x-litellm-call-id the cell generated, echoed in the response header and read back as metadata->>'litellm_call_id' on the LiteLLM_SpendLogs row

Row Cell (tests/integration/messages_endpoint/chat_bridge/test_chat_bridge_mid_stream_failure_bookkeeping_wire.py:: unless named) base 4417bf0 head 2709675 (run 1, run 2) Verdict
raw httpx, upstream drops after the content delta test_bridged_stream_failing_after_content_writes_one_failure_row[drop_after_content] red: no row for the call id in 60 s green, green: one failure row, provider error class, frame type and row error_code agree PASS
raw httpx, OpenAI error frame after the content delta test_bridged_stream_failing_after_content_writes_one_failure_row[error_frame_after_content] red green, green PASS
Anthropic SDK sync messages.stream test_anthropic_sdk_sync_stream_failing_after_content_raises_and_writes_one_failure_row red (raises, no row) green, green (raises APIStatusError after "Hello", one failure row) PASS
Anthropic SDK async test_anthropic_sdk_async_stream_failing_after_content_raises_and_writes_one_failure_row red green, green PASS
non-streaming, upstream 500 before any body (control) test_bridged_request_rejected_before_any_body_keeps_its_error_and_failure_row green green, green PASS
response-cache hit twin test_bridged_stream_served_from_the_response_cache_keeps_its_success_bookkeeping green green, green (one upstream POST, miss row then cache_hit True row) PASS
client leaves before the failure test_client_leaving_a_bridged_stream_before_its_failure_leaves_the_proxy_serving green green, green (follow-up stream and its row land, liveliness 200, no duplicate row) PASS
24 concurrent streams on two workers (8 drop, 8 error frame, 8 succeed) test_concurrent_bridged_streams_failing_after_content_each_land_one_row red: only the 8 success rows land green, green: all 24 call ids land exactly once with their status PASS
SDK without a proxy, litellm.anthropic.messages.acreate(stream=True) tests/integration/sdk/test_anthropic_messages_bridge_mid_stream_failure_sdk.py::test_sdk_bridged_stream_failing_after_content_reports_the_provider_error_once green green, green (one failure callback with the provider exception, no double dispatch) PASS
controls, once on ctl at 2709675 the 12 cells of test_chat_bridge_pre_content_retry_wire.py (pre-content retry per deployment and request budget, SDK retry, drop after content not retried, chat and responses bridges), test_anthropic_thinking_signature_logging_wire.py (anthropic and bedrock invoke pass-through streams), test_bedrock_passthrough_stream_wire.py::test_bedrock_passthrough_converse_stream_response_carries_event_stream_content_type, routing/test_deployment_num_retries_generic_routes_wire.py::test_vllm_passthrough_rejected_before_it_opens_is_retried_per_the_deployment_budget[stream], observability/test_passthrough_upstream_error_visibility.py::test_gemini_passthrough_streaming_200_relays_every_chunk n/a 17 green PASS

Pass-through routes never run the changed wrapper (it is built only by the chat adapter at adapters/transformation.py), so their streaming rows are controls. No inventory row is unverified

Type

🐛 Bug Fix

Caveats (if any)

Medium

  • With a buffering streaming guardrail, text the guardrail still holds when the provider fails mid-stream is dropped before the error frame; /v1/chat/completions behaves the same today. Not driven live

Low

  • The /v1/messages error frame text drops the litellm.MidStreamFallbackError: prefix and names the provider error; type and status are unchanged
  • The failure row and failure metrics for a bridged /v1/messages stream name the provider's exception class, where the chat route names MidStreamFallbackError
  • A failure callback that replaces the error with an HTTPException ends the stream with no error event, the same as every other streaming route today
  • SDK users now get their failure callbacks when the provider iterator itself raises mid-stream, with the provider's exception; a failure the chat stream wrapper already logged is still logged once
  • osv-scan is red on six next 16.3.6 advisories published 2026-10-07 20:30Z, after main's last green scan; the dashboard lockfile is identical on main, so main's next scan goes red the same way until the dependency bump lands

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

  • 2709675 passes /live-pr-risk

  • 8ce7974 passes /audit (ran at 2709675, the same product code)


Note

Medium Risk
Changes streaming error handling on a heavily used proxy route (observability and re-raise vs swallow); wire error message text and logged exception class shift slightly for bridged failures, with intentional behavior for router retry.

Overview
Fixes bridged /v1/messages streams that fail after content already started: the Anthropic chat adapter no longer treats mid-stream errors as “emit an SSE error frame and stop.” It now routes failures through LiteLLM’s logging boundary so spend logs and callbacks match /v1/chat/completions.

In async_anthropic_sse_wrapper, when the upstream iterator raises, _provider_error unwraps MidStreamFallbackError to the underlying provider exception (so router retry/fallback and error text stay correct). If on_detached_stream_failure is set (proxy-managed stream), that exception is re-raised for the proxy to record failure and still send the client-facing error. Otherwise the adapter calls dispatch_failure_handlers with the provider error, then yields the same Anthropic error SSE event; a broken failure handler cannot suppress the frame.

AnthropicStreamWrapper keeps a reference to litellm_logging_obj for this path. New integration, SDK, and unit tests cover failure rows, single callback dispatch, concurrent streams, and proxy vs standalone behavior.

Reviewed by Cursor Bugbot for commit 8ce7974. Bugbot is set up for automated code reviews on this repo. Configure here.

@CLAassistant

CLAassistant commented Oct 6, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@greptile-apps

greptile-apps Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Streaming error handling for Anthropic messages endpoint.

The PR appears safe to merge; no outstanding findings remain.

Findings

  1. P1 Sync failure callbacks are skipped ▶

Summary

The PR routes bridged /v1/messages mid-stream failures through proxy failure bookkeeping while preserving Anthropic error events and pre-content retry behavior.

  • Adds proxy-wire and SDK regression coverage for failure rows, callbacks, caching, and concurrent streams.

Reviews (4) · Last reviewed commit: "test(integration): cover mid-stream fail..." · Reviewed by Greptile

# and keep the client-facing Anthropic error frame.
if self.litellm_logging_obj is not None:
try:
await self.litellm_logging_obj.async_failure_handler(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Sync failure callbacks are skipped When a standalone async Messages stream fails, this calls async_failure_handler directly. That handler runs only async callbacks; unlike dispatch_failure_handlers, it does not schedule sync failure callbacks. SDK users who configure only litellm.failure_callback receive the error frame but never receive their failure callback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5d44f5d and a3e7fff: the adapter now calls dispatch_failure_handlers with prefer_async_handlers=True, which schedules the sync failure callbacks too.

Comment thread litellm/llms/anthropic/pass_through/adapters/streaming_iterator.py Outdated
Comment thread tests/unit/llms/anthropic/pass_through/messages/test_sse_wrapper.py Outdated
@codspeed

codspeed Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing D41910:fix/anthropic-messages-mid-stream-failure-bookkeeping (8ce7974) with main (bf9bd35)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (c38505d) during the generation of this report, so bf9bd35 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…failure boundary

AnthropicStreamWrapper.async_anthropic_sse_wrapper swallowed every upstream
exception and emitted its own Anthropic error frame, so a provider drop on a
streamed /v1/messages call never reached the proxy's streaming boundary — no
failure spend row, no failure callbacks.

Re-raise when the stream is proxy-managed (detached failure hook armed) so
async_streaming_data_generator runs post_call_failure_hook once and serializes
the error frame; when consumed standalone (SDK litellm.messages path), run the
logging object's async failure handler and keep the client-facing error frame.

Fixes BerriAI#44742
@Winoren
Winoren force-pushed the fix/anthropic-messages-mid-stream-failure-bookkeeping branch from 74088c3 to 125a25e Compare October 6, 2026 11:37
@Winoren

Winoren commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review, thanks for catching the real one.

Sync failure callbacks skipped (P1, line 1065): confirmed, fixed. async_failure_handler reads only litellm._async_failure_callback (litellm_logging.py:3933-3936); the sync litellm.failure_callback list is scheduled from dispatch_failure_handlers (litellm_logging.py:2248-2249). The adapter now calls dispatch_failure_handlers(..., prefer_async_handlers=True), which covers both lists. Ran a probe against a real Logging instance with only litellm.failure_callback configured: fired before the fix, does not fire now. prefer_async_handlers=True keeps it off the sync-SDK shortcut that would block the event loop.

Transformed failures lose error events (P1, line 1059): not a regression from this PR. That raise e on an HTTPException is upstream behaviour in the shared async_streaming_data_generator (common_request_processing.py:4011-4012), reached by every proxy streaming route including /chat/completions. A callback that returns an HTTPException ends the stream there by design. Fixing it would mean changing shared proxy error handling, so it is out of scope here. Happy to open a separate issue if useful.

Proxy outcome untested (P2, test_sse_wrapper.py): addressed, tests moved. The regression cases now live in the mapped file tests/unit/llms/anthropic/pass_through/adapters/test_streaming_iterator_mid_stream_error.py per AGENTS.md, and test_sse_wrapper.py is back to its main-branch content. The new cases assert both branches of the boundary: a proxy-managed stream re-raises so the proxy writes the failure row and serializes the frame, and a standalone stream routes through dispatch_failure_handlers and still emits the error frame. test_sse_wrapper.py had also lost its TestClient import, now restored.

Verification: 10 passed in that file, 375 passed across tests/unit/llms/anthropic/pass_through/adapters/. Reverting the implementation fails the two new cases and leaves the 8 pre-existing ones green. ruff finding count is unchanged from main (8, none new).

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

…elope so the router keeps its pre-content retry
…1/messages streams

Nine integration cells for the chat bridge: raw httpx and the Anthropic SDK (sync and async) streams that fail after the first content delta now land exactly one failure SpendLogs row carrying the provider's error class, the non-streaming 500, the response-cache twin and a client that leaves mid-stream keep their bookkeeping, a 24-stream burst lands every call id once, and the standalone SDK stream reports the provider error to failure callbacks once.
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 8ce7974. Configure here.

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the contribution!

@mateo-berri
mateo-berri merged commit 87e961f into BerriAI:main Oct 8, 2026
103 of 105 checks passed
mateo-berri added a commit that referenced this pull request Oct 9, 2026
…essages error frames (#45465)

#44800 re-raises the provider's own error from the chat adapter, so a bridged /v1/messages stream's error frame now reads litellm.RateLimitError without the litellm.MidStreamFallbackError: prefix that #44989's tests pinned shortly before #44800 merged. The three tests now expect the provider error text once and no sentinel, matching the chat_limited case and every other assertion in the two files

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: streamed /v1/messages call the provider drops mid-stream writes no spend log row and skips the failure hooks

3 participants