[dodoshell]

User description

Fixes #4407.

The completion call now always gets num_retries=0 and max_retries=0, so the handler's retry is the only layer. config.num_retries (default 4, 0 disables) is the number of same-model retries of a rejected request: a 429, a 5xx answer including 529, or a connection error. Each retry waits for a finite, positive Retry-After of up to 30 s, otherwise wait_random_exponential(multiplier=1, max=20). I read "capped at 30 s" together with "huge values must fall back to the jitter", so a Retry-After above 30 s falls back to the jitter instead of being clamped; happy to clamp instead if you prefer. Timeouts, MidStreamFallbackError, empty responses and 200s that fail to parse keep MODEL_RETRIES and are replayed at once, as before.

LiteLLM reports a 200 that fails to parse as a 5xx or APIConnectionError, the same types as a real failure. A real 5xx carries litellm_response_headers and a connection error has an httpx.TransportError in its cause chain, and in my checks on the OpenAI and Bedrock paths an unparsed 200 had neither.

Measured with an httpx-level mock, primary plus one fallback model:

error	Bedrock	OpenAI	num_retries = 3
429	10 calls, 20 s	10 calls, 13 s	8 calls
503 / 500	10 calls, 15-18 s	10 calls, 14-16 s	8 calls
connection error	20 calls	10 calls	16 / 8 calls
timeout	4 calls	4 calls	4 calls
400	2 calls	2 calls	2 calls

The extra Bedrock connection-error calls come from LiteLLM's HTTP client, which re-sends once on httpx.ConnectError; num_retries does not reach it.

Unit tests cover each error class, the bad Retry-After values and the stop and wait per class, and an autouse fixture keeps the backoff out of the test run. num_retries = 4 in configuration.toml is a new active key, so configuration_reference.md is regenerated and the key count goes to 296. The unit suite has no new failures; Ruff and pre-commit passed.

---

PR Type

Bug fix, Tests, Documentation

---

Description

• Centralize LiteLLM retries in the handler.

• Back off rejected requests before model fallback.

• Preserve separate timeout and empty-response retry controls.

• Update retry defaults, documentation, and regression tests.

---

Diagram Walkthrough

flowchart LR
  A["Model request"]
  B["Completion with client retries disabled"]
  C["429, answered 5xx, or connection failure"]
  D["Retry-After or exponential jitter"]
  E["Same-model retry budget"]
  F["Fallback models"]
  A --> B
  B -- "rejected request" --> C
  C --> E
  E -- "retries remain" --> D
  D --> B
  E -- "budget exhausted" --> F

Loading

File Walkthrough

	Relevant files
Bug fix	1 files litellm_ai_handler.py Centralize rejected-request retries with bounded backoff  +74/-14
Tests	5 files conftest.py Disable handler retry waits throughout unit tests                +13/-0    test_config_reference.py Update active configuration count for retry default            +1/-1      test_litellm_chat_completion_core.py Verify retry classification and disabled client retries    +46/-8    test_litellm_imds.py Verify throttling retries preserve ambient AWS credentials +3/-2      test_litellm_retry_config.py Cover retry budgets, error classification, and backoff      +111/-5
Documentation	2 files changing_a_model.md Explain same-model retries before fallback model selection +4/-1      configuration_reference.md Document retry default and rejected-request backoff behavior +1/-0
Configuration changes	1 files configuration.toml Enable four handler retries for rejected requests                +1/-1

---

---

⚙️ Agent run details

• Model: openai/gpt-6.1-sol
• Tokens: 13,211 in / 556 out / 13,767 total
• Time cost: 9.6s
• AI calls: 1

[the-pr-agent-bot]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis ✅ 4407 - PR Code Verified Compliant requirements: Always pass num_retries=0 and max_retries=0 to completion calls. Configure rejected-request retries with a default of 4 and support 0. Exclude the specified timeout, mid-stream, and parsing-error cases in the classifier. Implement bounded Retry-After handling and exponential backoff with jitter. Enable same-model rate-limit retries. Preserve the timeout and length retry controls and MODEL_RETRIES constant. Update the previous-behavior tests and add error-class and Retry-After coverage. Update the requested configuration comments, docstring, and documentation. Leave the explicitly excluded areas unchanged. Requires further human verification: Verify that the handler is the only effective retry layer across provider transports, particularly Bedrock connection failures. Verify that actual provider exception metadata reliably distinguishes genuine 5xx and connection failures from malformed successful responses.

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ No major issues detected

---

⚙️ Agent run details

• Model: openai/gpt-6.1-sol
• Tokens: 16,924 in / 895 out / 17,819 total
• Time cost: 28.0s
• AI calls: 1

[the-pr-agent-bot]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible bug	Enable retries for rejected requests Include _is_rejected_request in the retry eligibility check, not just the stop and wait callbacks. The unchanged _should_retry_same_model previously excluded rate-limit errors, so changing the retry budget and backoff alone does not enable the newly promised 429 retries. pr_agent/algo/ai_handlers/litellm_ai_handler.py [2870-2875] @retry( - retry=retry_if_exception(_should_retry_same_model), + retry=retry_if_exception( + lambda exc: _is_rejected_request(exc) or _should_retry_same_model(exc) + ), stop=_stop_same_model_retries, wait=_same_model_retry_wait, reraise=True, # surface the provider's error; RetryError hides the reason ) Suggestion importance[1-10]: 8 __ Why: The PR changes the retry budget and backoff but leaves _should_retry_same_model unchanged, including its exclusion of rate-limit errors. Adding _is_rejected_request to the eligibility predicate enables the promised 429 retries and aligns behavior with the updated tests.	Medium

---

⚙️ Agent run details

• Model: openai/gpt-6.1-sol
• Tokens: 23,120 in / 1,008 out / 24,128 total
• Time cost: 35.7s
• AI calls: 2

[dodoshell]

_should_retry_same_model no longer excludes RateLimitError (the diff drops it from the tuple), so 429s already reach the retry; test_rate_limit_retries_same_model and test_chat_completion_num_retries_sets_rate_limit_attempts cover it.

[IsmaelMartinez]

Thanks @dodoshell for taking the whole retry path in one PR. The behaviour matches the spec in #4407: the call counts for 429, 5xx, timeout and 400 reproduce, and odd Retry-After values fall back to the jitter. Keeping the jitter above 30 s is fine. One ask: the autouse fixture in conftest.py replaces retry.wait, so deleting wait=_same_model_retry_wait from the decorator still passes every test. Patching sleep instead and recording it in one test covers the wiring.

[IsmaelMartinez]

Suggested change

from tenacity import wait_none

[IsmaelMartinez]

Suggested change

	monkeypatch.setattr(handler_module.LiteLLMAIHandler._chat_completion_with_retry.retry, "wait", wait_none())
	async def no_sleep(seconds):
	pass
	monkeypatch.setattr(handler_module.LiteLLMAIHandler._chat_completion_with_retry.retry, "sleep", no_sleep)

[IsmaelMartinez]

Suggested change

	assert mock_call.call_count == expected_attempts
	assert mock_call.call_count == expected_attempts
	@pytest.mark.asyncio
	async def test_chat_completion_waits_retry_after_between_rate_limit_retries(monkeypatch):
	monkeypatch.setattr(litellm_handler, "get_settings", FakeSettings)
	sleeps = []
	async def record_sleep(seconds):
	sleeps.append(seconds)
	monkeypatch.setattr(litellm_handler.LiteLLMAIHandler._chat_completion_with_retry.retry, "sleep", record_sleep)
	error = _answered(litellm.RateLimitError("throttled", llm_provider="bedrock", model="m"))
	error.litellm_response_headers = httpx.Headers({"retry-after": "7"})
	with patch("pr_agent.algo.ai_handlers.litellm_ai_handler.acompletion", new_callable=AsyncMock) as mock_call:
	mock_call.side_effect = error
	handler = litellm_handler.LiteLLMAIHandler()
	with pytest.raises(litellm.RateLimitError):
	await handler.chat_completion(model="gpt-4o", system="sys", user="usr")
	assert sleeps == [7.0] * litellm_handler.DEFAULT_NUM_RETRIES

[dodoshell]

Thanks @IsmaelMartinez, good catch. Applied both suggestions in c06098d: the fixture now patches sleep, and the new test fails with [0.0, 0.0, 0.0, 0.0] == [7.0, 7.0, 7.0, 7.0] once wait=_same_model_retry_wait is removed from the decorator.

[IsmaelMartinez]

Thanks @dodoshell for applying both suggestions so quickly; the new test now fails without the backoff wiring, approving and merging.