[BUG] A single 429 becomes 27 identical requests: three retry layers multiply #8332

Prepared with an AI agent. Per CONTRIBUTING this should carry the llm-generated label, which I don't have permission to apply.

Description

On crewai==1.15.28, one 429 from the provider produced 27 identical HTTP requests in 27.3 s. Same
body, same tools, same user agent, none of them streamed. Three retry layers are nested and their counts
multiply:

3 × 3 × 3 = 27, and the request log matches that shape exactly.

Steps to Reproduce

1. crewai==1.15.28, Python 3.12, a single agent with one tool.
2. Point the LLM at an OpenAI-compatible server that always answers 429 (bench cell crewai/429 in
   https://github.com/simin-yuan/agent-double-exec-bench — the cell and the raw request log are both in
   that repo).
3. Run one task.
4. Count the requests the server sees: 27, over 27.3 s.

Expected behavior

One user-visible call should spend a number of requests bounded by the largest of the three budgets,
not by their product — and the outermost retry should wait before it re-sends.

Screenshots/Code snippets

The three places, on main:

# completion.py:377
max_retries: int = 2

# retry.py:35
_LLM_RATE_LIMIT_MAX_ATTEMPTS = 3

# agent/core.py:301-304
max_retry_limit: int | None = None
...
self.max_retry_limit = max_retry_limit if max_retry_limit is not None else 2

# agent/core.py:874 — the comment that states the re-entry outright
# A retry re-enters execute_task, which opens a new agent_execution_started scope

Operating System

Windows 11

Python Version

3.12

crewAI Version

1.15.28 (request log). The source locations above were read on main @ dec075b.

crewAI Tools Version

Not involved — the tool in the repro is a local counter, no crewai-tools package.

Virtual Environment

venv

Evidence

From results/e1/crewai/429/server.jsonl (27 lines):

• all POST /v1/chat/completions, stream=false, identical body hash (b7c7a67c…), identical UA
  (OpenAI/Python 2.54.0)
• the SDK's own counter cycles 0 → 1 → 2, nine times (x-stainless-retry-count = 0,1,2, 0,1,2, …)
• gaps inside a group: 1.0–1.06 s; gaps between groups: 1.11–1.92 s, except two of them: 0.02 s and
  0.03 s — that is the agent layer, which re-enters execute_task with no backoff at all
• the call ends in openai.RateLimitError

The second layer does not always engage, which matters if you tune this by hand. Same cell family:
9 requests for 429-hard and 500-text-429, but 27 for 429, 429-neutral-body and
500-ratelimit-text. _ThrottlingErrorClassifier decides which of the two you get, so changing
max_retry_limit alone adjusts one of three budgets that only sometimes apply.

Possible Solution

Keep the SDK's retries on the first attempt and turn them off on CrewAI's own replays — the provider builds
its client with with_options(max_retries=0) when the current attempt is a replay of a call that already
failed. The first attempt keeps the SDK's three tries, so nothing is lost on a transient blip. Expected on
the same cell: 3 × (3 + 1 + 1) = 15 requests instead of 27.

This does not remove the agent-level multiplication (the 0.02 s re-sends). Whether a failed task should
re-call the model at all is a design question, so it is reported here rather than patched.

Additional context

_times_executed is never reset. agent/core.py:242 declares
_times_executed: int = PrivateAttr(default=0); :886-887 increments it and compares it against
max_retry_limit. Nothing resets it when a task succeeds, so max_retry_limit is a budget for the agent
object's whole lifetime, not per task — an agent that has already retried once pays for that on every
later task. (This overlaps with #5822, which mentions the same non-reset while fixing #5802. It is listed
here because it is a separate defect and the two fixes do not conflict.)

Not the same as:

• Tool re-execution on task retry has no idempotency guard — duplicate payments, emails, trades possible #5802 / fix: add idempotency guard for tool re-execution on task retry #5822 / feat: add tool execution idempotency guard to prevent duplicate side effects on task retry #6048 — a tool runs twice after a retry (idempotency guard). Here the LLM
  request is sent 27 times; no tool needs to be involved at all.
• ToolUsage retries a failed tool call twice per outer attempt, not once (schema-args-filter fallback in _use/_ause) #7449 — same "two retry paths multiply" shape, but in ToolUsage._use/_ause, i.e. the tool path,
  not the provider call.
• Flow: persists across calls — no automatic reset #6639 — flow-level retry_count persisting across kickoff() calls; same failure mode as
  _times_executed, different object.

Disclosure: I directed this work. The source reading, the patch, its tests and the measurements were
produced by AI assistants I operate. Every number above comes from the server logs in
agent-double-exec-bench, and I will answer review comments on it.

Per CONTRIBUTING this should carry the llm-generated label; I'll add it if my account can, otherwise
please apply it.