bug(retry):Pydantic AI provider never retries on HTTP 429/5xx -- ModelHTTPError not recognized as retryable

[nskun]

Summary:

The Claude provider's retry logic silently skips retries for HTTP 429 (rate limit) and 5xx errors when using the Pydantic AI backend.
Agents fail immediately even with retry configured in workflow YAML.

Environment:

Conductor v0.1.31
pydantic-ai 2.x(latest)
Provider: claude (Pydantic AI backend)

Steps to Reproduce:

1.Configure an agent with retry

agents:
  -name: my_agent
    retry:
      max_attempts: 3
      backoff: exponential
      delay_seconds: 60
      retry_on: [provider_error]

2. Hit a 429 rate limit from the API.
3. Observe: agent fails immediately. Zero agent_retry events in the event log.

Root Cause:

Pydantic AI v2 wraps HTTP errors as pydantic_ai.exceptions.ModelHTTPError. In execute_with_retry(), the function _is_retryable_error() checks for:

• conductor.exceptions.ProviderError -- ModelHTTPError is not this
• Anthropic SDK class names (RateLimitError, APIStatusError) -- "ModelHTTPError" is not in the set
• anthropic.APIStatusError isinstance check -- ModelHTTPError is not this

All checks fail -> returns False -> error is immediately re-raised at L289 as ProviderError(is_retryable=False)
, before the retry loop ever executes.

Suggested Fix:

Add ModelHTTPError handling:

try:
    from pydantic_ai.exceptions import ModelHTTPError
except ImportError:
    ModelHTTPError = None

def _is_retryable_error(exception):
    # New: handle Pydantic AI's HTTP error type
    if ModelHTTPError is not None and isinstance(exception, ModelHTTPError):
        code = exception.status_code
        return code == 429 or 500 <= code < 600

Also worth considering:

_get_retry_after() should extract the wait time from ModelHTTPError.body when present, so Conductor can respect the server's requested delay instead of falling back to _calculate_delay().

Related:

max_delay default silently caps delay_seconds
A secondary issue compounds the problem. RetryConfig.max_delay defaults to 30 seconds.
When a user sets delay_seconds: 60, the delay is silently capped:

delay = base_delay * 2 ** (attempt - 1) # 60 * 1 = 60
delay = min(delay, config.max_delay) # min(60, 30) = 30 <- capped

For rate limits with a 60second window, the 30second wait is insufficient, causing repeated 429s even when retries do fire. The user's explicit delay_seconds should not be overridden by a lower default.

Suggestion:

Ensure max_delay >= delay_seconds when resolving the config.

[nskun]

Additional finding: max_delay not propagated to subworkflow child engines

Background

During further testing, I found that even when monkey-patching RetryConfig.max_delay to 60s, the actual retry delay stays at ~30s in subworkflows.

Root cause

There are two separate RetryConfig classes:

1. conductor/providers/claude.py L62: class RetryConfig(BaseModel) -- Pydantic BaseModel
2. conductor/providers/_pydantic_ai/retry.py L34: class RetryConfig -- plain Python class

When a subworkflow starts, the factory (factory.py L130) creates a new ClaudeProvider() without passing retry_config, so it defaults to RetryConfig() with max_delay=30.0.

Then in claude.py L834, _resolve_retry_config is called with:

PydanticRetryConfig(
    max_delay=self._retry_config.max_delay, # <- 30.0 from default
    base_delay=retry.delay_seconds, # <-60.0 from YAML
)

This means delay = min(60, 30) = 30 in _calculate_delay(), regardless of what the user set in delay_seconds.

Suggestion

Either:
1.Pass retry_config through to child engines in factory.py or
2.Ensure max_delay >= base_delay in _resolve_retry_config

[jrob5756]

Confirmed all three. Real regression, and the diagnosis is right.

Verified on main (0.1.32):

is_retryable(429) = False
raised: ProviderError | is_retryable=False | status=429
attempts made: 1

It's actually worse than the report suggests. _pydantic_ai/agent_builder.py:139 sets max_retries=0 on the Anthropic client, with a comment saying Conductor's execute_with_retry is the only retry layer. So the SDK's own 429/5xx retry (default 2) was switched off on the assumption the replacement worked. Since the replacement can't see ModelHTTPError, there's currently no 429/5xx retry anywhere in the claude provider.

This came in with #355 (08bf892). Before that the provider used the raw SDK, so RateLimitError (a subclass of APIStatusError) hit the isinstance check and retried fine. The retry helpers got mirrored across without accounting for pydantic-ai catching APIStatusError and re-raising it as ModelHTTPError (pydantic_ai/models/anthropic.py:283).

One thing that narrows the scope: APIConnectionError and APITimeoutError aren't APIStatusError subclasses, so pydantic-ai leaves them unwrapped and their class names still match retryable_names. Connection and timeout retries still work. Only status-code failures are broken.

On the suggested fix

The _is_retryable_error change is the right shape. Two caveats on the rest.

_get_retry_after can't be fixed the way you suggested. ModelHTTPError carries only status_code, model_name and body. pydantic-ai discards response headers, and Retry-After is a header rather than part of the body, so that value is already gone by the time the exception reaches us. Recovering it means wrapping the httpx transport (pydantic_ai.retries.wait_retry_after), which puts back exactly the double-retry layering #355 set out to remove. I'd lean toward accepting the loss and relying on backoff, but happy to be argued out of it.

The max_delay cap is real and worth fixing on its own. RetryPolicy.delay_seconds is Field(ge=0.0, le=300.0) and documented as 0.0-300.0, so anything above 30 silently breaks a bound the schema advertises.

Your subworkflow finding has a broader cause than described. factory.py:130 is the only production construction site for ClaudeProvider, and it never passes retry_config at all, so max_delay is 30 on every run rather than just in subworkflows. Clamping inside _resolve_retry_config (max_delay=max(default.max_delay, retry.delay_seconds)) covers every path and is smaller than threading config through the factory.

Why this got through

test_pydantic_ai_retry.py defines local MockRateLimitError / MockAPIStatusError classes whose names happen to match the hardcoded set, so the suite is exercising the legacy Anthropic contract rather than the one the current stack produces. A fix should assert against the real pydantic_ai.exceptions.ModelHTTPError and check attempt counts.

Also worth flagging: #421 adds a native OpenAI provider on this same retry module. That provider will never raise an Anthropic exception, so every HTTP failure there would be non-retryable. Probably worth landing this first.

[jrob5756]

Starting to take a look at this.

[nskun]

Thanks for the quick fix! I’ve confirmed that it works correctly now.