feat: coordinated rate-limit retry with per-model adaptive backoff

[tangym]

Summary

Add retry with coordinated per-model backoff for transient LLM failures (429 rate limits, 5xx provider errors). Previously, a single 429 or transient 5xx permanently failed the affected seed with no recovery.

What changed

p2m/core/model_client.py

Per-model adaptive rate limiter (_ModelRateLimiter):

• When any task hits 429, a shared cooldown is set for that model — all concurrent tasks pause, preventing thundering herd on fan-out workloads (e.g. 50 seeds × same model)
• Honors Retry-After header from the provider when available
• Adaptive escalation: doubles cooldown if 429 recurs after previous cooldown expired; absorbs concurrent in-flight 429s without escalating
• Per-task jitter on wake-up to prevent stampede at cooldown expiry
• Success resets escalation state

Retry wrapper (_with_retries):

• Wraps all three generate functions (generate, generate_structured, generate_with_tools)
• LLMRateLimitError (429) → coordinated cooldown retry (up to 5 attempts)
• LLMProviderError (5xx) → exponential backoff retry (1s × 2^attempt + jitter, capped at 120s)
• LLMAuthError (401/403) and LLMInputError (400/404/422) → propagate immediately, no retry

Error classification (_classify_llm_error):

• Added comprehensive docstring with full LiteLLM exception → HTTP status code mapping table (ref: https://docs.litellm.ai/docs/exception_mapping)
• Added PermissionDeniedError (403) → LLMAuthError
• Added UnprocessableEntityError (422) → LLMInputError
• Added APIResponseValidationError → LLMInputError
• Uses getattr(litellm, ..., ()) for new exception types for backward compat

LiteLLM config hardening:

• Raised _MAX_BACKOFF_S from 60 to 120 to fully honor server Retry-After headers
• Explicitly set litellm.num_retries = 0 to prevent double-retry (LiteLLM internal + our _with_retries)

tests/test_rate_limit_retry.py (new, 23 tests)

• ExtractRetryAfterTest: header extraction from cause chain
• ModelRateLimiterTest: cooldown, escalation, jitter, cross-model independence, retry-after clamping, success reset
• WithRetriesTest: success path, rate-limit retry, provider error backoff, non-retryable passthrough, max-retry exhaustion
• GenerateWithRetriesTest: integration test with mocked litellm

Reference

Compared against azure-ai-evaluation RetryPolicy — p2m's approach provides equivalent retry coverage with the addition of coordinated per-model cooldown, which is critical for fan-out eval workloads.

Testing

uv run pytest tests/test_rate_limit_retry.py -v  # 23 passed
uv run pytest -q                                  # 528 passed, 1 skipped (pre-existing docker permission issue)