Skip to content

Retry transient HTTP errors in blob IO - #336

Draft
higordearaujo-bsy wants to merge 3 commits into
SeequentEvo:mainfrom
higordearaujo-bsy:main
Draft

higordearaujo-bsy wants to merge 3 commits into
SeequentEvo:mainfrom
higordearaujo-bsy:main

Conversation

@higordearaujo-bsy

@higordearaujo-bsy higordearaujo-bsy commented Aug 20, 2026 •

Copy link
Copy Markdown

Fear not, dear reviewer, I come in peace! Although the diff is 1000+ lines, most of it is tests, comments and whitespace. The SDK itself has fewer than 300 lines of source code changed. I have also organized the initial push in three commits that can be reviewed one at a time.

Description

Blob chunk and commit requests now retry transient failures using the existing blob retry loops and default Retry. This fixes uploads failing on a single 503 Service Unavailable from Put Block List.

Old behavior

  • Every blob request failure other than 401/403 was wrapped in a plain ChunkedIOError, whose recover() returns False. Chunk and commit retry loops re-raised it immediately, even with a large caller-supplied Retry.

New behavior

  • 408, 425, 429, 500, 502, 503, 504 and TransportError are raised as the new TransientChunkedIOError (a ChunkedIOError subclass with status), whose recover() returns True. The existing loops retry it with the caller's Retry, or the default (3 attempts, exponential backoff).
  • Other errors are unchanged: 401/403 still renew the URL; anything else is still a non-recoverable ChunkedIOError with the same message.
  • A transient chunk failure retries the remaining transfer, as URL renewal already does; every worker pauses during the backoff. Per-chunk retries (paired with adaptive concurrency), jittered backoff, Retry-After and HEAD-probe retries follow in a separate PR.

API and compatibility

  • New public names: evo.common.exceptions.TRANSIENT_HTTP_STATUSES and evo.common.io.exceptions.TransientChunkedIOError. Callers catching ChunkedIOError keep working.
  • evo-sdk-common 0.5.29.
  • uv lock also normalized existing dependency markers and workspace metadata. No third-party or other workspace package versions changed.

Replay limits
Retries reuse block IDs and the commit payload. Safe replay assumes an unchanged source, the same upload destination after URL renewal, no competing writer, and storage that accepts same-ID block overwrites and repeated identical block-list commits. Retries don't guarantee exactly-once metadata, versions or events.

Checklist

  • I have read the contributing guide and the code of conduct

@CLAassistant

CLAassistant commented Aug 20, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@higordearaujo-bsy higordearaujo-bsy changed the title Add EvoAPIRetry class Retry transient HTTP errors in blob IO and add shared retry helpers Oct 13, 2026
@higordearaujo-bsy
higordearaujo-bsy requested a balanced review from Copilot October 13, 2026 19:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The intertwined retry, cancellation, concurrency, and request-replay behavior warrants final human review.

0 open findings

What changed in this PR

Adds shared transient retry infrastructure and applies it to blob HEAD, chunk, and commit operations.

Changes:

  • Adds retry helpers, jitter backoff, Retry-After support, and transient exceptions.
  • Introduces per-chunk retries, adaptive concurrency, and signed-URL-safe logging.
  • Re-exports compute exceptions, updates versions, lock metadata, and tests.
File Description
uv.lock Updates versions and lock metadata.
packages/​evo-sdk-common/​pyproject.toml Bumps common to 0.6.0.
packages/​evo-sdk-common/​src/​evo/​common/​exceptions.py Adds shared transient HTTP definitions.
packages/​evo-sdk-common/​src/​evo/​common/​utils/​retry.py Implements shared retry helpers and jitter.
packages/​evo-sdk-common/​src/​evo/​common/​utils/​__init__.py Exports retry APIs.
packages/​evo-sdk-common/​src/​evo/​common/​io/​exceptions.py Adds transient chunk errors.
packages/​evo-sdk-common/​src/​evo/​common/​io/​_retry.py Defines the default blob retry policy.
packages/​evo-sdk-common/​src/​evo/​common/​io/​_limiter.py Adds adaptive concurrency limiting.
packages/​evo-sdk-common/​src/​evo/​common/​io/​chunked_io_manager.py Adds independent per-chunk retries.
packages/​evo-sdk-common/​src/​evo/​common/​io/​http.py Retries HEAD requests and maps transient failures.
packages/​evo-sdk-common/​src/​evo/​common/​io/​storage.py Retries block commits by default.
packages/​evo-sdk-common/​src/​evo/​aio/​_helpers.py Documents transport replay limitations.
packages/​evo-sdk-common/​tests/​common/​test_exceptions.py Tests shared transient exceptions.
packages/​evo-sdk-common/​tests/​common/​utils/​test_retry.py Tests retry helpers and policies.
packages/​evo-sdk-common/​tests/​common/​io/​test_limiter.py Tests adaptive limiting.
packages/​evo-sdk-common/​tests/​common/​io/​test_chunked_io_manager.py Tests chunk retry behavior.
packages/​evo-sdk-common/​tests/​common/​io/​test_http.py Tests HTTP retries and redaction.
packages/​evo-sdk-common/​tests/​common/​io/​test_storage.py Tests block and commit retries.
packages/​evo-compute/​pyproject.toml Bumps compute and common dependency versions.
packages/​evo-compute/​src/​evo/​compute/​exceptions.py Re-exports common transient definitions.
packages/​evo-compute/​tests/​test_client.py Verifies exception re-exports.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Classify transient HTTP statuses and transport failures as recoverable blob errors, preserving existing messages and using the current chunk and commit retry loops. Add regression coverage for default retries, identical commit payloads, exception chains, and permanent failures.

Common suite: 595 passed, 37 skipped, one existing collection warning. Compute suite: 499 passed. Ruff check/format and uv lock --check passed.
Release the minimum transient blob retry fix and regenerate uv.lock. Retain uv-generated normalization of existing dependency markers and workspace metadata; no third-party package versions change.

Common suite: 595 passed, 37 skipped, one existing collection warning. Compute suite: 499 passed. Ruff check/format, uv lock --check, and the lock-source checker passed.
@higordearaujo-bsy higordearaujo-bsy changed the title Retry transient HTTP errors in blob IO and add shared retry helpers Retry transient HTTP errors in blob IO Oct 13, 2026
Move TransientAPIError and the transient HTTP status set to common while preserving compute import identity. Add factory, status-set, and compatibility regression tests; release compute 0.0.6 requiring common >=0.5.29 and regenerate the lockfile.

Common suite: 597 passed, 37 skipped, one existing collection warning. Compute suite: 500 passed. Ruff check/format, uv lock --check, and the lock-source checker passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants