Skip to content

Providers can't set a request timeout, so api_timeout_error retries almost never trigger #48

Open

Description

@jessicafalcon

Summary

At e1d4cc9 (v0.2.1), none of the providers let the caller set a request timeout, so every request uses the SDK default: up to 10 minutes for OpenAI and Anthropic, and no timeout at all for Gemini (its client defaults to httpx.Timeout(None)).

This means RetryPolicy(api_timeout_error=True), which the adapter accepts as a drop-in for TypeSafeClient, can't realistically trigger, and a stalled request hangs the caller instead of failing fast so it can fall back.

Reproduction

A local server that accepts connections but never replies, with RetryPolicy(max_retries=1):

import socket, threading, time
from typesafe_sdk import RetryPolicy
from system_one_adapter import Noul, SystemOneAdapterClient
from system_one_adapter.providers.openai import OpenAIProvider

server = socket.socket()
server.bind(("127.0.0.1", 0))
server.listen()
held = []
threading.Thread(target=lambda: [held.append(server.accept()) for _ in iter(int, 1)], daemon=True).start()

provider = OpenAIProvider("gpt-4o-mini", base_url=f"http://127.0.0.1:{server.getsockname()[1]}/v1", api_key="test")
client = SystemOneAdapterClient(structured_outputs=True, llm_answer_mode="discrete", retry=RetryPolicy(max_retries=1))

start = time.monotonic()
try:
    client.system_one(state="hi", questions={"q": Noul(instructions="Is it a greeting?")}, model=provider)
except Exception as error:
    print(f"{type(error).__name__} after {time.monotonic() - start:.2f}s")
  • Expected: a way to bound each request, failing with TypeSafeAPITimeoutError.
  • Actual on main: no output; I killed it after 30 s. Each attempt waits up to 10 minutes.
  • With the proposed fix (OpenAIProvider(..., timeout=0.5)): TypeSafeAPITimeoutError after 1.5s (two attempts plus backoff).

Proposed fix

An optional timeout= (seconds) on all six providers, following the existing max_tokens= option on AnthropicProvider:

client.system_one(state, questions, model=OpenAIProvider("gpt-4o-mini", timeout=2.0))
  • Omitted, it keeps today's behavior. None is never passed through, since the SDKs read it as "never time out".
  • Zero or negative values raise ValueError, like max_tokens.
  • Gemini receives it as sdk_configuration.timeout_ms.
  • README section (including that the timeout is per request, not per system_one call) and an Unreleased changelog entry.

The fix is one commit on my fork:

Happy to adjust the API (for example, also accepting an httpx.Timeout) or for a maintainer to take it from there.

Test plan

  • uv run pytest tests/test_provider_timeouts.py: new tests cover all six providers. The ones checking that the timeout reaches the SDK fail on main and pass with the fix.
  • uv run pytest: 445 passed.
  • uv run pyrefly check: 0 errors.
  • The reproduction above, run on main and on the branch.

Activity

  1. aoright commented on Sep 27, 2026

    @aoright

    I ran an independent verification of @jessicafalcon's patch (commit 5dbe207) on macOS with Python 3.14.3 and confirmed the timeout behavior end-to-end.

    Verification Summary

    1. Unit Tests:

      • uv run pytest tests/test_provider_timeouts.py: 21/21 passed (parameter passing across all providers, millisecond conversion for Gemini, rejection of non-positive timeouts).
      • Full suite passes: uv run pytest (453 passed), uv run pyrefly check (0 errors).
    2. End-to-End RetryPolicy Behavior:

      • Configured OpenAIProvider("gpt-4o-mini", timeout=0.2, ...) with RetryPolicy(max_retries=1, api_timeout_error=True) against an in-memory socket server that accepts connections without responding.
      • Initial attempt times out at ~0.2s, queues retry attempt, backoff delays ~0.8s, second attempt times out at ~0.2s, and successfully raises TypeSafeAPITimeoutError at total elapsed time ~1.46s. Both sync (SystemOneAdapterClient) and async (AsyncSystemOneAdapterClient) behave consistently.
    3. In-Suite Integration Testing & pytest-recording:

      • When running socket-level timeout integration tests directly under pytest, pytest-recording intercepts socket creation because pyproject.toml configures addopts = "--block-network -p no:tach".
      • Without an exception for local loopback, socket.connect() raises RuntimeError: Network is disabled from pytest_recording/network.py:124.
      • Adding "allowed_hosts": ["127.0.0.1"], to the vcr_config fixture in tests/conftest.py (as discussed in --block-network makes the whole test suite unrunnable on Windows #44) allows local loopback tests to run cleanly under pytest while keeping external provider calls strictly blocked.

    Integration Branch

    I added end-to-end timeout integration tests covering sync and async client behavior on branch:
    https://github.com/aoright/system-one-adapter-python/tree/test/verify-provider-timeout (commit 5b2a5bd)

    With "allowed_hosts": ["127.0.0.1"], all 455 tests pass cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions