Feature request: HTTP connection pooling for improved performance

[fede-kamel]

Problem

The SDK currently creates new HTTP connections for each API request. Each new connection requires a TCP handshake (~50-100ms) and TLS negotiation (~100-200ms), adding ~150-300ms overhead per request.

For applications making multiple sequential API calls (e.g., chat followed by embed, or multiple embed calls), this overhead adds up.

Proposed Solution

Configure httpx.Limits on the default httpx.Client and httpx.AsyncClient instances to enable connection reuse:

httpx.Limits(
    max_keepalive_connections=20,
    max_connections=100,
    keepalive_expiry=30.0
)

This is a ~16 line change across sync and async clients.

Expected Impact

• 15-30% reduction in latency for subsequent API calls
• Reduced server load from fewer connection establishments
• More predictable response times

Context

We use the Cohere SDK at Oracle for workloads involving multiple sequential API calls. Connection pooling is a standard optimization in HTTP client libraries and httpx supports it natively.

Implementation available in PR #697.

References

• httpx connection pooling docs: https://www.python-httpx.org/advanced/#pool-limit-configuration

[fern-support]

Thanks for reporting this issue. You should be able to configure whatever limit you want using the existing SDK without any changes like so:

# Create a custom httpx client with limits
custom_limits = httpx.Limits(
    max_keepalive_connections=20,
    max_connections=100,
    keepalive_expiry=30.0,
)
custom_httpx_client = httpx.Client(limits=custom_limits)

# Pass it directly to the Cohere client
client = cohere.ClientV2(api_key=api_key, httpx_client=custom_httpx_client)

Given that this behavior is already supported, we're going to close this issue and the linked PR.
If you still find this doesn't resolve your issue, please let us know. We would be happy to revisit this.

[fede-kamel]

Fair point. Since the SDK already uses HTTPX’s built-in pooling and already accepts a custom httpx_client, I agree this should not be framed as adding pooling support. I think the only remaining case for the PR would be if there is evidence that HTTPX’s default pool limits are not a good default for typical Cohere SDK workloads. Without that evidence, I agree the current extension point is probably sufficient.