You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[bug] AUTO half-open probes remain reserved after input errors #2144
After an AUTO model/provider circuit's cooldown, a half-open probe can receive a request/input error such as context_length_exceeded or HTTP 400 invalid tool schema. The middleware skips health recording for FailureScope.NONE, so the probe's half_open_in_flight reservation is never released.
Later valid AUTO requests cannot reach the affected model, although the health snapshot reports the route as healthy. With no other usable fallback, they fail with no healthy AUTO model is available. A provider-level reservation also blocks sibling models under that provider. Waiting longer does not release it; restarting the process or a successful explicit-model call can clear it.
Expected behavior
A request/input error should release the expired half-open reservation without quarantining the endpoint. A later valid AUTO request should execute normally. Context-length errors should retain their existing failover behavior; invalid request errors should still stop cross-model failover.
Steps to reproduce
Open a model or provider circuit after a health failure, then advance past its cooldown.
Make an AUTO request through SafeModelFailoverMiddleware; begin_attempt() reserves the half-open probe.
Have the handler raise RuntimeError("context_length_exceeded") or RuntimeError("HTTP 400 invalid tool schema") after retries are exhausted.
Send a valid AUTO request for the same route with no other usable fallback. The handler is never called because begin_attempt() continues returning False, even well beyond the cooldown. Provider-scope failures also block sibling routes.
The existing registry already releases this reservation in ModelHealthRegistry.record_failure(..., scope=FailureScope.NONE), but SafeModelFailoverMiddleware._record_failure() filters that call out. Both synchronous and asynchronous paths use this helper.
Environment
Octop: source checkout develop at 0d65f2ad4b855141572f4ddeb367db5538c5e92e (1.0.2b9)
Python: 3.12.13
OS: Windows
Install method: source, dependencies from uv.lock
LLM provider: offline mocked handlers; no live provider or credentials needed
Logs / screenshots
An offline regression matrix covering sync/async calls, model/provider circuits, and both input-error classes reproduces all eight cases on the unmodified implementation. Each subsequent valid request fails with:
RuntimeError: no healthy AUTO model is available
8 failed, 11 deselected
A minimal fix with regression coverage is prepared and will be linked in a PR.
Activity