Skip to content

[bug] AUTO half-open probes remain reserved after input errors #2144

Open

Description

@wxhking

What happened

After an AUTO model/provider circuit's cooldown, a half-open probe can receive a request/input error such as context_length_exceeded or HTTP 400 invalid tool schema. The middleware skips health recording for FailureScope.NONE, so the probe's half_open_in_flight reservation is never released.

Later valid AUTO requests cannot reach the affected model, although the health snapshot reports the route as healthy. With no other usable fallback, they fail with no healthy AUTO model is available. A provider-level reservation also blocks sibling models under that provider. Waiting longer does not release it; restarting the process or a successful explicit-model call can clear it.

Expected behavior

A request/input error should release the expired half-open reservation without quarantining the endpoint. A later valid AUTO request should execute normally. Context-length errors should retain their existing failover behavior; invalid request errors should still stop cross-model failover.

Steps to reproduce

  1. Open a model or provider circuit after a health failure, then advance past its cooldown.
  2. Make an AUTO request through SafeModelFailoverMiddleware; begin_attempt() reserves the half-open probe.
  3. Have the handler raise RuntimeError("context_length_exceeded") or RuntimeError("HTTP 400 invalid tool schema") after retries are exhausted.
  4. Send a valid AUTO request for the same route with no other usable fallback. The handler is never called because begin_attempt() continues returning False, even well beyond the cooldown. Provider-scope failures also block sibling routes.

The existing registry already releases this reservation in ModelHealthRegistry.record_failure(..., scope=FailureScope.NONE), but SafeModelFailoverMiddleware._record_failure() filters that call out. Both synchronous and asynchronous paths use this helper.

Environment

  • Octop: source checkout develop at 0d65f2ad4b855141572f4ddeb367db5538c5e92e (1.0.2b9)
  • Python: 3.12.13
  • OS: Windows
  • Install method: source, dependencies from uv.lock
  • LLM provider: offline mocked handlers; no live provider or credentials needed

Logs / screenshots

An offline regression matrix covering sync/async calls, model/provider circuits, and both input-error classes reproduces all eight cases on the unmodified implementation. Each subsequent valid request fails with:

RuntimeError: no healthy AUTO model is available
8 failed, 11 deselected
Image

A minimal fix with regression coverage is prepared and will be linked in a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions