Skip to content

fix(server): report an audio-only server healthy once its worker is ready - #1750

Merged
inureyes merged 1 commit into
mainfrom
fix/audio-only-health-readiness
Sep 10, 2026
Merged

inureyes merged 1 commit into
mainfrom
fix/audio-only-health-readiness

Conversation

@inureyes

Copy link
Copy Markdown
Member

Closes #1747.

An audio-only server answers every audio route correctly and never reports itself ready, so /health fails any container probe pointed at it.

The defect

health_check decides readiness with two arms: side_model_ready(&state) when embedding_serving_mode.blocks_generation(), and model_provider.is_loaded() otherwise. A server started with a primary audio checkpoint takes the second arm, because an audio-only mode does not set a generation-blocking embedding mode. In that configuration there is no chat worker to load: startup.rs detects ModelType::Whisper or ModelType::Kokoro and populates the audio slot instead, while the deliberately unused chat worker records a terminal state. is_loaded() therefore stays false for the life of the process and /health returns the b10621 loading envelope forever.

That contradicts two documented promises at once. docs/server-features.md says GET /health and /v1/health provide readiness, and docs/audio-api.md documents -m <whisper checkpoint> and -m <kokoro checkpoint> as the way to serve audio. A documented configuration cannot pass its documented readiness probe, so an orchestrator or load balancer in front of it restarts or drains a server whose transcription, translation and speech routes are all serving.

The fix

A third arm: when the chat provider is unavailable and an audio provider is installed, report ready.

This cannot produce a false ready. startup.rs sets audio_model to Some(..) only after WhisperSttProvider::load(..) or KokoroTtsProvider::load(..) returns Ok, and .with_audio_model(..) runs before the HTTP listener binds. When the audio load fails the slot stays None, the new arm does not fire, and the server correctly reports not ready.

It also cannot mask a chat worker crash. audio_model is populated only when the primary checkpoint is itself an audio model, which is the deliberate case rather than the failure case. exited_chat_worker_returns_503_without_channel_details covers the crash case and is unchanged by this PR.

Verification

  • audio_only_server_reports_healthy_after_its_worker_is_ready asserts 200 with exactly {"status": "ok"} for an audio-only state, alongside the existing audio_only_server_names_the_served_routes. The get helper it needs mirrors the post helper already in the file.
  • Negative control: with the new arm reverted and the test kept, the test fails. It guards the fix rather than restating it.
  • cargo test -p mlxcel --lib --features metal,accelerate availability passes 7 of 7 on an M1 Ultra, including the two pre-existing failure-path tests.
  • cargo clippy -p mlxcel --lib --tests -- -D warnings, the exact command the CI clippy job runs, is clean. cargo fmt --check is clean.

…eady

health_check fell through to model_provider.is_loaded() for a server whose
primary checkpoint is a Whisper or Kokoro model. There is no chat worker to
load in that mode, so is_loaded() stayed false for the life of the process
and /health returned the loading envelope forever, failing every container
probe while transcription, translation and speech all served correctly.

Report ready when the chat provider is unavailable and an audio provider is
installed. startup.rs fills the audio slot only after the provider loads and
before the listener binds, so a failed audio load leaves the slot empty and
the server still reports not ready; and the slot is filled only when the
primary checkpoint is itself an audio model, so a chat worker crash is not
masked.
@inureyes
inureyes merged commit 5004dfc into main Sep 10, 2026
13 checks passed
@inureyes
inureyes deleted the fix/audio-only-health-readiness branch September 10, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: An audio-only server never reports healthy, so /health blocks every container probe

1 participant