Repository navigation
fix(server): report an audio-only server healthy once its worker is ready - #1750
Merged
Merged
Conversation
…eady health_check fell through to model_provider.is_loaded() for a server whose primary checkpoint is a Whisper or Kokoro model. There is no chat worker to load in that mode, so is_loaded() stayed false for the life of the process and /health returned the loading envelope forever, failing every container probe while transcription, translation and speech all served correctly. Report ready when the chat provider is unavailable and an audio provider is installed. startup.rs fills the audio slot only after the provider loads and before the listener binds, so a failed audio load leaves the slot empty and the server still reports not ready; and the slot is filled only when the primary checkpoint is itself an audio model, so a chat worker crash is not masked.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1747.
An audio-only server answers every audio route correctly and never reports itself ready, so
/healthfails any container probe pointed at it.The defect
health_checkdecides readiness with two arms:side_model_ready(&state)whenembedding_serving_mode.blocks_generation(), andmodel_provider.is_loaded()otherwise. A server started with a primary audio checkpoint takes the second arm, because an audio-only mode does not set a generation-blocking embedding mode. In that configuration there is no chat worker to load:startup.rsdetectsModelType::WhisperorModelType::Kokoroand populates the audio slot instead, while the deliberately unused chat worker records a terminal state.is_loaded()therefore stays false for the life of the process and/healthreturns the b10621 loading envelope forever.That contradicts two documented promises at once.
docs/server-features.mdsaysGET /healthand/v1/healthprovide readiness, anddocs/audio-api.mddocuments-m <whisper checkpoint>and-m <kokoro checkpoint>as the way to serve audio. A documented configuration cannot pass its documented readiness probe, so an orchestrator or load balancer in front of it restarts or drains a server whose transcription, translation and speech routes are all serving.The fix
A third arm: when the chat provider is unavailable and an audio provider is installed, report ready.
This cannot produce a false ready.
startup.rssetsaudio_modeltoSome(..)only afterWhisperSttProvider::load(..)orKokoroTtsProvider::load(..)returnsOk, and.with_audio_model(..)runs before the HTTP listener binds. When the audio load fails the slot staysNone, the new arm does not fire, and the server correctly reports not ready.It also cannot mask a chat worker crash.
audio_modelis populated only when the primary checkpoint is itself an audio model, which is the deliberate case rather than the failure case.exited_chat_worker_returns_503_without_channel_detailscovers the crash case and is unchanged by this PR.Verification
audio_only_server_reports_healthy_after_its_worker_is_readyasserts 200 with exactly{"status": "ok"}for an audio-only state, alongside the existingaudio_only_server_names_the_served_routes. Thegethelper it needs mirrors theposthelper already in the file.cargo test -p mlxcel --lib --features metal,accelerate availabilitypasses 7 of 7 on an M1 Ultra, including the two pre-existing failure-path tests.cargo clippy -p mlxcel --lib --tests -- -D warnings, the exact command the CI clippy job runs, is clean.cargo fmt --checkis clean.