[inureyes]

Summary

Serves the Nemotron VoiceChat online session (#1378) over a WebSocket at /v1/realtime on mlxcel-server / mlxcel serve. The protocol uses JSON control messages, base64 PCM16 audio in both directions, and one active session at a time, with all MLX work on one dedicated engine thread. The PR also adds a WAV file client and a live microphone/speaker client.

What changed

• Engine (server::realtime_engine, server::realtime_session): one thread loads the checkpoint and owns the active VoiceChatStreamingSession. The engine accepts open / push / flush / cancel / close commands over a channel with one-shot replies. try_reserve admits exactly one connection. close cancels an unflushed session and clears the MLX buffer cache. A panicking session is contained and reported as an error.
• Protocol (server::realtime_protocol): client messages and server events as tabulated in the issue and in the mlx-vlm reference server. That covers session.created / updated, input_audio_buffer.append / commit / committed, session.cancel / response.cancel, session.ping / pong, the text, function, transcript and audio deltas, response.done / cancelled, and error with the four codes. It also includes PCM16 base64 decoding (odd byte counts rejected) and encoding with round(clip(x) * 32767).
• Route (server::routes::realtime): the WebSocket loop.
  • Appends are sliced into 1280-sample pieces, and each piece's events are forwarded before the next piece is sent.
  • A second connection gets server_busy and close code 1013.
  • The reservation is released on commit, cancel, or disconnect.
  • The route is registered only when the engine exists, behind the --api-key and --api-prefix layers.
  • Hyper connections now serve upgrades.
• Startup: a VoiceChat checkpoint spawns the engine and skips the chat worker, the scheduler, and the text warmup. The chat, completions, responses, and messages endpoints answer 501 with a pointer to /v1/realtime, and /health reports ready. The registry lists the family as generate + serve. The feat(nemotron_voicechat): load the VoiceChat checkpoint and run offline duplex inference (FastConformer+RNNT, Nemotron-H dual-head, EAR-TTS, codec) #1374 serve refusal is removed.
• Clients:
  • examples/voicechat_file_client.rs drives a WAV through the socket.
  • examples/voicechat_microphone.rs is a cpal capture/playback loop. It sends an append every 80 ms, plays the answer through a ring buffer, commits on Ctrl-C, and supports --list-devices, --input-device, and --output-device. cpal is an optional dependency behind the voicechat-mic feature (example required-features), so the library, the server, and CI never compile it.
• Docs: protocol tables, a websocat transcript, the single-session limit, the clients, and a README server entry.

Validation

Real checkpoint (4-bit), mlxcel-server -m ... , file client driving "What is the capital of France?" plus 3 s of silence, seed 0, system prompt "Be concise and answer in one sentence." (run before and again after the review fixes):

• Events: session.created, session.updated, 6 transcript deltas building "What is the capital of France", 7 text deltas building "The capital of France is Paris.", 61 response.audio.delta with 1764 samples each and frame_index 0..60, input_audio_buffer.committed, response.done, close 1000.
• The concatenated socket audio matches mlxcel generate --stream for the same input: 107604 samples each, every sample consistent with one shared float waveform; the integer WAVs differ only by the writers' scaling (the CLI WAV writer uses 32768, the wire format the reference's 32767), 107266 of 107604 identical.
• A second connection during a session gets server_busy and close 1013; after the first closes, a new connection configures and cancels normally. /v1/chat/completions answers 501 naming /v1/realtime; /health is ok.
• Local gate: fmt, clippy (lib+tests, examples, and the voicechat-mic example), cargo test --lib server:: (3268 passed), the registry/detection/voicechat unit tests, tests/realtime_ws.rs (9 passed with a fake engine: the seven issue cases plus abrupt disconnect and whole-float sample rate), cli_help_consistency, dead_doc_pointers, llama_compat_manifest. GitHub CI was not awaited (self-hosted runner down).

Not verified: live microphone exchange

The microphone client compiles, enumerates the audio devices (--list-devices), and uses the same protocol path the file client validated, but a live exchange could not be run here: in this non-interactive agent session, opening an input stream on this Mac blocks or fails (CoreAudio "Unknown property" on the default camera microphone, a hang on the USB microphone), consistent with the process lacking microphone permission. The acceptance item "microphone example holds a live exchange with headphones" therefore stays unchecked and needs a manual run: start the server, then cargo run --release --features voicechat-mic --example voicechat_microphone -- ws://127.0.0.1:8080/v1/realtime from a terminal that has microphone access.

Changes during review

• Server limits for the single slot: sessions capped at 600 s of audio (client values clamped, default applied), messages and frames at 1 MiB, 30 s to send session.update, and a 120 s idle limit after configuration (pings do not count; close 1008).
• A task dropped while its open is in flight still closes the engine session; /health reports a dead engine thread; whole-number floats are accepted for sample_rate and seed as the reference's int() does.
• Microphone client: sends its buffered tail before commit, a second Ctrl-C (or one before the session is ready) quits, and an error after commit stops waiting.
• Not changed: --cors-origins is not enforced on the WebSocket upgrade (browsers do not apply CORS to WebSockets); with a restricted origin list and no --api-key, any page can still take the slot. Tracked as a follow-up candidate.

Closes #1376