feat(nemotron_voicechat): full-duplex speech model (FastConformer + Nemotron-H + EAR-TTS + codec) with a /v1/realtime session #1372

Summary

NemotronLabs VoiceChat (mlx-community/NemotronLabs-VoiceChat-11B-{bf16,8bit,4bit}, model_type: "nemotron_voicechat", architectures: ["NemotronVoiceChatForConditionalGeneration"]) is a full-duplex speech-to-speech model. It consumes 16 kHz audio in 80 ms frames and, on every frame, emits an assistant text token, a function-channel token, a user-transcript update, and 80 ms of 22.05 kHz assistant speech, all on one shared timeline (silence and overlap are part of the timeline; there is no VAD). The model is four networks in one checkpoint: a cache-aware FastConformer speech encoder with an RNNT transcript branch, a 56-layer Nemotron-H hybrid LLM with two output heads, an EAR-TTS speech decoder (Gemma-3-style sliding-window backbone, character-aware text conditioning, mixture-of-Gaussians RVQ refinement), and a 31-codebook neural audio codec. mlxcel has nothing for this family; it does have Nemotron-H, Gemma-3 blocks, an audio worker thread, WAV I/O, a log-mel frontend, and an LSTM and STFT/iSTFT pair inside Kokoro that the sub-issues reuse.

Motivation

• mlxcel serves STT (Whisper) and TTS (Kokoro) as separate request/response endpoints (src/server/whisper_stt.rs, src/server/kokoro_tts.rs, src/server/audio_worker.rs). A duplex model is a different product surface: a stateful, bidirectional session where audio flows both ways continuously. Landing it establishes the streaming-session abstraction (per-connection model state, frame clock, aligned event stream) that any later speech-to-speech family will reuse.
• The checkpoint's 4-bit conversion is 9.2 GB and the per-frame work is bounded by caches, so a single Apple Silicon machine can hold a conversation in real time; this is the first mlxcel model where wall-clock throughput per frame is a correctness requirement, not just a benchmark number.
• The LLM is Nemotron-H (hybrid_override_pattern with M, -, * blocks), which src/models/nemotron_h.rs already runs with the fused Mamba2 kernels; reusing it keeps the largest component off the new-code path.

Sub-issues

1. feat(nemotron_voicechat): load the VoiceChat checkpoint and run offline duplex inference (FastConformer+RNNT, Nemotron-H dual-head, EAR-TTS, codec) #1374: load the checkpoint and run the four networks offline on a WAV file (speech encoder + RNNT, Nemotron-H with fused text/audio/function input and two heads, EAR-TTS warmup and per-step code generation, codec encode/decode). Delivers mlxcel generate -m <voicechat> --audio input.wav printing the transcript and assistant text and writing response.wav.
2. feat(nemotron_voicechat): cache-aware online session (80 ms frame clock, persistent caches, aligned events, profiler) #1378 (depends on 1): the stateful online session that turns arbitrary PCM chunks into 1280-sample frames and advances every network with persistent caches (streaming log-mel, FastConformer attention/conv caches, Nemotron-H sequence state, TTS KV caches, codec causal-conv and iSTFT caches), emitting aligned events with frame indices; includes the system-prompt prefill, flush/pad, cancel, and a per-frame stage profiler.
3. feat(server): /v1/realtime WebSocket endpoint for Nemotron VoiceChat sessions (PCM16 base64, one active session, dedicated worker) #1376 (depends on 2): the /v1/realtime WebSocket endpoint on mlxcel-server (session.created / session.update / input_audio_buffer.append / commit / cancel / ping, base64 PCM16 both ways, one active session, a dedicated MLX worker thread), plus a CLI microphone/speaker example.

Delivery order

1 -> 2 -> 3. Sub-issue 1 is validated offline against the checkpoint's published behavior; 2 is validated by first-frame parity with 1; 3 is validated by a WebSocket client driving a WAV file through the endpoint and comparing the concatenated audio events with the offline output.

Definition of done

• mlxcel-server -m models/NemotronLabs-VoiceChat-11B-4bit starts, a client connects to ws://host/v1/realtime, sends session.update with a system prompt, streams base64 PCM16 at 16 kHz in arbitrary chunk sizes, receives response.text.delta, conversation.item.input_audio_transcription.delta, response.function.delta, and response.audio.delta (22.05 kHz PCM16) events with monotonically increasing frame_index, and ends with response.done after input_audio_buffer.commit.
• A second concurrent connection is refused with server_busy while the first is active and accepted after it closes.
• The offline CLI path and the streaming path produce identical tokens and audio codes for the first frame of the same input (parity test), and the offline path transcribes and answers a spoken question in the checkpoint's built-in Aria voice.
• Per-frame latency is measured and reported by the profiler (perception, rnnt, language, tts, codec, total in ms, p50/p95), and the README for the family states the measured real-time factor on the validation machine for the 4-bit and 8-bit checkpoints.
• docs/supported-models.md lists the family under speech models; mlxcel list reports it.

Out of scope

• Voice cloning or any voice other than the checkpoint's Aria prompt latents.
• Acoustic echo cancellation in the microphone example.
• Batch size above 1 and more than one concurrent realtime session.
• Converting the original NeMo .nemo checkpoint; mlxcel loads the converted safetensors layout only.