Skip to content

fix: keep the VoiceChat TTS backbone stream f32 on CUDA - #2115

Merged
inureyes merged 2 commits into
mainfrom
fix/issue-2109-voicechat-tts-f32
Oct 5, 2026
Merged

inureyes merged 2 commits into
mainfrom
fix/issue-2109-voicechat-tts-f32

Conversation

@inureyes

@inureyes inureyes commented Oct 5, 2026

Copy link
Copy Markdown
Member

Summary

The CUDA promotion overlay (patches-cuda/dtype.cpp, kept unchanged) resolves bf16 with f32 to bf16, so stored bf16 tensors demoted the f32 EAR-TTS backbone stream on CUDA. This fixes all four sites inside the VoiceChat path:

  • bos_emb and audio_prompt_projection_W are promoted to f32 at load (promotes_to_f32).
  • Gemma3Backbone::widen_norms_to_f32 (called only by the VoiceChat loader) rebuilds each backbone norm (4 block norms, q_norm, k_norm, final norm) so it holds the stored-dtype 1 + w widened to f32. Exact for half-precision weights (proof in the doc comment); the shared GemmaRMSNorm in layers.rs is unchanged.
  • The gated fusion casts its bf16 text branch to the audio dtype. The issue did not list this site, but multiply(f32, bf16) there also returned bf16 on CUDA.
  • null_emb stays bf16, narrowing the issue's proposal: it is concatenated with the bf16 subword condition, so promoting it would change the text_proj input on non-CUDA builds and break identity. Its existing "not promoted" assertion is kept; bos_emb and audio_prompt_projection_W move to the promoted list.

Validation

  • New tts_dtype_tests::backbone_stream_is_f32_with_bf16_weights (all-bf16 tiny model): stream dtype after the projection and bos_emb assembly, after the fusion, after the final norm, and on a step. On 33c4505 with --features cuda it fails at the first assertion (left: 12 bf16, right: 10 f32).
  • New widened_norms_hold_the_stored_one_plus_w_in_f32: byte equality of every widened 1 + w against the cast stored one, including w = -1, -1 +/- 1 ulp, and large values, plus an f32 stream through a norm and the backbone.
  • GB10 --release --features cuda, --test-threads=1 under gpu-lock: models::nemotron_voicechat 28/28, models::gemma3_backbone 6/6, audio::f32_weights 4/4. clippy --lib --tests -D warnings and cargo fmt --check clean.
  • Non-CUDA identity is argued from the code, not measured (CPU-only Linux build is blocked by fix: CPU-only Linux build fails to link kv_inplace_write #2108): each change is the exact bf16 to f32 astype that upstream promotion already inserts inside matmul, add, multiply and fast::rms_norm, so the kernels read identical values.
  • No real VoiceChat checkpoint was used; none is available on GB10.

Closes #2109

inureyes added a commit that referenced this pull request Oct 5, 2026
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions priority:low Low priority area:models Model architectures, weights, loading, metadata modeltype:audio Audio or speech model platform:linux Linux (CUDA / packaging) specific status:done Completed labels Oct 5, 2026
The CUDA promotion overlay (patches-cuda/dtype.cpp, #636) resolves bf16 with f32 to bf16, so every stored bf16 tensor that met the f32 EAR-TTS stream demoted it while the reference runs f32. Four sites did: bos_emb (masked then added), audio_prompt_projection_W (matmul), the backbone's GemmaRMSNorm weights (1 + w built in bf16), and the gated fusion's bf16 text branch, which the issue did not list.

bos_emb and audio_prompt_projection_W join promotes_to_f32. Gemma3Backbone gains widen_norms_to_f32, called only by the VoiceChat loader, which rebuilds each norm so it holds the stored-dtype 1 + w cast to f32 (exact for half weights; the shared GemmaRMSNorm and the overlay are unchanged). The fusion casts its text branch to the audio dtype. null_emb stays bf16: it is concatenated with the bf16 subword condition, and promoting it would change the text branch on non-CUDA builds.

Every change is an exact widening that upstream promotion already performs inside the op, so non-CUDA output is unchanged. New tests check the stream dtype on an all-bf16 tiny model and the widened norm bytes. No real VoiceChat checkpoint was used.

Closes #2109
@inureyes
inureyes force-pushed the fix/issue-2109-voicechat-tts-f32 branch from d6da185 to bcb2a7d Compare October 5, 2026 12:07
@inureyes
inureyes merged commit 6f3fa7b into main Oct 5, 2026
26 checks passed
@inureyes
inureyes deleted the fix/issue-2109-voicechat-tts-f32 branch October 5, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata modeltype:audio Audio or speech model platform:linux Linux (CUDA / packaging) specific priority:low Low priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Nemotron VoiceChat TTS backbone runs bf16 on CUDA

1 participant