Repository navigation
fix: keep the VoiceChat TTS backbone stream f32 on CUDA - #2115
Merged
Merged
Conversation
inureyes
added a commit
that referenced
this pull request
Oct 5, 2026
3 tasks done
The CUDA promotion overlay (patches-cuda/dtype.cpp, #636) resolves bf16 with f32 to bf16, so every stored bf16 tensor that met the f32 EAR-TTS stream demoted it while the reference runs f32. Four sites did: bos_emb (masked then added), audio_prompt_projection_W (matmul), the backbone's GemmaRMSNorm weights (1 + w built in bf16), and the gated fusion's bf16 text branch, which the issue did not list. bos_emb and audio_prompt_projection_W join promotes_to_f32. Gemma3Backbone gains widen_norms_to_f32, called only by the VoiceChat loader, which rebuilds each norm so it holds the stored-dtype 1 + w cast to f32 (exact for half weights; the shared GemmaRMSNorm and the overlay are unchanged). The fusion casts its text branch to the audio dtype. null_emb stays bf16: it is concatenated with the bf16 subword condition, and promoting it would change the text branch on non-CUDA builds. Every change is an exact widening that upstream promotion already performs inside the op, so non-CUDA output is unchanged. New tests check the stream dtype on an all-bf16 tiny model and the widened norm bytes. No real VoiceChat checkpoint was used. Closes #2109
inureyes
force-pushed
the
fix/issue-2109-voicechat-tts-f32
branch
from
October 5, 2026 12:07
d6da185 to
bcb2a7d
Compare
18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The CUDA promotion overlay (
patches-cuda/dtype.cpp, kept unchanged) resolves bf16 with f32 to bf16, so stored bf16 tensors demoted the f32 EAR-TTS backbone stream on CUDA. This fixes all four sites inside the VoiceChat path:bos_embandaudio_prompt_projection_Ware promoted to f32 at load (promotes_to_f32).Gemma3Backbone::widen_norms_to_f32(called only by the VoiceChat loader) rebuilds each backbone norm (4 block norms,q_norm,k_norm, final norm) so it holds the stored-dtype1 + wwidened to f32. Exact for half-precision weights (proof in the doc comment); the sharedGemmaRMSNorminlayers.rsis unchanged.multiply(f32, bf16)there also returned bf16 on CUDA.null_embstays bf16, narrowing the issue's proposal: it is concatenated with the bf16 subword condition, so promoting it would change thetext_projinput on non-CUDA builds and break identity. Its existing "not promoted" assertion is kept;bos_embandaudio_prompt_projection_Wmove to the promoted list.Validation
tts_dtype_tests::backbone_stream_is_f32_with_bf16_weights(all-bf16 tiny model): stream dtype after the projection andbos_embassembly, after the fusion, after the final norm, and on a step. On 33c4505 with--features cudait fails at the first assertion (left: 12bf16,right: 10f32).widened_norms_hold_the_stored_one_plus_w_in_f32: byte equality of every widened1 + wagainst the cast stored one, includingw = -1,-1 +/- 1 ulp, and large values, plus an f32 stream through a norm and the backbone.--release --features cuda,--test-threads=1under gpu-lock:models::nemotron_voicechat28/28,models::gemma3_backbone6/6,audio::f32_weights4/4. clippy--lib --tests -D warningsandcargo fmt --checkclean.astypethat upstream promotion already inserts insidematmul,add,multiplyandfast::rms_norm, so the kernels read identical values.Closes #2109