Repository navigation
Reduce Qwen3-TTS, IndexTTS, and BreezeTTS memory allocations - #821
Merged
Merged
Conversation
This was referenced Oct 7, 2026
0xShug0
pushed a commit
that referenced
this pull request
Oct 7, 2026
* lfm2_audio: recommend Q8_0 or F16 for S2S With the Q4_0 package, a few S2S replies repeat a sentence until max_tokens: 2 of 30 replies to one English question on CUDA and 2 of 30 on the M3 Ultra CPU, 2 of 60 on the x86 CPU and none on Metal, while F16 and Q8_0 had none. The Validation section has these numbers, but the Run section, where a user picks a package, did not say so. Point S2S users at the Q8_0 package, the default, or F16 there. * lfm2_audio: text_temperature and text_top_k for S2S S2S text was always greedy. liquid-audio's generate_interleaved takes text_temperature and text_top_k next to the audio sampling arguments, and its _sample_text_token is greedy when the temperature is at most 0 or top_k is 1; otherwise it divides the logits by the temperature, keeps those at or above the k-th, applies softmax and draws. Add the two S2S request options with that rule, through the sampler the audio codes already use. Both default to 0, so text stays greedy as in liquid-audio's README and demo, and a request without them goes through the same lfm2_greedy call as before. Sampled text first checks its logits for NaN and infinity, as lfm2_greedy does, since the sampler would leave them out and draw from the rest. Text draws take a stream of their own, seeded from a fixed splitmix64 mix of the request seed, and the audio stream is the one it was. So with the same seed, a reply with sampled text is step for step the greedy-text reply, audio codes included, until a text token differs. liquid-audio draws both from torch's one stream, but a seed here does not reproduce its draws anyway. ASR and TTS reject the options. The spec lists them, and the docs add the option rows and the seed contract and say that the live route takes them in the query, which both server runtimes already forward. The TTS unit test checks the greedy rule on random logits, a tie at the k-th logit, the temperature on two logits, that sampled text rejects non-finite logits, and the separate streams; the session unit test checks that ASR rejects the options. On main the first does not build and the second fails. The S2S model test checks the step-for-step contract at three seeds, that text_top_k=1 gives the greedy reply and the rejections; the TTS model test checks the rejection. The lfm2_audio unit tests and model_spec_system pass on Linux CPU and CUDA builds (gcc 12) and a macOS Metal build. The S2S model test passes with the F16 and Q4_0 packages on CUDA and Metal and with F16 on the CPU, and the TTS model test with F16 on CUDA and Metal. Without the options, S2S replies through the CLI are byte for byte main's, text and WAV: 21 offline replies and 5 streams over the EN F16 and Q4_0 packages on CUDA and Metal and EN Q4_0 on the CPU, several requests per process. With text_temperature=0.7 and text_top_k=50, a seeded reply repeats in the same process and streams the text it gives offline, and text_top_k=1 gives the default reply. On the server's live route on CUDA, both runtimes pass the options on: the sampled reply's text is the CLI's for the same seed, and a negative value comes back as the session's error. * lfm2_audio: rebuild the WebUI page with the text sampling options The page audiocpp_server serves embeds model_specs, so it is rebuilt with npm ci and npm run build in webui/native for the two new lfm2_audio request options and the longer seed description. Main's committed page predates #821, which changed the breeze_tts, index_tts2 and qwen3_tts specs without rebuilding it, so those changes come in too: built from the sources before #821, main's page comes out byte for byte as committed. Built from this commit's sources, the page differs from a build of main's sources only in those lfm2_audio spec entries and in the bundle file name its imports use.
delebash
added a commit
to delebash/audio.cpp
that referenced
this pull request
Oct 15, 2026
Upstream's v0.9.1 (60 commits, 11 new model specs) under our 22 commits. Two conflicts: - README.md: upstream's new `reuse` row kept beside our Chatterbox Turbo row, which keeps our later wording (cloning packages from delebash/chatterbox-turbo-GGUF). - qwen3_tts talker.cpp and tokenizer_speech_decoder.cpp: upstream's 0xShug0#821 ("Reduce Qwen3-TTS, IndexTTS, and BreezeTTS memory allocations") makes the same two changes as our 8cdf121 — the talker prefill through ggml_gallocr with its inputs, outputs and every layer's K/V kept, and the old decoder graph freed before its replacement is built — and also moves the cached-step graph onto ggml_gallocr. Upstream's version taken whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Measured results
CUDA Q8 A/B measurements:
IndexTTS savings apply before loading emotion weights; sequences that include emotion synthesis showed essentially unchanged peaks.
The initial Qwen3 timing differences did not reproduce consistently in reversed-order measurements; affected cached-step timings showed no demonstrated regression. See the included report for both measurements and limitations.
All tested Qwen3-TTS and IndexTTS before/after C++ WAV pairs were byte-identical. Sampling math, weight/cache precision, and cache capacities are unchanged.