Repository navigation
fix(webui): no text box for lfm2_audio speech-to-speech - #828
Merged
Merged
Conversation
Owner
|
@ykhrustalev Could you resolve the conflict? I can't edit the fork directly. The fork reports |
The WebUI showed its text box for the lfm2_audio speech-to-speech entries and sent the text with /v1/tasks/run, where it becomes text_input and replaces the chat system prompt the checkpoints were trained with, `Respond with interleaved text and audio.` The page also keeps the text when the model changes, so after lfm2-audio-tts, whose box starts with "The next train leaves in ten minutes.", a question to lfm2-audio-s2s was answered with that sentence as the system prompt. The lfm2_audio S2S entries now have no text box, and the page sends no text for them, whatever the box held, so replies use liquid-audio's system prompt, as the CLI does without --text. liquid-audio's demo hard-codes that prompt, and its README says the system prompt should be set to it. Another prompt can still be tried with text on /v1/tasks/run or --text on the CLI. Other S2S entries, such as PersonaPlex, keep the box. The two catalog hints say which prompt the replies use instead of asking to leave the box empty, and the docs' WebUI sentence says the same. The WebUI has no unit tests; svelte-check reports no errors or warnings. In headless Chrome, against audiocpp_server with the EN and JP Q8_0 TTS and S2S entries, the page was opened on a TTS entry, Voice conversion then picked the S2S entry, and a question was run at seed 7. Main's page kept the TTS sentence in the box and sent it as text: the English reply to assets/resources/c.wav began "Oh hi, I'm Lin Samuae from Whole Life English Council.", and the reply to a Japanese question was garbled. This commit's page shows no box and sends no text, and each reply has the same audio, byte for byte, and the same text as the API's reply to that request with text set to liquid-audio's prompt. A PersonaPlex entry, listed but not run, still has its box. The lfm2_audio unit tests and model_spec_system_test pass on a Linux CPU build and on macOS.
The page audiocpp_server serves is rebuilt with npm ci and npm run build in webui/native, as webui/README.md describes, for the previous commit's page and catalog changes. The commit changes only webui/native/dist/index.html. Main's committed page equals a build of main's sources, so nothing else comes in. Built from this commit's sources, the page differs from a build of main's sources only in the two S2S hints, the new check that hides the box and drops the text, the bundle's content hash and the minifier's short names.
Contributor
Author
ykhrustalev
force-pushed
the
lfm2-audio-webui-s2s-box
branch
from
October 7, 2026 14:26
cf77855 to
6ac492d
Compare
Owner
|
@ykhrustalev Thanks! PR merged. I think LFM deserves a dedicated page like YuE2, with customized controls (https://github.com/0xShug0/audio.cpp/tree/main/webui/native/src/lib/models). But that can be handled in follow-up PRs after the next release. |
0xShug0
pushed a commit
that referenced
this pull request
Oct 10, 2026
* webui: let a studio panel run the page's request Model panels are registered per family, and the page always builds and sends the request itself. A conversation view has to send its own request and show the reply as it streams in, while it keeps the page's Run button, Ctrl+Enter, Cancel, elapsed time and status line. This adds that hook without changing what any existing panel does: - A panel entry may list the tasks it covers, such as tasks: ['s2s']. An entry without the list covers every task of its family, as before. - With requestMode 'panel', run() hands the request to the runner the panel registers through the page's registerPanelRunner, which the next commit passes to panels as setPanelRunner; registering returns the function that removes it. It passes the source audio, the resolved seed, Max tokens, Language, the request options and the abort signal that Cancel uses. The hand-off comes before anything is awaited, so the panel can start audio inside the Run click or key press. The reply the panel returns is shown in the Result column; these runs are not added to Run history. Missing source audio, a recording still running, or Run from a tab other than Studio is reported as a warning, like the page's other checks before a run. - With UI management the panel can ask for the entry in a given mode. This uses the page's ensureLoadedMode unchanged, after reloading an entry that is resident with another package or with imported settings that differ, as ensureLoaded does for the other runs. - api.ts gets taskStreamEvents, a reader for /v1/tasks/stream with "stream_format": "sse". It yields each event and then the result, throws the server's error message, and throws TaskStreamClosedError when the stream ends before task.stream.done. The done message carries the whole reply again and comes over many reads, so the reader keeps its pieces and joins them once, when the message ends. No panel uses requestMode 'panel' yet. * webui: LFM2.5-Audio conversation panel for speech-to-speech The LFM2.5-Audio speech-to-speech entries ran one question per Run, as a new conversation each time, and showed the reply only when it was done. The server can now carry a conversation (earlier turns sent as request artifacts) and stream a reply as server-sent events, so this gives the two S2S entries a page of their own, as asked on #828. The panel covers only the S2S entries; ASR and TTS keep the generic controls. It adds a conversation above the page's own Language, Seed, Max tokens, Source audio and Model parameters, which it uses as they are. Each Run: - uploads the recorded or chosen question, and sends it to /v1/tasks/stream with "stream_format": "sse" and return_codes, with each earlier turn as its question's upload path and the reply artifact that turn returned; - plays the reply as it streams in, its 80 ms chunks scheduled back to back on one Web Audio clock, and shows the reply text as it arrives; - adds the turn to a list with the question, the reply text and the reply audio, and clears the source picker for the next question. New conversation starts over, and Leave out drops an old turn from the next request, which is what the server asks for past its 8192-step limit. A failed or stopped turn, or one whose stream closed before its result, is never sent as history; the next Run takes its place. Starting a turn stops the previous reply, and Stop audio stops a reply that is still playing. If the browser does not let live audio start, the panel says so and the reply is in its player when it finishes. Finished turns are kept when the user opens another tab or entry and comes back (leaving the panel stops a turn still running), and a turn on the other checkpoint starts a new conversation. With UI management the entry is loaded in streaming mode when needed; on a server with a config file, an entry in offline mode is run through /v1/tasks/run and its reply shows when it is done. An S2S entry configured under an id that is not the catalog's gets the family's ASR parameters in Model parameters; turns leave those out and send the page's Max tokens. Each turn's players and its Leave out button are named with the turn number for screen readers, and its state badge is a polite live region. The page passes panels three more props for this: modelId, setPanelRunner, which is its registerPanelRunner, and clearSource, its own clearSourceFile. The other panels do not declare them and ignore them. model_params.json gets text_temperature and text_top_k for the two S2S entries, with the server's defaults, and the S2S hints and the model docs describe the conversation. * webui: rebuild the page Rebuilt from this branch's source with the lockfile (node 24.19.0, npm 11.17.0): cd webui/native npm ci npm run build The page on main was built from an older tree. It lacks changes to the model specs or the page from #824, #829, #830, #833, #834, #835, #846, #851, #853 and #854, so this rebuild also brings those into the page. Rebuilding main's own source the same way gives a page that differs from this one only in this branch's changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The WebUI shows no text box for the
lfm2_audiospeech-to-speech entries and sends no text for them.Problem
lfm2_audioS2S, a request's text becomestext_inputand replaces the chat system prompt the checkpoints were trained with,Respond with interleaved text and audio./v1/tasks/run, and keeps the text when the model changeslfm2-audio-tts, whose box starts with "The next train leaves in ten minutes.",lfm2-audio-s2sanswered with that sentence as the system prompt. At seed 7 its reply toassets/resources/c.wavbegan "Oh hi, I'm Lin Samuae from Whole Life English Council."; the JP entries replied with garbled textWhat this PR changes
+page.svelte: a newusesFixedChatPrompt(familylfm2_audio, tasks2s) hides the text box and keepstextout of the request, whatever the box heldRespond with interleaved text and audio., instead of asking to leave the box emptydocs/community_models/lfm2_audio.md: the WebUI sentence says speech-to-speech has no text boxwebui/native/dist/index.html. Main's committed page equals a build of main's sources, so nothing else comes inWhat changes for users
lfm2_audioS2S entries use liquid-audio's system prompt, as the CLI does without--text. liquid-audio's demo hard-codes that prompt, and its README says to set ittexton/v1/tasks/runor--texton the CLITesting
The browser checks below ran on main a08c7af. The branch is now on 7f8d739, after the v0.9.1 WebUI pass (#819), which changed
+page.svelteandmodels_catalog.json: this change is the same two conditions and two hints on top of it,svelte-checkreports no errors or warnings, and the page was rebuilt from these sources. The browser checks were not rerun on the new base.The WebUI has no unit tests. Headless Chrome drove the page each server embeds, main's and this branch's, with the EN and JP Q8_0 TTS and S2S entries at seed 7. It ran on a Linux x86 machine with an A10 (CUDA, and the CPU) and on an M3 Ultra (Metal, and the CPU). Each S2S run started on a TTS entry.
npm run check(svelte-check): 0 errors and 0 warnings on main and on this branchtextin the request, in all 9 S2S runs (EN and JP on CUDA, Metal and the Mac CPU; EN on the Linux CPU). Each reply has the same audio, byte for byte, and the same text as the API withtextset toRespond with interleaved text and audio.(9 of 9)/v1/audio/speechbody and get the same audio (7 of 7, EN and JP). Back on TTS after S2S, the box and its text are theremodel_spec_system_testpass on Linux (CPU and CUDA builds, withlfm2_audio_cuda_test) and on the M3 Ultra (Metal build).check_loader_catalog_syncpassesBuild and run
With
lfm2-audio-ttsandlfm2-audio-s2sinserver.json, open the TTS entry, then the S2S entry under Voice conversion: no text box.