Repository navigation
Add real-time voice chat with Deepgram Flux - #8
Open
dg-edcharbeneau wants to merge 6 commits into
Open
dg-edcharbeneau wants to merge 6 commits into
dg-edcharbeneau wants to merge 6 commits into
Conversation
Add a /voice-chat scenario that layers Deepgram voice around the unchanged AG-UI pipeline: Flux speech-to-text (/v2/listen) in, Flux TTS (/v2/speak) out, with barge-in. The browser connects to Deepgram directly using short-lived JWTs minted by a new /api/deepgram/token endpoint, so the API key stays on the server and no audio crosses the Blazor circuit. The voice UI follows the Deepgram voice UI best practices: an explicit state machine shown as an always-visible status, a mic meter driven in JS, AudioWorklet capture, gap-free playback, one TTS socket per session kept alive with a no-op Configure, preflight checks, friendly errors, and full teardown on stop. On barge-in the agent run is cancelled and SpeechTapChatClient rewrites the agent's history to what the user actually heard (SpeechInterrupted.text_spoken). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Give the real-time voice chat its own /voice_chat agent instead of sharing /agentic_chat. Its instructions are written for speech: short plain sentences with no Markdown, adapting to the user's tone as read from the transcript, and treating "[interrupted by the user]" as the point where the user stopped listening. Cut response latency for voice. With gpt-5-mini, silent reasoning took 2.3 to 4 seconds before the first token. The voice agent now requests minimal reasoning effort, which brought the first token to about 1.3 seconds. Microsoft.Extensions.AI has no minimal level, so it is set through ChatOptions.RawRepresentationFactory, and only for reasoning deployments (Foundry.SupportsReasoningEffort), because models such as gpt-4o-mini reject reasoning_effort with HTTP 400. Also: - Set Flux TTS expressivity to 1 for a slightly more animated voice. - Rename the scenario to "Real-time voice chat" in the UI and docs. - Add a Real-time voice chat card to the Home page. - Update the README and findings for the new endpoint and latency fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Remove the UserSecretsId and byte order mark that were committed to AgenticUI.AgentServer.csproj by mistake. The project file now matches main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add real-time voice chat with Deepgram Flux
Add docs/images/deepgram-voice-architecture.svg, in the style of the existing architecture diagram, showing how the browser, Deepgram Flux STT and TTS, the Blazor web app, the AG-UI agent server, and Microsoft Foundry connect. Reference it from a new "Real-time voice" section of the README's architecture overview. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Real-time voice chat scenario (
/voice-chat). You talk to an agent and it talks back: Deepgram Flux handles speech-to-text and Flux TTS handles text-to-speech. The scenario wraps the existing AG-UI pipeline without changing it. The only server-side addition is a dedicated/voice_chatagent./v2/listen,flux-general-en) decides when you've finished speaking. The transcript is then sent withAgentContext.SendMessageAsync.SpeechTapChatClientis aDelegatingChatClientthat sits insideFormattedChatClient. It splits the streamed reply into sentences, strips Markdown and skips code blocks. Each reply is spoken as one Flux TTS turn (/v2/speak,flux-haley-en, expressivity 1)./api/deepgram/tokenendpoint, passed as thebearersubprotocol. The API key stays on the server, and no audio goes through the Blazor circuit.Parameters:deepgram-api-keyAppHost secret or aDEEPGRAM_API_KEYenvironment variable. Without a key, the app runs normally and the page says how to enable voice.The voice agent
/voice_chatis a separate agent, so the Agentic chat prompt is untouched.[interrupted by the user]as the point where the user stopped listeninggpt-5-mini, the model reasoned silently for 2.3 to 4 s before its first token. The voice agent requestsminimalreasoning effort, which brought the first token to about 1.3 s; Agentic chat takes 3.5 to 4.1 s for the same question.ReasoningEfforthas no minimal level, so it's set withChatReasoningEffortLevel.MinimalthroughChatOptions.RawRepresentationFactory.Foundry.SupportsReasoningEffort), becausegpt-4o-minirejectsreasoning_effortwith HTTP 400.Voice UI best practices
This follows dg-edcharbeneau/voice-best-practices: