Add realtime (speech-to-speech) client support and a test playground - #158
Merged
Conversation
Wire Microsoft.Extensions.AI's IRealtimeClient into the AI client factory/provider stack, mirroring the existing IChatClient plumbing, and add a speech-to-speech "Realtime model test" playground to both sample hosts. Core: - IAIClientFactory.CreateRealtimeClientAsync and IAIClientProvider .GetRealtimeClientAsync (base throws NotSupported; OpenAI implemented via the packaged OpenAIRealtimeClient). - AIDeploymentPurpose.Realtime, AIProfileType.RealtimeChat, and AIProfile.RealtimeDeploymentName. Azure transport: - A custom Azure OpenAI GA realtime WebSocket transport under CrestApps.Core.AI.OpenAI.Azure/Realtime. The pinned Azure.AI.OpenAI is version-incompatible with the OpenAI SDK that Microsoft.Extensions.AI.OpenAI requires (AzureOpenAIClient.GetRealtimeClient throws MissingMethodException), so this connects directly to /openai/v1/realtime (GA session schema, api-key or Entra ID auth). Delete this folder once a compatible Azure.AI.OpenAI ships. Sample hosts (MVC + Blazor): - A shared server-side WebSocket bridge (RealtimeVoiceBridge in Startup.Shared) and a shared browser script (realtime-test.js in the CrestApps.AI.Resources RCL) captured/played as PCM16 @ 24 kHz. - "Realtime model test" page under a new Playground nav section in both hosts. Tests cover the factory dispatch, each provider's realtime capability, and the Azure realtime protocol mapping and session send/receive loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ealtime voice list - Merge origin/main (brings the WebCrawlers feature and new tests). - Implement IAIClientFactory.CreateRealtimeClientAsync on DataSourcePreemptiveRagHandlerTests.FakeAIClientFactory (added on main) so the test project compiles against the new interface member. - Factor the realtime voice <option> list into a single shared RealtimeVoices source (adds the newer gpt-realtime voices cedar and marin), used by both the MVC and Blazor test pages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add IAIClientProvider.GetRealtimeVoicesAsync (SpeechVoice[], mirroring GetSpeechVoicesAsync) and IRealtimeVoiceResolver/DefaultRealtimeVoiceResolver (mirroring ISpeechVoiceResolver, registered by default). OpenAI and Azure OpenAI return the fixed gpt-realtime voice set from OpenAIRealtimeVoices (adds cedar and marin); other providers return empty. Each voice carries a best-effort gender. - The MVC and Blazor test pages now resolve voices through the provider and group the selector by gender; the previously hardcoded shared voice list is removed. - Document the realtime client, IRealtimeVoiceResolver, RealtimeChat profile type, and Realtime deployment purpose in the 2.0.0 release notes and the canonical provider docs (providers/index, openai, azure-openai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CachingSource_WhenBuildExceedsBudget_ReportsPendingThenServesFromBackgroundBuild raced a 300ms build delay against a 20ms wait budget, so on a CPU-starved CI runner the build could finish before the first search checked and the expected DocumentationIndexPendingException was never thrown (intermittent CI failures). SlowCorpusSource now supports gating the build on a TaskCompletionSource the test controls, so the build cannot complete within the budget regardless of scheduling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds realtime (speech-to-speech) client support to the AI infrastructure and a test playground in both sample hosts, mirroring the existing
IChatClientplumbing.Core plumbing
IAIClientFactory.CreateRealtimeClientAsync(AIDeployment)andIAIClientProvider.GetRealtimeClientAsync(...).NotSupportedException(likeGetTextToSpeechClientAsync); providers without a realtime API (Ollama, Azure AI Inference, Azure Speech) inherit that.OpenAIRealtimeClient(fromMicrosoft.Extensions.AI.OpenAI).AIDeploymentPurpose.Realtime,AIProfileType.RealtimeChat, andAIProfile.RealtimeDeploymentName.Azure OpenAI GA realtime transport
The pinned
Azure.AI.OpenAItargets an olderOpenAISDK thanMicrosoft.Extensions.AI.OpenAIrequires, soAzureOpenAIClient.GetRealtimeClient()throwsMissingMethodExceptionat runtime. Until a compatibleAzure.AI.OpenAIships,CrestApps.Core.AI.OpenAI.Azure/Realtime/implements the MEAI realtime interfaces directly over aClientWebSocket:wss://<resource>.openai.azure.com/openai/v1/realtime?model=<deployment>(noapi-version).session.updateschema (type: "realtime", nestedaudio.input/audio.output,{type,rate}formats).Realtime/folder; remove it and route back through the SDK once upstream is fixed.Sample-host playground (MVC + Blazor)
RealtimeVoiceBridgeinStartup.Shared) and a shared browser script (realtime-test.jsin theCrestApps.AI.ResourcesRCL) — PCM16 @ 24 kHz capture/playback, server-VAD turn-taking, barge-in, live transcripts (assistant + user).How to try it
gpt-realtime) on an Azure OpenAI resource.Tests
2783 passing. New coverage: factory dispatch + NotSupported propagation, each provider's realtime capability, and the Azure realtime protocol mapping + session send/receive loop.Scope / follow-up
This delivers the realtime client, the Azure transport, and the test harness. It intentionally does not wire realtime into the orchestrator yet — tool calling, RAG/knowledge, and AI-Profile-driven behavior for
RealtimeChatprofiles are a separate, planned piece (a realtime execution path that reuses the orchestrator's preparation building blocks). The realtime session currently applies the system prompt only.🤖 Generated with Claude Code