Out-of-band speech: how can the resident daemon speak with no user turn? #3

Activity

1. Pierre-Mike

   added

   wayfinder:research

   on Aug 17, 2026
2. Pierre-Mike

   added a parent issue on Aug 17, 2026

   • Voice-driven Orca orchestrator: architecture #1
3. Pierre-Mike

   marked this as blocking on Aug 17, 2026

   • When the orchestrator is allowed to interrupt me #6
4. Pierre-Mike

   self-assigned this

   on Aug 17, 2026
5. Pierre-Mike commented on Aug 17, 2026

   

   Pierre-Mike

   on Aug 17, 2026

   OwnerAuthor

   More actions

   Decision: Out-of-band speech is reachable today with one frame class and one call, and the startup greeting already proves the mechanism. The constraint is not the frame, it is the toggle: with the session stopped there is no pipeline to queue into. A speak-only path that avoids the mic is structurally possible without restructuring VoiceSession, but it has to build a second output-only pipeline with its own TTS instance. Nothing about injected speech reaches the Claude Code subprocess's history, which is the opposite of contamination and a problem in its own right.

   Citations are relative to two roots: the project (~/Github/pipecat-voice-agent) and the vendored library (~/Github/pipecat-voice-agent/.venv/lib/python3.12/site-packages/pipecat, pipecat-ai 1.7.0). Library paths are prefixed pipecat/.

   ---

   1. The injection path

   Frame class: pipecat.frames.frames.TTSSpeakFrame (pipecat/frames/frames.py:790). Two fields: text: str and append_to_context: bool = True (frames.py:803-804).

   Call: PipelineWorker.queue_frame(frame) (pipecat/pipeline/worker.py:717-732). A downstream frame goes into self._push_queue (worker.py:730), which the push task drains and feeds to the head of the pipeline (worker.py:1104).

   This is not speculative. session.py:155 already speaks the startup greeting with exactly this call:

   await self._worker.queue_frame(TTSSpeakFrame(greeting))

   No transcription precedes the greeting, so out-of-band speech is a working feature of this build rather than a new capability.

   The route it takes. TTSSpeakFrame is a DataFrame, so it enters at transport.input() and traverses the whole chain rather than being handed straight to the TTS. Every stage forwards it:

   • VADProcessor pushes every frame through (pipecat/processors/audio/vad_processor.py:115)
   • neither STTService nor SegmentedSTTService has a TTSSpeakFrame branch
   • LanguageWatcher forwards untouched (voice_agent/processors.py:44)
   • the user aggregator has no TTSSpeakFrame branch either, so it falls through
   • ClaudeAgentLLMService.process_frame reaches its else-branch and forwards (voice_agent/services/claude_agent_llm.py:208-209)
   • TTSService.process_frame catches it at pipecat/services/tts_service.py:802, wraps it as AggregatedTextFrame(text, AggregationType.SENTENCE) and hands it to _push_tts_frames (tts_service.py:817-821), which routes to run_tts and out through transport.output()

   queue_frames (worker.py:734-753) is a loop over queue_frame, so it offers nothing extra here.

   What a POST /say would have to change. The daemon already holds app["session"] (daemon.py:144), so the handler is small. The blockers are both in VoiceSession: _worker is private, and it is None whenever the session is stopped (session.py:78-80, reset at session.py:193-196). So the change is a public VoiceSession.say(text) that checks self.running first, plus a route registered alongside the existing four (daemon.py:146-150). Both belong under _lock (daemon.py:40) if a stop can race a say.

   Library alternative, not wired here. Pipecat 1.7 ships a first-class out-of-band speak message: BusTTSSpeakMessage (pipecat/bus/messages.py:106-120), which PipelineWorker.on_bus_message converts into the same TTSSpeakFrame (worker.py:799-803). Nothing in this repo constructs a bus, so it adds a dependency and lands on the identical frame.

   ---

   2. Mic ownership

   GET /status right now returns {"running": false, ...}, so the stopped case is the live one, not the hypothetical.

   Speaking through the session pipeline reopens the mic. LocalAudioInputTransport.start() opens the PyAudio input stream unconditionally (pipecat/transports/local/audio.py:82-91). It never consults audio_in_enabled. That flag gates only the pipeline's audio task (pipecat/transports/base_input.py:257) and the audio push (base_input.py:201), so a pipeline that includes transport.input() with audio_in_enabled=False still opens the mic and still lights the indicator.

   A speak-only path can avoid it, and the current structure permits it. LocalAudioTransport.input() and .output() are lazily constructed and fully independent (local/audio.py:215-233). A pipeline that never calls .input() never constructs a LocalAudioInputTransport, so no input stream opens. LocalAudioOutputTransport.start() opens only the output stream (local/audio.py:155-162).

   What such a path has to construct:

   • its own LocalAudioTransport(LocalAudioTransportParams(audio_out_enabled=True, audio_out_sample_rate=24000)), calling only .output()
   • its own PocketTTSHttpService. Instances cannot be shared across pipelines: Pipeline._link_processors mutates _prev/_next through prev.link(curr) (pipecat/pipeline/pipeline.py:207-212), so reusing the session's TTS would rewire it and break the session. A second instance is cheap because it is a thin client over the daemon's shared aiohttp session (daemon.py:130).
   • a Pipeline([tts, transport.output()]), a PipelineWorker, and a WorkerRunner. The worker is not optional: BaseOutputTransport learns its sample rate and opens its stream from the StartFrame a running worker emits (worker.py:1092).

   So no restructuring of VoiceSession is needed. It is a sibling object, roughly the shape of VoiceSession minus the input half.

   One thing I cannot settle from source. LocalAudioTransport.__init__ constructs pyaudio.PyAudio() eagerly (local/audio.py:206), which calls Pa_Initialize and enumerates CoreAudio devices. Whether that alone touches the macOS recording indicator with no input stream open needs a live check. What the code establishes is that no input stream is opened; the indicator behaviour of PortAudio initialisation is an empirical question.

   Cost shape. A resident speak-only pipeline holds the PyAudio output stream for the daemon's life. Building it per utterance instead pays a pipeline build plus a StartFrame round trip before each one, which is the same order as the 0.09s to 0.2s toggle-on figure in RESEARCH.md.

   ---

   3. Conversation-history contamination

   The Claude Code history is not corrupted. It is not informed at all.

   Assistant text enters the SDK session only through the CLI subprocess's own turns, and those are triggered by exactly one call: client.query(prompt) at claude_agent_llm.py:256. That call is reached only from _run_turn, which is reached only from an LLMContextFrame (claude_agent_llm.py:200-201). A TTSSpeakFrame produces no LLMContextFrame, so it cannot write to SDK history.

   The Pipecat-side context does record it, and that record is inert. With append_to_context=True (the default, and what the greeting uses), TTSService sets push_assistant_aggregation=True when no LLM response is in flight (tts_service.py:815), emits LLMAssistantPushAggregationFrame when the TTSStoppedFrame goes by (tts_service.py:886-897), and the assistant aggregator commits the spoken text as an assistant message (pipecat/processors/aggregators/llm_response_universal.py:1920-1931). But ClaudeAgentLLMService reads that context only through _latest_user_text, which scans in reverse for role == "user" and ignores every other role (claude_agent_llm.py:291-304). The assistant entry never reaches a prompt.

   Consequence. No double-speak, and no false belief that it said something. The failure is the mirror image: the agent has no memory of the injected utterance. On the next user turn it can repeat the same information, or answer "what did you just say?" with nothing. For an orchestrator interrupting its human, that gap is the substantive constraint, because the human will reply to something the agent has no record of saying.

   Passing append_to_context=False keeps the Pipecat context clean and changes nothing on the Claude side either way.

   The obvious fix carries a real hazard. Implementing POST /say by calling app["claude"].query(...) so the utterance lands in SDK history would race the voice turn. ClaudeSDKClient.receive_messages iterates one shared anyio receive stream (claude_agent_sdk/client.py:282, claude_agent_sdk/_internal/query.py:951-960) with no lock and no fan-out. Two concurrent consumers split the stream between them, so a query issued from an aiohttp handler while _stream_reply is iterating receive_response() (claude_agent_llm.py:258) steals messages from the live turn and breaks both. Any design that writes to SDK history has to serialise against ClaudeAgentLLMService._generating (claude_agent_llm.py:163) or against a daemon-level lock.

   ---

   4. Interruption interaction

   Two outcomes, and the boundary is the user's turn start, not the whole span of their speech.

   Interruption fires once, at turn start. The user aggregator owns a UserTurnController and calls broadcast_interruption() from _on_user_turn_started when enable_interruptions is set (llm_response_universal.py:1207-1208), which defaults to True (pipecat/turns/user_start/base_user_turn_start_strategy.py:56). The default start strategies are [VADUserTurnStartStrategy(), TranscriptionUserTurnStartStrategy()] (pipecat/turns/user_turn_strategies.py:39), and session.py:127 takes the defaults by passing no params. VADProcessor itself only emits VAD frames and never interrupts (vad_processor.py:69-87).

   Case A: injected during the interruption. The frame is dropped, silently. InterruptionFrame is a SystemFrame (frames.py:1019), so it is processed on the input task immediately rather than queued (pipecat/processors/frame_processor.py:1040-1041) and it overtakes any queued DataFrame. _start_interruption cancels and recreates each processor's process task, and __reset_process_queue drops every non-UninterruptibleFrame item (frame_processor.py:869-889, 982-984). TTSSpeakFrame is a plain DataFrame with no UninterruptibleFrame mixin (frames.py:790), so a queued injected utterance goes with the rest. Audio already in flight goes too: TTSService._handle_interruption clears the sequencer and the serialisation queue and kills the audio contexts (tts_service.py:1005-1017), and the output transport drains its audio queue (pipecat/transports/base_output.py:551-568). No error, no retry, no notification to the caller.

   Case B: injected after turn start, before the reply. It talks over them. No second interruption fires for continued speech within the same turn, and nothing gates TTS on user-speaking state. The utterance synthesises and plays while the user is still talking.

   Second-order effect worth recording. There is no echo cancellation. TransportParams.audio_in_filter defaults to None (pipecat/transports/base_transport.py:77) and session.py:91-97 does not set one, and no STT branch suppresses transcription while the bot speaks. Injected audio reaches the open mic and can be transcribed as user speech. The startup greeting is the existing precedent for this and evidently survives it, so this is a mechanism to verify rather than a confirmed bug.

   If unprompted speech should win, there is a clean lever. Adding the UninterruptibleFrame mixin (frames.py:143-153) to an injected frame subclass makes both the queue reset and the output drain preserve it (pipecat/utils/frame_queue.py:83-88, base_output.py:561-568). That is a code change, not a call-site option, and it inverts barge-in for that frame, which is a product decision rather than a technical one.

   ---

   5. Language selection

   An injected utterance inherits the conversation's language, and the detector never sees it.

   PocketTTSHttpService._language_for reads self._conversation_language.current and falls back to detect_language(text) only when that is None (pocket_tts_http.py:130-133). Both the voice and the server key off that one decision (pocket_tts_http.py:135-143).

   LanguageWatcher observes TranscriptionFrame only (voice_agent/processors.py:38), and only the STT produces those. Injected text carries no vote, so it cannot move the conversation's language even when written in the other one.

   Three outcomes:

   • AGENT_LANGUAGE=fr or en: ConversationLanguage._current is pinned at construction (lang.py:182-183) and observe is inert (lang.py:203-204). The injected text gets the pinned voice and server.
   • auto, after the user has spoken clearly: current holds fr or en, and the injected text gets that voice regardless of what language it is actually in. French injected into an English conversation is read by alba on the English model at :8791, which is precisely the accent problem the per-language routing exists to prevent (config.py:26-33).
   • auto, before the user has said anything clear: current is None, so detect_language(text) decides from the injected text itself and defaults to en (lang.py:96-113). It needs two French signals, or one French marker in three words or fewer, to return fr. Short injected French such as "Trois heures." gets the English voice, which is the exact failure ConversationLanguage was built to fix for reply fragments.

   ConversationLanguage is constructed per session (session.py:106), so every toggle resets it to the forced value or None. An utterance injected right after a start behaves like the greeting.

   What would have to change for injected text to sound right in a mixed setting: either pass an explicit language alongside the text and give _language_for an override, or call language.observe(text) before queueing. The second is cheaper and wrong, because it lets text the user never spoke move the voice for the rest of the conversation.

   ---

   Feasibility summary

   Question	Supported today	Would have to change
   Injection path	TTSSpeakFrame + PipelineWorker.queue_frame, proven by the greeting	public VoiceSession.say(), a /say route, running guard
   Toggle OFF	nothing to queue into; _worker is None	either start a session (opens mic) or build a second output-only pipeline with its own TTS instance
   Claude history	untouched by injection, and unaware of it	any write path must serialise against the single SDK receive stream
   Mid-sentence	dropped at turn start, talks over the user after it	UninterruptibleFrame mixin to survive; a user-speaking gate to defer
   Language	inherits the conversation, or defaults to en	an explicit per-utterance language override

   Scope note

   Read-only investigation as scoped. Nothing implemented, no file in ~/Github/pipecat-voice-agent edited, no POST to :8792, no daemon or model server touched, no -m live tests run. The single live call was GET /status.
6. Pierre-Mike

   closed this as completedon Aug 17, 2026
7. Pierre-Mike

   added 5 commits that reference this issue on Sep 28, 2026

   perf(runtime): stop the expired-SSH-lease sweep from rescanning every…

   ...

   This issue will close once commit 6c4797c is merged into the 'main' branch.

   6c4797c

   fix(crash-reporting): sample system memory before the process is gone (…

   ...

   This issue will close once commit cc9e9ed is merged into the 'main' branch.

   cc9e9ed

   fix(agents): detect agent CLIs installed outside a version manager (s…

   ...

   This issue will close once commit 901c677 is merged into the 'main' branch.

   901c677

   perf(mobile): keep four bundle chunk reads in flight across the whole…

   ...

   This issue will close once commit 0c2514e is merged into the 'main' branch.

   0c2514e

   feat(mobile-web-bundle): gzipped 384 KiB ranges over a capability-neg…

   ...

   This issue will close once commit 996f9cc is merged into the 'main' branch.

   996f9cc