feat: low-latency voice pipeline — wall-clock pacing, Krisp denoiser, NAMO EOT, word-accurate barge-in + re-delivery#208
Merged
Conversation
…oundation) Adds the isolated, dual-SDK building block for the low-latency outbound path, ahead of wiring it into the stream handler: - Precomputed silence frames (mu-law 0xFF x160 / PCM16 zero) so an empty queue never stalls the clock and the first tick pays no encode cost (no cold start). - Fixed 20 ms / 160-byte (mu-law) re-framer of arbitrary enqueued audio, with a next_frame() 'NextSample' contract that pads a sub-frame remainder and returns silence when drained. - OutboundFramePacer: a drift-corrected wall-clock loop (monotonic deadline that advances one frame period per tick, not by work duration) that emits silence on an empty queue and bounds catch-up after an oversleep — avoiding the post-pause bursts that sank prior naive sleep(20ms) pacing attempts. getpatter/audio/pacer.py + src/audio/pacer.ts, 9 unit tests each. Not yet wired into the send path (follow-up); default behaviour unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
String-keyed denoiser selection (Agent.denoiser / AgentOptions.denoiser) mapping the real Krisp model ids krisp-viva-tel-v2 (NC/VIVA telephony) and krisp-bvc-o-pro-v3 (Background Voice Cancellation) to a resolved AudioFilter via a registry. NC vs BVC session-type handled (NcInt16 vs BvcInt16). Bring-your-own- license: ships zero Krisp binaries/models — resolves against the operator's own krisp_audio SDK + KRISP_VIVA_SDK_LICENSE_KEY + KRISP_MODELS_DIR, with clear fail-fast errors when SDK/license/model is missing. Reuses the existing audio_filter chain slot (explicit audio_filter wins). Default None = unchanged. TS mirrors the API surface + throws the same clear 'no Node Krisp SDK' guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
Adds NamoTurnDetector (NAMO Turn Detector v1, Apache-2.0, ONNX) as a clean-weights semantic end-of-turn detector consuming the rolling conversation transcript. Widens TurnDetectorProvider.predict backward-compatibly with a keyword-only transcript param (predict(pcm16_window, *, transcript=None) / predict(window, transcript?)) so the audio-native SmartTurnDetector keeps working byte-identically. The stream handler assembles the last ~4 turns + in-flight utterance and passes it on every predict call in the existing semantic hold/poll/backstop machinery. onnxruntime + HF tokenizer are optional/lazy with clear install errors; weights loaded at runtime (PATTER_NAMO_MODEL), never bundled. Opt-in via agent.turn_detector; default behaviour unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
Wire the (already-built, already-tested) OutboundFramePacer into the outbound carrier audio path as an opt-in mode. Default OFF is byte-identical to today's event-driven send — the pacer is never created and every send takes the current direct path. Opt-in flag: Agent.paced_output (Python) / AgentOptions.pacedOutput (TS), default false, threaded via the Agent object already handed to the handler. When ON (pipeline mode): - One OutboundFramePacer per call. Frame size / silence codec chosen from the sender's output format: native mu-law 8 kHz -> 160 B + mulaw silence, else PCM16 16 kHz -> 640 B + pcm16 silence. send_frame forwards to the existing audio_sender.send_audio / encodePipelineAudio+bridge.sendAudio so per-carrier transcode/envelope is unchanged. - Lifecycle: started as a tracked background task in start()/initPipeline once the output codec is settled (before the greeting, which also flows through the pacer); stopped + awaited in cleanup()/handleStop()/handleWsClose(). - Every pipeline send-site (synthesize_sentence, drain_sentence_entry, spec_send_chunk, first-message streaming + prewarm) enqueues instead of direct-sending. Realtime engines are unaffected (they arrive realtime-paced from the provider). - Barge-in / pause: added OutboundFramePacer.clear() and call it on every send_clear cancel path so the queued (cancelled) audio is dropped in the same beat and cancel latency stays low. - Playback tracking driven from the pacer's REAL emitted (non-silence) frames x 20 ms so the heard-prefix estimate stays exact under pacing; the byte-estimate path is retained unchanged for the unpaced default. TDD: mirrored Python/TS unit tests for clear(), fixed-frame 20 ms grid, silence gap-fill, barge-in drop, byte-identical default-OFF direct send, and emitted-frame playback tracking. Full suites green (Python 2957 passed, TS 2364 passed, tsc + ruff clean). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GJVkGyc6zocoh4Shiw6dBe
Truncate the interrupted turn's history to what the caller ACTUALLY heard — including mid-sentence — instead of rounding up whole sentences from a byte estimate. - Segment record extended to (text, start_s, dur_s): each RESPONSE sentence carries its own playout duration, finalized when the next sentence stamps; the last/in-flight sentence resolves its extent from the live playback total. Stamping centralized in one helper. - heard_response_split includes fully-heard sentences verbatim plus the heard fraction of the sentence in flight, split on a word boundary (proportional-by-word, rounded half up). heard_response_prefix keeps its stable (heard, everything) shape; the richer split exposes the unsaid remainder for the coming re-delivery feature. - Played-position ladder, highest tier first: (i) Twilio carrier- confirmed seg_<n> marks — emitted per sentence, advanced in on_mark without disturbing fm_* first-message flow control; (ii) paced emitted-frame cursor; (iii) byte estimate. Telnyx/Plivo have no reliable marks and fall to (ii)/(iii). Strict Python/TypeScript parity. No version bump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GJVkGyc6zocoh4Shiw6dBe
When the agent is interrupted mid-answer and the un-heard remainder still matters, inject a one-shot system nudge on the next turn so the LLM re-addresses the caller's new message first and then resumes the unfinished point naturally — instead of silently dropping what the caller never heard. - New Agent.redeliver_interrupted / AgentOptions.redeliverInterrupted (default false ⇒ zero behaviour change) plus an optional redelivery_policy / redeliveryPolicy hook. - services/redelivery: RedeliveryPolicy protocol (mirrors BargeInStrategy), default MinUnsaidWordsPolicy (gates on heard_everything=false AND >= DEFAULT_MIN_UNSAID_WORDS un-heard words), and build_redelivery_nudge. - Capture (heard, unsaid) on a confirmed barge-in right after the completed- turn truncation, before the playback cursor is reset; store on the handler. - Inject on the next turn via LLMLoop.set_redelivery_nudge / setRedeliveryNudge as a SECOND system message in _build_messages (same pattern as the rolling compaction summary); consumed on first build so it is strictly one-shot. - Pipeline-mode only; realtime/ConvAI have no heard/unsaid split and no-op. Strict Python/TypeScript parity. Tests: +18 Python, +17 TypeScript. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GJVkGyc6zocoh4Shiw6dBe
…r, EOT, barge-in) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Bandit B615 flagged AutoTokenizer.from_pretrained() in the NAMO turn detector for downloading without an immutable revision pin. Thread an optional revision through the loader (PATTER_NAMO_REVISION env var or revision= argument on load()) in both SDKs so a Hub repo id can be pinned to a commit/tag. The documented default source is a local, operator-provided model directory, not an unpinned Hub download, so a scoped nosec covers that residual case where revision is legitimately None.
Resolves the release-0.7.1 (#202) integration: keeps #208's Unreleased entries above the new 0.7.1 CHANGELOG section, adopts 0.7.1 across all version files (no additional bump), and regenerates package-lock.json from the merged package.json (Gemini deps + NAMO onnxruntime/transformers). Full suites green post-merge: Python 3066, TypeScript 2476, tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
nicolotognoni
added a commit
that referenced
this pull request
Jul 2, 2026
Integrates releases #202 (0.7.1) and #208 (low-latency pipeline) now on main. Keeps this PR's transfer-allowlist Unreleased entry alongside #208's entries under the single Unreleased section, adopts 0.7.1 across version files (no extra bump), and regenerates package-lock.json from the merged package.json. Full suites green post-merge: Python 3083, TypeScript 2493, tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
3 tasks
nicolotognoni
added a commit
that referenced
this pull request
Jul 3, 2026
Version files already bumped to 0.7.1 by #202; #206 (transfer allowlist) and #208 (low-latency voice pipeline) then merged onto main at 0.7.1 but their entries sat under Unreleased. Fold them into the 0.7.1 section so the published release notes match the code that ships. No version change (already 0.7.1); no code change. Merge this, then tag v0.7.1 to publish to npm + PyPI. Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five low-latency / turn-taking voice-pipeline improvements, all opt-in with byte-identical defaults so nothing changes for existing agents until enabled. Dual-SDK (Python + TypeScript) at strict parity.
paced_output/pacedOutput) — paces carrier audio onto a fixed 20 ms / 160-byte (μ-law) grid with a precomputed silence frame (no cold-start latency) and a drift-corrected monotonic-deadline loop (no post-pause bursts). Mirrors the standard telephony framing; the deliberate drift-correction avoids the pitfalls documented in the prior failed pacing attempts.denoiser="krisp-viva-tel-v2" | "krisp-bvc-o-pro-v3") — real Krisp model ids resolved to the existing pre-STTAudioFilterslot, NC vs BVC session handled. Ships zero Krisp binaries/models; loads the operator's ownkrisp_audioSDK + license, fail-fast with clear errors otherwise.NamoTurnDetector(NAMO Turn Detector v1, Apache-2.0, ONNX) over the rolling transcript.TurnDetectorProvider.predictgained a backward-compatible keyword-onlytranscriptarg; the audio-native smart-turn detector is untouched.redeliver_interrupted/redeliverInterrupted) — when the un-heard remainder still matters, a one-shot system nudge tells the LLM to answer the caller's new message first, then resume the unfinished idea only if still relevant. PluggableRedeliveryPolicy. Pipeline-only; no-op in realtime.Note: streaming input (
input_audio_buffer.appendper frame, not buffered to end-of-utterance) was already implemented — verified, no change needed.Implementation
audio/pacer.py/.ts(OutboundFramePacer+ precomputed silence),providers/denoiser.py/.ts(Krisp registry),providers/namo_turn_detector.py/.ts,services/redelivery.py/.ts.stream_handler.py/.ts: pacer lifecycle (start/stop/clear-on-barge-in), send-site → enqueue routing when paced, pacer-driven playback tracking, per-sentenceseg_<n>marks + mark-confirmed heard cursor, heard/unsaid split, re-delivery capture+one-shot inject.services/llm_loop.py/.ts: extra system-message injection for the re-delivery nudge (same pattern as the compaction summary).models.pyandtypes.ts; exports in__init__.py/index.ts.Feasibility / licensing notes (why these shapes)
.kefmodels are non-redistributable; LiveKit's Krisp path requires LiveKit Cloud. An OSS SDK cannot ship Krisp — hence bring-your-own-license pass-through, not bundled models.Breaking change?
No. Every feature is opt-in and defaults off; unpaced/no-denoiser/no-EOT/no-redelivery paths are byte-identical to
main.predict()gained a defaulted keyword-only arg (source- and behavior-compatible).Test plan
pytest tests/— 2980 passed, 8 skipped, 2 xfailed. New: pacer, paced-output, denoiser, NAMO, redelivery + extended barge-in/heard-prefix suites.npm test— 2385 passed, 9 skipped (151 files);npm run lint(tsc --noEmit) clean. Mirrored new suites.Docs updates
CHANGELOG.mdUnreleased (Added/Changed). No version bump (deferred to a separate release PR). Follow-up: docs pages + feature-inventory rows for the new opt-in knobs.🤖 Generated with Claude Code