feat(engines): GeminiLive/GeminiCascade/InworldRealtime markers + Gemini Live fixes (release 0.7.1)#202
Merged
Merged
Conversation
…+ fix Gemini Live thinking leak & 3.1 apiVersion
Follow-up to 0.7.0, which exported the Gemini pipeline factory + adapter but NOT
the GeminiLive engine marker (so `new GeminiLive({...})` / the engine import
failed at runtime), and never shipped a native InworldRealtime engine.
Engines (new public API):
- GeminiLive engine marker (engines/gemini.ts) + the richer GeminiLiveAdapter
(mulaw8<->PCM telephony transcode, apiVersion auto-detect,
GEMINI_LIVE_3_1_FLASH_PREVIEW), GeminiCascade engine + GeminiSTT/GeminiTTS,
integrated from the gemini-3.1-live-demo work onto the 0.7.0 baseline
(3-way merge; #200 barge-in + #199 Inworld pricing/exports preserved).
- InworldRealtime engine marker (engines/inworld.ts) + InworldRealtimeAdapter
(providers/inworld-realtime.ts), subclassing OpenAIRealtimeAdapter and
overriding only the transport (wss://api.inworld.ai/v1/realtime, Bearer/JWT;
Inworld's OpenAI-Realtime migration path). Lets the demo drop its
OpenAIRealtime monkey-patch. `import { InworldRealtime } from 'getpatter'`
resolves at runtime.
Fixes (live-call bugs found on the Giulia demo against 0.7.0):
- Gemini Live spoke the model's chain-of-thought before the reply. Now the Live
setup always sends thinkingConfig (voice default thinkingBudget:0, OFF), with
opt-in `thinking`/`thinkingBudget` on GeminiLiveOptions; and the adapter
defensively drops any `thought===true` part from BOTH the audio stream and
the text transcript (never logged — no PII).
- gemini-3.1-flash-live-preview dropped the call with a silent "session ready
timeout": it is a native-audio model served only on v1alpha but its id lacks
the 'native-audio' token, so the heuristic picked v1beta. New
geminiRequiresV1Alpha() selects v1alpha for native-audio AND flash-live-preview
(explicit `apiVersion` overrides); connect failures now throw a clear,
actionable error (model id + resolved apiVersion + root causes) instead of
dead air.
- Bump @google/genai peerDependency ^0.3.0 -> >=2.0.0 (kept optional) for the
2.x unified SDK; adds a ^2.8.0 devDependency so tsc resolves the literal
import in the stt/tts/cascade modules.
Backward compatible: OpenAIRealtime / OpenAIRealtime2 / Ultravox / ConvAI
unchanged; the instanceof OpenAIRealtimeAdapter gates are extended, not replaced
(InworldRealtimeAdapter passes them via subclassing). Tests: +23 authentic
(gemini-live-thinking, inworld-realtime, …), mock only the provider WebSocket.
lint green; 2334 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
Ships the GeminiLive/GeminiCascade/InworldRealtime engine markers + the Gemini Live thinking-leak and gemini-3.1 apiVersion fixes (this PR's feature commit). All four version files move together; the create-getpatter launcher is realigned to the SDK version (it pins the getpatter version it bootstraps to its own). - libraries/python/getpatter/__init__.py, libraries/python/pyproject.toml, libraries/typescript/package.json -> 0.7.1 - create-getpatter/package.json 0.6.9 -> 0.7.1 (lockstep restored) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
… (Indian langs), both SDKs Built by parallel agent teams (research -> per-provider modules -> serialized wiring -> adversarial verify), verified against the official docs/pricing. - Soniox v5 STT (update): SonioxModel gains stt-rt-v5 and it is now the default in both SDKs (v4/v3/v2 retained). v5 is wire-compatible with v4 (same WS URL, in-band api_key, token stream), so the only change to adopt it is the model id. New opt-in endpoint_sensitivity / endpoint_latency_adjustment_level, omitted when unset so the v4 wire config stays byte-identical. STT price unchanged (/bin/zsh.12/hr = /bin/zsh.002/min, soniox.com/pricing). - Soniox TTS (new): SonioxTTS over POST https://tts-rt.soniox.com/tts (Bearer, tts-rt-v1, default voice Adrian, native pcm_mulaw/8k carrier path), sharing the SONIOX_API_KEY credential. Provider + tts wrapper + root export + providers helper + _create_tts_from_config branch + pricing (modeled per char from the /bin/zsh.70/hr headline — Soniox publishes no per-char/token rate). - Sarvam AI TTS (new): SarvamTTS over POST https://api.sarvam.ai/text-to-speech (api-subscription-key header, bulbul:v3, 11 Indian languages via target_language_code, base64 audio), for Indian-language voice agents. Full wiring + a [sarvam] pyproject extra. Pricing matches sarvam.ai/api-pricing (Rs 30 / Rs 15 per 10k chars). Strict Python<->TS parity; backward compatible (existing Soniox/Cartesia/etc. unchanged); no PII (cost telemetry records only text length, never the text or audio). Authentic tests mock only the transport. Version stays 0.7.1. Python 2952 passed, TS 2380 passed, lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
…204) Closes #204. The carrier webhook HTTP routes were signature-validated, but the media-stream WS endpoints (/ws/stream/{call_id} Twilio, /ws/telnyx/stream, /ws/plivo/stream) accepted any peer with an attacker-chosen call_id — the only control was the DoS per-IP cap. An unauthenticated peer reaching the public host could open the socket, send a start frame, and drive a full STT->LLM->TTS session on the operator's provider keys (toll fraud), or converse to extract the agent's system prompt + tool list. Fix (both SDKs, all three carriers): - The signature-validated webhook that builds the stream URL/TwiML mints a high-entropy per-call token (secrets.token_urlsafe / crypto.randomBytes), stored in a per-server short-TTL map keyed by the call, and delivers it on each carrier's existing custom channel: Twilio <Parameter> -> customParameters, Telnyx URL query, Plivo extra_headers. - The WS handler constant-time-validates the token (hmac.compare_digest / timingSafeEqual) BEFORE opening any provider session; on missing/invalid/ expired it closes WS 1008 with no provider connect, no TTS, no callback/cost. Telnyx validates at accept; Twilio/Plivo in the start-frame branch. - Fail-closed by default via require_stream_auth / requireStreamAuth (opt-out for custom-TwiML operators, which logs a warning). Token never logged. Backward compatible for the standard serve() inbound + outbound path. - Hardened the forgeable X-Forwarded-For per-IP cap with a global concurrent-WS backstop (MAX_TOTAL_WS / MAX_WS_TOTAL = 500), documented DoS-cap-only. Polish over the workflow output: outbound Twilio now keys the token on a fresh per-call id (not the static 'outbound') so concurrent outbound dials don't overwrite each other's token; aligned the Python/TS backstop ceiling. Built by parallel Python+TS agent teams with an adversarial security-verify gate (re-ran the #204 repro: an unauthenticated peer is provably blocked before any provider session). Tests: tests/security/test_stream_auth.py (30) + TS stream-auth suite. Python 2982 passed, TS green, lint clean. Version stays 0.7.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
nicolotognoni
added a commit
that referenced
this pull request
Jul 2, 2026
Resolves the release-0.7.1 (#202) integration: keeps #208's Unreleased entries above the new 0.7.1 CHANGELOG section, adopts 0.7.1 across all version files (no additional bump), and regenerates package-lock.json from the merged package.json (Gemini deps + NAMO onnxruntime/transformers). Full suites green post-merge: Python 3066, TypeScript 2476, tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
nicolotognoni
added a commit
that referenced
this pull request
Jul 2, 2026
Integrates releases #202 (0.7.1) and #208 (low-latency pipeline) now on main. Keeps this PR's transfer-allowlist Unreleased entry alongside #208's entries under the single Unreleased section, adopts 0.7.1 across version files (no extra bump), and regenerates package-lock.json from the merged package.json. Full suites green post-merge: Python 3083, TypeScript 2493, tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
3 tasks
nicolotognoni
added a commit
that referenced
this pull request
Jul 3, 2026
Version files already bumped to 0.7.1 by #202; #206 (transfer allowlist) and #208 (low-latency voice pipeline) then merged onto main at 0.7.1 but their entries sat under Unreleased. Fold them into the 0.7.1 section so the published release notes match the code that ships. No version change (already 0.7.1); no code change. Merge this, then tag v0.7.1 to publish to npm + PyPI. Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.7.1 — Gemini/Inworld engine markers + Gemini Live fixes
Follow-up to 0.7.0. 0.7.0 shipped the pipeline barge-in fix (#200) but did NOT ship the
GeminiLiveengine marker (sonew GeminiLive({...})/ the engine import threw at runtime — the demo had to vendor an old.tgz), and a live-call Gemini "thinking" leak surfaced. This lands the markers + fixes and prepares 0.7.1.npm publishis intentionally NOT run — it's human-gated (publish happens via thev0.7.1tag after merge).Engines (new public API)
GeminiLiveengine marker + richerGeminiLiveAdapter(mulaw8↔PCM telephony transcode, apiVersion auto-detect,GEMINI_LIVE_3_1_FLASH_PREVIEW),GeminiCascadeengine,GeminiSTT/GeminiTTS— integrated from thegemini-3.1-live-demowork onto the 0.7.0 baseline (3-way merge; fix(pipeline): stop false barge-in from self-interrupting the user-turn LLM (agent goes mute) #200 barge-in + feat(inworld): complete TTS pricing + config helper + Realtime Router LLM preset #199 Inworld pricing/exports preserved, verified).InworldRealtime— native engine marker +InworldRealtimeAdapterthat subclassesOpenAIRealtimeAdapterand overrides only the transport (wss://api.inworld.ai/v1/realtime, Bearer/JWT — Inworld's OpenAI-Realtime migration path). Lets the demo drop itsOpenAIRealtimemonkey-patch..tgz:require('getpatter')and ESM named imports both resolveGeminiLive,GeminiCascade,InworldRealtimeas functions (the exact 0.7.0 bug, now fixed).Fixes (live-call bugs on the Giulia demo vs 0.7.0)
thinkingConfig(voice defaultthinkingBudget: 0, OFF; opt-inthinking/thinkingBudgetonGeminiLiveOptions), and the adapter defensively drops anythought===truepart from both the audio stream and the text transcript (never logged — no PII).gemini-3.1-flash-live-preview→ silent "session ready timeout" (call dropped with dead air). It's a native-audio model served only onv1alphabut its id lacks thenative-audiotoken, so the heuristic chosev1beta. NewgeminiRequiresV1Alpha()selectsv1alphafor native-audio ANDflash-live-preview(explicitapiVersionoverrides); connect failures now throw a clear, actionable error (model id + resolved apiVersion + root causes) instead of dead air.Changed
@google/genaipeerDep^0.3.0→>=2.0.0(kept optional) for the 2.x unified SDK —npm install getpatterwith@google/genai@^2.xno longer needs--legacy-peer-deps.create-getpatterlauncher realigned to the SDK version (lockstep) — it pins thegetpatterversion it bootstraps to its own version. It had drifted to 0.6.9 (sonpm create getpatterwas provisioning 0.6.9); bumped to 0.7.1.Backward compatibility
OpenAIRealtime / OpenAIRealtime2 / Ultravox / ConvAI unchanged. The
instanceof OpenAIRealtimeAdaptergates are extended, not replaced (InworldRealtimeAdapterpasses them via subclassing; the Geminifunction_calldispatch gate is widened). #200 barge-in fix confirmed intact.Versioning
All four version files move to 0.7.1 together (Python
__init__.py+pyproject.toml, TSpackage.json, launchercreate-getpatter/package.json). Per the "0.7.x" ask this is tagged a patch; note it does add new public API (engine markers), which is strictly a minor — call it 0.8.0 instead if you prefer stricter semver.Test plan
npm run lint(tsc --noEmit) green;npm testgreen — 2334 passed, 9 skipped (+23 authentic tests: gemini-live-thinking, inworld-realtime, …; mock only the provider WebSocket)..tgz(npm pack→getpatter-0.7.1.tgz):GeminiLive/GeminiCascade/InworldRealtimeresolve at runtime (CJS + ESM).giulia-three-variants) verifies against the local.tgz, deletesGiulia/vendor/getpatter-0.6.7.tgz+ the Inworld monkey-patch, reverts theimport type { GeminiLive }workaround to a value import.v0.7.1→release.ymlpublishes npm + PyPI + the realigned launcher.🤖 Generated with Claude Code
https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
Also in this PR: media-stream WebSocket authentication (Closes #204)
Fixes a second security report from the same reporter: the media-stream WS endpoints (
/ws/stream/…Twilio,/ws/telnyx/stream/…,/ws/plivo/stream/…) were unauthenticated — the signed carrier webhook validated HTTP, but any peer reaching the public host could open the WS with an attacker-chosencall_id, send astartframe, and drive a full STT→LLM→TTS session on the operator's keys (toll fraud) or extract the agent's system prompt.<Parameter>→customParameters, Telnyx URL query, Plivoextra_headers). The WS constant-time-validates it before opening any provider session; unauthenticated → WS 1008, no provider connect, no TTS.require_stream_auth/requireStreamAuth), opt-out for custom-TwiML operators (logs a warning). Token never logged. Backward compatible for the standardserve()inbound+outbound path.X-Forwarded-Forper-IP cap (documented DoS-cap-only).tests/security/test_stream_auth.py(30) + TSstream-authsuite. Adversarial security-verify pass (re-ran the Security: media-stream WebSocket endpoints are unauthenticated — toll fraud & system-prompt extraction #204 repro — an unauthenticated peer is provably blocked). Python 2982 / TS green.Closes #204.