Skip to content

feat(engines): GeminiLive/GeminiCascade/InworldRealtime markers + Gemini Live fixes (release 0.7.1)#202

Merged
nicolotognoni merged 4 commits into
mainfrom
feat/gemini-inworld-engines
Jul 2, 2026
Merged

feat(engines): GeminiLive/GeminiCascade/InworldRealtime markers + Gemini Live fixes (release 0.7.1)#202
nicolotognoni merged 4 commits into
mainfrom
feat/gemini-inworld-engines

Conversation

@nicolotognoni

@nicolotognoni nicolotognoni commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Release 0.7.1 — Gemini/Inworld engine markers + Gemini Live fixes

Follow-up to 0.7.0. 0.7.0 shipped the pipeline barge-in fix (#200) but did NOT ship the GeminiLive engine marker (so new GeminiLive({...}) / the engine import threw at runtime — the demo had to vendor an old .tgz), and a live-call Gemini "thinking" leak surfaced. This lands the markers + fixes and prepares 0.7.1. npm publish is intentionally NOT run — it's human-gated (publish happens via the v0.7.1 tag after merge).

Engines (new public API)

  • GeminiLive engine marker + richer GeminiLiveAdapter (mulaw8↔PCM telephony transcode, apiVersion auto-detect, GEMINI_LIVE_3_1_FLASH_PREVIEW), GeminiCascade engine, GeminiSTT/GeminiTTS — integrated from the gemini-3.1-live-demo work onto the 0.7.0 baseline (3-way merge; fix(pipeline): stop false barge-in from self-interrupting the user-turn LLM (agent goes mute) #200 barge-in + feat(inworld): complete TTS pricing + config helper + Realtime Router LLM preset #199 Inworld pricing/exports preserved, verified).
  • InworldRealtime — native engine marker + InworldRealtimeAdapter that subclasses OpenAIRealtimeAdapter and overrides only the transport (wss://api.inworld.ai/v1/realtime, Bearer/JWT — Inworld's OpenAI-Realtime migration path). Lets the demo drop its OpenAIRealtime monkey-patch.
  • Verified at runtime from the built .tgz: require('getpatter') and ESM named imports both resolve GeminiLive, GeminiCascade, InworldRealtime as functions (the exact 0.7.0 bug, now fixed).

Fixes (live-call bugs on the Giulia demo vs 0.7.0)

  • Gemini Live spoke the model's chain-of-thought before the reply (every turn, native-audio). The Live setup now always sends thinkingConfig (voice default thinkingBudget: 0, OFF; opt-in thinking/thinkingBudget on GeminiLiveOptions), and the adapter defensively drops any thought===true part from both the audio stream and the text transcript (never logged — no PII).
  • gemini-3.1-flash-live-preview → silent "session ready timeout" (call dropped with dead air). It's a native-audio model served only on v1alpha but its id lacks the native-audio token, so the heuristic chose v1beta. New geminiRequiresV1Alpha() selects v1alpha for native-audio AND flash-live-preview (explicit apiVersion overrides); connect failures now throw a clear, actionable error (model id + resolved apiVersion + root causes) instead of dead air.

Changed

  • @google/genai peerDep ^0.3.0>=2.0.0 (kept optional) for the 2.x unified SDK — npm install getpatter with @google/genai@^2.x no longer needs --legacy-peer-deps.
  • create-getpatter launcher realigned to the SDK version (lockstep) — it pins the getpatter version it bootstraps to its own version. It had drifted to 0.6.9 (so npm create getpatter was provisioning 0.6.9); bumped to 0.7.1.

Backward compatibility

OpenAIRealtime / OpenAIRealtime2 / Ultravox / ConvAI unchanged. The instanceof OpenAIRealtimeAdapter gates are extended, not replaced (InworldRealtimeAdapter passes them via subclassing; the Gemini function_call dispatch gate is widened). #200 barge-in fix confirmed intact.

Versioning

All four version files move to 0.7.1 together (Python __init__.py + pyproject.toml, TS package.json, launcher create-getpatter/package.json). Per the "0.7.x" ask this is tagged a patch; note it does add new public API (engine markers), which is strictly a minor — call it 0.8.0 instead if you prefer stricter semver.

Test plan

  • npm run lint (tsc --noEmit) green; npm test green — 2334 passed, 9 skipped (+23 authentic tests: gemini-live-thinking, inworld-realtime, …; mock only the provider WebSocket).
  • Built .tgz (npm packgetpatter-0.7.1.tgz): GeminiLive/GeminiCascade/InworldRealtime resolve at runtime (CJS + ESM).
  • Adversarial verification pass (independent re-build + re-test + checklist).
  • Demo (giulia-three-variants) verifies against the local .tgz, deletes Giulia/vendor/getpatter-0.6.7.tgz + the Inworld monkey-patch, reverts the import type { GeminiLive } workaround to a value import.
  • Publish is human-gated — merge → tag v0.7.1release.yml publishes npm + PyPI + the realigned launcher.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM


Also in this PR: media-stream WebSocket authentication (Closes #204)

Fixes a second security report from the same reporter: the media-stream WS endpoints (/ws/stream/… Twilio, /ws/telnyx/stream/…, /ws/plivo/stream/…) were unauthenticated — the signed carrier webhook validated HTTP, but any peer reaching the public host could open the WS with an attacker-chosen call_id, send a start frame, and drive a full STT→LLM→TTS session on the operator's keys (toll fraud) or extract the agent's system prompt.

  • Per-call token: the signature-validated webhook mints a high-entropy token (short-TTL, keyed to the call) and delivers it on each carrier's existing custom channel (Twilio <Parameter>→customParameters, Telnyx URL query, Plivo extra_headers). The WS constant-time-validates it before opening any provider session; unauthenticated → WS 1008, no provider connect, no TTS.
  • Fail-closed by default (require_stream_auth / requireStreamAuth), opt-out for custom-TwiML operators (logs a warning). Token never logged. Backward compatible for the standard serve() inbound+outbound path.
  • Also added a global concurrent-WS backstop for the forgeable-X-Forwarded-For per-IP cap (documented DoS-cap-only).
  • Both SDKs, all three carriers. Tests: tests/security/test_stream_auth.py (30) + TS stream-auth suite. Adversarial security-verify pass (re-ran the Security: media-stream WebSocket endpoints are unauthenticated — toll fraud & system-prompt extraction #204 repro — an unauthenticated peer is provably blocked). Python 2982 / TS green.

Closes #204.

nicolotognoni and others added 4 commits June 30, 2026 19:52
…+ fix Gemini Live thinking leak & 3.1 apiVersion

Follow-up to 0.7.0, which exported the Gemini pipeline factory + adapter but NOT
the GeminiLive engine marker (so `new GeminiLive({...})` / the engine import
failed at runtime), and never shipped a native InworldRealtime engine.

Engines (new public API):
- GeminiLive engine marker (engines/gemini.ts) + the richer GeminiLiveAdapter
  (mulaw8<->PCM telephony transcode, apiVersion auto-detect,
  GEMINI_LIVE_3_1_FLASH_PREVIEW), GeminiCascade engine + GeminiSTT/GeminiTTS,
  integrated from the gemini-3.1-live-demo work onto the 0.7.0 baseline
  (3-way merge; #200 barge-in + #199 Inworld pricing/exports preserved).
- InworldRealtime engine marker (engines/inworld.ts) + InworldRealtimeAdapter
  (providers/inworld-realtime.ts), subclassing OpenAIRealtimeAdapter and
  overriding only the transport (wss://api.inworld.ai/v1/realtime, Bearer/JWT;
  Inworld's OpenAI-Realtime migration path). Lets the demo drop its
  OpenAIRealtime monkey-patch. `import { InworldRealtime } from 'getpatter'`
  resolves at runtime.

Fixes (live-call bugs found on the Giulia demo against 0.7.0):
- Gemini Live spoke the model's chain-of-thought before the reply. Now the Live
  setup always sends thinkingConfig (voice default thinkingBudget:0, OFF), with
  opt-in `thinking`/`thinkingBudget` on GeminiLiveOptions; and the adapter
  defensively drops any `thought===true` part from BOTH the audio stream and
  the text transcript (never logged — no PII).
- gemini-3.1-flash-live-preview dropped the call with a silent "session ready
  timeout": it is a native-audio model served only on v1alpha but its id lacks
  the 'native-audio' token, so the heuristic picked v1beta. New
  geminiRequiresV1Alpha() selects v1alpha for native-audio AND flash-live-preview
  (explicit `apiVersion` overrides); connect failures now throw a clear,
  actionable error (model id + resolved apiVersion + root causes) instead of
  dead air.

- Bump @google/genai peerDependency ^0.3.0 -> >=2.0.0 (kept optional) for the
  2.x unified SDK; adds a ^2.8.0 devDependency so tsc resolves the literal
  import in the stt/tts/cascade modules.

Backward compatible: OpenAIRealtime / OpenAIRealtime2 / Ultravox / ConvAI
unchanged; the instanceof OpenAIRealtimeAdapter gates are extended, not replaced
(InworldRealtimeAdapter passes them via subclassing). Tests: +23 authentic
(gemini-live-thinking, inworld-realtime, …), mock only the provider WebSocket.
lint green; 2334 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
Ships the GeminiLive/GeminiCascade/InworldRealtime engine markers + the Gemini
Live thinking-leak and gemini-3.1 apiVersion fixes (this PR's feature commit).
All four version files move together; the create-getpatter launcher is realigned
to the SDK version (it pins the getpatter version it bootstraps to its own).

- libraries/python/getpatter/__init__.py, libraries/python/pyproject.toml,
  libraries/typescript/package.json -> 0.7.1
- create-getpatter/package.json 0.6.9 -> 0.7.1 (lockstep restored)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
… (Indian langs), both SDKs

Built by parallel agent teams (research -> per-provider modules -> serialized
wiring -> adversarial verify), verified against the official docs/pricing.

- Soniox v5 STT (update): SonioxModel gains stt-rt-v5 and it is now the default
  in both SDKs (v4/v3/v2 retained). v5 is wire-compatible with v4 (same WS URL,
  in-band api_key, token stream), so the only change to adopt it is the model id.
  New opt-in endpoint_sensitivity / endpoint_latency_adjustment_level, omitted
  when unset so the v4 wire config stays byte-identical. STT price unchanged
  (/bin/zsh.12/hr = /bin/zsh.002/min, soniox.com/pricing).
- Soniox TTS (new): SonioxTTS over POST https://tts-rt.soniox.com/tts (Bearer,
  tts-rt-v1, default voice Adrian, native pcm_mulaw/8k carrier path), sharing the
  SONIOX_API_KEY credential. Provider + tts wrapper + root export + providers
  helper + _create_tts_from_config branch + pricing (modeled per char from the
  /bin/zsh.70/hr headline — Soniox publishes no per-char/token rate).
- Sarvam AI TTS (new): SarvamTTS over POST https://api.sarvam.ai/text-to-speech
  (api-subscription-key header, bulbul:v3, 11 Indian languages via
  target_language_code, base64 audio), for Indian-language voice agents. Full
  wiring + a [sarvam] pyproject extra. Pricing matches sarvam.ai/api-pricing
  (Rs 30 / Rs 15 per 10k chars).

Strict Python<->TS parity; backward compatible (existing Soniox/Cartesia/etc.
unchanged); no PII (cost telemetry records only text length, never the text or
audio). Authentic tests mock only the transport. Version stays 0.7.1.
Python 2952 passed, TS 2380 passed, lint clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
…204)

Closes #204. The carrier webhook HTTP routes were signature-validated, but the
media-stream WS endpoints (/ws/stream/{call_id} Twilio, /ws/telnyx/stream,
/ws/plivo/stream) accepted any peer with an attacker-chosen call_id — the only
control was the DoS per-IP cap. An unauthenticated peer reaching the public host
could open the socket, send a start frame, and drive a full STT->LLM->TTS
session on the operator's provider keys (toll fraud), or converse to extract the
agent's system prompt + tool list.

Fix (both SDKs, all three carriers):
- The signature-validated webhook that builds the stream URL/TwiML mints a
  high-entropy per-call token (secrets.token_urlsafe / crypto.randomBytes),
  stored in a per-server short-TTL map keyed by the call, and delivers it on
  each carrier's existing custom channel: Twilio <Parameter> -> customParameters,
  Telnyx URL query, Plivo extra_headers.
- The WS handler constant-time-validates the token (hmac.compare_digest /
  timingSafeEqual) BEFORE opening any provider session; on missing/invalid/
  expired it closes WS 1008 with no provider connect, no TTS, no callback/cost.
  Telnyx validates at accept; Twilio/Plivo in the start-frame branch.
- Fail-closed by default via require_stream_auth / requireStreamAuth (opt-out
  for custom-TwiML operators, which logs a warning). Token never logged.
  Backward compatible for the standard serve() inbound + outbound path.
- Hardened the forgeable X-Forwarded-For per-IP cap with a global concurrent-WS
  backstop (MAX_TOTAL_WS / MAX_WS_TOTAL = 500), documented DoS-cap-only.

Polish over the workflow output: outbound Twilio now keys the token on a fresh
per-call id (not the static 'outbound') so concurrent outbound dials don't
overwrite each other's token; aligned the Python/TS backstop ceiling.

Built by parallel Python+TS agent teams with an adversarial security-verify gate
(re-ran the #204 repro: an unauthenticated peer is provably blocked before any
provider session). Tests: tests/security/test_stream_auth.py (30) + TS
stream-auth suite. Python 2982 passed, TS green, lint clean. Version stays 0.7.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
@nicolotognoni nicolotognoni merged commit 5c7b848 into main Jul 2, 2026
14 checks passed
nicolotognoni added a commit that referenced this pull request Jul 2, 2026
Resolves the release-0.7.1 (#202) integration: keeps #208's Unreleased
entries above the new 0.7.1 CHANGELOG section, adopts 0.7.1 across all
version files (no additional bump), and regenerates package-lock.json from
the merged package.json (Gemini deps + NAMO onnxruntime/transformers).
Full suites green post-merge: Python 3066, TypeScript 2476, tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
nicolotognoni added a commit that referenced this pull request Jul 2, 2026
Integrates releases #202 (0.7.1) and #208 (low-latency pipeline) now on main.
Keeps this PR's transfer-allowlist Unreleased entry alongside #208's entries
under the single Unreleased section, adopts 0.7.1 across version files (no
extra bump), and regenerates package-lock.json from the merged package.json.
Full suites green post-merge: Python 3083, TypeScript 2493, tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM
@github-actions github-actions Bot deleted the feat/gemini-inworld-engines branch July 3, 2026 07:21
nicolotognoni added a commit that referenced this pull request Jul 3, 2026
Version files already bumped to 0.7.1 by #202; #206 (transfer allowlist) and
#208 (low-latency voice pipeline) then merged onto main at 0.7.1 but their
entries sat under Unreleased. Fold them into the 0.7.1 section so the published
release notes match the code that ships. No version change (already 0.7.1); no
code change. Merge this, then tag v0.7.1 to publish to npm + PyPI.


Claude-Session: https://claude.ai/code/session_01D52f6rBsvzicXg45C6MKVM

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Security: media-stream WebSocket endpoints are unauthenticated — toll fraud & system-prompt extraction

1 participant