Skip to content

feat(google): add gemini-3.8 TTS models and expressive mode support - #2576

Merged
tinalenguyen merged 3 commits into
mainfrom
feat/gemini-tts-3-8-expressive
Sep 23, 2026
Merged

tinalenguyen merged 3 commits into
mainfrom
feat/gemini-tts-3-8-expressive

Conversation

@tinalenguyen

Copy link
Copy Markdown
Member

Port of livekit/agents#7425.

  • Gemini TTS: add gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts. On these models the expression marker rides per-part speech_metadata.style (one request part per style change), instructions move into the style instead of a spoken preamble, headerless PCM is requested explicitly, and a two-speaker config (speakers + speaker) is supported.
  • provider_format: add the gemini expr dialect (instructions, sound vocabulary and aliases, steering table). Sounds, breaks and emphasis lower to inline tags; the expression marker is left for the plugin.
  • tts.StreamAdapter: normalize and convert markup when expressive is on, snapshot the expressive flag per stream, pass the dialect and _setExpressive through to the wrapped TTS.
  • Expressive gate: allow any TTS with a dialect that the framework lowers for (inference.TTS, or non-streaming / StreamAdapter), not only the gateway.
  • Transcript stripping: a marker heading a line takes the space after it, and a marker ending the text takes the space before it.

Description

Changes Made

Pre-Review Checklist

  • Build passes: All builds (lint, typecheck, tests) pass locally
  • AI-generated code reviewed: Removed unnecessary comments and ensured code quality
  • Changes explained: All changes are properly documented and justified above
  • Scope appropriate: All changes relate to the PR title, or explanations provided for why they're included
  • Video demo: A small video demo showing changes works as expected and did not break any existing functionality using Agent Playground (if applicable)

Testing

  • Automated tests added/updated (if applicable)
  • All tests pass
  • Make sure both restaurant_agent.ts and realtime_agent.ts work properly (for major changes)

Additional Notes


Note to reviewers: Please ensure the pre-review checklist is completed before starting your review.

Port of livekit/agents#7425.

- Gemini TTS: add gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts. On
  these models the expression marker rides per-part speech_metadata.style
  (one request part per style change), instructions move into the style
  instead of a spoken preamble, headerless PCM is requested explicitly, and
  a two-speaker config (speakers + speaker) is supported.
- provider_format: add the gemini expr dialect (instructions, sound
  vocabulary and aliases, steering table). Sounds, breaks and emphasis
  lower to inline tags; the expression marker is left for the plugin.
- tts.StreamAdapter: normalize and convert markup when expressive is on,
  snapshot the expressive flag per stream, pass the dialect and
  _setExpressive through to the wrapped TTS.
- Expressive gate: allow any TTS with a dialect that the framework lowers
  for (inference.TTS, or non-streaming / StreamAdapter), not only the
  gateway.
- Transcript stripping: a marker heading a line takes the space after it,
  and a marker ending the text takes the space before it.
@tinalenguyen
tinalenguyen requested a review from a team as a code owner September 23, 2026 17:44
@changeset-bot

changeset-bot Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0b77057

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-azure Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-krisp Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-meta Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-protoface Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-soniox Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugins-test Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

@chenghao-mou chenghao-mou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar python issue and one nit. It works well in my testing.

Comment thread plugins/google/src/beta/gemini_tts.ts Outdated
}

function assertKnownSpeaker(speaker: string, speakers: Record<string, string>): void {
if (!(speaker in speakers)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex:

js gotcha:

const speakers = { Sienna: 'Kore', Comanchero: 'Puck' };

'Sienna' in speakers;   // true — configured speaker
'Nobody' in speakers;   // false
'toString' in speakers; // true — inherited method

@tinalenguyen
tinalenguyen merged commit 38f65f8 into main Sep 23, 2026
7 checks passed
@tinalenguyen
tinalenguyen deleted the feat/gemini-tts-3-8-expressive branch September 23, 2026 18:53
vznh added a commit to vznh/agents-js that referenced this pull request Sep 24, 2026
Resolve StreamAdapterWrapper against livekit#2576, which added an `expressive`
snapshot beside this branch's `error` getter; both are kept. The API
report is regenerated from the merged tree and differs from main only by
this branch's three internal `error` getters.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants