Repository navigation
feat(ai-components): add VoiceWave and VoiceGlow voice-agent visualizations - #3339
Merged
Merged
Conversation
…ations Port the voice agent visualizations from stream-chat-android (GetStream/stream-chat-android#6848, #6849, #6851) to the ai-components entry point, with the same motion maths. - VoiceWave: a string carrying a wave packet, drawn on a 2D canvas. - VoiceGlow: a soft globe drawn by a WebGL shader, with a 2D canvas fallback chosen by `rendering`. - useAudioLevel: a pull-based level getter for a MediaStream, for the `audioLevel` prop. - Both follow the theme through the voice-color and voice-backdrop tokens, honour prefers-reduced-motion, pause off screen and are labelled for assistive technology in all 12 locales. The ai-chatbot example gains a hands-free voice mode built on them, and its send path moves into a shared useSendAIMessage hook.
…ow to the globe The glow is all soft gradients, so a 2x backing store filled four times the pixels for no visible difference; it now draws at no more than 1x, while the wave keeps up to 2x for a crisp line. The Canvas glow clips its color masses to the globe instead of painting past it and masking the excess away, and computes the orbit angles once a frame.
`rendering='css'` composes the glow the way the shader does, as CSS: an opaque color field (a base and the masses mixed over it in order, with the shader's smoothstep falloffs, diffuse shade and fresnel rim) seen through an alpha that only the first two masses carry, a faint static body plus one mask riding each of their orbits, with inverse turns inside each mask keeping the colors still. Every gradient is painted once and only transform and opacity animate, so the compositor runs the motion: it keeps moving while the main thread is busy, needs no canvas or WebGL, and runs no script per frame unless an audio level is given. The palette is computed with the shader's color math and follows the theme; each state sets playback rates, so state changes ease instead of jumping. WebGL stays the default for `auto`.
…h, speech rhythm and fade-in - Only the shape takes the pointer: the root lets events through to whatever is behind it, and a hit area the size of the globe or the string takes them and hands them to the root's handlers. - The wave and the Canvas glow animate speech with the shader's simplex envelope, ported in float32, instead of a 1-D CPU noise, so every renderer and iOS share the same rhythm. - The visualizations fade in over 0.3s once their first frame is drawn (the CSS glow once its palette is set), instantly with reduced motion.
…ghter CSS glow `rendering` is now 'auto' | 'webgl' | 'css'. Where WebGL cannot start or loses its context, VoiceGlow falls back to the CSS renderer instead of a Canvas2D drawing: closer to the shader in look, and it keeps moving while the main thread is busy. Standing in for WebGL, the GPU is likely off and the compositor blends every layer on the CPU, so the fallback draws a lighter CSS glow: the still body and the body riding the first orbit, without the stretches, shade and rim. With GPU acceleration off it holds 60 fps at about half the CPU of the full CSS glow, close to the old Canvas2D cost, while leaving the main thread nearly idle. An explicit rendering='css' keeps the full look. Both skip masses the palette leaves empty. The Canvas2D renderer and the 1-D noise only it used are removed.
…t at up to 3x Stroke the wave at 0.6% of the shorter side with Android's 1.5 device-pixel floor, instead of a 1.5 CSS-pixel floor that made it heavier in boxes under 250px, and draw it at the screen's own density up to 3x, as iOS does. Each visualization now sets its own pixel-ratio cap. Give the ai-chatbot voice mode's wave a square box, since the wave is sized from the shorter side.
In loud speech the wave's crests rose into near-vertical spikes and its height swung sharply with every syllable. The crests now spread out instead of steepening past a slope of 6, with the cap on their spacing eased so it breathes, and the speech height rises quickly and settles slowly, with shallower syllables. While idle, connecting or thinking the wave is still exactly the Android and iOS wave. useAudioLevel now measures over 2048 samples (about 43 ms) instead of 256, a few pitch periods of a voice, so the level holds steady through a vowel. Renderers get the time since the last frame so the smoothing runs at the same speed at any frame rate. The voice-mode demo uses the default expressivity, and the CSS glow stylesheet uses list.nth in place of the deprecated global nth.
Contributor
|
Size Change: +49.1 kB (+4.34%) Total Size: 1.18 MB 📦 View Changed
ℹ️ View Unchanged
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release-v14 #3339 +/- ##
===============================================
+ Coverage 85.89% 86.45% +0.55%
===============================================
Files 524 540 +16
Lines 16649 17516 +867
Branches 5229 5401 +172
===============================================
+ Hits 14301 15143 +842
- Misses 2348 2373 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ow and fix review findings - Every WebGL glow now draws with one shared offscreen context and copies its frame onto its own 2D canvas, so the shader compiles once and many glows stay under the browser's context limit. A lost context drops every glow to CSS; the last glow to unmount releases it. - The CSS glow has one composition, the former fallback: no stretches, shade, rim or second riding mass. Off screen its animations pause and its theme and audio reads stop. - `rendering` is `'auto' | 'css'`; `'webgl'` behaved exactly like `'auto'`. - Colors the browser rejects are skipped instead of read as black, and an unreadable backdrop is no longer taken for dark. - Both glows share the theme reading, the pointer conversion and the driver's easing constants. - The driver computes its color slots once per palette and eases its own copies in place. - The canvas follows a change of pixel ratio alone, as on a move to another monitor. - Ripple timers are dropped once they fire. - The voice-mode demo ends a turn the agent is never seen working on after 20 seconds.
MartinCupela
approved these changes
Oct 22, 2026
oliverlaz
added a commit
that referenced
this pull request
Oct 22, 2026
Resolve the ai-components README conflict with #3339 (VoiceWave and VoiceGlow): keep both sides' exports in the component, hook, context and type lists, and both hook sections (useHeardText and useAudioLevel).
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 22, 2026
## [14.14.0](v14.13.3...v14.14.0) (2026-10-22) ### Features * **ai-components:** add VoiceWave and VoiceGlow voice-agent visualizations ([#3339](#3339)) ([da6f72f](da6f72f)), closes [GetStream/stream-chat-android#6848](GetStream/stream-chat-android#6848) [GetStream/stream-chat-android#6849](GetStream/stream-chat-android#6849) [GetStream/stream-chat-android#6851](GetStream/stream-chat-android#6851) [GetStream/stream-chat-swift#4338](GetStream/stream-chat-swift#4338) * **Message:** add SpokenMessageText for voice-paced AI message text ([#3343](#3343)) ([22abe90](22abe90)), closes [GetStream/Vision-Agents#1128](GetStream/Vision-Agents#1128)
|
🎉 This PR is included in version 14.14.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
oliverlaz
added a commit
that referenced
this pull request
Oct 22, 2026
…ardText hook (#3344) ### 🎯 Goal `stream-chat-react/ai-components` must work standalone, without the core SDK. `SpokenMessageText` (added in #3343) rendered through the core `Message` components (`MessageText`, the `StreamedMessageText` fallback, and the message and channel contexts), so it doesn't belong in the plugin. This keeps only the utility hook. The drop-in component becomes a recipe in the docs (GetStream/getstream.io#761). This is the last commit of #3343 (`57e84ca2e`), which was pushed just before the PR was merged and didn't make it in. >⚠️ **14.14.0 already shipped `SpokenMessageText`, `SpokenTextProvider`, `SpokenTextContext` and `useSpokenTextContext`** (with the message-based `useHeardText`) from `stream-chat-react/ai-components`. This PR removes those exports and changes `useHeardText`'s signature, and ships as a fix (14.14.1): they were released minutes ago without docs, so it is treated as correcting a mistaken release rather than a major change. ### 🛠 Implementation details - **Removed:** `SpokenMessageText`, `SpokenTextProvider`, `SpokenTextContext`, `useSpokenTextContext`, their types and their tests. - **`useHeardText`** now lives in `src/plugins/AIComponents/hooks/useHeardText.ts`. Its only import is `react`; I checked this in the source, the built ESM and the CJS bundle. - `useHeardText({ text, heard, subscribe?, generating?, live? }) => { heardText, isPaced, skip }`. - `heard(now)` returns `{ chars, done } | undefined`, where `chars` counts Unicode code points. - `heard` and `subscribe` are read through refs, so new function identities on each render don't restart pacing. - Types: `HeardTextProgress`, `HeardTextFunction`, `HeardTextSubscribe`, `UseHeardTextParams`, `UseHeardTextResult`. - **Behaviour:** - Reveals the first `floor(chars)` code points and never goes backwards; surrogate pairs are never split. - Ticks on `requestAnimationFrame` only while paced and not done. - The full text appears once `done && !generating`, when `live` is false, or after `skip()`. - Falls back to the full text when unpaced. - **README:** documents the hook with a plain-element example and points to the docs for the drop-in. The VoiceWave/VoiceGlow docs from #3339 are untouched. - **Checks:** 21 `renderHook` tests. Full suite 3175/3175 passes, as do `yarn types`, `yarn lint`, `yarn build` and `validate-cjs` / `validate-esm` / `validate-entry-isolation`. In the built output, `ai-components` exports `useHeardText` and nothing `Spoken*`, and the main entry exports neither. - **Verified** end to end in a Next.js app against a local Stream Agents router, using the docs' `SpokenMessageText` recipe on this hook: each voice reply is revealed as its audio plays. ### 🎨 UI Changes None.
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 22, 2026
## [14.14.1](v14.14.0...v14.14.1) (2026-10-22) ### Bug Fixes * **ai-components:** replace SpokenMessageText with a standalone useHeardText hook ([#3344](#3344)) ([ad16506](ad16506)), closes [#3343](#3343) [GetStream/getstream.io#761](https://github.com/GetStream/getstream.io/issues/761) [#3343](#3343) [#3339](#3339)
oliverlaz
added a commit
that referenced
this pull request
Oct 26, 2026
…ations Port of #3339 from the v14 line: VoiceWave, VoiceGlow and useAudioLevel in stream-chat-react/ai-components, with v15 namespaced i18n keys (aiComponents.voiceVisualization.*), and a voice mode in the ai-chatbot example built on v15's channel and composer APIs.
This was referenced Oct 26, 2026
oliverlaz
added a commit
that referenced
this pull request
Oct 26, 2026
…ations (#3347) ### Goal Port the voice-agent visualizations from the v14 line (#3339, released in v14.14.0) to v15, so `stream-chat-react/ai-components` on `master` ships `VoiceWave`, `VoiceGlow` and `useAudioLevel` with the same API, motion and theming as v14, and the `ai-chatbot` example gets the voice mode. ### Implementation details The components, hook, styles and tests come over from #3339 unchanged: `VoiceWave`, `VoiceGlow` (`rendering?: 'auto' | 'css'`), `useAudioLevel(stream)`, the types `VoiceAgentState`, `VoiceGlowRendering`, `VoiceVisualizationProps`, `VoiceWaveProps` and `VoiceGlowProps`, `voice-visualization.scss`, `voice-glow-css.scss` and the new `_tokens.scss` entries. See #3339 for how they work. Adapted to v15: - **i18n.** The accessible name now uses namespaced keys with inline English defaults: `aiComponents.voiceVisualization.{connecting,listening,ready,speaking,thinking}.text` and `aiComponents.voiceVisualization.voiceAgent.ariaLabel` (`Voice agent: {{ state }}`). `keys.ts` and the catalog fixture are regenerated. The v14 locale JSON changes are dropped, as v15 bundles English only. - **Migration docs.** The six v14 keys (`Connecting`, `Listening`, `Ready`, `Speaking`, `Thinking`, `aria/Voice agent: {{ state }}`) are added to the key table in the AI components README and to `ai-docs/i18n-v15-key-map.json`. - **Tests.** The voice tests use the default translator and `mockT` instead of v14's `en.json`. `i18n.test.tsx` checks both the requested keys and the English defaults. - **ai-chatbot example.** `useSendAIMessage` is the send path shared by the composer and voice mode, rebuilt on master's flow: `useChannel()`, `channel.ensureWatched()` for a new conversation, `composer.send()`, and the summary in `data.custom.summary`, now written in the background so it never delays or fails a send. `ChatContainer` wraps master's `<Channel channel={channel}>` in `VoiceModeProvider`. - **Voice mode fixes over v14.** Spoken turns send through an unregistered `MessageComposer` of their own (drafts off), so they no longer replace the typed draft or send its attachments. A failed send stops voice mode with an error instead of waiting out the answer timeout. The agent's answer is read from `channel.messagePaginator.latestMessage`, which does not depend on the window the list shows. Opening or closing voice mode refreshes the composer's rate-limit banner. The port adds API and breaks nothing. ### UI Changes The same visuals as #3339. In the `ai-chatbot` example, a waveform button in the composer opens voice mode, which needs a browser with `SpeechRecognition` (Chrome, Edge or Safari) and microphone access. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added `VoiceWave` and `VoiceGlow` visualizations that respond to voice-agent states and audio levels, with theme-aware colors and accessible state labels. * Added a CSS rendering option for the glow, including fallback when WebGL is unavailable and support for reduced-motion preferences. * The AI chatbot example now includes hands-free voice conversations with pause, resume, and end controls. Voice mode requires browser speech recognition and microphone access. * **Documentation** * Added guidance on voice mode and voice visualization options. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
oliverlaz
added a commit
that referenced
this pull request
Oct 26, 2026
### Goal Port #3343 and #3344 from `release-v14` to v15. Together they add `useHeardText` to `stream-chat-react/ai-components`: a standalone hook (React only, no chat SDK imports) that reveals a text at the pace a voice AI agent speaks it. Neither PR was ported to `master`. #3344 removes the `SpokenMessageText` / `SpokenTextProvider` / `SpokenTextContext` exports that #3343 added, so `master` never had them, and the port is the net result: the hook, its tests, the export and the README section. A review of the port found issues in the hook itself (shared with v14), which the second commit fixes. ### Implementation details **Commit 1, the port** (`feat(ai-components): add useHeardText for voice-paced text`) - `src/plugins/AIComponents/hooks/useHeardText.ts` and `__tests__/useHeardText.test.ts` are byte-identical to `release-v14`. - `src/plugins/AIComponents/index.ts` exports the hook and its types (`HeardTextProgress`, `HeardTextFunction`, `HeardTextSubscribe`, `UseHeardTextParams`, `UseHeardTextResult`). - README: the `useHeardText` section matches v14; the intro sentence and the export lists mention the hook. The VoiceWave / VoiceGlow / `useAudioLevel` entries that sit next to it on v14 are left to the open port of #3339 (#3347), which edits the same README lines, so whichever lands second has a small conflict there. **Commit 2, fixes from the review** (`fix(ai-components): restart useHeardText for a new text and treat Infinity as fully heard`) - A component that renders the next reply with the same hook instance (for example the README's `SpokenReply` without a `key`) carried over the length already shown, `skip()` and `live`, so the new reply appeared partly or fully revealed before any audio. The reveal now starts over when `text` is replaced by a different text. Growing or truncating the text keeps the reveal, capped at the new length, so a regrowth after a cut does not reuse the old length. The reset happens during render, so the new text never renders with the previous reveal. - `chars: Infinity` revealed nothing, contrary to the type doc ("a value at or above the text's code point length reveals the whole text"). It now reveals the whole text; `NaN`, negative and non-numeric counts still mean nothing heard. - Docs (JSDoc and README): `heard` should return `undefined` only for a text that is not spoken. The old wording also suggested it for a text "not known to the voice source yet", which shows the full text and then never paces it, because the reveal does not move backwards. Also documented: `subscribe` is bound once (remount to switch speech sources) and is the only wake-up for a settled hook, and `heard` is called during render, so it must be free of side effects. - Left as is: re-subscribing on a new `subscribe` identity (it would churn every frame with inline arrows, which the hook explicitly supports), and polling unpaced texts (every non-spoken message would tick forever). Rebuilding the code-point offsets per streamed chunk costs about 20 ms over a whole 4k-character reply in 1k chunks, so it was not optimized. `release-v14` still has the issues fixed in commit 2; they should be ported back. Checks: `yarn test src/plugins/AIComponents` (101 tests, 27 for the hook, 6 of them new), eslint `--max-warnings 0`, prettier, `tsc -p tsconfig.lib.json --noEmit`, `yarn build`, `validate-cjs`, `validate-esm` and `validate-entry-isolation`. The built `useHeardText` module imports only `react`, and the main entry does not include it. ### UI Changes None. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added a hook that reveals text in sync with speech progress, with support for skipping the reveal and handling Unicode text. Text is shown immediately when pacing is unavailable and fully revealed when speech or generation is complete. * **Documentation** * Added usage guidance covering the hook’s inputs, results, pacing behavior, and an example. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 of 6 tasks
oliverlaz
added a commit
to GetStream/stream-chat-react-native
that referenced
this pull request
Oct 28, 2026
…ations (#3895) Ticket: https://linear.app/stream/issue/RN-468 ## 🎯 Goal Android (GetStream/stream-chat-android#6848, GetStream/stream-chat-android#6849, GetStream/stream-chat-android#6851), iOS (GetStream/stream-chat-swift#4338) and web (GetStream/stream-chat-react#3339) added two voice-agent visualizations to their AI components. This brings them to `stream-chat-react-native/ai-components` (and `stream-chat-expo/ai-components`) with the same motion, so the platforms look and move alike, and shows them in a new voice mode in `ExpoChatGPTSample`. ## 🛠 Implementation details **Public API** (from the `ai-components` entry point): - `VoiceWave`: a single string carrying a wave packet that erupts with speech, its colors flowing along it. - `VoiceGlow`: a soft globe whose color masses orbit inside it and swell with speech. - `useDictationAudioLevel()`: the microphone's live level while dictation listens, as a Reanimated `SharedValue<number | null>`. - Types `VoiceAgentState`, `VoiceVisualizationProps`, `VoiceWaveProps`, `VoiceGlowProps`, `DictationLevel`; `useDictationState` and `useDictationTranscript` are now exported too. - An optional `voiceVisualization` theme key: `colors` (default palette, the accent color) and `backdropColor` (light or dark look; the system color scheme when unset). Both components take `state` (`idle | connecting | listening | thinking | speaking`), `colors` (one to five colors), `expressivity` (0 to 2) and `audioLevel` (a `SharedValue`, read once a frame on the UI thread; `null` animates speech on its own). Other `View` props go to the root. **How it's built:** - One motion driver, ported from Android's `VoiceVisualizationDriver` (per-state motion table, frame-rate-independent easing, reduced motion at 0.07x, tap bounce, HSV palette, and the `Steady`/`Smooth` level responses for glow and wave). It's plain objects and worklets, advanced in a Reanimated `useFrameCallback`, so the motion runs on the UI thread and keeps going while JS is busy streaming a reply. - `VoiceGlow` is a Skia `RuntimeEffect`: Android's AGSL shader is SkSL, so it's ported nearly verbatim and runs on every Android version. Every glow shares one compiled effect. There is no Canvas/CSS fallback and no `rendering` prop. - `VoiceWave` is a Skia path with a 13-stop gradient, as Android draws it; its speech envelope uses the shader's float32 simplex noise (as on iOS and the web). - Drawn at most 2x for the glow and 3x for the wave, as on iOS. - Touches are only observed on a hit area the size of the shape, so a parent `Pressable` still fires and touches beside the shape pass through. Hover ripples with a mouse or pen. - Accessibility follows native: "Voice agent" as the label and the state as the value; Reduce Motion is honoured. - Where native and web differ, this follows native (e.g. no web-only wave smoothing). **Native dictation:** - New `AIDictationOnLevel` event (no codegen spec change; it goes through the existing `NativeEventEmitter`). Android maps the recognizer's `onRmsChanged`; iOS measures the audio tap in dBFS, throttled to ~30 Hz, tuned on an iPhone for `.measurement` mode (-60 to -35 dBFS). - Fix: iOS dictation now remembers the `AVAudioSession` configuration it found and restores it when dictation ends. It used to leave the session in the `.record` category, which muted all playback in the app after the first dictation and kept other apps' audio ducked. **Sample:** `ExpoChatGPTSample` gets a full-screen voice mode (`VoiceGlow` for the agent, `VoiceWave` on the microphone level). It listens with dictation, sends the transcript, follows the agent's AI state, reads the reply aloud with `expo-speech` and listens again. The send path moved into a shared `useSendAIMessage`, and the demo backend is now called with `platform: 'openai'`, as in the React example (its Anthropic key currently fails `/summarize`). **Entry isolation:** everything lives under `src/ai-components`; `validate-entry-isolation` passes. ## 🎨 UI Changes Voice mode in `ExpoChatGPTSample`, on a physical iPhone 15 Pro Max and the iOS simulator. Screenshots and a recording to be added. ## 🧪 Testing - Unit tests: Android's 22 driver tests and the bounce tests ported; palette, input clamping, simplex noise against GPU readback values, hit areas, wave geometry, rendering, accessibility label and value, theme fallbacks, reduced motion, and the dictation level hook. `yarn test:unit`, `yarn lint`, `yarn typecheck` and `yarn build` pass. - The SkSL shader compiles and renders under CanvasKit, matching Android's Paparazzi snapshot. - iOS simulator: both visualizations render and animate, and the voice-mode loop runs (listening, thinking, speaking, back to listening). Speech recognition itself fails on iOS 27 simulators ("Failed to initialize recognizer"), so transcription was faked there. - Physical iPhone: dictation, the mic level driving the wave, the spoken reply and the full voice loop work. - Not yet tested: Android on a device (the Android level mapping is untuned), SampleApp. ## ☑️ Checklist - [ ] I have signed the [Stream CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform) (required) - [x] PR targets the `develop` branch - [x] Documentation is updated (AI components README; the React Native v9 AI integration docs are in a separate getstream.io PR) - [ ] New code is tested in main example apps, including all possible scenarios - [ ] SampleApp iOS and Android - [ ] Expo iOS and Android (iOS done, Android pending)
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 28, 2026
## [15.0.0-rc.3](v15.0.0-rc.2...v15.0.0-rc.3) (2026-10-28) ### ⚠ BREAKING CHANGES * channel.watch() resolves with the channel instance to use (the channel, or the instance that superseded it), not the server response: read channel.data, channel.state and channel.messagePaginator instead. - channel.watch() sends no query for a channel already watched and loaded unless it asks for options the current watch was not sent with, so it no longer refreshes one: refresh with channel.reload(), load other messages through channel.messagePaginator. - A channel.watch() while another with the same options is in flight joins it instead of sending a request. * **MessageList:** useMessageListScrollManager no longer takes onScrollToTop. Loading older messages keeps the message the reader was looking at in place, also when loading starts at the very top, instead of pinning the newly loaded page to the top. - InfiniteScrollPaginator's loadNextDebounceMs now rate-limits the edge check while scrolling (the first check runs at once, then at most once per loadNextDebounceMs) instead of waiting until scrolling stops for that long. Older and newer pages start loading before the reader reaches the edge. - MessageList and VirtualizedMessageList pass showNotification to NewMessageNotification only when a message from someone else arrives while the list is scrolled up. Being on an older message set no longer shows it; ScrollToLatestMessageButton signals that. - In a channel, ScrollToLatestMessageButton shows the current user's unread count from the channel's read state instead of counting message.new events, so messages a reconnect brought in are included. In a thread it counts new replies from 0. * marker: the removals are v14 API, so the branch has a `fix!:` commit with a `BREAKING CHANGE:` footer, per the AGENTS.md convention. The footer to keep in the squash commit: * `MessageList` no longer accepts `messages`, `hasMore`, `jumpToLatestMessage`, `loadingMoreNewer` or `messageLimit`, and `VirtualizedMessageList` no longer accepts `messages`; the lists render and paginate the message paginator. `CustomClasses.chatContainer` and the `ComponentContext.LoadingErrorIndicator` slot are removed. Inside `ChatView` the channel list no longer renders its own `str-chat <theme>` scope. See ai-docs/v15-breaking-changes.md (BC-019, BC-046, BC-047, BC-073, BC-074). Verification: `yarn lint`, `tsc -p tsconfig.lib.json --noEmit`, `yarn types:scripts`, `yarn validate-translations` and `yarn test` (281 files, 3090 tests) pass. `yarn build`, `validate-cjs`, `validate-esm`, `validate-entry-isolation`, and the tutorial and vite example builds passed before the rebase onto the guides branch, which brought in only docs and the api-surface script. The ai-chatbot example build was not verified: it hung locally in a sandboxed environment. The theme-scope change was not checked visually in a browser. ### UI Changes No intended visual change inside `ChatView`. Outside `ChatView`, the channel list picks up the design tokens again, as in v14. The `MessageList` loading indicator now appears while older messages load. No screenshots: the examples need Stream credentials to run. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Gallery items can display message-specific timestamp overrides. * Reply-count buttons honor a message’s custom thread-opening behavior. * Standalone channel lists now inherit the chat theme, and channel-detail actions use workspace navigation. * `useMessagePaginator` and `useIsCooldownActive` are available from the package entry point. * Message timestamps can render with an explicit message outside a message context. * **Bug Fixes** * Message-list loading indicators and pagination behavior now reflect active page loads more accurately. * Workspace navigation warns when navigation actions are used without a provider. * **Breaking Changes** * Removed legacy message-list props and the `LoadingErrorIndicator` component override slot. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ### Bug Fixes * correct issues found during the comment cleanup ([#3334](#3334)) ([7f87026](7f87026)) * **examples:** translate the voice visualization keys in the vite example ([#3352](#3352)) ([c88ab33](c88ab33)), closes [#3347](#3347) * **MediaRecorder:** stop the recording timer when the preview unmounts ([#3351](#3351)) ([3c8b30a](3c8b30a)) * **MessageList:** keep the reader's place on older pages and count only arrived messages as new ([#3342](#3342)) ([ab00d67](ab00d67)) * remove dead v15 surface and restore v14 behaviors found by the API audit ([#3328](#3328)) ([d721f40](d721f40)), closes [#3327](#3327) [#3305](#3305) [#3237](#3237) ### Features * **ai-components:** add useHeardText for voice-paced text ([#3348](#3348)) ([ec7c956](ec7c956)), closes [#3343](#3343) [#3344](#3344) [#3344](#3344) [#3343](#3343) [#3339](#3339) [#3347](#3347) * **ai-components:** add VoiceWave and VoiceGlow voice-agent visualizations ([#3347](#3347)) ([216d00d](216d00d)), closes [#3339](#3339) [#3339](#3339) [#3339](#3339) ### Refactors * call Channel.watch(), which replaces ensureWatched() in stream-chat ([#3346](#3346)) ([0100d98](0100d98))
6 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Android (GetStream/stream-chat-android#6848, GetStream/stream-chat-android#6849, GetStream/stream-chat-android#6851) and iOS (GetStream/stream-chat-swift#4338) added two visualizations of a voice agent to their AI components. This brings them to
stream-chat-react/ai-componentson v14, with the same motion, so the three platforms look and move alike, and shows them in a new voice mode in theai-chatbotsample.Implementation details
Public API, all from
stream-chat-react/ai-components:VoiceWave: a single string carrying a wave packet that erupts with speech, its colors flowing along it.VoiceGlow: a soft globe whose color masses orbit inside it and swell with speech.rendering?: 'auto' | 'css'(defaultauto).useAudioLevel(stream): a stable getter for the live level of aMediaStream(a microphone or a remote track), for theaudioLevelprop.VoiceAgentState,VoiceGlowRendering,VoiceVisualizationProps,VoiceWaveProps,VoiceGlowProps.Both components take
state(idle | connecting | listening | thinking | speaking),colors(one to five CSS colors, defaulting to the theme accent),expressivity(0 to 2) andaudioLevel(a getter read once a frame, so a changing level re-renders nothing;nullanimates speech on its own). Any otherdivprop goes to the root.Shared motion. A port of Android's driver: the per-state motion table, frame-rate-independent easing, reduced motion slowing time to 0.07x, the tap bounce and the HSV palette. The speech envelope runs the shader's simplex noise in float32, so the CPU-drawn wave and the GPU glow speak with the same rhythm as iOS.
VoiceWave is a Canvas2D port of Android's
WaveDrawing. Its line is 0.6% of the shorter side with Android's 1.5 device-pixel floor, drawn at the screen's density up to 3x as on iOS. In speech it is smoothed on the web: the crests spread out rather than steepen past a slope of 6, and the height rises quickly and settles slowly. While idle, connecting or thinking it is exactly the native wave, and it matches Android's Paparazzi snapshots down to antialiasing.VoiceGlow draws with a GLSL ES 1.00 port of the AGSL shader, at 1x (it is all soft gradients). Every glow on a page draws with one shared offscreen WebGL context, as iOS shares one renderer, and copies its frame onto its own 2D canvas. The shader compiles once, and a page with many glows stays under the browser's limit on live contexts. If the context is lost, every glow falls back to CSS and the next glow to mount starts a fresh context; the last glow to unmount releases it.
rendering='css', also the fallback where WebGL is unavailable or lost, is one CSS composition of the shader's glow: a color field seen through a mask riding the first mass's orbit, without the shader's fine marbling or its second, fainter riding mass. Its gradients are painted once and onlytransformandopacityanimate, so the browser's compositor runs the motion and it keeps moving while the main thread is busy. Off screen, its animations pause and its theme and audio reads stop. With GPU acceleration disabled, it holds 60 fps at about 18 to 22% CPU at 300px in headless Chrome.Behavior shared with iOS:
onClickstill fires.--str-chat__ai-voice-colorand--str-chat__ai-voice-backdrop, re-read about once a second.role="img"with a name such as "Voice agent: Speaking", translated in all 12 locales.useAudioLevelmeasures RMS on demand over the last 2048 samples (about 43 ms), so the level holds steady through a vowel. The audio graph is torn down when the stream changes and on unmount.Sample app.
ai-chatbotgets a voice mode: a full-panel overlay with the glow for the agent and the wave for the user's microphone. It usesuseSpeechToTextfor dictation andspeechSynthesisto read replies aloud, and returns to listening hands-free. A turn the agent is never seen working on ends after 20 seconds, with its answer if one came in. The send path moved into a shareduseSendAIMessage. The sample's README andimplementation.mddescribe it.Known differences from native:
Tests:
useAudioLevel.UI Changes
Screenshots and a recording of voice mode are to be added. In the meantime, the sample's voice mode shows both components: run
yarn build, thenyarn start:ai-chatbot, and start voice mode from the composer. Microphone access needs Chrome, Edge or Safari.