feat(voice): talk to existing threads with GPT-Live

[quentinsteinke]

What Changed

Adds experimental GPT-Live voice conversation to an existing thread on web, desktop, and native mobile. Users can speak with GPT-Live, send coding requests to the thread's selected agent, hear agent updates, and read captions during the call.

The environment server creates the Live session using its OPENAI_API_KEY and handles client delegation through the existing thread command path. Audio travels directly between the client and OpenAI over WebRTC. The API key stays on the server, and voice uses separate OpenAI API billing.

Calls stay attached to one environment and thread. Mute pauses microphone input; End call releases voice resources while accepted coding work continues. Navigation, backgrounding, or connection loss ends the call. Existing provider approvals remain in the thread. Mobile microphone ownership prevents overlapping Live capture and dictation.

Why

T3 already lets users direct their coding agents across devices. This adds a spoken interface to the same thread without adding voice methods to each provider adapter or replacing existing dictation.

Unlike #8324, GPT-Live manages the conversation and delegates to T3's existing command flow; this does not negotiate Codex-native realtime sessions or require a Codex-specific voice protocol. Related prior voice proposals include #6206 and #3997. This is still an experimental feature contribution; the different implementation does not assume maintainer approval of the product scope.

Verification

• Real GPT-Live WebRTC call with prerecorded synthetic input: connected, transcribed speech, answered from recent thread context, delegated a coding request to the existing thread, and returned spoken progress/completion with captions. The coding agent created the requested fixture file with the expected contents. Mute and end-call controls were exercised. Temporary synthetic-input plumbing was removed before committing.
• Fixed an integration failure where the authenticated WebSocket layer disposed voice handlers before the connection ran. A regression test now exercises that real HTTP/WebSocket route.
• Server protocol/session/authorization tests: 33 passed. Authenticated WebSocket lifecycle regression: 1 passed.
• Shared controller: 15 passed. Contracts: 2 passed. Browser transport/hooks: 16 passed. Native transport: 6 passed; existing mobile dictation tests: 16 passed.
• Scoped server, web, mobile, and shared-client typechecks passed. Targeted formatting/lint passed. No repository-wide checks were run.
• iOS Debug Simulator build succeeded using Xcode 27 beta. The resulting app installed on the dedicated iOS 18 Simulator, and Metro produced its iOS JavaScript bundle successfully.
• Local browser flow exercised; private Tailscale HTTPS pairing was exercised from another machine.

Remaining draft checks:

• Physical microphone capture and audible playback are not verified. The browser integration test used synthetic audio; hardware capture stalled in the automation environment.
• Native iOS runtime is not verified: Simulator migration failed, the tooling issued overlapping launch requests, and subsequent Simulator queries hung. The process exited before the app UI could be verified; available logs do not establish an app crash. Android and packaged Electron runtime have not been tested.
• Relay/tunnel behavior has not been exercised.
• Browser screenshots have been captured locally, but GitHub attachment upload requires signing into Safari Technology Preview; Chrome automation could not attach. Native before/after images and an interaction video are still outstanding. No PR-only assets are committed.

Focused server commands (from apps/server):

vp test run src/voice/liveProtocol.test.ts src/voice/VoiceSession.test.ts src/auth/RpcAuthorization.test.ts
vp test run src/server.test.ts -t 'keeps voice handlers alive after the authenticated WebSocket upgrade'

Checklist

• This PR is small and focused
• I explained what changed and why
• I included before/after screenshots for any UI changes
• I included a video for animation/interaction changes

Built with GPT-6 in the Codex harness.

[mujuni88]

Hi @quentinsteinke, I updated the voice UI and rebased the branch onto current main.

Key changes:

• Voice transcript is now in a collapsible panel above the composer.
• Users can still type and use model/access controls during voice mode.
• Replaced the modal controls with a smaller mic/equalizer/end control.
• The control changes based on who is speaking, idle, or muted.
• Added 94 focused tests and all affected typechecks pass.

Branch: https://github.com/mujuni88/t3code/tree/jbuza/voice-inline-prototypes

If this direction looks good, please pull it into feat/gpt-live-voice.

[t3dotgg]

Note

🤖 GPT-6 Astra responding on behalf of Theo

We're closing this as part of a cleanup of stale PRs. Thank you for the time and care you put into this contribution.

A local merge check found conflicts in 8 files against the current target branch. The latest implementation commit is from 2026-09-14, and there has been no contribution update in over a week.

At this point, regenerating a focused patch against the current intended base would be easier to review and maintain than updating this PR. We appreciate the work and will keep this thread as a reference. If you revisit it, please check the current direction with us and link the fresh PR back here.