feat(desktop): add local voice input

[t3-code]

what changed

• adds desktop-only local voice input to the composer
• records 16 khz mono pcm with @picovoice/pvrecorder-node
• transcribes with the latest stable transcribe-cpp@0.2.1
• downloads Moonshine Streaming Tiny Q8 on first use after confirmation
• verifies the 48 MiB model with its pinned size and sha-256 before an atomic rename
• inserts the transcript at the current composer cursor without submitting
• adds escape cancellation, a two-minute recording cap, level feedback, and model removal in Settings
• packages native dependencies outside asar and adds the macOS microphone description and entitlement

why

this gives the desktop app private speech-to-text without a cloud service or persisted microphone audio. the implementation borrows the useful parts of pi-transcribe: native 16 khz capture, loading the model while recording, verified Hugging Face downloads, one active transcription job, and deterministic cleanup.

web and mobile remain unchanged because their microphone and local inference paths need separate implementations. Windows arm64 reports unsupported because transcribe.cpp does not publish a matching native Node package yet.

ui changes

after

this is a new control, so there is no equivalent before state.

verification

• contracts: 271 tests passed
• desktop: 583 tests passed
• web: 2,744 tests passed
• focused speech and packaging suite: 57 tests passed
• contracts, web, and desktop typechecks passed
• formatting and lint passed, with only existing repository warnings
• web and desktop production builds passed
• Electron 41 loaded both native packages successfully
• real transcribe-cpp@0.2.1 inference transcribed the 11-second JFK fixture in 171 ms on CPU
• real Electron app launched and rendered the microphone control

not run locally: the final AppImage artifact build, because this environment does not have Cargo. packaging configuration tests passed, and CI remains the cross-platform packaging check.

request provenance

• requested by: @UtkarshUsername

---

Note

Medium Risk
Introduces native microphone capture, model download/verification, and asar-unpacked native modules plus macOS mic entitlements. Packaging or native-load failures would break desktop builds or voice input, not core auth.

Overview
Adds local voice input to the desktop composer: record, transcribe on-device, and insert text at the cursor without sending.

The Electron main process captures 16 kHz PCM (pvrecorder-node), downloads Moonshine Streaming Tiny (~48 MiB) from Hugging Face on first use after confirmation, verifies size/sha256, then transcribes with transcribe-cpp. Audio stays in memory; Windows arm64 is unsupported. IPC exposes start/stop/cancel/status/remove-model plus live events (level, progress, transcript).

The composer shows a mic control with level feedback, Escape/cancel, and a 2-minute cap. Settings → General can remove the model. Packaging unpacks the native modules from asar and adds macOS microphone usage text and the audio-input entitlement.

^{Reviewed by Cursor Bugbot for commit 2956e79. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add local on-device voice input to desktop app

• Adds DesktopSpeech context service in the main process that orchestrates microphone capture via @picovoice/pvrecorder-node and transcription via transcribe-cpp, with state machine covering missing-model, downloading, ready, recording, transcribing, and error states
• Model download is streamed to a .part file and verified by size and SHA256 before atomic rename; removeSpeechModel lets users delete the model from disk
• Exposes speech controls (getStatus, start, stop, cancel, removeModel) and event forwarding over new IPC channels; renderer gets window.desktopBridge.speech through preload.ts
• Adds ComposerSpeechButton in ChatComposer.tsx for start/stop/cancel with live level metering, and a settings row for model state and removal; useDesktopSpeechInput hook manages recording ownership, auto-stop at 120s, and Escape-to-cancel
• Build config unpacks native modules (transcribe-cpp, koffi, pvrecorder-node) from asar and adds macOS NSMicrophoneUsageDescription plus com.apple.security.device.audio-input entitlement
• Risk: native koffi build is now allowed via pnpm-workspace.yaml; Windows arm64 is unsupported per voice-input.md. Reviewers should verify asarUnpack globs in build-desktop-artifact.ts cover the installed native package paths

^{Macroscope summarized 2956e79.}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	13.4 KiB	13.4 KiB	−10 B (−0.1%)	15.1 KiB	✅
Codex	Thread snapshot wire	6.9 KiB	6.9 KiB	0 B (0.0%)	7.3 KiB	✅
Codex	Live turn WebSocket wire	6.5 KiB	6.5 KiB	−10 B (−0.1%)	7.8 KiB	✅
Codex	Live turn WebSocket decoded	55.0 KiB	55.0 KiB	0 B (0.0%)	66.4 KiB	✅
Codex	Live turn messages	16	16	0 (0.0%)	21	✅
Claude	Total thread wire	13.4 KiB	13.5 KiB	+20 B (+0.1%)	15.1 KiB	✅
Claude	Thread snapshot wire	6.9 KiB	6.9 KiB	+3 B (+0.0%)	7.3 KiB	✅
Claude	Live turn WebSocket wire	6.6 KiB	6.6 KiB	+17 B (+0.3%)	7.8 KiB	✅
Claude	Live turn WebSocket decoded	55.8 KiB	55.8 KiB	0 B (0.0%)	66.4 KiB	✅
Claude	Live turn messages	16	16	0 (0.0%)	21	✅

Baseline: b1670ac · PR result: 2956e79 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 109.4 KiB
• Claude decoded thread snapshot: 110.1 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[macroscopeapp]

Reviewed the new speech subsystem against the Effect service conventions. The desktop-side speech feature is wired as a module-global Promise singleton rather than an Effect service, so its dependencies and lifecycle are invisible to the runtime. Three findings, all in apps/desktop/src/ipc/methods/speech.ts.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Reviewed the new web UI surfaces (ComposerSpeechButton, the composer wiring, the General settings row, and the speech hook) against the shared primitives and interaction conventions. Five findings, all in changed lines: a tooltip that cannot open in the states it describes, a locally rebuilt spinner, an unwired cancel affordance, an app-wide Escape capture, and a settings row missing from the search catalog.

^{Posted via Macroscope — UI Consistency}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR introduces a new native microphone and local-transcription workflow, including model downloads, persistent storage, IPC, UI behavior, native packaging, and macOS microphone permissions. Unresolved findings also raise risks around cancellation during model download, model-removal failures, and redirect handling for the model URL.

Not approved because:

• 2 blocking correctness issues found at or above your repo's Minimum Blocking Severity

^{Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.}

[macroscopeapp]

🟠 High speech/useDesktopSpeechInput.ts:38

Unmounting during the first-use model download leaves the desktop operation running, so the pending bridge.start() can begin microphone capture after this hook has been disposed; recording then continues without its UI until the timeout. Because cleanup only calls bridge.cancel() when activeRef.current is already true, it misses this in-flight start. Track and cancel the pending start/download through the bridge’s cancellation support so unmount cannot begin capture later.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @apps/web/src/speech/useDesktopSpeechInput.ts around line 38:

Unmounting during the first-use model download leaves the desktop operation running, so the pending `bridge.start()` can begin microphone capture after this hook has been disposed; recording then continues without its UI until the timeout. Because cleanup only calls `bridge.cancel()` when `activeRef.current` is already `true`, it misses this in-flight start. Track and cancel the pending start/download through the bridge’s cancellation support so unmount cannot begin capture later.

[macroscopeapp]

🟡 Medium settings/SettingsPanels.tsx:1938

Clicking “Remove model” produces an unhandled promise rejection when speech.removeModel() fails, so the model stays installed and the user receives no error feedback. Handle the rejection and show an error toast instead of only chaining .then(setSpeechStatus).

-                    void window.desktopBridge?.speech?.removeModel().then(setSpeechStatus)
+                    void window.desktopBridge?.speech
+                      ?.removeModel()
+                      .then(setSpeechStatus)
+                      .catch((error: unknown) => {
+                        toastManager.add(
+                          stackedThreadToast({
+                            type: "error",
+                            title: "Could not remove voice model",
+                            description:
+                              error instanceof Error ? error.message : "Voice model removal failed.",
+                          }),
+                        );
+                      })

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @apps/web/src/components/settings/SettingsPanels.tsx around line 1938:

Clicking “Remove model” produces an unhandled promise rejection when `speech.removeModel()` fails, so the model stays installed and the user receives no error feedback. Handle the rejection and show an error toast instead of only chaining `.then(setSpeechStatus)`.

[macroscopeapp]

UI consistency review of the web-side voice input additions. The five findings from the previous run (tooltip-blocking disabled, ad-hoc spinner, missing cancel affordance, Escape swallowing, missing settings-search entry) are all resolved. Two smaller issues remain in ComposerSpeechButton.

^{Posted via Macroscope — UI Consistency}

[macroscopeapp]

One convention finding on the new DesktopSpeech service: the model directory is read from the Electron app singleton inside make instead of being acquired from the Effect environment. Everything else (inline Context.Service interface, make + layer, typed DesktopSpeechOperationError with structural attributes and preserved cause, IPC handlers acquiring DesktopSpeech from the environment, subscribe mirroring PreviewManager.subscribeStateChanges) matches the repository's Effect service conventions.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

One finding on the new composer voice-input control: the recording level pulse has no reduced-motion path. Everything else (aria-disabled + click guard instead of disabled, the shared Spinner, the discard affordance, the settings-search catalog entry) now matches the repo's primitives and conventions.

^{Posted via Macroscope — UI Consistency}

[macroscopeapp]

One finding: the speech model downloader reaches for global fetch instead of the HttpClient service, so the network dependency stays invisible to DesktopSpeech.layer. Everything else (service tag with inline interface, make/layer, environment-acquired DesktopAppIdentity/HostProcess*, Schema.TaggedErrorClass with structural attributes, scoped subscribe, typed IPC handlers) matches the conventions.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Two consistency findings in the new web UI. The earlier notes on the recording-state stop button and the level overlay scale look addressed at this head.

^{Posted via Macroscope — UI Consistency}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7c038f5. Configure here.}

[cursor]

Model download skips redirects

High Severity

requestModel uses Undici-backed HttpClient.get without HttpClient.followRedirects(). That replaces the prior fetch call that set redirect: "follow". Hugging Face /resolve/ URLs return a 302 to the CDN, so filterStatusOk rejects the redirect and first-use model download fails.

 

^{Reviewed by Cursor Bugbot for commit 7c038f5. Configure here.}

[macroscopeapp]

One finding: the recording level overlay's reduced-motion opt-out does not actually stop the motion, because the animated value is an inline style.

^{Posted via Macroscope — UI Consistency}

[macroscopeapp]

One interaction-consistency finding on the new composer control; everything else (variant/size choices, aria-disabled + click guard, Spinner primitive, reduced-motion opt-out, settings row + search catalog entry) now matches the shared component system.

^{Posted via Macroscope — UI Consistency}

[t3dotgg]

Note

🤖 GPT-5.6 Sol responding on behalf of Theo

We're closing this PR as we clean up the T3 Code backlog. Thank you for taking the time to put this together.

This combines microphone capture, local speech recognition, model downloads, native IPC, desktop settings, and composer insertion across 30 files. Voice input needs a clear native dependency, privacy, download, and support plan before we accept an implementation.

If you believe we closed this in error, please reopen the PR and leave a comment explaining what we missed.