Skip to content

feat: Hex 2.2.0 — faster dictation and the Meetings beta - #133

Merged
kitlangton merged 36 commits into
mainfrom
feat/hex-voice-meetings
Oct 16, 2026
Merged

kitlangton merged 36 commits into
mainfrom
feat/hex-voice-meetings

Conversation

@kitlangton

@kitlangton kitlangton commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Long dictations were slow on a busy Mac: the Parakeet decoder ran 8 spin-waiting CPU threads, so under build load a 30 s dictation could take 12.8 s to transcribe, and nothing was transcribed until release. Meetings existed only as a debug prototype, with echo duplicating remote speech as You on speakers and no way for agents to read past calls. This PR ships both as Hex 2.2.0.

What Changes

Faster dictation

  • Decoder threads are bounded to 1 on GPU sessions, and the inference thread runs at user-interactive QoS. Benchmark under synthetic load: 30 s clip 12.8 s → 0.7 s, 90 s clip 17 s → 1.8 s.
  • Transcribe during pauses (Settings › Behavior, on by default): completed speech is transcribed in the background at pauses; release transcribes only the tail and pastes once. Any missing or stale piece falls back to full-clip transcription. Release-to-text on 77–99 s clips: 0.7–3.3 s → 0.05–0.2 s.

Dictation accuracy

  • Capture keeps 200 ms of audio after the physical release, so the last word isn't clipped. The release boundary itself stays exact.
  • A final piece shorter than 3 s is transcribed together with the piece before it, so a lone word after a pause keeps its context ("…an amazing world" instead of "…an amazing").
  • Piece joins drop false periods at mid-clause pause seams (after an article, "of", a hesitation, or a repeated word) and lowercase the continuing word, instead of splitting one sentence in two.
  • Near-silent tails skip inference, and near-silent captures are discarded instead of hallucinating "thank you" (Inserting 'thank you' at the end of dictation sometimes #108).

Fixes

Meetings (beta, opt-in)

  • Settings toggle with Screen Recording health; a focus-free offer capsule for Zoom, Meet, Slack, Teams, and FaceTime calls; a menu-bar recording timer; a keep/stop prompt when the call app releases the mic. Recording never starts automatically.
  • Rolling local transcription during the call, in overlapping chunks, with echo suppression by sentence plus a mic-vs-system energy check.
  • Apple Intelligence (on-device) writes a provisional title while recording, then a final title and a summary of at most two sentences.
  • Redesigned Meetings pane: search, day groups, notes first, a You/Them conversation view, live tail, rename, delete to Trash, and Markdown export to ~/Documents/Hex Meetings.
  • hex mcp: a read-only stdio MCP server (list_meetings, search_transcripts, get_transcript, get_meeting_notes, Markdown resources). No writes, no network, no model calls. scripts/sync-meetings-mirror.sh mirrors another Mac's meetings for local MCP clients.

Scope

This branch combines perf/decoder-threads, perf/incremental-transcription, fix/triage-batch, test/dictation-corpus, and the meetings stack. Semantic transcript search, calendar titles, and the native macOS shell (docs/plans/native-macos-app.md) are follow-ups. The version bump to 2.2.0 (build 20200) lands separately on main as the release commit.

Verification

cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test
git diff --check
  • macOS cargo test: 584 passed, 13 ignored, plus the stdio MCP smoke test. Linux CI (test and nix) on this PR.
  • Speed numbers come from a transcribe.cpp harness and an opt-in real-model test on the Mac Studio, not from live microphone capture. The release-boundary and short-final-piece results come from the dictation corpus harness (scripts/dictation-corpus.sh).
  • Echo suppression was tuned on a real 17-minute speaker call: mic twins in the live view 105 → 2, with all 67 genuine You lines kept.
  • Kit used a signed build of this branch on his MacBook Pro before approving the release. hex mcp was exercised over SSH from an OpenCode client: initialize 0.33 s, list_meetings 18 ms, search_transcripts 12 ms.
  • Not individually verified: Dvorak – QWERTY ⌘ paste and clipboard-manager skipping.

GGUF sessions on GPU devices now request one CPU thread instead of
transcribe.cpp's default min(8, ncpu). Parakeet's per-token TDT decoder
runs on CPU even with Metal encoding, and ggml's spin-waiting worker pool
stalls it under CPU load. CPU-only devices (Linux Backend::Auto fallback)
keep the library default so the encoder is not serialized.

The macOS inference thread moves from user-initiated to user-interactive
QoS. ggml creates workers with plain pthread_create, which starts them at
QOS_CLASS_DEFAULT rather than inheriting (confirmed with a probe); with one
thread no workers are created and decoding stays on this thread.

transcribe-cpp 0.2.4 bench, Unified English Q8, Metal, Mac Studio (18
cores, already loaded by other work), user-interactive QoS, median ms for
3/30/90 s clips:

  + 36 `yes` procs: 1 thread 170/703/1819, 2 threads 136/1907/4940,
                    4 threads 178/4304/13519
  no extra load:    1 thread 83/364/1631,  2 threads 64/705/2889

Not verified physically on the MacBook Pro or on Linux.
Clipboard managers such as Maccy and Raycast recorded every dictation
because the paste write carried only plain text. Publish the transcript
as one pasteboard item that also carries org.nspasteboard.TransientType
and org.nspasteboard.AutoGeneratedType so conforming managers skip it.
Generation-safe restoration of the previous clipboard is unchanged.

Refs kitlangton/Hex#223
On "Dvorak – QWERTY ⌘" the plain layer puts V on keycode 47, but with
Command held that key is ⌘. — so synthetic ⌘V posted ⌘. and paste beeped.
Build a second snapshot map with UCKeyTranslate's cmdKey modifier state
during initialize_layout and resolve Command shortcuts against it first,
falling back to the plain snapshot. Headless lookups do the same.

Refs kitlangton/Hex#162
Holding the shortcut without speaking made Cohere (and Whisper-style
models) paste a hallucinated "thank you". Measure the loudest 30 ms
window of the captured 16 kHz audio, DC removed and excluding model
padding, and discard captures below -50 dBFS as empty: no inference,
paste, History, or last-result update. The level and threshold are
logged. The threshold is conservative so whispered speech is kept.

Covers macOS hotkey, voice-delimited, and Voice Action captures and the
Linux hotkey loop. Local API dictation and host-audio transcription
deliberately keep returning raw model output.

Refs #108
With Release when idle, the first hold after minutes of idle failed with
the error tone. The cold stream reported "A buffer underrun or overrun
occurred" — cpal 0.18's ErrorKind::Xrun from CoreAudio's processor
overload listener, which does not stop the stream — and HEX treated it as
fatal: the owner closed the device without retrying and recognition
suspended the still-held gesture.

- Log and ignore Xrun and RealtimeDenied stream errors; real sample gaps
  still fail the capture as discontinuities.
- In release-when-idle mode, a stream failure before the hold threshold
  returns the press to pending and reopens, so release-before-ready
  discards and readiness captures from the original press. Later
  failures still fail the capture rather than paste a fragment.
- Cold opens retry the selected microphone twice, 100 ms apart, before
  the existing automatic fallback.

Refs #130
…e-perf-batch

# ------------------------ >8 ------------------------
# Do not modify or remove the line above.
# Everything below it will be ignored.
#
# Conflicts:
#	src/dictation.rs
…ppression

Stream each track through a 16 kHz resampler into pause-aware chunks
(first pause after 20 s, hard cut at 30 s with a 2 s overlap) and
transcribe them while recording on a utility-QoS session that waits for
dictation capture and inference. Tracks that fall behind are rebuilt from
their WAV after Stop. Seams drop repeated words, microphone utterances
that echo overlapping remote speech are removed, and each finished meeting
is written as Markdown to the configured folder. Remote speech is labeled
Them.
…ding

Meetings (beta) in Settings enables detection, the Meetings pane, and
meeting paste in release builds, with a Screen Recording permission row
and a transcript folder. Offers appear as a focus-free capsule with a
countdown and stay snoozed for the call. The menu bar shows elapsed
recording time with Stop & Transcribe, and a capsule asks Keep Recording
or Stop & Transcribe when the call app leaves the microphone.
…Intelligence

Add a weak-linked FoundationModels Swift bridge (availability, context
size, guided-generation notes at utility priority) and a meeting notes
worker that runs after final publication: bounded map/reduce chunking
sized from the model context, a total deadline, manifest storage, and an
in-place marker-delimited Markdown update that keeps the file name.
Settings gains 'Summarize with Apple Intelligence' (default on) with a
one-line availability; the Meetings pane and ready capsule show the
generated title, notes, and a Regenerate Notes action. Debug CLI:
hex meeting summarize <id|file>.
…eed detection

Rolling chunks fell behind (packet-bounded queue) and the pane then showed the raw
Moonshine draft without any echo suppression, so microphone twins of remote speech
appeared as You. Coalesce rolling audio into one-second blocks, keep transcribed
entries of a track that fell behind, judge the live draft with the same echo rules,
split timed words into sentences, and classify microphone segments by comparing
their energy with delayed system audio so misheard bleed is dropped and double-talk
is kept.
…earch, rename, and delete

List rows show the generated title, time, length, app, and a status chip, grouped
by day. The detail leads with notes, then Them/You turns on opposite sides that
follow the live tail, under one action row: Copy Notes, Copy Transcript, Reveal
Markdown, Regenerate Notes, Rename, and a two-step Delete that moves the Markdown
and recording to the Trash. Adds a deterministic meetings preview.
… summaries

Name a recording from its opening (~90 s of speech or 600 words) with one
title-only Apple Intelligence call, refreshed once at ten minutes, shown in
the menu bar, Meetings pane, manifest, and exported file. After publication,
write the final title and a summary of at most two sentences with one call
over a bounded excerpt (opening, sampled middle, ending) instead of
map-reduce. Remove action items, decisions, and chunking; older manifests
still read. get_meeting_notes now returns the title and summary and points
agents at the transcript.
One periodic rsync (scripts/sync-meetings-mirror.sh, LaunchAgent template)
replaces an SSH session per MCP client, which exhausted the Pro's sshd.
A copy of the binary named hex-meetings-mcp always runs hex mcp.
…te the instance-lock test

Add a resampled delayed-release regression test (44.1/48 kHz, segmented and
not) and an opt-in Unified English release-boundary harness. The instance
lock test now uses a temporary directory instead of the real Application
Support folder, which accumulated test-<pid>.lock files.
@kitlangton kitlangton changed the title feat: faster dictation, meeting notes, and a meetings MCP server feat: Hex 2.2.0 — faster dictation and the Meetings beta Oct 16, 2026
@kitlangton
kitlangton marked this pull request as ready for review October 16, 2026 16:59
@kitlangton
kitlangton merged commit df2edf8 into main Oct 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant