Repository navigation
fix: skip audio with no detected speech before Whisper, and drop output that only repeats the hints - #194
Conversation
Model::capabilities() now returns a Result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Downloads and verification now work on a GGUF artifact, so the speech detector reuses the same resumable, checksummed path. Its download is best effort and never fails the model download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loads the pinned Silero VAD on the CPU on first use, answers whether 16 kHz audio contains speech, and returns None while the detector is missing or broken so callers keep their level-only silence check. A missing file starts one background download per process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capture above the -50 dBFS level gate now also goes through the Silero speech detector when it is installed; a key click, the start tone or a noisy room no longer reaches the model and comes back as an invented phrase. Without the detector the gate works as before. Preparation prewarms an installed detector and never downloads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rolling transcription, the Stop & Transcribe fallback and meeting echo now ask the Silero speech detector before each chunk. A chunk without speech becomes an empty transcript without inference, and accepting it still advances the track's coverage, so the draft and echo suppression keep moving. On a 63-minute call the detector rejected all 152 invented microphone entries and kept 32 of 33 real ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion hints Whisper reads the hints as its initial prompt and returns them verbatim on silence or noise. Segments equal to the hints after normalization are removed together with their words, so meetings do not rebuild them into sentences; speech that contains a hint is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep a single recognition hint: the echo filter needs at least two hints, and commas, semicolons and line breaks now separate words. - Lower the Silero threshold to 0.4 so a whisper at -43 dBFS counts as speech. - Log audio_ms and vad_ms at debug level when a dictation passes the detector. - Download the detector in the background when a model downloads or activates, so an installed model activates at once offline. - Fix documentation drift (local API, Linux stop tone, prewarm, figures). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: - Cargo.toml, Cargo.lock: dev's dependencies, with transcribe-cpp =0.4.0. - dictation.rs: dev's near-silent evidence log, then the speech detector. - gguf_session.rs: dev's job cancel token, then the prompt-echo filter. - meeting_transcript.rs: dev's quitting check, then the speechless check. The speechless tests use Chunk::tail and settled_ms. - meetings.md, AGENTS.md, main.rs: both sides. transcription_benchmark.rs boxes Parakeet: with dev's new fields and transcribe-cpp 0.4.0 it trips clippy::large_enum_variant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dev now pins transcribe-cpp =0.4.1 (anomalyco#217), so the branch takes dev's Cargo.toml and Cargo.lock and no longer changes the dependency. - gguf_session.rs: dev's keep_repetition_partial and capabilities check, then the prompt-echo filter. - speech_detector.rs: ModelOptions gained no_mmap in 0.4.1; the detector keeps the default (mapped), as dev's model loads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thdxr
left a comment
There was a problem hiding this comment.
Thanks @0xCaso, this is a careful, well-measured fix, and the write-up made it easy to review. I reviewed it against current dev and tested it on Linux (apollo: Ryzen AI Max+ 395, Radeon 8060S, Vulkan).
What I checked
- Vulkan shader trimming: no problem. The detector loads with
Backend::Cpuand is F32, so it never touches theGGML_VULKAN_SHADER_TYPESsubset. - Binary size: release
hexon Linux grows from 47,080,224 to 47,085,728 bytes (+5.5 KB). Downloading the 1.2 MB model (pinned revision, size + SHA-256, same receipt/promote path as the transcription models) is the right call over bundling it. - Latency: the real detector takes about 2–3 ms for 3–4.5 s clips, 92 ms for 150 s and 155 ms for a 5-minute hold, on CPU. It runs in the finish step after the level gate, not in the audio callback: the Linux audio channel is unbounded, and on macOS the recognition thread is downstream of the audio owner. So capture is never blocked, and normal dictations add a few ms.
- Real model, real clips (Whisper large-v3-turbo on Vulkan, hints
T3 Code, Ottobit, Hex):- digital silence: Whisper returned
www.t3.org(a hint echo); the level gate already drops it. - brown noise above the -50 dBFS gate: detector says no speech, so it's discarded.
- normal speech, and
say -v Whisperat a loudest 30 ms window of -40 and -44 dBFS: detector hears it, and Whisper transcribes it correctly. - the same whisper at -48 dBFS: the level gate keeps it and Whisper transcribes it correctly, but the detector rejects it. So there's a narrow band (about -45 to -50 dBFS loudest window) where very quiet whispers are now dropped. I think that's an acceptable trade for removing the hallucinations, and it matches your 0.4 threshold reasoning. Just flagging it as a known limit.
- your three ignored real-detector tests pass with these clips.
- digital silence: Whisper returned
- All models vs Whisper only: I'd keep it for all models. Parakeet/Cohere also emit "thank you"-style text on noise sometimes (#108), and there's one simple policy. The prompt-echo filter is correctly Whisper-only.
- Meetings: the check in
ChunkModel::transcribebefore engine use, with coverage still advancing, looks right. Tests cover the overlap case.
Small fixes I pushed to your branch (maintainer edits; hope that's OK):
- Linux clippy failed with
-D warnings. On Linux,is_verified,verification_receipt_matches,write_verification_receipt,verify_file_with_cancelandspeech_detector::prewarmbecame dead code once downloads went throughartifact_receipt_matches. I gated the receipt helpers likeverify_installed. I also callspeech_detector::prewarm()when the Linux listener starts, so Linux loads the detector at launch instead of in the first dictation's finish. - Retry after a failed download. The PR description says the next dictation retries a failed download, but
DOWNLOAD_STARTEDwas never reset, so it only retried after a restart. The download thread now clears it on failure. I updated the docs to match. - Removed
speech_detector_cost_is_logged_when_speech_passes. It failed once in a full parallelcargo testrun on Linux: the tracing callsite interest races other tests even withrebuild_interest_cache. It only asserted a log line, so I dropped it rather than keep a flaky test.
After merging current dev, Linux fmt, clippy (capture,onnx-benchmark, -D warnings), Windows x64 cross-clippy and cargo test are all clean for this change. The only failures are every_persisted_setting_has_a_cli_key and the two Rune golden tests, and those also fail on dev itself (from #303), not from this PR.
Your remaining questions:
- Download vs bundle: download, as you did.
- Background download vs "never silently download": that rule covers menu-bar model selections. The detector is a small companion to a model the user chose to download. It's fine as is, and nothing blocks on it.
- THIRD_PARTY_NOTICES: not needed for a downloaded model.
CI on the new head agrees: fmt and clippy pass on macOS and Windows, and the only test failures are those same dev tests. Approving and squash-merging.
— from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊
Fixes #193.
What changes
VadSessionfromtranscribe-cpp0.4.1, whichdevpins since perf(transcription): bump transcribe.cpp to 0.4.1, cutting model memory from ~900 MB to ~160 MB #217.handy-computer/silero-vad-v6.2-ggufat revisiona9b2d61, with size and SHA-256 checks. It downloads in the background, so a model that is already installed still activates at once when the computer is offline. If the download fails, dictation and meetings keep their current behavior, and the next dictation or meeting chunk starts it again. Preparation at launch (installed_only) never downloads it.speech_detectormodule. It loads the detector once, on the CPU: about 65 ms on CPU, against 7.6 s for the first Metal start on my M1 Pro.contains_speechreturnsNonewhile the detector is missing or broken, and then every caller keeps its current behavior.process.logrecordsaudio_msandvad_ms(at debug level when the capture passes).meeting echoskip chunks with no speech. Sincec0ed5c7chunks run on the dictation model, so the check runs inChunkModel::transcribe, before the model is used. It comes afterdev's -60 dB silent-chunk check, which room and keyboard noise passes. A skipped chunk still advances the track's coverage, so the draft and echo suppression keep moving.The local API's programmatic dictation stays ungated, as before. Only the prompt-echo filter applies there, because it runs in the shared GGUF session.
Results on my recording
Hardware: one 63-minute Italian call, M1 Pro, Whisper large-v3-turbo with hints. Microphone entries were classified by hand.
main(transcript.ndjsonfrom 2.3.0)transcribe-cpp0.4.0 (before #217), threshold 0.4,min_speech_ms160meeting echo --transcribe(real chunk path)main. None of them is a closing phrase, the hints, or a fragment on quiet audio.Tests
dev6757e12:scripts/check(fmt, clippy withcapture,onnx-benchmarkand-D warnings, 1565 tests passed),git diff --check. The three real-model tests below pass on 0.4.1.speech_detector_discards_loud_captures_without_speech,speechless_chunks_skip_the_model_and_still_advance_coverage,a_speechless_chunk_after_an_overlap_keeps_earlier_entries,a_speechless_chunk_skips_the_shared_model,speech_detector_cost_is_logged_when_speech_passes,prompt_echo_is_dropped,speech_containing_the_hints_is_kept,only_the_echo_segment_is_removed,single_hint_dictation_is_kept,hints_without_spaces_after_commas_are_matched,a_corrupt_detector_file_is_not_loaded,an_installed_model_does_not_wait_for_the_speech_detector, and artifact tests intranscription_models.real_detector_rejects_silence_and_noise,real_detector_hears_speechandreal_detector_hears_quiet_whispers(HEX_VAD_MODEL,HEX_VAD_SPEECH_WAV,HEX_VAD_WHISPER_WAV). The whisper clip is macOSsay -v Whisperat -43 dBFS.hex listen, Whisper large-v3-turbo, Italian, with hints): a spoken sentence, a whispered sentence and a lone "Ok" were pasted. Three 1.5–3.3 s presses without speech, at -48 dBFS (so above the -50 dBFS gate), were discarded by the detector in 15–31 ms (vad_ms).clippy -D warningsalone.Not in this PR
Questions
THIRD_PARTY_NOTICES.mdlists only vendored assets and no downloaded model, so I did not add it there. Tell me if you want a section.🤖 Generated with Claude Code