Skip to content

fix: skip audio with no detected speech before Whisper, and drop output that only repeats the hints - #194

Merged
thdxr merged 13 commits into
anomalyco:devfrom
0xCaso:fix/vad-silence
Oct 28, 2026
Merged

thdxr merged 13 commits into
anomalyco:devfrom
0xCaso:fix/vad-silence

Conversation

@0xCaso

@0xCaso 0xCaso commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #193.

What changes

  1. No dependency change. The detector uses VadSession from transcribe-cpp 0.4.1, which dev pins since perf(transcription): bump transcribe.cpp to 0.4.1, cutting model memory from ~900 MB to ~160 MB #217.
  2. The speech detector downloads in the background with every transcription model. The file is the pinned Silero v6.2 GGUF (1.2 MB) from handy-computer/silero-vad-v6.2-gguf at revision a9b2d61, with size and SHA-256 checks. It downloads in the background, so a model that is already installed still activates at once when the computer is offline. If the download fails, dictation and meetings keep their current behavior, and the next dictation or meeting chunk starts it again. Preparation at launch (installed_only) never downloads it.
  3. speech_detector module. It loads the detector once, on the CPU: about 65 ms on CPU, against 7.6 s for the first Metal start on my M1 Pro. contains_speech returns None while the detector is missing or broken, and then every caller keeps its current behavior.
  4. Dictation. After the existing -50 dBFS level gate, a capture with no speech of at least 160 ms is discarded. The speech threshold is 0.4, not Silero's 0.5: at 0.5 a whisper at -43 dBFS, which the level gate keeps on purpose, was not heard. process.log records audio_ms and vad_ms (at debug level when the capture passes).
  5. Meetings. Rolling transcription, the Stop & Transcribe fallback and meeting echo skip chunks with no speech. Since c0ed5c7 chunks run on the dictation model, so the check runs in ChunkModel::transcribe, before the model is used. It comes after dev's -60 dB silent-chunk check, which room and keyboard noise passes. A skipped chunk still advances the track's coverage, so the draft and echo suppression keep moving.
  6. Prompt echo. A Whisper segment whose text equals the recognition hints after normalization is removed, with its words. It applies only with two or more hints, so dictating the one listed name still pastes it. Speech that contains a hint among other words is kept.

The local API's programmatic dictation stays ungated, as before. Only the prompt-echo filter applies there, because it runs in the shared GGUF session.

Results on my recording

Hardware: one 63-minute Italian call, M1 Pro, Whisper large-v3-turbo with hints. Microphone entries were classified by hand.

Invented mic entries Real mic entries Real system entries
main (transcript.ndjson from 2.3.0) 152 33 343
per entry, Silero via transcribe-cpp 0.4.0 (before #217), threshold 0.4, min_speech_ms 160 1 kept 32 kept 341 kept
this branch, meeting echo --transcribe (real chunk path) 1 33 entries 345 entries, at most 1 invented
  • The one invented microphone entry left is a "Grazie a tutti." over 16 s of silence. At threshold 0.5 it is skipped, but the whisper above is lost too.
  • Per entry, the one real microphone entry lost is a 40 ms "Certo." at a chunk boundary. The chunk path cuts the audio differently, so its 33 entries do not map one to one onto the 33 of main. None of them is a closing phrase, the hints, or a fragment on quiet audio.
  • System track: the 7 "Grazie" / "Grazie a tutti" during remote silence and the 2 hints repeats are gone. 3 "Grazie" remain: one is the real goodbye at the end of the call, one follows a real reply, and one opens a chunk that also holds real speech (out of scope, see below).
  • With Silero's default 250 ms, 3 short real replies ("Ok", "Ciao") were lost on the system track; 160 ms keeps them.
  • The detector scans the 63 minutes in 1.6 s on the CPU.

Tests

  • Merged with dev 6757e12: scripts/check (fmt, clippy with capture,onnx-benchmark and -D warnings, 1565 tests passed), git diff --check. The three real-model tests below pass on 0.4.1.
  • New: speech_detector_discards_loud_captures_without_speech, speechless_chunks_skip_the_model_and_still_advance_coverage, a_speechless_chunk_after_an_overlap_keeps_earlier_entries, a_speechless_chunk_skips_the_shared_model, speech_detector_cost_is_logged_when_speech_passes, prompt_echo_is_dropped, speech_containing_the_hints_is_kept, only_the_echo_segment_is_removed, single_hint_dictation_is_kept, hints_without_spaces_after_commas_are_matched, a_corrupt_detector_file_is_not_loaded, an_installed_model_does_not_wait_for_the_speech_detector, and artifact tests in transcription_models.
  • Ignored, with the real model: real_detector_rejects_silence_and_noise, real_detector_hears_speech and real_detector_hears_quiet_whispers (HEX_VAD_MODEL, HEX_VAD_SPEECH_WAV, HEX_VAD_WHISPER_WAV). The whisper clip is macOS say -v Whisper at -43 dBFS.
  • Manual, on macOS (hex listen, Whisper large-v3-turbo, Italian, with hints): a spoken sentence, a whispered sentence and a lone "Ok" were pasted. Three 1.5–3.3 s presses without speech, at -48 dBFS (so above the -50 dBFS gate), were discarded by the detector in 15–31 ms (vad_ms).
  • Not tested on Windows or Linux; that relies on CI. The detector loads its model memory-mapped, like the transcription models since perf(transcription): bump transcribe.cpp to 0.4.1, cutting model memory from ~900 MB to ~160 MB #217; I did not test replacing a mapped detector file on Windows.
  • Please squash: the first commits do not pass clippy -D warnings alone.

Not in this PR

Questions

  • Should the detector run for every model, or only for Whisper? Here it runs for every model. I tested only Whisper.
  • Download with the model (this PR), or ship the 1.2 MB file in the app?
  • The detector downloads in the background at first use. Is that fine with the rule to never download a model silently, or should Settings show it?
  • Silero VAD is MIT. THIRD_PARTY_NOTICES.md lists only vendored assets and no downloaded model, so I did not add it there. Tell me if you want a section.

🤖 Generated with Claude Code

0xCaso and others added 7 commits October 22, 2026 11:35
Model::capabilities() now returns a Result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Downloads and verification now work on a GGUF artifact, so the speech
detector reuses the same resumable, checksummed path. Its download is
best effort and never fails the model download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loads the pinned Silero VAD on the CPU on first use, answers whether
16 kHz audio contains speech, and returns None while the detector is
missing or broken so callers keep their level-only silence check. A
missing file starts one background download per process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capture above the -50 dBFS level gate now also goes through the
Silero speech detector when it is installed; a key click, the start
tone or a noisy room no longer reaches the model and comes back as an
invented phrase. Without the detector the gate works as before.
Preparation prewarms an installed detector and never downloads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rolling transcription, the Stop & Transcribe fallback and meeting echo
now ask the Silero speech detector before each chunk. A chunk without
speech becomes an empty transcript without inference, and accepting it
still advances the track's coverage, so the draft and echo suppression
keep moving. On a 63-minute call the detector rejected all 152 invented
microphone entries and kept 32 of 33 real ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion hints

Whisper reads the hints as its initial prompt and returns them verbatim
on silence or noise. Segments equal to the hints after normalization
are removed together with their words, so meetings do not rebuild them
into sentences; speech that contains a hint is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep a single recognition hint: the echo filter needs at least two hints,
  and commas, semicolons and line breaks now separate words.
- Lower the Silero threshold to 0.4 so a whisper at -43 dBFS counts as speech.
- Log audio_ms and vad_ms at debug level when a dictation passes the detector.
- Download the detector in the background when a model downloads or
  activates, so an installed model activates at once offline.
- Fix documentation drift (local API, Linux stop tone, prewarm, figures).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- Cargo.toml, Cargo.lock: dev's dependencies, with transcribe-cpp =0.4.0.
- dictation.rs: dev's near-silent evidence log, then the speech detector.
- gguf_session.rs: dev's job cancel token, then the prompt-echo filter.
- meeting_transcript.rs: dev's quitting check, then the speechless check.
  The speechless tests use Chunk::tail and settled_ms.
- meetings.md, AGENTS.md, main.rs: both sides.

transcription_benchmark.rs boxes Parakeet: with dev's new fields and
transcribe-cpp 0.4.0 it trips clippy::large_enum_variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0xCaso and others added 5 commits October 27, 2026 21:07
dev now pins transcribe-cpp =0.4.1 (anomalyco#217), so the branch takes dev's
Cargo.toml and Cargo.lock and no longer changes the dependency.

- gguf_session.rs: dev's keep_repetition_partial and capabilities check,
  then the prompt-echo filter.
- speech_detector.rs: ModelOptions gained no_mmap in 0.4.1; the detector
  keeps the default (mapped), as dev's model loads do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@thdxr thdxr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @0xCaso, this is a careful, well-measured fix, and the write-up made it easy to review. I reviewed it against current dev and tested it on Linux (apollo: Ryzen AI Max+ 395, Radeon 8060S, Vulkan).

What I checked

  • Vulkan shader trimming: no problem. The detector loads with Backend::Cpu and is F32, so it never touches the GGML_VULKAN_SHADER_TYPES subset.
  • Binary size: release hex on Linux grows from 47,080,224 to 47,085,728 bytes (+5.5 KB). Downloading the 1.2 MB model (pinned revision, size + SHA-256, same receipt/promote path as the transcription models) is the right call over bundling it.
  • Latency: the real detector takes about 2–3 ms for 3–4.5 s clips, 92 ms for 150 s and 155 ms for a 5-minute hold, on CPU. It runs in the finish step after the level gate, not in the audio callback: the Linux audio channel is unbounded, and on macOS the recognition thread is downstream of the audio owner. So capture is never blocked, and normal dictations add a few ms.
  • Real model, real clips (Whisper large-v3-turbo on Vulkan, hints T3 Code, Ottobit, Hex):
    • digital silence: Whisper returned www.t3.org (a hint echo); the level gate already drops it.
    • brown noise above the -50 dBFS gate: detector says no speech, so it's discarded.
    • normal speech, and say -v Whisper at a loudest 30 ms window of -40 and -44 dBFS: detector hears it, and Whisper transcribes it correctly.
    • the same whisper at -48 dBFS: the level gate keeps it and Whisper transcribes it correctly, but the detector rejects it. So there's a narrow band (about -45 to -50 dBFS loudest window) where very quiet whispers are now dropped. I think that's an acceptable trade for removing the hallucinations, and it matches your 0.4 threshold reasoning. Just flagging it as a known limit.
    • your three ignored real-detector tests pass with these clips.
  • All models vs Whisper only: I'd keep it for all models. Parakeet/Cohere also emit "thank you"-style text on noise sometimes (#108), and there's one simple policy. The prompt-echo filter is correctly Whisper-only.
  • Meetings: the check in ChunkModel::transcribe before engine use, with coverage still advancing, looks right. Tests cover the overlap case.

Small fixes I pushed to your branch (maintainer edits; hope that's OK):

  1. Linux clippy failed with -D warnings. On Linux, is_verified, verification_receipt_matches, write_verification_receipt, verify_file_with_cancel and speech_detector::prewarm became dead code once downloads went through artifact_receipt_matches. I gated the receipt helpers like verify_installed. I also call speech_detector::prewarm() when the Linux listener starts, so Linux loads the detector at launch instead of in the first dictation's finish.
  2. Retry after a failed download. The PR description says the next dictation retries a failed download, but DOWNLOAD_STARTED was never reset, so it only retried after a restart. The download thread now clears it on failure. I updated the docs to match.
  3. Removed speech_detector_cost_is_logged_when_speech_passes. It failed once in a full parallel cargo test run on Linux: the tracing callsite interest races other tests even with rebuild_interest_cache. It only asserted a log line, so I dropped it rather than keep a flaky test.

After merging current dev, Linux fmt, clippy (capture,onnx-benchmark, -D warnings), Windows x64 cross-clippy and cargo test are all clean for this change. The only failures are every_persisted_setting_has_a_cli_key and the two Rune golden tests, and those also fail on dev itself (from #303), not from this PR.

Your remaining questions:

  • Download vs bundle: download, as you did.
  • Background download vs "never silently download": that rule covers menu-bar model selections. The detector is a small companion to a model the user chose to download. It's fine as is, and nothing blocks on it.
  • THIRD_PARTY_NOTICES: not needed for a downloaded model.

CI on the new head agrees: fmt and clippy pass on macOS and Windows, and the only test failures are those same dev tests. Approving and squash-merging.

— from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊

@thdxr
thdxr merged commit e68d4e6 into anomalyco:dev Oct 28, 2026
0 of 5 checks passed
@0xCaso
0xCaso deleted the fix/vad-silence branch October 28, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Whisper invents text on silence in meetings and short dictations; the -50 dBFS gate does not catch it

3 participants