Skip to content

WP-004: build the push-to-talk audio pipeline #5

Open

Description

@Circuit-Overtime

Parent: #1
Blocked by: #4

Scope

Add live/WAV capture, VAD, offline STT, response chunking, PocketTTS, pronunciation normalization, cancellable playback, audio fixtures, and latency instrumentation. Use pluggable adapters so laptop and SBC backends share contracts.

Acceptance criteria

  • WAV fixtures and a real laptop microphone can complete speech to response to captured/live audio.
  • sherpa-onnx and Vosk candidates are benchmarked; primary and fallback choices are recorded.
  • PocketTTS streams the first speakable audio chunk.
  • Numbers, units, acronyms, URLs, IPs, and commit SHAs have deterministic pronunciation tests.
  • Endpoint decision is under 500 ms p95 and cached short-command STT under 1.2 s p95 on the reference laptop.
  • TTS first audio is under 500 ms p95 after the first speakable chunk.
  • A 30-minute audio soak loses no buffers and shows no resource growth.
  • ARM deployment path benchmarks sherpa-onnx PocketTTS before hardware release.

Activity

  1. Circuit-Overtime commented on Oct 3, 2026

    @Circuit-Overtime
    CollaboratorAuthor

    WP-004 foundation is implemented across b576422, 3f0da16, 09bcf11, and 073f633. Oreo now has bounded/redacted PCM contracts, explicit push-to-talk lifecycle, strict WAV fixtures, cancellation-aware streaming STT/TTS/output interfaces, capture and transcript limits, a 300 ms fallback endpoint, streamed response chunking, deterministic pronunciation, and bounded p95 gates for endpoint/cached-STT/TTS-first-audio. The package has 20 focused tests and targeted Clippy is clean. Next implementation checkpoint is the CPAL laptop microphone/output adapter; this host currently lacks the ALSA development package required to compile CPAL on Linux.

  2. Circuit-Overtime commented on Oct 3, 2026

    @Circuit-Overtime
    CollaboratorAuthor

    Live laptop audio checkpoint completed in 1f296e8. CPAL 0.18.2 now provides bounded default microphone/speaker adapters with preallocated non-blocking callback pools, cancellation, redacted failures, and loss/error counters. Host smoke evidence: ALSA default input/output at 48 kHz stereo; 1-second capture produced 48 chunks with 0 dropped chunks and 0 stream errors; 1-second speaker tone completed with 0 stream errors and 1 startup underrun callback. Focused audio/CLI tests: 22 passed; targeted Clippy clean.

  3. Circuit-Overtime commented on Oct 3, 2026

    @Circuit-Overtime
    CollaboratorAuthor

    Audio format bridge completed in 84488e7 and acb8f85. Oreo now converts 8-48 kHz mono/stereo device or WAV PCM to a fixed speech-engine format outside real-time callbacks using bounded sinc processing, trims startup delay and final padding to exact duration, and exposes a backpressured ConvertingSource for direct use by streaming STT. Cancellation discards queued PCM; format/state/capacity failures remain redacted. Focused audio tests: 28 passed; targeted Clippy clean. Next checkpoint is the reproducible sherpa-onnx/Vosk fixture benchmark runner and result/provenance schema; model downloads and long benchmark invocations remain operator-run.

  4. Circuit-Overtime commented on Oct 3, 2026

    @Circuit-Overtime
    CollaboratorAuthor

    Lightweight English-only STT cache checkpoint completed in eb13c79. The repo now pins sherpa-onnx English 20M streaming Zipformer INT8 as primary and Vosk small en-US 0.15 as fallback/benchmark baseline. models/cache is local and git-ignored; the operator-run fetcher validates archive paths and exact byte sizes, records local SHA-256 digests, removes archives/staging data, and retains only the roughly 44 MB Sherpa runtime subset. No model was downloaded by this commit. Next checkpoint: common fixture benchmark runner and engine adapters.

  5. 6 remaining items

  6. Circuit-Overtime commented on Oct 4, 2026

    @Circuit-Overtime
    CollaboratorAuthor

    Cost-control and laptop-STT finalization tooling landed across abd994f and 10076cb. Oreo now has a 256-token/4-round/4-tool bounded voice profile, private-safe agent metrics, fixed behavioral fixtures, a live-catalog Pollen cost runner, Vosk mean/minimum word-confidence diagnostics, and an in-process production-adapter soak gate (stable hypothesis, cached p95 <1.2 s, <=8 MiB end-RSS growth after warm-up). Fast validation: 40 default tests, 36 Vosk-enabled CLI/audio tests, benchmark self-test, strict feature-enabled Clippy, and formatting all pass. The operator base WAV transcribed in 766 ms with mean confidence 0.9837, minimum 0.7066, 18 words. Remaining operator evidence: paid three-model harness comparison and 1/30-minute soak; broader noisy corpus and AArch64 remain production-image gates. WP-004 stays open because TTS/playback acceptance is still pending.

  7. Circuit-Overtime commented on Oct 4, 2026

    @Circuit-Overtime
    CollaboratorAuthor

    Operator acceptance and final laptop decisions recorded in 5457c9f. Release-mode Vosk soak: 2,515 turns over 30 minutes, 0 hypothesis mismatches, 758 ms p95 (under 1.2 s), baseline RSS 163,916 KiB, end RSS 102,664 KiB, 0 KiB end growth; mean/min word confidence 0.9838/0.7079 across 18 words. Vosk small en-US is now final for the laptop profile; broader noisy-corpus and AArch64 measurements remain production-image gates. Agent comparison correction: Nano refusal failures were a fixture lexical miss for the curly apostrophe in “can’t”; manual probe confirmed a safe refusal and corrected rerun passed 15/15. GPT-5.4 Nano is pinned as default at 0.0000686025 mean Pollen and 5,709 ms p95. Qwen passed 15/15 but cost 0.000186458 and reached 46,971 ms p95 due high billed reasoning/output tokens; Nemotron passed 9/15. WP-004 remains open for PocketTTS, spoken playback latency, and the complete audio soak.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    audioVoice and audio pipelinework-packageOrdered implementation package

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions