Batched encoder: run N clips through one fused ggml graph

CMakeLists.txt

@@ -76,6 +76,7 @@ set(PARAKEET_SRC
     src/joint.cpp
     src/tdt.cpp
     src/rnnt.cpp
+    src/transducer_batch.cpp
     src/tokenizer.cpp
     src/search.cpp
     src/transcription.cpp)

README.md

@@ -173,6 +173,29 @@ The `parakeet-cli` binary lands at `build/examples/cli/parakeet-cli`.
 
 ---
 
+## Batching
+
+Single-clip transcription is the default and needs no flags: every `transcribe` call runs one clip at a time, byte-for-byte identical to before. Batching is an opt-in path for decoding several clips together, which matters when you serve many concurrent requests on a GPU.
+
+The win is on the **decode** side. A transducer (TDT/RNN-T) decodes autoregressively with tiny per-step prediction-LSTM and joint GEMMs; one clip launches hundreds of these matvec-sized kernels and leaves the GPU mostly idle between launches. Decoding N clips together coalesces each step into one batched GEMM, so the device stays busy. On the NVIDIA GB10 this reaches about **10-12x** at batch size 16 (CPU about 3-5x); the encoder is already compute-bound, so batching it gives no throughput win. CTC has no autoregressive decode, so batching does not apply to standalone CTC models. The batched path is bit-identical to running the clips one by one (greedy decode is deterministic). Full numbers and per-model tables are in [`benchmarks/BENCHMARK.md`](benchmarks/BENCHMARK.md#batched-decode-throughput).
+
+Measure it yourself:
+
+```bash
+# Decode-only: serial vs batched decode of one clip replicated B times (the win in isolation).
+parakeet-cli bench-decode --model <model.gguf> --audio <wav> [--batch-sizes 1,4,8,16] [--threads N] [--reps R] [--json <out>]
+
+# Full transcribe (encoder + decode) over a manifest at several batch sizes.
+parakeet-cli bench-batch --model <model.gguf> --manifest <file> [--decoder ctc|tdt] [--threads N] [--batch-sizes 1,4,8] [--json <out>]
+```
+
+To batch from code, use the batched entry points (single-clip B=1 is just N=1):
+
+- C++ (`src/model.hpp`): `Model::transcribe_16k_batch(pcms16k, decoder)` and `transcribe_16k_batch_with_timestamps(...)` take N clips of 16 kHz mono float PCM and return N results.
+- C-API (`include/parakeet_capi.h`): `parakeet_capi_transcribe_pcm_batch(...)` (N transcripts) and `parakeet_capi_transcribe_pcm_batch_json(...)` (one JSON array of N `{text,words,tokens}` objects). These are what LocalAI's `parakeet-cpp` backend calls to coalesce concurrent requests; it leaves batching off by default and exposes a `batch_max_size` option to opt in.
+
+---
+
 ## C-API (`libparakeet.so`)
 
 `include/parakeet_capi.h` defines a flat, exception-free C-API meant for `dlopen` / FFI / LocalAI integration. Build the shared library with `-DPARAKEET_SHARED=ON`:

benchmarks/BENCHMARK.md

@@ -84,6 +84,38 @@ Averaged over all models (LibriSpeech). Size is the mean GGUF size as a fraction
 
 > f32 is the faithful reference (agreement ≈ 0). q8_0 is near-lossless; K-quants (q6_k→q4_k) shrink the model further at a small, monotonic accuracy cost. See the per-model quant plots below.
 
+## Batched decode throughput
+
+Decode batching coalesces the per-step prediction-LSTM and joint-network GEMMs across several utterances into single batched ops, so one decode loop advances B clips at once. It applies to **transducer (TDT/RNN-T) models only** — CTC has no autoregressive decode to batch. Speedup is `serial_ms / batched_ms`: the wall-clock of B independent single-clip decodes divided by one batched decode over the same B copies (encoder cost is paid once and excluded). The `clips/s @B=16` column is the batched decode throughput at B=16.
+
+**CPU** (cpu, q5_k, 8 threads), best-of-5, one clip replicated B times.
+
+
+| Model | B=1 | B=4 | B=8 | B=16 | clips/s @B=16 |
+|---|---|---|---|---|---|
+| rnnt-0.6b | 0.98× | 2.81× | 3.91× | 5.01× | 378.9 |
+| rnnt-1.1b | 1.01× | 2.27× | 3.54× | 4.19× | 366.1 |
+| rt-eou-120m-v1 | 1.00× | 2.32× | 3.00× | 3.36× | 638.1 |
+| tdt-0.6b-v2 | 0.90× | 2.78× | 4.12× | 5.19× | 458.8 |
+| tdt-0.6b-v3 | 0.82× | 3.26× | 5.90× | 4.88× | 253.2 |
+| tdt-1.1b | 0.98× | 2.55× | 3.74× | 4.57× | 483.2 |
+| tdt_ctc-1.1b | 1.00× | 2.59× | 3.80× | 4.73× | 458.8 |
+| tdt_ctc-110m | 1.03× | 2.46× | 3.12× | 3.61× | 834.5 |
+
+**GPU** (CUDA0, f16, 8 threads), best-of-5, one clip replicated B times.
+
+
+| Model | B=1 | B=4 | B=8 | B=16 | clips/s @B=16 |
+|---|---|---|---|---|---|
+| rnnt-0.6b | 1.00× | 3.32× | 6.46× | 11.44× | 585.8 |
+| rnnt-1.1b | 1.00× | 3.31× | 6.30× | 11.17× | 594.1 |
+| rt-eou-120m-v1 | 0.98× | 3.11× | 5.87× | 10.25× | 951.1 |
+| tdt-0.6b-v2 | 1.00× | 3.61× | 6.88× | 12.42× | 712.0 |
+| tdt-0.6b-v3 | 1.00× | 3.45× | 6.49× | 11.45× | 614.5 |
+| tdt-1.1b | 1.00× | 3.55× | 6.72× | 12.17× | 817.9 |
+| tdt_ctc-1.1b | 1.00× | 3.45× | 6.50× | 11.47× | 746.9 |
+| tdt_ctc-110m | 1.01× | 3.29× | 6.33× | 10.89× | 1259.7 |
+
 ## Plots
 
 ### RTFx per model — NeMo vs ours (all dtypes), LibriSpeech

benchmarks/results/decode_batch/cpu/rnnt-0.6b.json

@@ -0,0 +1,50 @@
+{
+  "model": "rnnt-0.6b-q5_k.gguf",
+  "decoder": "ctc",
+  "backend": "cpu",
+  "dtype": "q5_k",
+  "threads": 8,
+  "reps": 5,
+  "clip_frames": 131,
+  "d_model": 1024,
+  "batch_sizes": [
+    1,
+    4,
+    8,
+    16
+  ],
+  "rows": [
+    {
+      "B": 1,
+      "serial_ms": 13.32,
+      "batched_ms": 13.65,
+      "speedup": 0.98,
+      "serial_cps": 75.1,
+      "batched_cps": 73.3
+    },
+    {
+      "B": 4,
+      "serial_ms": 54.99,
+      "batched_ms": 19.56,
+      "speedup": 2.81,
+      "serial_cps": 72.7,
+      "batched_cps": 204.5
+    },
+    {
+      "B": 8,
+      "serial_ms": 103.03,
+      "batched_ms": 26.35,
+      "speedup": 3.91,
+      "serial_cps": 77.7,
+      "batched_cps": 303.6
+    },
+    {
+      "B": 16,
+      "serial_ms": 211.7,
+      "batched_ms": 42.23,
+      "speedup": 5.01,
+      "serial_cps": 75.6,
+      "batched_cps": 378.9
+    }
+  ]
+}

benchmarks/results/decode_batch/cpu/rnnt-1.1b.json

@@ -0,0 +1,50 @@
+{
+  "model": "rnnt-1.1b-q5_k.gguf",
+  "decoder": "ctc",
+  "backend": "cpu",
+  "dtype": "q5_k",
+  "threads": 8,
+  "reps": 5,
+  "clip_frames": 131,
+  "d_model": 1024,
+  "batch_sizes": [
+    1,
+    4,
+    8,
+    16
+  ],
+  "rows": [
+    {
+      "B": 1,
+      "serial_ms": 11.36,
+      "batched_ms": 11.26,
+      "speedup": 1.01,
+      "serial_cps": 88.0,
+      "batched_cps": 88.8
+    },
+    {
+      "B": 4,
+      "serial_ms": 45.26,
+      "batched_ms": 19.92,
+      "speedup": 2.27,
+      "serial_cps": 88.4,
+      "batched_cps": 200.8
+    },
+    {
+      "B": 8,
+      "serial_ms": 94.43,
+      "batched_ms": 26.68,
+      "speedup": 3.54,
+      "serial_cps": 84.7,
+      "batched_cps": 299.9
+    },
+    {
+      "B": 16,
+      "serial_ms": 183.17,
+      "batched_ms": 43.7,
+      "speedup": 4.19,
+      "serial_cps": 87.4,
+      "batched_cps": 366.1
+    }
+  ]
+}

benchmarks/results/decode_batch/cpu/rt-eou-120m-v1.json

@@ -0,0 +1,50 @@
+{
+  "model": "realtime_eou_120m-v1-q5_k.gguf",
+  "decoder": "ctc",
+  "backend": "cpu",
+  "dtype": "q5_k",
+  "threads": 8,
+  "reps": 5,
+  "clip_frames": 132,
+  "d_model": 512,
+  "batch_sizes": [
+    1,
+    4,
+    8,
+    16
+  ],
+  "rows": [
+    {
+      "B": 1,
+      "serial_ms": 5.08,
+      "batched_ms": 5.07,
+      "speedup": 1.0,
+      "serial_cps": 196.9,
+      "batched_cps": 197.3
+    },
+    {
+      "B": 4,
+      "serial_ms": 20.31,
+      "batched_ms": 8.75,
+      "speedup": 2.32,
+      "serial_cps": 196.9,
+      "batched_cps": 457.4
+    },
+    {
+      "B": 8,
+      "serial_ms": 42.16,
+      "batched_ms": 14.07,
+      "speedup": 3.0,
+      "serial_cps": 189.8,
+      "batched_cps": 568.6
+    },
+    {
+      "B": 16,
+      "serial_ms": 84.35,
+      "batched_ms": 25.07,
+      "speedup": 3.36,
+      "serial_cps": 189.7,
+      "batched_cps": 638.1
+    }
+  ]
+}

benchmarks/results/decode_batch/cpu/tdt-0.6b-v2.json

@@ -0,0 +1,50 @@
+{
+  "model": "tdt-0.6b-v2-q5_k.gguf",
+  "decoder": "tdt",
+  "backend": "cpu",
+  "dtype": "q5_k",
+  "threads": 8,
+  "reps": 5,
+  "clip_frames": 131,
+  "d_model": 1024,
+  "batch_sizes": [
+    1,
+    4,
+    8,
+    16
+  ],
+  "rows": [
+    {
+      "B": 1,
+      "serial_ms": 10.89,
+      "batched_ms": 12.12,
+      "speedup": 0.9,
+      "serial_cps": 91.9,
+      "batched_cps": 82.5
+    },
+    {
+      "B": 4,
+      "serial_ms": 43.97,
+      "batched_ms": 15.83,
+      "speedup": 2.78,
+      "serial_cps": 91.0,
+      "batched_cps": 252.8
+    },
+    {
+      "B": 8,
+      "serial_ms": 89.54,
+      "batched_ms": 21.71,
+      "speedup": 4.12,
+      "serial_cps": 89.3,
+      "batched_cps": 368.5
+    },
+    {
+      "B": 16,
+      "serial_ms": 181.13,
+      "batched_ms": 34.88,
+      "speedup": 5.19,
+      "serial_cps": 88.3,
+      "batched_cps": 458.8
+    }
+  ]
+}

benchmarks/results/decode_batch/cpu/tdt-0.6b-v3.json

@@ -0,0 +1,50 @@
+{
+  "model": "tdt-0.6b-v3-q5_k.gguf",
+  "decoder": "tdt",
+  "backend": "cpu",
+  "dtype": "q5_k",
+  "threads": 8,
+  "reps": 5,
+  "clip_frames": 131,
+  "d_model": 1024,
+  "batch_sizes": [
+    1,
+    4,
+    8,
+    16
+  ],
+  "rows": [
+    {
+      "B": 1,
+      "serial_ms": 17.43,
+      "batched_ms": 21.17,
+      "speedup": 0.82,
+      "serial_cps": 57.4,
+      "batched_cps": 47.2
+    },
+    {
+      "B": 4,
+      "serial_ms": 126.46,
+      "batched_ms": 38.81,
+      "speedup": 3.26,
+      "serial_cps": 31.6,
+      "batched_cps": 103.1
+    },
+    {
+      "B": 8,
+      "serial_ms": 262.68,
+      "batched_ms": 44.54,
+      "speedup": 5.9,
+      "serial_cps": 30.5,
+      "batched_cps": 179.6
+    },
+    {
+      "B": 16,
+      "serial_ms": 308.27,
+      "batched_ms": 63.18,
+      "speedup": 4.88,
+      "serial_cps": 51.9,
+      "batched_cps": 253.2
+    }
+  ]
+}

benchmarks/results/decode_batch/cpu/tdt-1.1b.json

@@ -0,0 +1,50 @@
+{
+  "model": "tdt-1.1b-q5_k.gguf",
+  "decoder": "tdt",
+  "backend": "cpu",
+  "dtype": "q5_k",
+  "threads": 8,
+  "reps": 5,
+  "clip_frames": 131,
+  "d_model": 1024,
+  "batch_sizes": [
+    1,
+    4,
+    8,
+    16
+  ],
+  "rows": [
+    {
+      "B": 1,
+      "serial_ms": 9.2,
+      "batched_ms": 9.42,
+      "speedup": 0.98,
+      "serial_cps": 108.7,
+      "batched_cps": 106.2
+    },
+    {
+      "B": 4,
+      "serial_ms": 37.35,
+      "batched_ms": 14.62,
+      "speedup": 2.55,
+      "serial_cps": 107.1,
+      "batched_cps": 273.6
+    },
+    {
+      "B": 8,
+      "serial_ms": 76.22,
+      "batched_ms": 20.36,
+      "speedup": 3.74,
+      "serial_cps": 105.0,
+      "batched_cps": 393.0
+    },
+    {
+      "B": 16,
+      "serial_ms": 151.37,
+      "batched_ms": 33.11,
+      "speedup": 4.57,
+      "serial_cps": 105.7,
+      "batched_cps": 483.2
+    }
+  ]
+}