Batched encoder: run N clips through one fused ggml graph - #1

Merged

mudler merged 41 commits into

worktree-batched-encoder

Jun 1, 2026

Commits on May 31, 2026

feat(encoder): add MelBatch + forward_batch (B=1 per-item loop)

mudler
and
claude
committed
9b89a3f
feat(model): transcribe_pcm_batch + extract decode_enc_out helper

mudler
and
claude
committed
ab1b458
fix(model): guard invalid sample_rate in transcribe_pcm_batch (parity with single-clip)

mudler
and
claude
committed
2719230
feat(capi): parakeet_capi_transcribe_pcm_batch

mudler
and
claude
committed
e786266
fix(capi): null out[] on entry so error paths leave a clean, uniform contract

mudler
and
claude
committed
bc582d4
test: batch API B=1 equivalence smoke

mudler
and
claude
committed
4786ca7
feat(subsampling): batched build_graph_batched; build_graph adapts B=1

mudler
and
claude
committed
8c76c1a
fix(subsampling): release-surviving guard for batched-causal; dedupe valid_out_len

mudler
and
claude
committed
fc391a3
test(subsampling): batched-vs-standalone per-item equivalence

mudler
and
claude
committed
eab74e6
fix(subsampling): per-stage per-item trailing-pad input masking for batched conv (matches NeMo MaskedConvSequential)

mudler
and
claude
committed
d355250
feat(conformer): batch axis through build_conv_module (B=1 unchanged)

mudler
and
claude
committed
b9f1cb1
feat(attention): batched build_graph_batched with 4D rel-shift

mudler
and
claude
committed
4efe312
test(attention): batched rel-shift equivalence + padding invariance

mudler
and
claude
committed
5943984
feat(conformer): build_graph_batched ([D,T,B]); build_graph adapts B=1

mudler
and
claude
committed
7a8a8a3
test(conformer): batched-vs-standalone per-item equivalence + padding invariance

mudler
and
claude
committed
b5bede7
fix(conformer): per-item depthwise conv for B>1 (ggml 1D im2col requires ne[3]==1); B=1 byte-identical

mudler
and
claude
committed
3a89768
feat(encoder): fused single-graph batched forward_batch

mudler
and
claude
committed
2257e67
test(encoder): fused batched equivalence + padding invariance

mudler
and
claude
committed
0b1f409
bench: add bench-batch CLI subcommand for batched-encoder throughput

mudler
and
claude
committed
50d2cfc
docs(encoder): correct forward_batch comment (fused graph, not internal loop)

mudler
and
claude
committed
3982fa7
feat(model): transcribe_pcm_batch_with_timestamps + extract decode_enc_out_with_timestamps

mudler
and
claude
committed
7cc5942
refactor(model): share build_mel_batch across batch paths; restore CTC span rationale

mudler
and
claude
committed
d2b76a5
feat(capi): parakeet_capi_transcribe_pcm_batch_json (batched timestamps, ABI bump)

mudler
and
claude
committed
9bd7407
docs(capi): state sum(n_samples) precondition for batch_json as caller-must-uphold

mudler
and
claude
committed
78f84f5
refactor(decode): share argmax/max_prob_conf in decode_common.hpp

mudler
and
claude
committed
b9ca3f6
feat(prediction): step_batch (batched LSTM, [H,N])

mudler
and
claude
committed
fc19481
docs(prediction): explain the 4H gate-slice stride in step_batch

mudler
and
claude
committed
e5846ef
feat(joint): step_logits_batch (batched joint, [V_plus,N])

mudler
and
claude
committed
cc9834d
feat(decode): transducer_greedy_batch (batched RNNT+TDT greedy, bit-exact parity)

mudler
and
claude
committed
093aacc
refactor(decode): extract commit_state lambda in transducer_greedy_batch; document max_symbols assumption

mudler
and
claude
committed
782a801
feat(model): batched transducer decode in transcribe_pcm_batch[_with_timestamps]

mudler
and
claude
committed
df4c857
bench: add bench-decode (batched vs serial transducer decode timing)

mudler
and
claude
committed
52cffea
perf(decode): cache prediction-net g across rounds in transducer_greedy_batch (bit-exact; recovers B=1, fewer LSTM calls)

mudler
and
claude
committed
85feda9
refactor(model): extract batch_enc_to_row_major helper (dedup batched-decode transpose)

mudler
and
claude
committed
c3161d2
docs(decode): correct header comment to describe the g_valid cache

mudler
and
claude
committed
da38ea1

Commits on Jun 1, 2026

Merge pull request #3 from mudler/batched-decode
mudler
authored
543af2c
refactor(subsampling): restore v1 2-D scalar build_graph for B=1
mudler
committed
98bd731
refactor(conformer,attention): restore v1 2-D scalar builders for B=1
mudler
committed
0c42127
feat(bench): bench-decode --json + BENCHMARK.md batched-decode section

mudler
and
claude
committed
0b47f3a
docs(bench): add batched-decode throughput tables (CPU + GPU)

mudler
and
claude
committed
efc061c
docs(readme): add Batching section (CLI, C-API, when to use)

mudler
and
claude
committed
5e9bb6f