Remove unprintable characters from vocab list by beiller · Pull Request #25 · ggml-org/llama.cpp

beiller · 2023-03-11T21:34:03Z

Fixes #11

This fixes a Japanese prompt I was attempting to run

EG:

./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -n 128 -n 512 -p $'人生の意味は'

Output before change:

So it is outputting some characters but some �

Output after change:

人生の意は、一人が一人ということであります。は安部が立していたので、去からは一人の人にれるのはにとどまったのですが、そう

* Added ggml_tensor_printf() to debug tensors. Not sure if all cases work, it was only tested a bit. Example for the ggml_repeat2 dst tensor after it was computed: +======================+======================+======================+======================+ | ggml_compute_forward_repeat2_f32:9497 | node_1233 +----------------------+----------------------+----------------------+----------------------+ | Dimensions | Quantization | Layer id | Backend | | 3 | f32 | 31 | CPU | +----------------------+----------------------+----------------------+----------------------+ | Elements | Src0 | Src1 | Operation | | 64 x 2 x 71 | 64 x 2 x 1 | 64 x 2 x 71 | REPEAT2 | +----------------------+----------------------+----------------------+----------------------+ | Src0 name: | node_1232 | | Src1 name: | leaf_17 | +----------------------+----------------------+----------------------+----------------------+ +-------------------------------------------------------------------------------------------+ | Content of src0 "node_1232" (3 dim) Layer 0 | -0.019758 0.772589 0.000000 | | 0.772589 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | +-------------------------------------------------------------------------------------------+ Layer 1 | 0.001423 -1.063233 0.000000 | | -1.063233 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | +-------------------------------------------------------------------------------------------+ Layer 2 | -0.042461 -0.936166 0.000000 | | -0.936166 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | +-------------------------------------------------------------------------------------------+ +-------------------------------------------------------------------------------------------+ | Content of src1 "leaf_17" (3 dim) Layer 0 | 0.000000 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | +-------------------------------------------------------------------------------------------+ Layer 1 | 0.000000 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | +-------------------------------------------------------------------------------------------+ Layer 2 | 0.000000 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | | 0.000000 0.000000 0.000000 | +-------------------------------------------------------------------------------------------+ +-------------------------------------------------------------------------------------------+ | Content of dst "node_1233" (3 dim) Layer 0 | -0.019758 -0.019758 -0.019758 | | 0.772589 0.772589 0.772589 | | -0.019758 -0.019758 -0.019758 | +-------------------------------------------------------------------------------------------+ Layer 1 | 0.001423 0.001423 0.001423 | | -1.063233 -1.063233 -1.063233 | | 0.001423 0.001423 0.001423 | +-------------------------------------------------------------------------------------------+ Layer 2 | -0.042461 -0.042461 -0.042461 | | -0.936166 -0.936166 -0.936166 | | -0.042461 -0.042461 -0.042461 | +-------------------------------------------------------------------------------------------+ +======================+======================+======================+======================+ * typo stride>n_elem - sample print is probably still bugged * added strides and boolean info flags --------- Co-authored-by: John <nolife+git@gmail.com>

Add information on compiler flags

QVAC-6093: Stream shards. Fixup for gradle.

https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md

- Experiment #25: QJL helps turbo4 by +0.3 PPL. Without QJL, turbo4 ≈ turbo3. - Experiment #25b: Sign+magnitude encoding is neutral (decode is memory-bound). - Experiment #25c: Long-context PPL validates turbo3 competitive with q8_0. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…bug 2) Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l-org#25 bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…bug 2) Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l-org#25 bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…moe) Enables proper graph-construction-time fusion of the o-projection MUL_MAT with the inpSA residual ADD, bypassing the post-hoc op-mutation aliasing bug from Phase 1. Changes: - build_attn(llm_graph_input_attn_kv*, ...) gains an optional `residual` parameter. When safe (no LoRA, no wo_s, no wo_b, F32, shape match), uses ggml_mul_mat_add_residual so the allocator sees src[2] from the start and assigns a fresh output slot. Falls back to explicit ADD otherwise so callers can treat the result as "already residual-added". - qwen3moe.cpp passes inpSA as the residual (gated by GGML_FUSE_ATTN_RES=1 while we validate). Skips fusion on the last layer (inp_out_ids slicing) and when wo_s is set (outer scale would rescale residual). - repack.cpp: added apply_residual_chunk and per-chunk fusion inside forward_mul_mat. This is the critical fix — repacked weights use their own kernel and don't go through ggml_compute_forward_mul_mat, so the original Phase 1 code path silently produced garbage when the weight was in the repack buffer type. Per-chunk fusion needs no extra barriers (each thread's chunk is disjoint). - ggml-cpu.c: removed the mm_fused guard that was skipping extra_compute _forward for fused ops, since the repack path now handles them. Correctness (GGML_NUMA_WEIGHTS=1, 48t, NPS4, PPL over 20 chunks of wikitext): fusion OFF: 10.4006 fusion ON: 10.4006 (bit-exact) Throughput (48t, NPS4, NUMA_WEIGHTS=1): fusion OFF: pp128=296.26 tg128=39.41 t/s fusion ON: pp128=293.92 tg128=39.43 t/s Gain is within run-to-run noise on decode; fusion primarily saves one barrier per layer + one allocated tensor. Ship as env-gated so future work (RMS_NORM+MUL_MAT, attention-internal fusions) has a validated infrastructure to build on. Resolves task ggml-org#25. Task ggml-org#22 (CCD pool teardown hang) also resolved — non-reproducible on 4x24t and 4x48t concurrent benches, apparently fixed by the earlier cpuset commits (0ade7bd4d + 69b4c3fa4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…bug 2) Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l-org#25 bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…bug 2) Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l-org#25 bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…bug 2) Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l-org#25 bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…gml-org#25) Tree verify buffers (parent_ids_gpu, SSM intermediates) are allocated on GPU 0. When the model is split across devices, recurrent layers on other GPUs can't read these tensors, causing the scheduler to abort. Disable tree mode on multi-GPU and fall back to the regular SSM_CONV + GATED_DELTA_NET kernels. The verify batch is still processed in a single llama_decode call — for linear chains the results are identical. Also stops tree buffer allocation failure from spamming logs every verify cycle by setting disabled=true on first failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…bug 2) Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l-org#25 bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…gml-org#25)

PR ggml-org#25 review follow-up. The "batched im2col" rewrite of fm_causal_conv1d (from _f32_n1 + per-batch concat to a single _f32_nB call, added in 904edf3) was shown to give no measurable CPU benefit in two independent REPEAT=10 A/B runs against 32bb0d8: - baselines/ab_32bb0d8_vs_fc347ab_2026-04-23_1116.md (A→B, 32bb0d8 → no-1A) - baselines/ab_fc347ab_vs_32bb0d8_2026-04-23_1121.md (B→A) Both rounds land inside the ±2% system-noise band; neither direction of the swap showed a consistent p50 change on total / t2m.compute / voc.compute. Keeping the more invasive _nB form costs explanatory surface (new im2col reshape/permute bookkeeping around the [Cout, T_out, B] layout) with no offsetting speed win, so the revert is on the safer side. Changes: - fm_causal_conv1d_im2col_f32_nB → removed. - Two call sites in fmCausalConv1d go back to a per-batch loop: im2col + mul_mat + reshape, concat along batch at the end. - Matches the shape of the code as it lived before 904edf3 on the public _n1 helper; 1.B (tree-concat), 1.C (no-cont on KV) and Phase 2.1 (CUDA graphs) are all untouched. Bit-exactness: both A/B reports confirmed MD5 equality of the generated wav across the revert; no audio change expected. Cherry-picked from review branch review/pr25-no-1A (fc347ab), rebased onto opt/t2w HEAD (4a2a310) via git cherry-pick with automatic context fix-up. Made-with: Cursor

PR ggml-org#25 review follow-up. The optimization work in 904edf3 / 32bb0d8 also carried along three pieces of "Phase 0" profiler scaffolding that have nothing to do with the optimization itself: - ggml/include/ggml-cpu.h +7 lines (extern decl) - ggml/src/ggml-cpu/ggml-cpu.c +124 lines (per-op wall-time profiler) - tools/omni/CMakeLists.txt +19 lines (OMNI_T2W_TRACK_OPS option + CUDAToolkit nvtx include) - tools/omni/token2wav/token2wav-profile.h +35 lines in class ScopeTimer (NVTX range push/pop, weak symbol hookup to the ggml-cpu profiler) These are genuinely useful for *internal* profiling (will keep driving the next rounds of op-attribution work — see sibling branch opt/t2w-profile), but they shouldn't block the core optimization PR: 1. ggml-cpu.c per-op profiler is an upstream ggml touch that will invite its own review conversation (statics sized by GGML_OP_COUNT, getenv + atexit, label stack model). 2. OMNI_T2W_TRACK_OPS adds new build knobs and a conditional CUDAToolkit lookup that expands the diff surface without moving the perf needle. 3. NVTX + weak-symbol hookup in ScopeTimer depends on (1) being present to be more than a no-op, and pulls in an <nvtx3/nvToolsExt.h> include path that complicates PR build envs. What's kept: - The 8 `ScopeTimer` *call sites* in token2wav-impl.cpp (voc.build_alloc, voc.upload, voc.compute, voc.download, t2m.upload, t2m.feed_noise, t2m.compute, t2m.download). They use the base `ScopeTimer(const char *, bool)` constructor that already lives in e654026 ("add t2w profiling"), so they continue to produce the p50/p95/p99 stage breakdown that baselines/run_ab.sh depends on — just without the NVTX push/pop and the ggml-cpu label forwarding. Net diff: 1 insertion, 184 deletions; strictly subtracts surface area from the PR diff, doesn't touch 1.B / 1.C / 2.1 / Phase 2.2 / scheme E code. bit-exact preserved (per-op profiler was a dormant no-op when OMNI_T2W_OP_PROFILE was unset, which it always is in release). Companion branch: opt/t2w-profile = opt/t2w HEAD + these three patches re-applied. Used for day-to-day dev profiling; not intended for submission. Kept so that any future optimization round can just git checkout opt/t2w-profile and have the full Phase-0 observability back. Made-with: Cursor

opt: token2wav structural optimization

PR ggml-org#25 review follow-up. The "batched im2col" rewrite of fm_causal_conv1d (from _f32_n1 + per-batch concat to a single _f32_nB call, added in 904edf3) was shown to give no measurable CPU benefit in two independent REPEAT=10 A/B runs against 32bb0d8: - baselines/ab_32bb0d8_vs_fc347ab_2026-04-23_1116.md (A→B, 32bb0d8 → no-1A) - baselines/ab_fc347ab_vs_32bb0d8_2026-04-23_1121.md (B→A) Both rounds land inside the ±2% system-noise band; neither direction of the swap showed a consistent p50 change on total / t2m.compute / voc.compute. Keeping the more invasive _nB form costs explanatory surface (new im2col reshape/permute bookkeeping around the [Cout, T_out, B] layout) with no offsetting speed win, so the revert is on the safer side. Changes: - fm_causal_conv1d_im2col_f32_nB → removed. - Two call sites in fmCausalConv1d go back to a per-batch loop: im2col + mul_mat + reshape, concat along batch at the end. - Matches the shape of the code as it lived before 904edf3 on the public _n1 helper; 1.B (tree-concat), 1.C (no-cont on KV) and Phase 2.1 (CUDA graphs) are all untouched. Bit-exactness: both A/B reports confirmed MD5 equality of the generated wav across the revert; no audio change expected. Cherry-picked from review branch review/pr25-no-1A (fc347ab), rebased onto opt/t2w HEAD (4a2a310) via git cherry-pick with automatic context fix-up. Signed-off-by: Halcao <nyz1500@gmail.com>

PR ggml-org#25 review follow-up. The optimization work in 904edf3 / 32bb0d8 also carried along three pieces of "Phase 0" profiler scaffolding that have nothing to do with the optimization itself: - ggml/include/ggml-cpu.h +7 lines (extern decl) - ggml/src/ggml-cpu/ggml-cpu.c +124 lines (per-op wall-time profiler) - tools/omni/CMakeLists.txt +19 lines (OMNI_T2W_TRACK_OPS option + CUDAToolkit nvtx include) - tools/omni/token2wav/token2wav-profile.h +35 lines in class ScopeTimer (NVTX range push/pop, weak symbol hookup to the ggml-cpu profiler) These are genuinely useful for *internal* profiling (will keep driving the next rounds of op-attribution work — see sibling branch opt/t2w-profile), but they shouldn't block the core optimization PR: 1. ggml-cpu.c per-op profiler is an upstream ggml touch that will invite its own review conversation (statics sized by GGML_OP_COUNT, getenv + atexit, label stack model). 2. OMNI_T2W_TRACK_OPS adds new build knobs and a conditional CUDAToolkit lookup that expands the diff surface without moving the perf needle. 3. NVTX + weak-symbol hookup in ScopeTimer depends on (1) being present to be more than a no-op, and pulls in an <nvtx3/nvToolsExt.h> include path that complicates PR build envs. What's kept: - The 8 `ScopeTimer` *call sites* in token2wav-impl.cpp (voc.build_alloc, voc.upload, voc.compute, voc.download, t2m.upload, t2m.feed_noise, t2m.compute, t2m.download). They use the base `ScopeTimer(const char *, bool)` constructor that already lives in e654026 ("add t2w profiling"), so they continue to produce the p50/p95/p99 stage breakdown that baselines/run_ab.sh depends on — just without the NVTX push/pop and the ggml-cpu label forwarding. Net diff: 1 insertion, 184 deletions; strictly subtracts surface area from the PR diff, doesn't touch 1.B / 1.C / 2.1 / Phase 2.2 / scheme E code. bit-exact preserved (per-op profiler was a dormant no-op when OMNI_T2W_OP_PROFILE was unset, which it always is in release). Companion branch: opt/t2w-profile = opt/t2w HEAD + these three patches re-applied. Used for day-to-day dev profiling; not intended for submission. Kept so that any future optimization round can just git checkout opt/t2w-profile and have the full Phase-0 observability back. Signed-off-by: Halcao <nyz1500@gmail.com>

opt: token2wav structural optimization

… path (+35.6% tg) Cherry-picks only the ggml/src/ggml-cpu/spacemit/ portion of SpacemiT 0.1.7 (commits 5a23f07 ggml-org#25 and 6ad6d85 ggml-org#27). The remainder of those commits is the ONNX media / smt-mtmd / Qwen3-TTS wrapper path, which this build does not enable, so taking them whole would add a large untested surface for no gain. Corrects an earlier assumption in MODELS.md that the newer spacemit commits were ONNX-only. That holds for ggml-org#19/ggml-org#20/ggml-org#21 but not 0.1.7, which adds a gemm_m == 1 (decode) fast path for q4_0/q8_0 that streams wide Q8 output heads straight through TCM, replaces the even/odd pair-barrier lockstep, and adds RVV GELU/TANH/GEGLU kernels wired in as GGML_OP_UNARY/GGML_OP_GLU. Applied with git apply -3, no conflicts. Verified our backend patches survived: patch 20's GGML_OP_CONT bypass in get_tensor_traits, the patch 14 dispatch and supports log hooks, and patch 28's max_perfer_threads clamp. Same-session A/B (llama-bench -t 8 -p 128 -n 128 -mmp 0 -fa 1 -ub 128 -r 3), stashing and rebuilding between measurements so there is no cross-session drift: gemma-4-E2B_q4_0 pp 116.39 -> 136.55 (+17.3%) tg 11.42 -> 15.49 (+35.6%) Qwen3-4B-Q4_K_M pp 57.63 -> 57.58 tg 9.20 -> 9.20 q4_K is unchanged within noise, as expected: the fast path is if constexpr gated to q4_0/q8_0. This matters because it is a decode win, which patch 17 found the IME2 HP tile could not deliver (M=1 is bandwidth-bound). Output coherence spot-checked on both quants at --temp 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…rg#30 q4_0 HP overflow fix Patch 31 was audited against the v0.1.7 tag only, which missed a commit. SpacemiT moved their active branch: spacemit/spacemit-mtmd is still spacemit/HEAD but is stale at 2026-07-20, while development continues on spacemit/mtmd-backend, whose tip (2026-08-05) is newer than the v0.1.7 tag. Future backend syncs should diff against that branch, not the tag. Re-audited every commit touching ggml/src/ggml-cpu/spacemit/ or tools/smt-mtmd/ since v0.1.6: ggml-org#25 and ggml-org#27 were taken in patch 31, ggml-org#28 is CI-only, and ggml-org#30 is this one. ggml-org#30 matters because of patch 31. quantize_a_row_i8_hp and friends stored the block scale as _Float16 while computing 1.0f / scale_avg; for small scales that reciprocal exceeds fp16 range and corrupts the activation quantization. The fix rescales both sides by 0.1 so the product is preserved with less range pressure, and adds the zero-guard the _ref variant was missing. Patch 31 routes more q4_0 decode traffic through exactly that path, so the two belong together. Re-measured after the fix: gemma-4-E2B_q4_0 pp 136.24, tg 15.51, versus 136.55 and 15.49 for patch 31 alone -- identical within noise, so the fix is free. Coherence re-checked at --temp 0. The overflow itself was not reproduced (no qwen2.5-3b q4_0 locally); taken on the upstream report plus code reading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

beiller added 2 commits March 11, 2023 16:32

Remove unprintable characters from vocab list

e236dbb

More robust unprintable character check

5e625ea

beiller closed this Mar 11, 2023

beiller deleted the feature/remove_unprintable branch March 11, 2023 22:09

rooprob pushed a commit to rooprob/llama.cpp that referenced this pull request Aug 2, 2023

Merge pull request ggml-org#25 from wsmoses/master

0e4076c

Add information on compiler flags

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

codayon mentioned this pull request May 6, 2025

Misc. bug: [SYCL] Unexpected "setvars.sh has already been run" warning #13333

Closed

jesusmb1995 pushed a commit to jesusmb1995/llama.cpp that referenced this pull request Sep 30, 2025

Merge pull request ggml-org#25 from jesusmb1995/jmb/fixup_include

9009b2b

QVAC-6093: Stream shards. Fixup for gradle.

uttampc1 mentioned this pull request Nov 18, 2025

Throughput improvement for small batch sizes #17342

Open

sainnhe mentioned this pull request Jan 25, 2026

Eval bug: coredump due to ops of discontinuous tensor memory #19078

Closed

rururush pushed a commit to USTC-ADSL/llama.cpp that referenced this pull request Mar 16, 2026

opt mulmat base on official doc (ggml-org#25)

ff033e1

https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md

freelinuxer-cn mentioned this pull request May 9, 2026

Misc. bug: Meta backend ggml_context pool exhaustion with --split-mode tensor #22404

Closed

oobabooga added a commit to oobabooga/llama.cpp that referenced this pull request Jun 14, 2026

Show PR titles in automated release notes (follow-up to ggml-org#24) (g…

a4a0463

…gml-org#25)

zhsy12345689 pushed a commit to zhsy12345689/llama.cpp-omni that referenced this pull request Jul 30, 2026

Merge pull request ggml-org#25 from Halcao/opt/t2w

96efae5

opt: token2wav structural optimization

zhsy12345689 pushed a commit to zhsy12345689/llama.cpp-omni that referenced this pull request Jul 30, 2026

Merge pull request ggml-org#25 from Halcao/opt/t2w

e3d0a32

opt: token2wav structural optimization

1234562026 mentioned this pull request Aug 14, 2026

Double-free crash in console::history_t::~history_t() on Termux (Android aarch64 bionic libc) #27065

Open

cwriter mentioned this pull request Aug 22, 2026

Eval bug: SYCL dual-GPU stuck during model load #27547

Open

Copilot AI mentioned this pull request Aug 26, 2026

[WIP] Fix code scanning alert(s) in repository kerrrang9214-tech/llama.cpp#6

Merged

6 tasks

Navigation Menu

Uh oh!

Remove unprintable characters from vocab list - #25

Remove unprintable characters from vocab list#25
beiller wants to merge 2 commits into
ggml-org:masterfrom
beiller:feature/remove_unprintable

beiller commented Mar 11, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

beiller commented Mar 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

beiller commented Mar 11, 2023 •

edited

Loading