ggml-cpu: add RVV vec dot kernels for quantization types by taimur-10x · Pull Request #7 · riseproject-dev/llama.cpp

taimur-10x · 2026-01-12T14:57:32Z

Summary

This PR adds RVV vector dot kernels for a number of quantization types.

Key Changes

Added the following RVV kernels:

Kernel	VLEN
ggml_vec_dot_iq1_s_q8_K	256
ggml_vec_dot_iq1_m_q8_K	256
ggml_vec_dot_iq2_s_q8_K	128, 256
ggml_vec_dot_iq3_s_q8_K	256
ggml_vec_dot_tq1_0_q8_K	256
ggml_vec_dot_tq2_0_q8_K	256

Testing

Kernels were functionally tested through test-quantize-fns for 128-bit and 256-bit VLENs.

Benchmarking Results

End-to-end benchmarking on BananaPI-BPI F3 (VLEN=256) with llama-bench.

IQ1_S

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	3.08	10.23
Tinyllama 1.1B	64	3.08	10.21
Tinyllama 1.1B	128	3.09	10.27
Tinyllama 1.1B	256	3.07	10.22
Tinyllama 1.1B	512	3.09	10.24

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	2.44	7.08
Tinyllama 1.1B	16	2.53	7.04
Tinyllama 1.1B	32	2.65	7.14
Tinyllama 1.1B	64	2.58	7.11
Tinyllama 1.1B	100	2.54	7.11

IQ1_M

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	1.96	8.01
Tinyllama 1.1B	64	1.95	8.05
Tinyllama 1.1B	128	1.96	8.04
Tinyllama 1.1B	256	1.96	8.05
Tinyllama 1.1B	512	1.94	7.98

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	1.71	5.62
Tinyllama 1.1B	16	1.73	5.59
Tinyllama 1.1B	32	1.73	5.64
Tinyllama 1.1B	64	1.72	5.63
Tinyllama 1.1B	100	1.73	5.65

IQ2_S

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	1.17
Tinyllama 1.1B	64	7.57	1.16
Tinyllama 1.1B	128	8.78	1.14
Tinyllama 1.1B	256	8.57	1.2
Tinyllama 1.1B	512	8.68	1.95

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	1.18
Tinyllama 1.1B	16	3.45	1.02
Tinyllama 1.1B	32	3.25	1.06
Tinyllama 1.1B	64	3.27	1.06
Tinyllama 1.1B	100	3.15	1.04

IQ3_S

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	1.19
Tinyllama 1.1B	64	7.57	1.25
Tinyllama 1.1B	128	8.78	1.21
Tinyllama 1.1B	256	8.57	1.18
Tinyllama 1.1B	512	8.68	1.12

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	1.22
Tinyllama 1.1B	16	3.45	1.14
Tinyllama 1.1B	32	3.25	1.13
Tinyllama 1.1B	64	3.27	1.13
Tinyllama 1.1B	100	3.15	1.12

TQ1_0

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	2.71
Tinyllama 1.1B	64	7.57	2.72
Tinyllama 1.1B	128	8.78	2.75
Tinyllama 1.1B	256	8.57	2.63
Tinyllama 1.1B	512	8.68	2.68

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	2.50
Tinyllama 1.1B	16	3.45	2.41
Tinyllama 1.1B	32	3.25	2.46
Tinyllama 1.1B	64	3.27	2.51
Tinyllama 1.1B	100	3.15	2.37

TQ2_0

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama 1.1B	32	8.42	4.37
Tinyllama 1.1B	64	7.57	3.89
Tinyllama 1.1B	128	8.78	3.81
Tinyllama 1.1B	256	8.57	3.81
Tinyllama 1.1B	512	8.68	3.79

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama 1.1B	10	3.11	4.22
Tinyllama 1.1B	16	3.45	3.60
Tinyllama 1.1B	32	3.25	3.51
Tinyllama 1.1B	64	3.27	3.16
Tinyllama 1.1B	100	3.15	3.11

Future Work

Subsequent PRs plan to extend existing RVV kernels for quantization types to other VLENs.

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

taimur-10x · 2026-01-14T15:44:58Z

@luhenry, @xctan, could this be reviewed please? Thank you.

) * ggml: backend-agnostic tensor parallelism * support for GPT-OSS, Qwen 3 MoE * partial Vulkan fix * add support for 4/8 GPUs * unconditional peer access * re-use buffers + ggml contexts * fix output pattern * NCCL support * GGML: HIP: add RCCL support * Remove shfl and AllReduce from backend interface * move allocation workaround out of ggml-alloc.c * 2d tensor set/get support * Fix the seg fault without NCCL * Apply suggestion from JohannesGaessler * support for tensor dims % n_devs != 0 * fix view_offs scaling * arbitrary num. of GPUs/tensor split * fix compilation * better granularity estimate * Support device-specific host buffer types if all underlying backends expose the same type. This allows using pinned memory instead of pageable memory for CUDA. Fix compilation errors. * partial Qwen 3 Next support * Fix qwen3 30b (#8) * Fix crash with Qwen-30B-A3B Q4_0 Qwen-30B-A3B Q4_0 has an intermediate dimension of 768. Using a granularity of 256 forces an uneven split between GPUs, which is not supported by the current implementation. * Decide block size based on tensor quantization type * Fix crashes due to KV cache serialization (#9) KV cache serialization requires non-zero offsets on the tensor. Add support in the meta backend to set/get a tensor with a non-zero offset. * metal : fix build (#7) * static memory allocations, fix usage count * fix tensor granularity * more even memory distribution * use BF16 for allreduce * rebase fixup * better error message for unsupported architectures * Fix device mismatch during scatter of allReduce. (#11) There is a mismatch between the dst buffer device and the backend device, causing the use of sync copies * Enable the previous allreduce implementation. It is better in both perf and stability (#12) * delay AllReduce for Moe for less I/O * build : clean-up compile warnings * backend : move most of the meta backend API to ggml-backend-impl.h * cont : hide unused public API in the implementation * llama : use llama_device + remove ggml_backend_dev_is_meta() * ggml-backend : remove unused alloc include * minor : remove regex include * ggml : introduce ggml-ext.h for staging new APIs * rebase fixup * fix tests * llama : more robust logic for determining Meta devices (ggml-org#16) * llama : more robust logic for determining Meta devices * cont : fix devs size check Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * cont : fix log type Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * disable roundtrip for meta backend * fix arch selection * Qwen 3.5 support * fix Gemma 4 MoE * fix OpenVino, SYCL * fix test-llama-archs for CPU-only builds * Fix Qwen 3.5 MoE * disable meta backend tests for WebGPU * tests : filter CPU-based devices from the Meta backend tests (ggml-org#17) * meta : formatting, naming, indentation (ggml-org#18) * formatting : llama-model.cpp * formatting : ggml-ext.h * formatting : ggml-backend-meta.cpp * meta : add TODO * add documentation * better error messages * fix GPT-OSS --------- Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* spec: support MTP * fix batch size * rename files * cont : simplify (#7) * MTP: clean-up (#9) * MTP: clean-up * review: use llama_context_type instead of llama_graph_type * review: remove llama_model_has_mtp * review: fix convert issues * convert: fix pycheck * review: formatting * use `mtp-` for identifying mtp models * convert: fix mtp conversion * mtp -> draft-mtp * remove unused llama_arch * add need_embd in speculative * llama: allow partial seq_rm for GDN models for speculative decoding Currently speculative checkpoint needs to restart from a checkpoint after some draft tokens are not accepted, this leads to some wastage in running the target again. This PR adds the ability to rollback upto `draft_max` by storing the GDN intermediates. * fix pending state * vulkan: add GDN partial rollback * meta: extend check to axis 1 * metal: add GDN partial rollback Extend the gated delta net kernel to store intermediate states for partial rollback support on the Metal backend. - Add K (snapshot slot count) as a function constant - Read input state from slot 0 of the 3D state tensor - Write intermediate states to different slots during token loop - For K=1, maintain backward-compatible single-slot behavior Ref: ggml-org@8c05923 Assisted-by: llama.cpp:local pi * delta_net_base: use ggml_pad instead of new_tensor * review: add need_rs_seq * review: rename part_bounded to n_rs * review: deslop comments * review: rename, add asserts * server : adjust checkpoint logic (#11) * server : adjust checkpoint logic * cont : rm asserts * server-context: fix early exit * spec : fix compatibility with n-gram and add TODOs (#13) * metal : cleanup * llama : fix faulty bitwise check in recurrent memory * server : disable RS-based MTP in combination with other spec types * spec : add TODOs * cont : fix comment * cont : update comment * common : fix logic for ngram + mtp compat * llama-memory: enable checkpointing with partial rollback * cont: add test-case for loading into a dirty ctx * llama-memory-recurrent: clear rs_idx in clear * download: fix mtp path * llama-arch: fix enorm op * docs: update docs * conversion: fix type annotations --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

@pcuenca

* Get started with Onyx * Add architecture * Skip keys handled in super() * Loading tensors * Shorten * Graph * Apply suggestion from @pcuenca * Remove norm now embedding in transformers weights * Add eot * Explicit output_multiplier * Handle post_norm_eps * No super call; unhardcode eot. The pattern `self._set_vocab_gpt2()` seems preferred throughout the codebase, and it allows `set_vocab()` to be called from a different part of the Python class hierarchy: the drafter model converter that we may need eventually. * Register for drafting * DFlash: inherit rope type from the linked target. Another option would be to store it in the gguf file itself. * mmproj conversion Note: some fields to be renamed after the implementation works. We are keeping compatibility with the reference Meta gguf for testing purposes. * "clip" header declarations * Load mmproj * Pre-processing * Graph * Go back to using delimiters. Otherwise our generations are worse. Transformers does not use them. We need to trace inputs to verify whether they are equivalent. * downsample_factor -> merge_size * Add vision graph lol, forgot from a previous commit * Additional renames, align with llama.cpp / transformers * Prefer _size instead of independent _h and _w * Fix token layout Co-authored-by: Young Han <younghan@fb.com> * onyx: bring the chat parser onto the onyx branch common/chat.cpp on this branch has no Onyx handling, so a converted model serves malformed chat: the assistant preamble leaks into content ("to=self<|message|>...") and tool calls fail with HTTP 500 "The model produced output that does not match the expected peg-native format" common_chat_params_init_onyx exists on onyx-fair-patch, added there by 8bb73dd3d. It was never on this branch, so this is not a regression -- the two lines developed independently. The code here is taken verbatim from that commit. It is the clean side of `git merge origin/onyx-fair-patch`: chat.cpp is one of the files that merges without conflict. The full merge is not viable -- it produces 13 conflicts, including add/add on conversion/onyx.py and src/models/onyx.cpp where the q_norm-folding and metadata-scale approaches contradict each other, and #4/#7 are stacked on this branch's side of that. Verified on this branch: builds with 0 errors, converts an Onyx checkpoint, and serving it gives "4" for "What is 2+2?" plus a correct get_weather {"city":"Paris"} tool call, where the unported branch gives the two failures above. No converter or runtime changes are included, so this should not interact with the q_norm work. Co-authored-by: Beto de Paola <betodepaola@meta.com> * Less params, bilinear pos-emb interpolation as a graph op instead of CPU * Map to symbolic V_MMPROJ instead of strings * Make a couple params explicit * Patchify via build_inp() * No param for rope_theta * Small cleanup * Restore blank line * Unpermute, to adapt to the latest transformers checkpoint * Apply norm after token embeddings This follows the latest transformers approach. * Remove duplicated function * build_vit * onyx: use the model rope theta on sliding-window layers * DFlash: conversion from transformers drafter * Revert rope_type derivation from target NOTE: this breaks compatibility with Meta's distributed DFlash GGUFs, as the Q/K are stored in "NEOX" (rotated half) format, like in transformers. * Apply suggestion from @pcuenca * Set model type * Remove comment that will become obsolete * Hardcode post_norm_rms_eps instead of new param * Derive SWA+RoPE pattern from gguf array or scalar * Fix model type <-> number of layers * Reorder * Rename * Fix typo * DFlash: seed the draft KV cache from multimodal embedding batches `common_speculative_impl_draft_dflash::process()` returned early on any batch carrying embeddings, so an image prefill never had its target-layer features fused through the DFlash encoder and injected into the draft's KV cache. That left a hole spanning the image's positions, and the next injection at a post-image position failed to initialize its batch: ``` decoding image batch 1/1, n_tokens_batch = 256 decode: failed to initialize batch llama_decode: failed to decode, ret = -1 process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=17, offset=0) srv decode: failed to process speculative batch ``` Every image request with `--spec-type draft-dflash` failed with HTTP 500. Text-only was unaffected, since those batches carry token ids and were let through. Restore the earlier condition, which admits a batch that is either tokens or embeddings and skips only the degenerate neither/both cases. The rest of `process()` is already layout-agnostic -- it gathers features via `llama_get_embeddings_layer_inp()` and indexes `batch_in.pos[]` / `batch_in.seq_id[]`, none of which assume token ids -- so this is the whole fix. Validated against `muse-glimmer-30B-bf16.gguf` + `mmproj-muse-glimmer-30B-bf16.gguf` + a DFlash draft head, on an image describe-the-shapes request: - before: HTTP 500, `failed to process speculative batch` - after: HTTP 200, draft acceptance 0.34012 (167 accepted / 491 generated), mean len 3.04 Output equivalence holds, which is the property that matters: at temperature 0 the drafted response is byte-identical to the same request served with no draft attached (1213/1213 chars), so the draft is drafting correctly through the image context rather than merely not crashing. * Conversion: prefer rewrite to mapping * Revert "Conversion: prefer rewrite to mapping" This reverts commit a92d0ac. * fix lint * sliding_window metadata is not optional * disable state save/load * Apply suggestion from @pcuenca --------- Co-authored-by: Young Han <younghan@fb.com> Co-authored-by: Beto de Paola <betodepaola@meta.com> Co-authored-by: Daniel Han <michaelhan2050@gmail.com> Co-authored-by: ruanrms <ruanslv@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

taimur-10x marked this pull request as draft January 12, 2026 14:57

github-actions Bot added the ggml label Jan 12, 2026

taimur-10x and others added 2 commits January 12, 2026 20:01

ggml-cpu: add rvv vec_dot for iq3_s

fc2a2dd

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

ggml-cpu: add rvv vec_dot for tq1_0, tq2_0

d432cf5

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

taimur-10x force-pushed the 10x/riscv-quant-vec-dot-kernels branch from ef71fd4 to d432cf5 Compare January 12, 2026 15:08

taimur-10x self-assigned this Jan 12, 2026

taimur-10x force-pushed the 10x/riscv-quant-vec-dot-kernels branch from 716d818 to 85ecce6 Compare January 13, 2026 14:23

ggml-cpu: add rvv vec_dot for iq1_s, iq1_m

8814361

Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>

taimur-10x force-pushed the 10x/riscv-quant-vec-dot-kernels branch from 85ecce6 to 8814361 Compare January 14, 2026 11:20

taimur-10x marked this pull request as ready for review January 14, 2026 12:13

taimur-10x requested review from david-baker-808 and luhenry January 14, 2026 12:13

xctan approved these changes Jan 19, 2026

View reviewed changes

taimur-10x changed the base branch from master to 10x/riscv-quant-vec-dot January 20, 2026 10:43

taimur-10x merged this pull request into 10x/riscv-quant-vec-dot Jan 20, 2026
58 of 76 checks passed

Navigation Menu

Uh oh!

ggml-cpu: add RVV vec dot kernels for quantization types - #7

ggml-cpu: add RVV vec dot kernels for quantization types#7
taimur-10x merged 3 commits into
10x/riscv-quant-vec-dotfrom
10x/riscv-quant-vec-dot-kernels

taimur-10x commented Jan 12, 2026 •

edited

Loading

Uh oh!

taimur-10x commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

taimur-10x commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Testing

Benchmarking Results

IQ1_S

Tokens / Second

IQ1_M

Tokens / Second

IQ2_S

Tokens / Second

IQ3_S

Tokens / Second

TQ1_0

Tokens / Second

TQ2_0

Tokens / Second

Future Work

Uh oh!

taimur-10x commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

taimur-10x commented Jan 12, 2026 •

edited

Loading