Repository navigation
sycl: add grouped MoE XMX GEMM - #29245
Conversation
github-actions
Bot
added
ggml
|
Hm, I see that you have cases for IQ4_NL + IQ3_S weights. Is there some way to generalize the kernel to accept more quant types? |
arthw
left a comment
arthw
left a comment
There was a problem hiding this comment.
It's good idea!
I didn't get the benefit with Qwen2-1.5Moe.Q4_K_M.gguf.
Could you share your test method (gguf, parameter) to reproduce the result?
What's your Intel GPU in the test?
Thank you!
|
Hi
There is, I have implemented some more types (including Q8_0), although the effects are much more limited there. I thought it more useful to add the biggest gains now without blowing up the commit. If you prefer, I can update the commit with more types, but we're talking about ~2% for the remaining types for the IQ model I'm tuning. The trade-off will be the size of the commit.
I'm using 3x Arc Pro B60 on an IQ model |
ecce82b to
3a2f173
Compare
github-actions
Bot
added
the
documentation
|
@arthw If I remember correctly, you're on an A770 16GB, right?
@Titaniumtown I've added all IQ types for now. Would it be ok to do other types in a later commit? |
A MUL_MAT_ID over a MoE layer ran one library GEMM per expert. With 128 experts
and 8 active, each is a sliver, so launches dominate and the XMX units idle.
This groups every expert of a node into one launch, laying out the work-groups
on the host from the row offsets it already has. Weights are dequantized inside
the GEMM, gathered into the XMX A tiles instead of being written out to f16 and
read back; activations are packed once into VNNI f16.
Nine weight formats have an A stage: iq4_nl, iq3_s, iq4_xs, iq3_xxs, iq2_xxs,
iq2_xs, iq2_s, iq1_s, iq1_m. Which of them are covered decides almost the whole
result, so GGML_SYCL_XMX_GATHER_TYPES selects them as a bitmask, one bit per
format, all set by default; 0 disables the paths and is the baseline to measure
against. Quant names do not give coverage away: unsloth's UD quants mix per
tensor, so a file named IQ3_XXS holds IQ3_XXS, IQ3_S and IQ4_XS experts, and a
Q4_K_M model has no IQ tensors at all and cannot benefit.
Both paths compute in f16 on XMX. The grouped MUL_MAT_ID path therefore trades
some precision against the per-expert library GEMM it replaces, which in a
default build (GGML_SYCL_F16=OFF) computes in f32. The plain MUL_MAT half only
engages when built with GGML_SYCL_F16=ON, as it sits inside that f16 branch.
One Arc Pro B60, unsloth/Qwen3-30B-A3B-GGUF UD-IQ3_XXS, -fa on, 4 interleaved
passes after a warm-up:
GGML_SYCL_XMX_GATHER_TYPES pp512 pp2048 tg128
0 (off) 671.1 677.2 49.3
iq4_nl|iq3_s (42/144) 720.1 725.3 49.4 +7.3%
all formats (144/144) 1016.9 1006.9 49.3 +51.5%
Token generation is unchanged, as expected for a prefill path.
test-backend-ops -o MUL_MAT_ID and -o MUL_MAT against the CPU backend: 125 and
191 iq* cases, zero failures with every bit set, identical to the disabled arm.
Assisted-by: Claude Opus 5
3a2f173 to
950f1c4
Compare
There was a problem hiding this comment.
It shows good performance on B60:
| GGUF | Test | fa | Base t/s | Primary t/s | Increase Rate (Primary vs Base) |
|---|---|---|---|---|---|
| Qwen3-30B-A3B-UD-IQ3_XXS.gguf | pp512 | 0 | 611.26 | 918.15 | 50.21% |
| Qwen3-30B-A3B-UD-IQ3_XXS.gguf | pp512 | 1 | 641.75 | 990.48 | 54.34% |
| Qwen3-30B-A3B-UD-IQ3_XXS.gguf | tg128 | 0 | 46.80 | 46.85 | 0.11% |
| Qwen3-30B-A3B-UD-IQ3_XXS.gguf | tg128 | 1 | 49.00 | 49.01 | 0.02% |
The tile was hard-coded to f16 8x16x16 with sub-group 16, so devices without that combination (Arc A770 and other Xe-HPG parts) never took the path. The kernels, the B packing and the grouped schedule are now templated on the shape, and each device uses the first shape in a fixed preference order that it lists in matrix_combinations. Shapes: f16 8x16x16, 16x16x16, 32x64x16, 32x64x32 (sub-group 16), f16 8x8x16 (sub-group 8, Xe-HPG) and tf32 8x16x8 (promoted, f32 data rounded to tf32). GGML_SYCL_XMX_GATHER_SHAPES masks them, one bit per shape, to force or exclude one. AOT: ocloc fails on a joint_matrix kernel whose sub-group size the target lacks, so a spir64_gen GGML_SYCL_DEVICE_ARCH build compiles only the shapes of the matching size, and intel_gpu_* targets compile each tile body only where it fits. Before this, an AOT build for dg2 failed with an IGC internal error. Assisted-by: Claude Opus 5.5
Rewrite the GGML_SYCL_XMX_GATHER_TYPES entry in SYCL.md: an example that enables two formats at once, which GPUs and models benefit, and the exact conditions under which an enabled format actually takes the path. Rename the dispatch result from `ok` to `launched`. Assisted-by: Claude Opus 5.5
The joint_matrix combination was picked once per device and assumed f16 A and B. It is now picked per call from the src1 type, among the combinations the device lists and GGML_SYCL_XMX_GATHER_SHAPES allows. The rules, in order: a tile that does not spill, the fastest matrix type (f16 and bf16 before tf32), a B type equal to the src1 type, a B type at least as precise as f16, the device's native DPAS tile (8x16x16 on Xe2), then the largest M x K. On B60 the default is now f16 8x16x16. The kernels are templated on the A and B types separately. bf16 8x16x16 and 8x8x16 are added as bits 64 and 128, off by default. The plain MUL_MAT path hands src1 to the pack in its own type (F32, F16 or BF16) instead of running a separate to-f16 pass first, so an f16 src1 is packed as a plain copy. The same ops take the XMX paths as before. Assisted-by: Claude Opus 5.5
|
I've added a patch that generalizes the kernels into different shapes (and types). One key change was necessary to the CMakeLists.txt so the compiler would not panic in device-specific AOT, there is some bookkeeping around not compiling for non-targeted applications. Other targets (like AMX) should also be possible to add, but I unfortunately do not have the suitable hardware for this. |
|
Here is my test result on Intel GPUs. Qwen3-30B-A3B-UD-IQ3_XXS.gguf on B580x2
On Arc770:
On Arl-h: Intel Arc 140T GPU (16GB) Windows 11, Qwen3.5-35B-A3B-UD-IQ2_XXS.gguf debug log:
|
|
@cwriter As the guide, is that Thank you! |
|
The XMX dequant-GEMM paths always used f16 operands, so an activation above 65504 became inf. GGML_SYCL_DYNAMIC_PRECISION selects the operand type: F16, BF16 or TF32 (f32 range), or F32, which turns the XMX paths off and keeps the library GEMM and dmmv from converting src1 to f16. The default is F16 with GGML_SYCL_F16 and F32 otherwise, so a build without GGML_SYCL_F16 no longer takes the XMX paths unless asked. GGML_SYCL_XMX_GATHER_SHAPES now allows all combinations by default and masks those of the selected type. The src1 precision request of an op [TAG_GGML_PREC] is met in every mode. F16, Q8 and Q4 allow any type. BF16 needs the f32 range: such an op runs on the mode type if it has it, else bf16, else tf32, else the library f32 GEMM. F32 runs on the library f32 GEMM, unless GGML_SYCL_DYNAMIC_REQUIRED_PRECISION allows tf32 (TF32) or tf32 and bf16 (BF16); F16 there ignores the requests, for testing. The library GEMM and dmmv do not convert src1 to f16 for a BF16 or F32 request. The XMX paths accumulate in f32, so they take any accumulator request. The precision rule follows 54acc96 (GGML_SYCL_ENFORCE_PREC), always on and without reporting MUL_MAT_ID as unsupported. Assisted-by: Claude Opus 5.5
The grouped MUL_MAT_ID path now also takes Q8_0, Q4_K, Q5_K and Q6_K expert weights, so Q4_K_M, Q5_K_M and Q8_0 MoE models get the grouped launch. New GGML_SYCL_XMX_GATHER_TYPES bits: 512 Q8_0, 1024 Q4_K, 2048 Q5_K, 4096 Q6_K. K must be a multiple of 32 for Q8_0 and 256 for the k-quants. The plain MUL_MAT path keeps these types on the library GEMM: it decodes the weights again for every 32 columns of the batch, which for these formats costs more than the library's one dequantization. The k-quant experts can be in the reorder (SoA) layout of reorder_qw() after a decode step. Both layouts are decoded, from the same per-sub-block decode. Assisted-by: Claude Opus 5.5
The grouped dequant-GEMM decodes the reorder (SoA) layout of Q4_K, Q5_K and Q6_K experts faster than the canonical one, but only the decode path (mmvq) installed it. So the first prefill, and any prefill without a decode before it, used the slower layout. Install it in the grouped path too when the XMX paths can take the type. Every reader of these tensors already handles the reorder, as it happens after the first decode step anyway. Assisted-by: Claude Opus 5.5
|
Provided the update. There is a small caveat: Unlike for the IQ kernels, the QK kernels make no sense for dense With the updated patch, for Qwen3-30B-A3B, I get the following performance: (pp512, tg is unchanged, running on 1-2 B60 (Q8_0 does not fit a single card))
The precision change is also included. The patch makes SYCL respect the set tensor precision from the GGML core. However, ggml is missing the TF32 rank.
As the table shows, even nominally same-performance bf16 takes a significant hit. TF32 is particularily slow even on supporting hardware, where the library gemm beats it significantly (I assume it's due to kernel spilling, but I did not want to further investigate it for now - there really is limited gains). I'll be happy to add more improvements, but I think that this PR is already growing out of control. |
|
@cwriter This PR involve the XMX direct programming (joint_matrix) firstly. Thank you for the good job! |
arthw
left a comment
arthw
left a comment
There was a problem hiding this comment.
It's good job!
Involve the XMX directly programming (based on joint_matrix).
Thank you!
arthw
added
the
merge ready
|
@Titaniumtown Would you like to do a review before merging? |
|
I will look over the changes now. Been busy with work IRL. |
Assisted-by: Claude Opus 5.5
|
Thank you for your contribution! |
Co-authored-by: cwriter <cwriter@localhost> (cherry picked from commit 847f447)
…and gate fusion Upstream merged its own versions of work this branch carries (ggml-org#29375 Q5_K reorder-layout MMVQ and fused GLU, ggml-org#29245 grouped MoE XMX GEMM, ggml-org#29608 pinned upload staging, ggml-org#29687 delta-net alpha gate fusion), and the commits before this one were replayed onto it with their own side taken in every conflicting hunk. This commit puts the upstream side back where both belong: - common.hpp, fusion.cpp, the op tests: both sides' additions (tile schedule of the grouped GEMM, the fused unary predicate, test_top_k_inf) - mul_mat_vec_q_reorder_ncols takes both new template flags (has_tiles, shared_weights); the Q5_K multi-column dispatch is upstream's (weights shared and rows paired on Xe2) - ggml_sycl_op_mul_mat_sycl tries upstream's fused dequant GEMM first and keeps the bf16 src0 route after it - the gate / up / GLU fusion uses upstream's reorder pairs (same-type Q4_K, Q5_K on Xe2 up to 5 columns) and keeps this branch's rule that pairs both reordered run unfused - MUL_MAT_ID keeps this branch's sorted loop; upstream's grouped GEMM is built but not called from it Qwen3.8-27B on two Arc Pro B70 against the tip before the rebase (llama-bench, depth 0, three mirrored runs each): 2 to 32 rows equal or 1-4% faster (inside today's 2-4% run-to-run spread), 48 / 64 rows 190 -> 160 / 200 -> 168 ms per batch (the fused dequant GEMM), 1024 rows equal. Perplexity unchanged on the 27B (one card and two, 1024-row and 8-row batches) and on Ornith. 17/17 op suites clean (MUL_MAT 2711, FLASH_ATTN_EXT 4211). Assisted-by: Claude Opus 5.5
…and gate fusion Upstream merged its own versions of work this branch carries (ggml-org#29375 Q5_K reorder-layout MMVQ and fused GLU, ggml-org#29245 grouped MoE XMX GEMM, ggml-org#29608 pinned upload staging, ggml-org#29687 delta-net alpha gate fusion), and the commits before this one were replayed onto it with their own side taken in every conflicting hunk. This commit puts the upstream side back where both belong: - common.hpp, fusion.cpp, the op tests: both sides' additions (tile schedule of the grouped GEMM, the fused unary predicate, test_top_k_inf) - mul_mat_vec_q_reorder_ncols takes both new template flags (has_tiles, shared_weights); the Q5_K multi-column dispatch is upstream's (weights shared and rows paired on Xe2) - ggml_sycl_op_mul_mat_sycl tries upstream's fused dequant GEMM first and keeps the bf16 src0 route after it - the gate / up / GLU fusion uses upstream's reorder pairs (same-type Q4_K, Q5_K on Xe2 up to 5 columns) and keeps this branch's rule that pairs both reordered run unfused - MUL_MAT_ID keeps this branch's sorted loop; upstream's grouped GEMM is built but not called from it Qwen3.8-27B on two Arc Pro B70 against the tip before the rebase (llama-bench, depth 0, three mirrored runs each): 2 to 32 rows equal or 1-4% faster (inside today's 2-4% run-to-run spread), 48 / 64 rows 190 -> 160 / 200 -> 168 ms per batch (the fused dequant GEMM), 1024 rows equal. Perplexity unchanged on the 27B (one card and two, 1024-row and 8-row batches) and on Ornith. 17/17 op suites clean (MUL_MAT 2711, FLASH_ATTN_EXT 4211). Assisted-by: Claude Opus 5.5
4 participants
Overview
The current approach to handling experts in MoE models use one GEMM per expert. These activations are relatively narrow, and cannot use the compute hardware effectively (neither ALU0/1 nor the XMX tiles). Additionally, the small inputs result in particularly inefficient dequantization.
This can be fixed by grouping many small experts into one kernel, and handling them in one operation.
This patch proposes grouped and optionally fused GEMM kernels tuned on XMX (
joint_matrix).Approach:
Additional information
Model:
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS.The boost happens during pp.
Requirements