vulkan: small M matrix optimizations for qwen

[jeffbolznv]

Overview

Optimize m=1 by swapping A and B matrices. codex rediscovered this then told me ggml-cuda already did this in #26171.

Optimize small m (e.g. m=32) by changing tile size selection heuristic and allowing split_k.

These changes target these buckets which appear in recent qwen models:

before:
MUL_MAT f32 m=1 n=512 k=2048: 440 x 60.885 us = 26789.7 us (34.4356 GFLOPS/s)
MUL_MAT f32 m=32 n=512 k=2048: 660 x 61.496 us = 40587.4 us (1091 GFLOPS/s)

after:
MUL_MAT f32 m=1 n=512 k=2048: 440 x 8.57 us = 3771.04 us (244.633 GFLOPS/s)
MUL_MAT f32 m=32 n=512 k=2048: 660 x 21.535 us = 14213.3 us (3115.46 GFLOPS/s)

before

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 0 -p 512 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\Qwen3.6-35B-A3B-UD-Q3_K_M.gguf -m c:\models\Qwen3.8-27B-UD-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2v
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           pp512 |    11950.14 ± 103.16 |
| qwen35moe 35B.A3B Q3_K - Medium |  15.45 GiB |    34.66 B | Vulkan     |  -1 |   1 |           pp512 |      8995.91 ± 92.09 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | Vulkan     |  -1 |   1 |           pp512 |      2950.64 ± 11.72 |

build: 427291b5b (10816)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 0 -p 512 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\Qwen3.6-35B-A3B-UD-Q3_K_M.gguf -m c:\models\Qwen3.8-27B-UD-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           pp512 |      7914.64 ± 88.56 |
| qwen35moe 35B.A3B Q3_K - Medium |  15.45 GiB |    34.66 B | Vulkan     |  -1 |   1 |           pp512 |      5833.99 ± 62.46 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | Vulkan     |  -1 |   1 |           pp512 |       2231.80 ± 5.48 |

build: 427291b5b (10816)

after

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 0 -p 512 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\Qwen3.6-35B-A3B-UD-Q3_K_M.gguf -m c:\models\Qwen3.8-27B-UD-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2v
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           pp512 |    11920.52 ± 267.66 |
| qwen35moe 35B.A3B Q3_K - Medium |  15.45 GiB |    34.66 B | Vulkan     |  -1 |   1 |           pp512 |     9648.96 ± 131.71 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | Vulkan     |  -1 |   1 |           pp512 |      3437.53 ± 11.49 |

build: 22c8a29da (10818)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 0 -p 512 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\Qwen3.6-35B-A3B-UD-Q3_K_M.gguf -m c:\models\Qwen3.8-27B-UD-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           pp512 |      7902.34 ± 91.36 |
| qwen35moe 35B.A3B Q3_K - Medium |  15.45 GiB |    34.66 B | Vulkan     |  -1 |   1 |           pp512 |      7324.52 ± 98.68 |
| qwen35 27B Q4_K - Medium       |  15.32 GiB |    27.32 B | Vulkan     |  -1 |   1 |           pp512 |      2323.58 ± 10.16 |

build: 22c8a29da (10818)

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, paired with codex.

[0cc4m]

A bunch of ARGSORT tests failed completely on the T4 CM2 CI, please check.

[jeffbolznv]

A bunch of ARGSORT tests failed completely on the T4 CM2 CI, please check.

This seems to be affecting multiple PRs. Maybe related to the driver update? I haven't been able to reproduce it locally so far.

[0cc4m]

Very likely the driver, yes. CM1 passed on 580 GB10, the 615 T4 failed.