Feature Request: Vulkan: retune coopmat/MMQ tiles for RDNA3.5 / Strix Halo prefill

[Ankk98]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add a Strix Halo (gfx1151 / RDNA3.5) specific tuning for the Vulkan KHR_cooperative_matrix (coopmat) matrix-multiply tiles, to make prompt processing (prefill) faster on models like Qwen3.8-27B.

Today the AMD coopmat path uses one generic large tile for all AMD cards; Strix Halo is an integrated GPU with its own cache sizes and wavefront behavior, so it should get its own tile sizes, picked by measuring.

Measurements / observations

Setup: Radeon 8050S (32 CU), RADV / Mesa 26.1.8, uma:1, Qwen3.8-27B
UD-Q4_K_S, -p 128 -n 32 -b 128 -ub 128 -r 3 -fa on -ngl 99:

• pp128: 194.13 ± 1.45 t/s, tg32: 12.86 ± 0.10 t/s
• Prefill is >90% GEMM (MUL_MAT), all on the coopmat/MMQ path:

shape	iq4_xs	q4_K	q5_K
m=17408, k=5120 (FFN up/gate)	69x 2087us (10.9 TFLOPS)	17x 2084us (11.0 TFLOPS)	14x 2045us (11.2 TFLOPS)
m=5120, k=17408 (FFN down)	29x 2587us (8.8 TFLOPS)	12x 2533us (9.0 TFLOPS)	14x 2718us (8.4 TFLOPS)

Path checks (same flags, prove which code runs):

• GGML_VK_DISABLE_COOPMAT=1 → pp128 falls to 106.21 (-45%), tg32
  unchanged. So prefill runs on coopmat, decode does not.
• GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 → pp128 falls to 174.53
  (-10%), tg32 unchanged. So the integer-dot MMQ path is active in
  prefill with a modest share.

History: PR #18749 (generic AMD large tile {256,128,128}) gave
+35-75% prefill on 8060S. That tile reuses medium TM/TN/TK values and
was never specialized for gfx1151. The card reports only 16x16x16
fp16/int8 coopmat shapes (verified with a Vulkan probe), so there is
room left in workgroup geometry, subgroup size, and LDS traffic.

Related files and functions

All in ggml/src/ggml-vulkan/ (line numbers at HEAD 002a12ad2):

• ggml-vulkan.cpp:4540-4545 — AMD+coopmat large-tile override
  (this is where a gfx1151 branch goes)
• ggml-vulkan.cpp:4490-4498 — tm/tn/tk from probed coopmat_m/n/k
• ggml-vulkan.cpp:4568-4627 — shared-memory guards that silently
  disable tiles that do not fit (must check when adding tiles)
• ggml-vulkan.cpp:4166-4289 — ggml_vk_matmul_shmem_support /
  ggml_vk_matmul_int_shmem_support (LDS budget math)
• ggml-vulkan.cpp:7320-7327 — AMD tile enables (mul_mat_id_l
  stays false, see Eval bug: vulkan: enable large warptile for MUL_MAT_ID on AMD with coopmat #23106)
• vulkan-shaders/mul_mm.comp:43-47,161-192,383-386 — LOAD_VEC_A,
  spec constants, hardcoded A-RowMajor / B-ColumnMajor coop loads
• vulkan-shaders/mul_mmq.comp:103-104 — LOAD_VEC_A/B for int path
• vulkan-shaders/mul_mmq_funcs.glsl — int-dot dequant + dot
• Unmerged prototype branch: 0cc4m/vulkan-coopmat-int8
  (mul_mmq_cm1.comp, int8 coopmat MMQ, tip 918f32dcf)

Proposed behavior

1. Detect gfx1151 narrowly (AMD + coopmat + not proprietary driver +
   uma + 64 KiB shared memory + STRIX_HALO device name) and give it
   its own l/m_warptile_mmq* values, found by measurement: first
   BM/BN/BK geometry (including non-square), then subgroup 32 vs 64,
   then LOAD_VEC_A / LDS bank padding per quant type.
2. Generic AMD path and all other vendors/drivers stay exactly as now.
3. Longer term: rebase and evaluate the int8-coopmat prototype branch,
   starting with NVFP4 (simple dequant, already in the upstream MMQ
   type lists) before attempting iq4_xs.

Notes

• Assisted by a coding AI agent
• Current Commit: 002a12a

Motivation

Prefill at n=128 is compute-bound (about 380 FLOP/byte vs a machine balance of about 325), while decode (n=1) is memory-bandwidth-bound near the ~200 GB/s wall. So tile tuning moves prefill/TTFT without being able to hurt decode much and the per-shape table above shows prefill GEMM at ~9-11 TFLOPS against a ~65 TFLOPS WMMA roof, i.e. mostly headroom. Prior art (#18749) already proved tiles alone can move Strix Halo prefill by tens of percent.

Possible Implementation

• Step 1: sweep l_mmq BM/BN/BK ({256,128,128} vs {256,64,128}
  vs {128,128,128} vs {256,128,64}) and m_mmq at n=128;
  keep s tiles stable; verify each candidate survives the
  matmul_shmem_support guard via GGML_VK_PERF_LOGGER=1.
• Step 2: clone the winner with WARP 64 -> 32 (RDNA3 is natively
  wave32; current tiles use 64).
• Step 3: LOAD_VEC_A 4 -> 8 for q4_K/q5_K/iq4_xs int paths,
  SHMEM_STRIDE_PAD 8 vs 4/16 on the winning tile.
• Step 4: measure (do not assume) B-operand layout per dtype; expect
  ~0% on fp16.
• Step 5: rebase 0cc4m/vulkan-coopmat-int8, test-backend-ops all
  covered quants, measure pp128/512/2048 + tg32; NVFP4 first, then
  consider iq4_xs (43.8% of prefill, covered nowhere today).
• Bench rule: matched -b/-ub flags, cool-downs, tg32 guard on every
  step, test-backend-ops on every tile change. Tiles must stay
  bit-identical (same accumulation order) or pass greedy exact-match
  (--temp 0 --top-k 1 --seed 42).

[tczank]

First-party gfx1100 (RDNA3) datapoint for the generic-AMD-coopmat-tile retune, using your path-check methodology:

• GPU: AMD Radeon RX 7900 XTX (RADV NAVI31), KHR_coopmat matrix cores, uma:0 (discrete), shared memory 65536, warp 64
• Model: Qwen3.8-27B GSQ-RCO IQ3_XXS, 3-bit imatrix, 9.72 GiB / 27.32 B params, draft-mtp
• Build: llama.cpp ce8caa6 (b11065), Vulkan0, -ngl 99 -t 16 -ctk q4_0 -ctv q4_0 -fa on -p 512 -n 512 -r 2

coopmat path A/B (identical flags, same device):

	pp512 (prefill)	tg512 (decode)
coopmat ON	962.5 ± 0.5 t/s	53.5 ± 0.7 t/s
GGML_VK_DISABLE_COOPMAT=1	608.0 ± 0.4 t/s (−36.8%)	53.7 ± 0.0 t/s (flat)

Findings:

1. Prefill is carried by the coopmat matmul path — a −37% prefill drop with coopmat off, decode unchanged. gfx1100 shows the same coopmat-carried-prefill / decode-unaffected split as your gfx1151 case (yours −45%).
2. At 962.5 t/s prefill is ~52.7 TFLOPS ≈ 43% of the ~121 TFLOPS fp16 matrix-core roof — the same "mostly headroom" picture, on a discrete RDNA3 (gfx1100) rather than the Strix Halo iGPU (gfx1151).
3. The generic AMD tile {256,128,128} (PR Vulkan: Optimize Matmul parameters for AMD GPUs with Coopmat support #18749) is therefore also untuned on gfx1100; a gfx1100 retune is a second RDNA3 datapoint for the same work.

Reference datapoint only, not a PR.

[tczank]

Follow-up to my gfx1100 datapoint above — the retune is now measured with a correctness check, and I found a hard invariant the tile must satisfy. A naive "bigger tile" retune silently corrupts.

Setup: RX 7900 XTX (gfx1100, RADV NAVI31), llama.cpp b11065 (ce8caa6) + a local env-var override for the l-tile (GGML_VK_TILE_L=BM:BN:BK, rewrites l_warptile_mmq and l_mmq_wg_denoms in lockstep), Qwen3.8-27B IQ3_XXS (MTP), q4_0 KV, 16 threads, llama-bench -p 512 -n 512 -r 2; gaming env vars (MANGOHUD/RADV_PERFTEST/AMD_VULKAN_ICD/DXVK_ASYNC) cleared for all runs.

The invariant (coopmat D-write in mul_mm.comp): warps map onto the tile with warp_r = warp_i % (BM/WM), warp_c = warp_i / (BM/WM). Full coverage requires (BM/WM)·(BN/WN) == BLOCK/WARP. On gfx1100: subgroupSize = 64 → WARP = WM = WN = 64, BLOCK = 256 → 4 warps, so (BM/64)·(BN/64) == 4:

tile	(BM/64)·(BN/64)	result
128:128 (AMD default)	2×2	valid
64:256	1×4	valid
256:64	4×1	valid
256:128	4×2	invalid — only cols 0..63 of each 128-wide D tile are written; cols 64..127 are never written (D has no init, no fallback) → stale data

The 256:128 case is not just slow, it is corrupting: stale D → garbage hidden states → garbage logits → sampled token == n_vocab exactly (248320 for Qwen3.8) → rejected at the next batch init (init: invalid token[k] = 248320). That signature is byte-identical to #28158 — my case is a controlled, known bad-logits source that reproduces it. llama-bench cannot catch any of this (no correctness check): my first 256:128 bench reported +42.5% pp, which was just "half the matmul computed on garbage."

Measured results (valid tiles; correctness = token-identical vs baseline on a ~634-token prompt, temp 0 greedy):

config	pp512	tg512	correctness
baseline (128:128)	914.90	50.25	ref
tile 64:256:32	933.33 (+2.0%)	50.00 (−0.5%)	byte-identical
GGML_VK_ALLOW_GRAPHICS_QUEUE=1	919.13 (+0.5%)	53.32 (+6.1%)	—
tile 64:256:32 + gfx queue	962.73 (+5.8%)	53.56 (+6.8%)	byte-identical

256:64:32 is also byte-identical but slower (−14.5% pp). Decode is unaffected by the tile (the override only touches the l-tile; decode uses the s-tile) — the tg gain comes from the graphics-queue flag, not the tile.

Takeaways for the RDNA3 retune:

1. The only valid non-default l-tiles for this warptile layout are 64:256 and 256:64 — same total FLOPs as the default; expect single-digit % from tile shape, not double-digit.
2. GGML_VK_ALLOW_GRAPHICS_QUEUE=1 is the biggest single lever on this card (+6% decode).
3. If the warptile tables get retuned, add a sanity check (BM/WM)·(BN/WN) == BLOCK/WARP — a bad tile silently produces stale D columns and no benchmark can detect it. (The Intel Xe1 branch already satisfies the invariant by raising BLOCK to 512, so the pattern is in-tree; the AMD table just was never validated against it.)

Happy to share the exact benchmark/verify scripts.