ggml-hrx: the f32_f32 WMMA cores accumulate in f32 (engine #284)

[bong-water-water-bong]

A fix for 1bit-MONSTER/engine#284.

Root cause

ggml_mul_mat_id_f32_f32_wmma and the other f32_f32 WMMA cores carried the
whole K accumulation in an f16 accumulator vector (vector<8xf16>), which
lowers to v_wmma_f16_16x16x16_f16. On gfx11 that accumulator keeps about
8 mantissa bits: it is not f16 rounding. Simulating a bf16 accumulation of the
same 16-wide steps reproduces the issue's numbers exactly, while a true f16
accumulation of 0.375 steps is exact.

From the issue's setup (MXFP4 weights of 0x77 = 0.0234375, 1.0 inputs,
output_size 96, 1 token, 1 route):

K	exact	f16 accumulator (issue)	bf16 model (this PR's diagnosis)
1152	27.0	27.0	27.0
1440	33.75	34.3125	34.0
2048	48.0	53.3125	53.0
2304	54.0	61.3125	61.0
2816	66.0	77.3125 (+17%)	77.0
2880	67.5	79.3125	79.0

Change

Use the f32 accumulator of the same 16x16x16 WMMA. gfx11 wave64 f16->f32 uses
a 4xf32 accumulator payload (rdna3_wmmar3_f32_16x16x16_f16_w64,
v_wmma_f32_16x16x16_f16), so vector<8xf16> becomes vector<4xf32> and the
finished tile is staged in a plain row-major f32 view. The publish path already
consumed f32, so the vector.extf widen is gone (the swiglu cores dropped
their gate/up narrow->widen too).

Covers the core the issue reports plus the two it names as possibly the same
root cause, and the routed-expert swiglu that shares the motif:

• motifs/mul_mat_id_f32_f32_wmma_core.loom — common.mul_mat_id.f32_f32_wmma (the reported kernel)
• motifs/mul_mat_f32_f32_wmma_core.loom — common.mul_mat.f32_f32_wmma
• ops/mul_mat_swiglu_f32_f32_wmma.loom — common.mul_mat_swiglu.f32_f32_wmma
• ops/mul_mat_id_swiglu_f32_f32_wmma.loom — routed-expert swiglu

Verification done here

Link + compile each affected export for gfx11-generic, then disassemble:

export	before	after
ggml_mul_mat_id_f32_f32_wmma	v_wmma_f16_16x16x16_f16 x8	v_wmma_f32_16x16x16_f16 x8
ggml_mul_mat_f32_f32_wmma	x8 f16	x8 f32
ggml_mul_mat_swiglu_f32_f32_wmma	x16 f16	x16 f32
ggml_mul_mat_id_swiglu_f32_f32_wmma	x16 f16	x16 f32
ggml_mul_mat_id_postops_f32_f32_wmma	x8 f16	x8 f32
ggml_mul_mat_bias_f32_f32_wmma	x8 f16	x8 f32
ggml_mul_mat_add_f32_f32_wmma	—	x8 f32
ggml_mul_mat_bias_add_f32_f32_wmma	—	x8 f32
llm_attention_v_matmul_set_rows_f32_f32_wmma	x8 f16	x8 f32

All 21 exports that depend on the four changed files were compile-checked; every
one that compiled before still compiles (the handful that fail do so identically
on the base commit, from template-specialization constraints on the small
synthetic configs used to compile them standalone).

Not verified here

No gfx1151 box was available, so this PR has no numeric run on Strix Halo.
Before merge someone should re-run the issue's calib.sh K sweep to confirm the
values become exact, and llama-bench to measure the cost of the f32
accumulator (RDNA3 may issue f16-accumulate WMMA at a higher rate). If it costs
too much, the alternative is periodic f16->f32 flushes, but the f32 accumulator
is the straightforward correct fix.

[bong-water-water-bong]

Adding the check that matters most: the same link+compile+disassemble for gfx1151, the part in engine ggml-org#284.

loom-compile <linked mul_mat_id export> --backend=amdgpu-hal --target=gfx1151 \
  --config=ggml.mul_mat_id.input_size=2048 --config=ggml.mul_mat_id.output_size=4096 \
  --config=ggml.mul_mat_id.expert_count=2 --config=ggml.mul_mat_id.route_count=1 \
  --config=ggml.mul_mat_id.input_route_count=1 --config=ggml.mul_mat_id.weight_format=39 \
  --config=ggml.workload.token_capacity=512

target	base (f5b7f4a)	this PR
gfx1100	v_wmma_f16_16x16x16_f16 x8	v_wmma_f32_16x16x16_f16 x8
gfx1151	v_wmma_f16_16x16x16_f16 x8	v_wmma_f32_16x16x16_f16 x8

gfx1201 rejects the kernel, which is expected: the export's target record is
gfx11-generic, so the corpus is gfx11-family only.

This is still instruction selection, not a numeric run — a Strix Halo box is
needed for that.

[bong-water-water-bong]

One more build-level check, plus where existing coverage lives.

The corpus build step itself. The .loom sources are embedded at configure
time by ggml/src/ggml-hrx/tools/generate_kernel_corpus.py (source format
binary, so it loom-links every source). Running the real command over the
whole loom-libs manifest with these changes:

generate_kernel_corpus.py --source-format binary \
  --manifest .../loom-libs/manifest.json --corpus-dir .../loom-libs \
  --loom-link loom-link --loom-format loom-format
-> embedded kernel corpus: source_files=87 source_bytes=2322655 source_format=binary

Existing numeric coverage on hardware. tests/test-backend-ops.cpp already
has enabled test_mul_mat cases at the issue's shape —
GGML_TYPE_MXFP4, GGML_TYPE_F32, 2880, 32, 2880 and the Q4_0/Q8_0 twins — and
test_mul_mat_id's long-K cases exist but are under #if 0 ("A matrix > 4GB,
too slow by default"). So test-backend-ops -b HRX0 and hrx-backend-test
should show this on Strix Halo; I just cannot run them here.

Left alone deliberately. mul_mat_id_f16_f16_wmma_core.loom and the q4k/q5k/
q6k projection motifs keep their f16 accumulators. They feed f16-output paths
where the f16 accumulator is the existing contract; if they should move too,
that is a separate change with its own accuracy measurement.

[bong-water-water-bong]

Static cost of the change, since I could not benchmark on hardware. Same link,
same gfx11-generic target, same configs (input_size=2048, output_size=4096, expert_count=2, route_count=1, weight_format=39, token_capacity=512), from the
loom-compile --compile-report=text-summary for the mul_mat_id export:

	instructions	code bytes	vgpr (final)	spills	occupancy
base f5b7f4a	1165	6284	112	0	25%
this PR	1109	5996	112	0	25%

The f32 accumulator is smaller: −56 instructions (−4.8%) and −288 bytes, at
identical register pressure, no spills and the same occupancy. Dropping the
extf widen and the f16 staging round trip pays for the wider accumulator.

This says nothing about the runtime rate of v_wmma_f32_16x16x16_f16 versus
v_wmma_f16_16x16x16_f16 on RDNA3, which still needs a llama-bench run on
Strix Halo.

[bong-water-water-bong]

Verified on strixhalo (gfx1151, balanced power mode 85 W). As posted, 85f5983 gives wrong results. I pushed a fix, b5669e0, onto this branch.

The bug in 85f5983. The change removed encoding.layout.strided [1, 16] from the result fragment views. The fragment is stored as [channel, token], but the publish path reads the staged tile as [token][channel], so every published vector read the wrong elements. A constant case with all 16 tile tokens live hides this. With fewer live tokens it shows up. b5669e0 restores the strided layout on the f32 views in all four files.

Known-answer cases. Run with iree-test-loom --sanitizer=access, linked with the export's manifest recipe. Weights are MXFP4 0x77 (0.0234375) and inputs are 1.0, so the expected output is K x 0.0234375 (swiglu: silu(g) x u). mul_mat and mul_mat_swiglu use 6 tokens; mul_mat_id and mul_mat_id_swiglu use 1 token and 1 route. K = 1152, 2048, 2816, 2880, 4096, 5120, 2 repeats each.

kernel	base f5b7f4a	85f5983	b5669e0
mul_mat_id	1/6 exact (K=2816: 77.3125, not 66)	0/6 (element 1 = 0)	6/6 exact
mul_mat	1/6 (K=2816: 77.3125)	0/6 (element 6 = 0)	6/6
mul_mat_swiglu	1/6 (K=2816: 5977.2, not 4356)	0/6 (element 6 = 0)	6/6
mul_mat_id_swiglu	1/6	0/6 (element 1 = 0)	6/6

The access sanitizer passes for all four kernels.

Models (base vs f5b7f4a + 85f5983 + b5669e0):

• gpt-oss-20b, KLD vs CPU (c512, b512, 8 chunks, -fa on): 0.028448 -> 0.026932. Same top 87.75% -> 89.61%. Max KLD 1.99 -> 2.92. The PR value was the same on 2 runs.
• Qwen3.8-27B UD-Q4_K_XL, KLD vs BF16 logits (c512, b512, 20 chunks): 0.007122 -> 0.006949. Same top 96.27% -> 96.32%. Max KLD 1.59 -> 1.20.
• Qwen3.8-27B UD-IQ3_S, chat "capital of France", greedy, 3 requests each:
  • base: "Paris", top logprob -0.0066;
  • PR: "Paris", -0.0065;
  • neither run logged NaN.
  • The first-token NaN reported on dd74f6b does not reproduce at f5b7f4a, even without this PR.

Prompt speed. llama-bench, 3 interleaved rounds, medians:

model	base	PR	change
gpt-oss-20b pp512	1006.54	967.48	-3.9%
gpt-oss-20b tg64	22.45	22.02	-1.9%
Qwen3.8-27B UD-Q4_K_XL pp512	342.09	353.17	+3.2%

All samples: gpt-oss pp base 1011.6/1006.5/1006.2, PR 971.1/965.8/967.5. 27B pp base 344.0/336.8/342.1, PR 353.2/354.5/349.2.

[bong-water-water-bong]

Follow-up on the UD-IQ3_S check: that prompt does run the kernels this PR changes.

I reran the same chat request ("capital of France", greedy, 1 request) with GGML_HRX_LOG_DISPATCH=1 on both builds. Both builds dispatch the same f32 WMMA kernels:

• common.mul_mat.f32_f32_wmma
• common.mul_mat_postops.f32_f32_wmma
• common.mul_mat_swiglu.f32_f32_wmma
• llm.attention_qkv_matmul_postprocess.f32_f32_wmma

build	answer	top logprob	NaN-logit lines
base f5b7f4a	"Paris"	-0.0066	0
f5b7f4a + 85f5983 + b5669e0	"Paris"	-0.0065	0

So the matching IQ3_S result in the comment above is a real check of the changed kernels on a model, not a path that skips them.

[bong-water-water-bong]

Conflicts with #85, which fixed the same issue (engine ggml-org#284) in mul_mat_id_f32_f32_wmma_core.loom by folding an f16 accumulator per 256-wide K tile into f32. This PR takes the true f32-accumulator approach in all four WMMA files. Resolving in favour of this PR, then re-verifying on the current tip (known-answer, KLD vs CPU, pp/tg A/B) before merging.

[bong-water-water-bong]

Merged 1bit/hrx-vulkan-patched (86eee89, with #85, #82, #83) into this branch as d5a9932. The only conflict was motifs/mul_mat_id_f32_f32_wmma_core.loom. I kept this PR's version, so the f32 accumulator replaces #85's per-tile f16 fold.

On the pinned hrx-system 51b1739, the JIT compiles the vector<4xf32> accumulator to v_wmma_f32_16x16x16_f16 on gfx1151. That holds for all 93 mul_mat and 72 mul_mat_id specializations test-backend-ops built; the base build uses v_wmma_f16_16x16x16_f16.

Checks on strixhalo, performance mode, base 86eee89 vs merge:

• Known-answer cases (MXFP4 0x77, exact K x 0.0234375, K 1152-5120, 6/17-token and 1-token tiles, --sanitizer=access, 2 repeats): merge 84/84, base 28/84 (base mul_mat_id passes through ggml-hrx: fold the MUL_MAT_ID WMMA f16 accumulator into f32 at each 256-wide K tile (engine#284) #85).
• test-backend-ops -b HRX0: MUL_MAT 322/322, MUL_MAT_ID 108/108, MUL_MAT_VEC_FUSION 45/45, 2 runs each.
• gpt-oss-20b KLD vs CPU (c512, 8 chunks): 0.029296 -> 0.026932; same top 87.89% -> 89.61%.
• GLM-4.7-Flash Q4_K_M vs BF16 (32 chunks): 0.1236 -> 0.1253, within noise.
• 5 identical greedy requests on gpt-oss-20b and Qwen3-Coder-30B: byte-identical.
• llama-bench pp512 (medians of 3, interleaved): gpt-oss-20b 1045.8 -> 995.7 (-4.8%), Qwen3.8-27B UD-Q4_K_XL 355.3 -> 367.0 (+3.3%), Qwen3-Coder-30B 2060.8 -> 2058.8. tg128 unchanged on all three.
• The gpt-oss pp loss comes from the MoE core alone: with ggml-hrx: fold the MUL_MAT_ID WMMA f16 accumulator into f32 at each 256-wide K tile (engine#284) #85's core restored and this PR in the other three files, pp512 is 1038.9 (base 1042.3) and KLD 0.027798. Follow-up: profile the f32 MoE core (result staging went from 512 to 1024 B per wave).