HRX MLA decode path loses accuracy past ~512 KV cells (GLM-4.7-Flash, deepseek2)

[bong-water-water-bong]

Found 2026-10-05 by the GLM decode work (strixhalo, performance mode). GLM-4.7-Flash Q4_K_M on the HRX decode path (-ub 1) at c4096: PPL 11.79 vs CPU 10.60. On the same build with the new decode kernels, the result is 10.64. So the existing HRX MLA decode path (absorbed MLA, dk 576 / dv 512) drifts past about 512 KV cells. Every MLA model that still takes that path is affected (deepseek2-family, GLM-4.7-Flash without the new kernels).

Related, also found there: in a flash-attention PV product, masked keys (P exactly 0) still changed the output by about 1e-6 through their V values, so stale KV rows from earlier requests leaked into decode and identical requests diverged. Other HRX flash-attention kernels may have the same determinism problem.

Next: bisect which op in the old path drifts (decode-split FA at dk 576 vs the MLA absorb matmuls), and add a repeated-request check with random data in masked KV cells.

[bong-water-water-bong]

The measurement recipe for this issue is recovered (it was never actually lost)

I previously recorded that this issue's PPL invocation and corpus were "not recoverable". That was wrong — I was looking in the wrong place. Both are on the box, and the command is in a script.

Exact invocation — ~/wt/loom-glm-decode-run/ppl.sh <tag> <bin> <disable> [args]:

GGML_HRX_DISABLE_DISPATCH=$D LD_LIBRARY_PATH=$B systemd-run --user --scope -q -p MemoryMax=40G \
  $B/llama-perplexity -m ~/models/GLM-4.7-Flash-Q4_K_M.gguf \
  -f ~/models/qwen38/wikitext-2-raw/wiki.test.raw -ngl 99 "$@"   # defaults: c4096 x 2 chunks

• model present (18,312,339,808 B) and corpus present (1,290,590 B), both verified on strixhalo
• the CPU comparison is the other path: -dev none -ngl 0 -nr -t 4 under ~/lb/thermal-run.sh

Two original logs survive, and they corroborate this issue:

log	recorded
final/ppl.cpu-c4096.log	PPL 10.5965 ± 0.49092 (the "CPU 10.60" quoted above)
final/ppl.b512-base.log	PPL 10.7084 ± 0.49726

The second is the old HRX path at 512 KV cells, and it is healthy — 10.71 vs 10.60 CPU. That is exactly what the "drifts past about 512 KV cells" claim predicts, and it is independent corroboration of the shape of the defect.

What is not recoverable: the two numbers this issue is built on — 11.79 (old path) and 10.64 (new decode kernels), both at c4096 — were never written to disk. Only the two logs above exist anywhere under ~/wt; the other 11.79/10.64 matches I found are coincidental substrings in unrelated logs. So those two stand as reported-but-unlogged, while 10.60 and the 512-cell data point now have file-level evidence.

I have not re-run the c4096 measurement yet, deliberately: engine CI is currently executing on strixhalo itself (the self-hosted runner now carries the ubuntu-24.04 label), and starting an 18 GB-model PPL run against a box already serving CI is how earlier CI jobs got cancelled at the 15-minute mark. The recipe above is now complete, so the measurement can be run as soon as the runner is idle.

Still outstanding for this issue, unchanged: bisect which op in the old path drifts (decode-split FA at dk 576 vs the MLA absorb matmuls), and add the repeated-request check with random data in masked KV cells.

[bong-water-water-bong]

The decode path is verified fixed at c4096 — and it reproduces this issue's own number

This issue rests on two measurements (11.79 old path, 10.64 new kernels) that were never written to disk. I have now
reproduced one of them, using this issue's own specification: -ub 1, c4096, GLM-4.7-Flash-Q4_K_M, wikitext-2-raw.

condition	PPL
CPU reference (from the surviving ppl.cpu-c4096.log)	10.5965 ± 0.49092
new decode kernels, decode path, -c 4096 -ub 1 --chunks 2	10.6375 ± 0.49452
this issue's claim for that same case	10.64
this issue's claim for the old path	11.79

10.6375 against a claimed 10.64 — a match to within 0.003. The result sits 0.4% from the CPU reference and shows
none of the ~11.79 drift. Per-chunk: [1]11.2472, [2]10.6375. So the fix in gpu-kernels gk-glm does what this issue
says it does.

Note the method point: my first attempt at this measured without -ub 1, which is the prefill path — the wrong path
entirely, since this issue is specifically about the decode path. It returned 10.7084 and meant nothing. Re-running with
-ub 1 is what produced the match above.

What I could not reproduce, and why

The old-path side (11.79) is not runnable as-is. Every build without the add-on also lacks the CLI tools, and every
build with tools carries the new kernels:

build	glm_mla strings	has llama-perplexity
pin-fork-hrx/build (ONEBIT_GPU_PRIVATE empty)	0	NO
pin-fork-hrx/build-priv (add-on on)	39	NO
loom-glm-decode-run/bin-v1/-v2/-v7	29	yes
loom-glm-decode-run/bin-v10	37	yes
loom-glm-decode-run/bin-v13/-v16	39	yes

So to reproduce 11.79 one must either build the tools in a non-add-on tree, or disable the new dispatches at runtime. The
registered dispatch names are hip.glm_mla.{attn,part_ml,part_o,prep,qkv_a,qkv_a_counter,qn_q8_1_x4,rope_freq}, but the
literal GGML_HRX_DISABLE_DISPATCH string the original A/B used was never written down anywhere I can find. Inventing one
would produce exactly the kind of invalid experiment this issue has already suffered from, so I did not.

What is therefore verified: the fixed state reproduces this issue's own fixed number. What is not verified is
the before/after delta, because the before side is not runnable without that build or that string.

A correction to my own earlier claim

I previously recorded that ppl.b512-base.log corroborated "the old path at 512 KV cells is still healthy". That was
wrong. Its per-chunk values [1]11.4529, [2]10.7084 are numerically identical to a 2-chunk c4096 run, so despite its
tag it never measured 512 cells. Claim withdrawn; both surviving logs are c4096 2-chunk runs, and both measure the prefill
path.

Still open on this issue

• The drift bisect (decode-split FA at dk 576 vs the MLA absorb matmuls) needs the old-path side, so it needs either
  a tools build in a non-add-on tree or the recovered disable string.
• The masked-V determinism item (masked keys changing output by ~1e-6, so identical requests diverge) is separate and
  unaffected by any of the above.

A same-build repeat is running as a stability control; I will add its number if it differs materially from 10.6375.

[bong-water-water-bong]

Control result: the 10.6375 reproduces exactly

I said I would report the stability control only if it differed. It did not — recording that for completeness, since a
promised control that goes unmentioned is a claim I did not check.

run	binary dir	result
decode path, -c 4096 -ub 1 --chunks 2	loom-glm-decode-run/bin-v16	10.6375 ± 0.49452 [1]11.2472,[2]10.6375
identical rerun	loom-glm-decode/build/bin	10.6375 ± 0.49452 [1]11.2472,[2]10.6375

Byte-identical, including error bars and per-chunk values. That also settles a suspicion from the build survey: those two
directories are the same configuration (both report 39 glm_mla strings), so this is a true repeat rather than an A/B.

Also recorded, since it is the gate standard this project uses for decode-shaped perplexity:

• -c 128 -ub 1 --chunks 4: PPL 11.0762 ± 2.28309, per-chunk [1]9.4502,[2]12.0834,[3]11.3060,[4]11.0762

The large error bar there is expected at 128 tokens with 4 chunks; it is a different measurement from the c4096 figure and
should not be compared against it.

Nothing here changes the verdict: the decode path at c4096 measures 10.6375 versus this issue's claimed 10.64 and a
CPU reference of 10.5965, so the fix holds and the ~11.79 drift is absent. The old-path side still is not runnable
(no build without the add-on has CLI tools), so the before/after delta stays unverified while the fixed state is reproduced
twice.

[bong-water-water-bong]

At the pinned head the masking moved to the score, and the votes are gone

Reading the corpus that engine#329 pins (b04c4e95), the decode-split FA moved to motifs/flash_attention/decode_partials.loom and the masked-key handling changed shape:

• A masked score is %negative_large = -1e+30, so expf(score - block_max) underflows to exactly 0, and %probability = vector.select %query_valid, %raw_probability, %c0_f32x4 zeroes invalid query heads explicitly. A masked key therefore contributes P = 0 to the PV product, rather than having its V forced to +0 as the old kernel did.
• The two kernel.subgroup.vote ops and the key_tile_all_visible / key_tile_any_visible branch are gone from the whole corpus (engine#314 records that), so the masked path no longer branches on group visibility.

That is why the "masked keys changed the output through their V values by ~1e-6" item looks addressed at this pin: for finite stale V, 0 * V is exactly zero and the product is deterministic, with no group-vote state to diverge.

This is a code reading, not the check the issue asks for. attn-masked-kv-check - seeded random stale values in non-live cells, two seeds, first random run repeated, bit-identical required - still wants a run on the bumped pin, and that needs the GPU. The only remaining piece after that is the old-path drift bisect, which the issue's own 10.64 number already brackets.

[bong-water-water-bong]

At the synced pin AMD's attention export refuses GLM's shape; repointing it to our pre-merge kernel makes the prompt run at 872 t/s

Following the masking analysis above: the decode path isn't just different at this pin, it doesn't start, and I traced why.

ggml_flash_attention_f32_f16_wmma at the synced pins resolves to AMD's ops/flash_attention/prefill_f32_f16_wmma.loom, which declares

config.decl @ggml.flash_attention.qk_head_size : %value: index where [range(%value, 16, 512), mul(%value, 16)]

GLM-4.7-Flash's qk_head_size is 576, so Loom root linking fails before anything runs:

CONFIG/INVALID: symbol '@ggml.flash_attention.qk_head_size' value 576 violates constraint 'range'

Our pre-merge kernel (ops/flash_attention_f32_f16_wmma.loom, one of the 83 files the AMD sync deleted) declares the same symbol with range(%value, 16, 36864) - and the two export records are otherwise the same interface: bindings query,key,value,mask,gate,output (6), launch/workload query_token_count,key_value_token_count (2), identical names and counts.

So I repointed the export at our kernel (restored under ours/, library_sources emptied because our kernel is self-contained and AMD's four motifs carry a narrower publish_output - 21 args against our 23) and rebuilt:

GLM-4.7-Flash-Q4_K_M | HRX | HRX0 | pp512 | 872.47 t/s

That is the first time this model's prompt has run at the synced pin (the pre-sync build did 536.35 t/s, and AMD's kernel refused it outright). Relaxing AMD's range alone is not enough - the kernel then fails at TARGET/059 ... rejected 'index.assume', so the bound was load-bearing.

Decode still fails, and for a different reason: -p 512 -n 0 passes while -p 512 -n 32 does not, and the server shows the ABI/planner class on shared kernels - ggml_binary_f32, ggml_mul_mat_vector..., ggml_rmsnorm_binary..., all with workload=dynamic, compiled ABI does not match manifest. So this issue's decode path has two blockers at this pin: the attention symbol cap (fixed here by repointing) and a metadata/specialisation mismatch that stops the decode graph.

Artifact: port-flash-attn/ in the workspace (our kernel, the manifest patch, and a README). Nothing was committed to the engine or the forks.

[bong-water-water-bong]

HRX decode vs CPU on the current stack: no measurable loss — and it is the CPU -ub 1 arm that is broken

Ran the comparison properly (fresh build directory, the engine's own ExternalProject arguments — AMD's core plus our ggml-hrx; GLM-4.7-Flash Q4_K_M, wiki.test.raw, -c 4096 --chunks 2, -fa 1):

arm	PPL
HRX, -ub 1 — the decode path this issue is about	10.7208 +/- 0.49991
CPU, default ubatch	10.7414 +/- 0.50110
CPU, -ub 1	264.9719 +/- 17.62758
CPU, -ub 1, without -fa	387.5591 +/- 25.19623

HRX's decode path and the CPU reference agree to 0.02 PPL, far inside the +/-0.50 bars. So the degradation this issue reports (HRX 11.79 vs CPU 10.60) is not reproducible on the current stack at 4096 tokens. Caveats stated plainly: two chunks only, and the CPU reference is at the default ubatch — because the CPU -ub 1 arm is unusable, which is the second finding.

The CPU -ub 1 arm is broken for this model. At -ub 1 the CPU gives 264.97 (387.56 without -fa), while the same build's CPU at the default ubatch gives 10.74 — and the same -ub 1 setting on a small model is fine: MiniCPM5-1B-Q4_K_M, CPU, -ub 1 -> 26.93 (+/- 1.31), a plausible figure for a 1B model. So it is specific to GLM-4.7-Flash (deepseek2/MLA) rather than the ubatch setting, and it is CPU-only: HRX at -ub 1 on the same model is 10.72.

That matters for anyone repeating this measurement: the natural A/B is "HRX at -ub 1 versus CPU at the same settings", and at those settings the reference is the broken arm, so the comparison silently implies HRX is healthy when nothing has been checked. Filed separately, because it is a different defect from this issue.

[bong-water-water-bong]

Longer context: HRX perplexity falls (10.72 -> 7.50), it does not degrade

Doubling the context to 8192 tokens (-c 4096 --chunks 4), same model and corpus, HRX on the decode path:

context	HRX -ub 1 PPL
4096 tokens (2 chunks)	10.7208 +/- 0.49991
8192 tokens (4 chunks)	7.4988 +/- 0.23237

PPL falls as context grows — which is what a healthy decode path does — while the issue reports a rise (HRX 11.79 against CPU 10.60). At twice the context the degradation is not there, and the interval halves as the scored-token count doubles, which is also the expected behaviour.

One caveat, and it is my error rather than the build's: the CPU arm in that same run carried -fa 1, which engine#337 shows corrupts CPU perplexity — it produced 919.78 there against 10.7414 without the flag. So the CPU half of this table is invalid and is being re-run without -fa. The HRX half stands either way, since HRX is stable with and without the flag (10.7208 both times).

If the intent is to chase the drift further, the useful next step is the op-level bisect the issue itself suggests — decode-split FA at dk 576 versus the MLA absorb matmuls — rather than more end-to-end PPL. At this point the end-to-end number says the current stack is fine.

[bong-water-water-bong]

Final: the HRX decode path matches the CPU reference at both context lengths

The valid CPU arm is in, so the comparison is complete. GLM-4.7-Flash Q4_K_M, wiki.test.raw, -fa off on the CPU side (engine#337: that flag corrupts CPU perplexity).

context	HRX -ub 1 (decode path)	CPU, default ubatch	delta
4096 tokens (2 chunks)	10.7208 +/- 0.49991	10.7414 +/- 0.50110	0.02
8192 tokens (4 chunks)	7.4988 +/- 0.23237	7.4936 +/- 0.23212	0.005

At twice the context the two agree to 0.005 PPL — about 45x inside the confidence interval — and both fall with context, which is what a healthy decode path does. The issue reports the opposite pattern (HRX 11.79 against a CPU 10.60) and I cannot reproduce it on the current stack at either length.

Retiring my own earlier framings on this issue as well: the reported 11.79-versus-10.60 gap does not reproduce; no ubatch workaround is needed for accuracy (the hybrid is correct at -ub 512 and on the decode path); and the run in my round-84 pair that looked like a sane CPU reference was simply the -fa-free configuration — which is the one worth using, confirmed independently by engine#337.

If this needs chasing further, the op-level bisect the issue itself proposes (decode-split FA at dk 576 versus the MLA absorb matmuls) is the right next step. End-to-end, the current stack is fine at twice the context where the drift was reported.

[bong-water-water-bong]

Measured: no accuracy loss past 512 KV cells on the HRX path

Reporting the measurement that answers this, since the issue is still open. HRX MLA decode, GLM-4.7-Flash, -ub 1, with CPU as the reference on the same prompts:

length	HRX	CPU	delta
4096 tok	10.7208 ± 0.49991	10.7414 ± 0.50110	0.021
8192 tok	7.4988 ± 0.23237	7.4936 ± 0.23212	0.005

At 8192 — well past the ~512-cell threshold in the title — HRX is within 0.005 perplexity of CPU, with overlapping error bars. No degradation is visible.

One measurement caveat worth recording, because it gave me a wrong number first: on the CPU arm, default ubatch with -fa off is the only trustworthy configuration — -fa 1 gives 1014.1151 at default ubatch and 264.9719 at -ub 1, both also ~30x slower. HRX is stable either way. If this is re-measured, fix the CPU arm first.

Both arms were run under the merged pin (f2099e9b7 — AMD's core with our ggml-hrx).