Repository navigation
HRX: intermittent NaN logits on the first sample after a ~4.8K-token prompt (Qwen3-0.6B, fork tip) #313
Description
Found 2026-10-05 by the masked-KV determinism audit (strixhalo, performance mode). Not a masked-V leak: every shipped attention path gives the same bits whatever stale K/V the masked cells hold (public Loom fork #72 and the add-on fix are both clean under a 118+25-case check).
What happens. Qwen3-0.6B Q4_K_M, fresh llama-server on HRX0 (llama.cpp fork at or after d5de4b1, i.e. with #72), one short request (~180 tokens, 16 out), then a 4,798-token prompt with cache_prompt: false. The first sample after prefill is NaN and the server aborts. 8/21 runs fail; when it does not fail the output is bit-identical (db5701f7).
What changes it
- graph replay off: 1/5 fail
- any
GGML_HRX_DISABLE_DISPATCHvalue set, even one that matches nothing: 0/12 fail - NPU: the build's own lax kernels are the default; Qwen3.6-35B-A3B through Lemonade #72 reverted: 0/9 fail
- an extra request between the two: passes
An env var that changes nothing semantically hiding it points at an uninitialized host-side value or a host/device timing race, not at kernel math. Same class as #123.
Related: graph replay on vs off gives different but each-stable output bits on the 0.6B (db5701f7 vs f6f810ca, 5/5 each).
Repro on strixhalo: ~/.cache/vaudit/repro.sh TAG BINDIR d0,t250f0 (logs under ~/.cache/vaudit/). Next: run the repro under ASan / an uninitialized-read sanitizer on the HRX plan/bind path, then bisect by code change (Loom prefill FA vs the rest) rather than the env var.
Activity
bong-water-water-bong commentedon Oct 5, 2026
Root cause found (2026-10-05). The NaN is a Loom compiler hazard on GFX11 wave64 at hrx-system 51b1739: in the prompt FA kernel compiled for a 190-row ubatch (the 4,798-token prompt's last ubatch), the SGPR pair s[28:29] is overwritten (s_mov_b64 … exec / s_lshl_b32 s28) right after a VALU read it as a lane mask (v_cndmask_b32_e64) with no s_waitcnt_depctr in between; a wave then reads V from a wrong address and one 16x16 output block is garbage/NaN. Binary patch bisected to a single site: depctr sa_sdst(0) after the overwrite -> 0/750 vs 7/750 control; a wait after only the FIRST overwrite (upstream's placement) is not enough on gfx1151 (12/1000). Upstream fixed the class in ROCm/hrx-system ee77c8f7f + 336037562, but that pin gives wrong Qwen3 output and changes the loom-jit API; a conservative backport exists on a local hrx-system branch (0/1000 in the FA harness, server repro 24/24 clean, test-backend-ops 1073/1073, bench flat on 0.6B/8B, Coder-30B pp512 maybe -6% to re-measure). Shipping it needs an hrx-system pin change — owner's call. Two host bugs found on the way are in fork PR 1bit-MONSTER/llama.cpp (graph-input writeback = #319; non-replay upload race). The env-var sensitivity was timing: the hazard is a race, and an extra request changes the last ubatch size and therefore the compiled kernel.
Bump HRX: llama.cpp 522dab4789ce (long-prompt fix); hrx-system moves …
bong-water-water-bong commentedon Oct 5, 2026
Fixed in the engine since #320 (main 11d6e2f): hrx-system now comes from our fork 1bit-MONSTER/hrx-system (1bit/main 98d05d94 = ROCm 51b1739 + the GFX11 wave64 lane-mask drain backport in Loom), and the two host-side bugs found on the way are in llama.cpp pin 522dab47 (fork #90). Verified: the repro (Qwen3-0.6B, ~180-token request then the 4,798-token prompt, cache_prompt false) 20/20 with no NaN and the right answer; standalone FA harness 0/1000 vs 7/1000 on the old Loom; test-backend-ops clean; no prompt-speed cost on 0.6B/8B/Coder-30B/GLM/ZAYA. Still open elsewhere: the same hazard pattern exists in other Loom kernels (mul_mat WMMA, decode-split next_q8, MoE table, set_rows, rope_set_rows) — the backport drains after every lane-mask overwrite function-wide, so they are covered by the same fix; GLM decode reads 1-2.5% slower with the fork hrx-system, to be A/B'd per commit.
315: the device probe BUILT and is running; and another agent already…
315: if the staging path is cold - the surviving signature is exact: …
315: the corruption is WAVE-GRANULAR - starts on a wave64 boundary in…
315: CORRECTION - the measured shape is a contiguous 256-byte-aligned…
HRX hardware correctness gate on Strix Halo (hw-gate) (#403)