Repository navigation
HRX: wrong first token after a long distractor when a slot reuses a cached prefix (fused qwen.attention publish path) #318
Description
Found 2026-10-05 while bisecting a determinism gate on Qwen3-Coder-30B-A3B Q4_K_M (strixhalo, HRX0, performance mode). Public Loom path: it reproduces with every private HIP matcher disabled (GGML_HRX_DISABLE_DISPATCH=hip.) and with them on.
What happens. llama-server, sequence: q2, d1, q2, d2, q2, d3 (a 720-token distractor on slot 1), then q2 again reusing a 14-token cached prefix and prefilling 27 tokens. Token 0 goes from p≈1.0 ("```") to "Here" at −0.455 — a wrong token, not drift. A full prefill right afterwards is correct.
What points at the publish path. It goes away with GGML_HRX_DISABLE_DISPATCH=hip.,qwen.attention, i.e. with the fused qwen.attention_* KV-publish/postprocess path off; the standalone FA kernels with the same shapes pass the masked-KV check (0 differing elements, cells 256–2048, holes up to 1400), so attention itself is cleared. Suspect: handling of non-contiguous KV cells after a prefix reuse.
Related, same session. With --cache-ram 0, requests that reuse their own slot (server truncates the generated tokens and re-evaluates one token) produce "" or repeated </div> from the first token — 3/4 distinct — with HIP on and off.
Gate note. The 4-identical-request gate's 2/4 results on Coder/Qwen3-0.6B/GLM come from llama-server's prompt cache sending requests 3 and 4 down the one-token re-eval path, which drifts (max |Δlogprob| 0.06–0.08 in every config) and flips near-ties; with cache_prompt:false HRX decode is bit-identical. The gate now sends cache_prompt:false; cache-hit requests need their own check.
Repros on strixhalo: ~/wt/loom-detfix/probe.sh + probe.py (results ~/wt/loom-detfix/probes/log); ~/wt/loom-detfix/mygates.sh with SRV_EXTRA="--cache-ram 0" (runs/ncr-*).
Activity
bong-water-water-bong commentedon Oct 5, 2026
Re-tested on the current pin: the wrong first token is gone
This looks fixed by the pin move, and the candidate change is named in the PR that landed it. Re-ran the issue's own repro and sequence against the current pins (
hrx-system 98d05d94,llama.cpp 522dab4, HRX0, Qwen3-Coder-30B-A3B Q4_K_M), with the private HIP matchers on — i.e. the configuration in which it broke:Step 7 is the case this issue is about, and it now returns the same token with the same probability as steps 1/3/5. The report was "token 0 goes from p≈1.0 (
```) toHereat −0.455 — a wrong token, not drift". It no longer does. The related step 8 (self-slot reuse withcache_prompt:false) is also clean.Why this is believable rather than a lucky run, unlike #315: this manifestation was deterministic — a fixed wrong token at a fixed step — so a single clean run is meaningful evidence, whereas #315's intermittency needs repeats. And the fix lines up: fork PR #90 (
1bit/kv-nan-rootcause, now the pin) changed exactly this area — "on the current tip every multi-ubatch Qwen prompt gives wrong output… Such bindings now carry agraph_inputflag: uploaded, never downloaded", i.e. the host writeback that was overwriting the next ubatch's fresh inputs.Suggest closing this; a second confirming run would make it airtight, and
GGML_HRX_DISABLE_DISPATCH=hip.,qwen.attentionshould still be the isolation if anyone wants to re-derive it.bong-water-water-bong commentedon Oct 6, 2026
Closing as fixed on the current pin (
hrx-system 98d05d94,llama.cpp 522dab4). The cached-prefix reuse case (step 7) now returns the same token and probability as the uncached steps, the deterministic repro the issue reported no longer reproduces, and fork PR #90 ("graph inputs are never written back to host") is the named change in the same KV/ubatch area. Evidence in the comment above. Re-open if a fresh run disagrees.HRX hardware correctness gate on Strix Halo (hw-gate) (#403)