Eval bug: SYCL F16: NaN output on dense Qwen3.8-27B Q4_K_M since #29107 (IQ3 code reorder); fused GEMM off fixes it (Arc Pro B70) #30892

Activity

1. baustin27 commented on Oct 21, 2026

   

   baustin27

   on Oct 21, 2026

   More actions

   Independent reproduction on a different driver / compiler / tag combination than the one in the report — which is what makes "F16/XMX is the cause, not the driver" hard to dismiss.

   Our environment

   	this report	ours
   llama.cpp	b11755 (704366d)	b11756 (d806193)
   compiler	IntelLLVM 2026.1.0	oneAPI 2026.2 (/opt/intel/oneapi/compiler/2026.2/bin/icpx)
   compute runtime (NEO)	26.22.38646.6	26.27.39122.11
   IGC	2.36.3	2.38.2
   Level Zero	1.28.6	1.32.0
   build flags	-DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DGGML_SYCL_DNN=OFF	-DGGML_SYCL=ON -DGGML_SYCL_TARGET=INTEL -DGGML_SYCL_F16=ON -DGGML_SYCL_DNN=ON -DGGML_SYCL_GRAPH=ON -DGGML_SYCL_HOST_MEM_FALLBACK=ON -DGGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON -DGGML_OPENMP=OFF -DCMAKE_BUILD_TYPE=Release (oneDNN 2026.1)
   GPU	Intel Arc Pro B70 (BMG-G31, 32 GB)	Intel Arc Pro B70 (32,656 MiB, PCI 0000:03:00.0)
   model	unsloth/Qwen3.8-27B-UD-Q4_K_M.gguf (Q4_K/Q5_K/Q6_K)	mradermacher/Holo4-27B-i1-GGUF → Holo4-27B.i1-IQ4_XS.gguf (IQ4_XS; general.architecture=qwen35, general.base_model.0.name=Qwen3.8 27B)

   So: a different compute runtime, a different IGC, a different Level Zero loader, a different compiler, a different tag, GGML_SYCL_DNN=ON instead of OFF, and a different quant — and the same failure, at the same prompt lengths, with the same signature.

   Result

   Garbage from the first token onward at every prompt length, including a 26-token prompt ("What is 2 + 2? Reply with just the number."):

   [short, 26 prompt tokens]  |        #       </s>  </s>           </s>  d  #  ##  #  #
   

   which is the same shape as the !!!! / token-0 signature in the report (garbage rather than a hard NaN — !!!! is itself one particular NaN-logits spelling).

   Four further data points that are the same for us, from the same build:

   • The model is not the variable: a second artifact of the same family (the AutoRound W4A16 arcitech-psp/Holo4-27B-AutoRound-W4A16, a VLM on the same Qwen3.8-27B base) fails identically.
   • The quant is not the variable: five separate Holo4-27B GGUFs (IQ4_XS, IQ3_XXS, IQ3_S, IQ3_M, Q3_K_M) fail identically.
   • KV type is not the variable: -ctk/-ctv f16/f16, q8_0/q8_0 and q4_0/q4_0 all fail identically.
   • The failure is length-independent, so on a SYCL build with GGML_SYCL_F16=ON no prompt-length measurement means anything — including the "fails even a 2+2" observation, which is this bug and not a context result.

   Questions

   1. Does b11756 + GGML_SYCL_F16=OFF also give correct output? The table above shows F16=OFF on b11755 is correct to 14K tokens. b11756 is one commit later and is the tag we reproduce on, so b11755+F16=OFF vs b11756+F16=OFF isolates the tag delta from the F16 axis. If b11756+F16=OFF were also broken, the first-bad-commit bisect would need widening past b11636.
   2. Do the two escape hatches also work on a NEO 26.27 / oneAPI 2026.2 / GGML_SYCL_DNN=ON combination? You measured GGML_SYCL_XMX_GATHER_TYPES=0 → correct and GGML_SYCL_DYNAMIC_PRECISION=F32 → correct on NEO 26.22 / IntelLLVM 2026.1.0 / DNN=OFF. Our combination differs on all three axes, so it is worth knowing whether the workaround is version-specific. Both switches exist in the b11756 tree without a rebuild — ggml/src/ggml-sycl/ggml-sycl.cpp:107,428 and ggml/src/ggml-sycl/fused-gemm.cpp:36 — so the A/B is a two-line change and a rebuild, no source edit.
   3. Does GGML_SYCL_DNN=ON matter? Your build had it off; ours has it on and fails the same way. That rules one more axis out for us.

   One more axis worth ruling out, separately

   oneAPI 2026.2's release notes say: "L0 v2 is used by default on Xe2 and newer GPUs (e.g. Battlemage…). When using the L0 v2 adapter, only accepts in-order make_queue(). Out-of-order command lists result in undefined behavior." We export SYCL_UR_USE_LEVEL_ZERO_V2=0 for every SYCL process to avoid it. That is a different axis from F16/XMX and we did not vary it, but it is worth ruling out because it is the same class as the sparse-FA memset-ordering bug (#30624 → #30812), and it is a default that differs between oneAPI 2026.1 (yours) and 2026.2 (ours) — i.e. exactly the kind of difference that could make the two of us disagree about whether an escape hatch works.

   Measured on our own hardware (a single B70, shared with other work); the investigation and this write-up were done with Claude Code (Anthropic).
2. athenagaliotos-pixel commented on Oct 21, 2026

   

   athenagaliotos-pixel

   on Oct 21, 2026

   Last edited by athenagaliotos-pixel

   More actions

   Thanks for investigating this. I can confirm a similar issue on an Intel Arc Pro B60 (24 GB) using llama.cpp's SYCL backend.

   Environment

   • GPU: Intel Arc Pro B60, 24 GB

   • Backend: llama.cpp SYCL

   • Source commit: 2238bc057403d0184b48a465caee9e50ab2bb491 (experimental build incorporating additional PRs)

   • Models: Qwen3.8-27B UD Q4_K_XL and GSQ/RCO IQ3_S-MTP

   • Context: 131,072

   • Q8_0 K/V cache, FlashAttention enabled, all 66 layers GPU-offloaded

   • MTP depth 1

   Observed behaviour

   During testing with approximately 100K-token prompts, cached multi-turn follow-ups produced incorrect or incoherent output.

   The problem also occurred with MTP disabled. A fresh-server standard-chat request reproduced the incorrect output, suggesting the problem was not limited to retained KV-cache state.

   We investigated request formatting and scoring separately. Although our harness had issues that needed correction, those issues did not fully explain the backend output corruption.

   Workaround

   Setting:

   GGML_SYCL_XMX_GATHER_TYPES=0

   restored coherent cached follow-up responses in bounded tests for both models.

   Results with MTP depth 1:

   Model | KV tokens reused | Drafts accepted | Result -- | -- | -- | -- GSQ/RCO IQ3_S | 99,988 | 74/76 | PASS UD Q4_K_XL | 99,995 | 56/62 | PASS

   The GSQ/RCO case was tested with the same canonical requests and serving settings, changing only the XMX gather type mask. The UD confirmation used the corrected request protocol and fallback, but was not a separate one-variable A/B test.

   Possible connection

   Our source inspection also identified a potential reordered-weight layout mismatch in the fused GEMM dispatch path. This appears consistent with the issue reported here, although we have not isolated the exact offending kernel or validated a source-level fix.

   These were limited correctness checks, not a full stability or performance qualification.

   Sharing this in case the additional B60 reproduction and Qwen3.8 results are useful. Happy to provide a smaller reproducer or specific logs if that would help.

   Thanks for maintaining the SYCL backend!
3. RGarrett93 commented on Oct 22, 2026

   

   RGarrett93

   on Oct 22, 2026

   More actions

   Confirmed workaround — Intel Core Ultra 7 255H integrated Arc GPU

   I can confirm that the GGML_SYCL_XMX_GATHER_TYPES=0 workaround also resolves the repeated-character corruption on my UGREEN iDX6011 Pro (Intel Core Ultra 7 255H, integrated Intel Arc Graphics).

   Environment:

   • llama.cpp b11753 (d8d92973c)
   • IntelLLVM 2026.1.1
   • Intel oneAPI 2026.1 runtime
   • SYCL / Level Zero (SYCL0)
   • Model: Qwen3.8-27B-UD-Q4_K_M.gguf from Unsloth

   With default SYCL settings, the model loaded but consistently produced repeated characters (333333... or ////////...).

   With the following environment variable:

   GGML_SYCL_XMX_GATHER_TYPES=0

   The same model now generates correct output.

   Validation performed:

   • Short greeting: Hello! — passed, finish_reason: stop.
   • Longer Python coding request: 400 coherent tokens, including a correct prime-number function and explanation.
   • Longer test generation speed: 3.72 tokens/sec.
   • No repeated-character corruption observed.

   The longer response ended with finish_reason: length because I deliberately limited generation to 400 tokens.

   For comparison, the standard Qwen3.8-27B-Q4_0.gguf quantisation also generates correctly without the workaround on the same backend.

   Switching the failing UD-Q4_K_M model from the oneAPI 2026.1 runtime to the 2026.2 SYCL runtime did not resolve the issue by itself.

   These findings independently support the suspected fused XMX kernel regression and demonstrate that the workaround is effective on an integrated Intel Arc GPU, in addition to the discrete Arc GPUs already reported here.

   Thanks for investigating this issue!
4. RGarrett93 commented on Oct 22, 2026

   

   RGarrett93

   on Oct 22, 2026

   More actions

   Update to my previous report — Core Ultra 7 255H / Intel Arc SYCL

   Following my earlier report, I've completed further testing on my UGREEN iDX6011 Pro (Intel Core Ultra 7 255H, integrated Intel Arc Graphics).

   1. Confirmed workaround on b11753

   Using llama.cpp b11753 (d8d92973c) with Intel oneAPI 2026.1.1, the original Qwen3.8-27B-UD-Q4_K_M.gguf produced repeated-character corruption under the default SYCL settings.

   Setting:

   GGML_SYCL_XMX_GATHER_TYPES=0

   restored correct generation. I confirmed this with both a short greeting and a longer Python coding response (400 coherent tokens, approximately 3.72 tokens/sec).

   2. Compiled and tested b11798

   I subsequently built llama.cpp from tag b11798, commit 9febe63, using IntelLLVM 2026.1.1 with:

   GGML_SYCL=ON
   GGML_SYCL_TARGET=INTEL
   GGML_SYCL_F16=OFF
   GGML_SYCL_DNN=ON
   GGML_SYCL_GRAPH=ON
   GGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON
   

   With GGML_SYCL_XMX_GATHER_TYPES unset, the original UD-Q4_K_M model now produces correct output.

   Tests completed:

   • Short greeting: PASS (Hello!)
   • Python coding task: PASS (337 coherent tokens, finish_reason: stop)
   • Generation performance: approximately 3.74 tokens/sec

   Important caveat: The original b11753 build's compile-time FP16 setting is unknown, whereas b11798 was explicitly compiled with GGML_SYCL_F16=OFF. Therefore, this does not establish that the regression has been fixed between those versions.

   These results are consistent with the existing findings that disabling FP16 or the fused XMX path avoids the corruption.

   I'll retain both builds for additional comparisons and can test an FP16-enabled b11798 build if that would help isolate the problem further.