Repository navigation
Eval bug: SYCL: bad output on Qwen3.6 35B A3B #28728
Description
Name and Version
root@db159f83e6c2:/app# ./llama-cli --version
version: 0.4.0-dev (build 10902, commit df03399b8)
built with IntelLLVM 2025.3.3 for Linux x86_64
Operating systems
Linux
GGML backends
SYCL
Hardware
B580 + Ryzen 7 5700X3D + 48GB at 3133MHz
Models
Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
Problem description & steps to reproduce
./llama-server --host 0.0.0.0 --port 8080 --model /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --n-gpu-layers 99 --n-cpu-moe 99 --load-mode none --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 9000 --ubatch-size 9000 --ctx-size 262144 --no-cache-prompt --ctx-checkpoints 0 --parallel 1podman run --interactive --tty --rm localhost/llama-benchy:latest --base-url http://host.docker.internal:1235/v1 --model Qwen/Qwen3.6-35B-A3B --pp 244625 --tg 1024 --exact-tg --runs 1 --no-warmup(my own llama-benchy container, should produce the same results on native llama-benchy)- Start a new chat conversation
- Watch the model have a meltdown
First Bad Commit
No response
Relevant log output
Logs
davi@davi:~/git/llama.cpp$ podman run --interactive --tty --device=/dev/dri/renderD128 --volume=/home/davi/AI/models/:/models/ --volume=/media/davi/HD/AI/models/:/models-hdd/ --entrypoint=/bin/bash --publish=1235:8080 localhost/test:latest
root@db159f83e6c2:/app# ./llama-server --host 0.0.0.0 --port 8080 --model /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --n-gpu-layers 99 --n-cpu-moe 99 --load-mode none --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 9000 --ubatch-size 9000 --ctx-size 262144 --no-cache-prompt --ctx-checkpoints 0 --parallel 1
0.00.276.767 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.277.235 W srv llama_server: -----------------
0.00.277.238 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.277.238 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.277.238 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.277.238 W srv llama_server: -----------------
0.00.278.492 I srv load_model: loading model '/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf'
0.01.470.449 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
0.01.470.459 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
0.01.470.471 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring
0.01.470.476 W model has unused tensor blk.40.attn_k.weight (size = 1114112 bytes) -- ignoring
0.01.470.481 W model has unused tensor blk.40.attn_v.weight (size = 1114112 bytes) -- ignoring
0.01.470.498 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring
0.01.470.503 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.01.470.508 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.01.470.513 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 1048576 bytes) -- ignoring
0.01.470.519 W model has unused tensor blk.40.ffn_down_exps.weight (size = 220200960 bytes) -- ignoring
0.01.470.529 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 184549376 bytes) -- ignoring
0.01.470.534 W model has unused tensor blk.40.ffn_up_exps.weight (size = 184549376 bytes) -- ignoring
0.01.470.540 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 4096 bytes) -- ignoring
0.01.470.545 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring
0.01.470.550 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring
0.01.470.555 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring
0.01.470.561 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring
0.01.470.567 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
0.01.470.572 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
0.01.470.588 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring
0.24.408.600 I cmn init: llama threadpool init, n_threads = 8
1.13.994.447 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 262144, kv_unified = 'false'
1.13.999.353 W srv init: chat template supports preserving reasoning, it is enabled by default (may use more tokens, disable via --no-reasoning-preserve)
1.13.999.385 I srv llama_server: model loaded
1.13.999.390 I srv llama_server: listening on http://0.0.0.0:8080
1.13.999.391 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release
1.13.999.391 W srv llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508
1.31.028.875 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
1.31.028.944 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
1.31.916.912 I slot print_timing: id 0 | task 0 | prompt eval time = 887.94 ms / 30 tokens ( 29.60 ms per token, 33.79 tokens per second)
1.31.916.916 I slot print_timing: id 0 | task 0 | eval time = 0.00 ms / 1 tokens ( 0.00 ms per token, 0.00 tokens per second)
1.31.916.917 I slot print_timing: id 0 | task 0 | total time = 887.94 ms / 31 tokens
1.31.916.919 I slot print_timing: id 0 | task 0 | graphs reused = 1
1.31.916.951 I slot release: id 0 | task 0 | stop processing: n_tokens = 30, truncated = 0
1.31.920.564 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 27528155819
1.32.001.033 I slot launch_slot_: id 0 | task 2 | processing task, is_child = 0
1.33.094.808 I slot print_timing: id 0 | task 2 | prompt eval time = 1093.76 ms / 35 tokens ( 31.25 ms per token, 32.00 tokens per second)
1.33.094.812 I slot print_timing: id 0 | task 2 | eval time = 0.00 ms / 1 tokens ( 0.00 ms per token, 0.00 tokens per second)
1.33.094.813 I slot print_timing: id 0 | task 2 | total time = 1093.76 ms / 36 tokens
1.33.094.813 I slot print_timing: id 0 | task 2 | graphs reused = 1
1.33.094.829 I slot release: id 0 | task 2 | stop processing: n_tokens = 35, truncated = 0
1.33.097.412 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 27529333697
1.33.157.639 I slot launch_slot_: id 0 | task 4 | processing task, is_child = 0
1.37.317.281 I slot print_timing: id 0 | task 4 | prompt eval time = 275.40 ms / 23 tokens ( 11.97 ms per token, 83.52 tokens per second)
1.37.317.285 I slot print_timing: id 0 | task 4 | eval time = 3884.23 ms / 100 tokens ( 39.23 ms per token, 25.49 tokens per second)
1.37.317.285 I slot print_timing: id 0 | task 4 | total time = 4159.62 ms / 123 tokens
1.37.317.286 I slot print_timing: id 0 | task 4 | graphs reused = 99
1.37.317.302 I slot release: id 0 | task 4 | stop processing: n_tokens = 122, truncated = 0
1.38.477.226 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 27533556169
1.38.551.616 I slot launch_slot_: id 0 | task 105 | processing task, is_child = 0
1.46.022.058 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 9000, progress = 0.04, t = 7.47 s / 1204.75 tokens per second
1.52.787.759 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 18000, progress = 0.07, t = 14.24 s / 1264.39 tokens per second
2.00.011.865 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 27000, progress = 0.11, t = 21.46 s / 1258.14 tokens per second
2.07.647.458 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 36000, progress = 0.15, t = 29.10 s / 1237.29 tokens per second
2.15.898.504 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 45000, progress = 0.18, t = 37.35 s / 1204.92 tokens per second
2.24.241.036 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 54000, progress = 0.22, t = 45.69 s / 1181.89 tokens per second
2.46.908.026 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 63000, progress = 0.26, t = 68.36 s / 921.64 tokens per second
3.07.058.006 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 72000, progress = 0.29, t = 88.51 s / 813.50 tokens per second
3.16.930.122 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 81000, progress = 0.33, t = 98.38 s / 823.35 tokens per second
3.27.547.796 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 90000, progress = 0.37, t = 109.00 s / 825.72 tokens per second
3.38.042.343 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 99000, progress = 0.40, t = 119.49 s / 828.52 tokens per second
3.48.880.075 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 108000, progress = 0.44, t = 130.33 s / 828.68 tokens per second
4.00.144.396 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 117000, progress = 0.48, t = 141.59 s / 826.31 tokens per second
4.11.880.815 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 126000, progress = 0.52, t = 153.33 s / 821.76 tokens per second
4.27.340.752 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 135000, progress = 0.55, t = 168.79 s / 799.81 tokens per second
4.39.967.919 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 144000, progress = 0.59, t = 181.42 s / 793.75 tokens per second
4.52.976.806 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 153000, progress = 0.63, t = 194.43 s / 786.94 tokens per second
5.07.681.655 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 162000, progress = 0.66, t = 209.13 s / 774.64 tokens per second
5.21.637.686 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 171000, progress = 0.70, t = 223.09 s / 766.52 tokens per second
5.38.332.746 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 180000, progress = 0.74, t = 239.78 s / 750.68 tokens per second
5.53.490.558 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 189000, progress = 0.77, t = 254.94 s / 741.35 tokens per second
6.09.544.126 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 198000, progress = 0.81, t = 270.99 s / 730.65 tokens per second
6.27.187.511 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 207000, progress = 0.85, t = 288.64 s / 717.17 tokens per second
6.43.129.838 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 216000, progress = 0.88, t = 304.58 s / 709.18 tokens per second
6.59.317.404 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 225000, progress = 0.92, t = 320.77 s / 701.45 tokens per second
7.15.920.330 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 234000, progress = 0.96, t = 337.37 s / 693.60 tokens per second
7.33.192.825 I slot print_timing: id 0 | task 105 | prompt processing, n_tokens = 243000, progress = 0.99, t = 354.64 s / 685.20 tokens per second
7.45.335.606 I slot print_timing: id 0 | task 105 | n_gen = 100, tg = 10.82 t/s, tg_3s = 10.93 t/s
7.48.337.197 I slot print_timing: id 0 | task 105 | n_gen = 135, tg = 11.03 t/s, tg_3s = 11.66 t/s
7.51.382.906 I slot print_timing: id 0 | task 105 | n_gen = 171, tg = 11.19 t/s, tg_3s = 11.82 t/s
7.54.408.933 I slot print_timing: id 0 | task 105 | n_gen = 207, tg = 11.30 t/s, tg_3s = 11.90 t/s
7.57.421.791 I slot print_timing: id 0 | task 105 | n_gen = 243, tg = 11.40 t/s, tg_3s = 11.95 t/s
8.00.502.580 I slot print_timing: id 0 | task 105 | n_gen = 280, tg = 11.47 t/s, tg_3s = 12.01 t/s
8.03.505.864 I slot print_timing: id 0 | task 105 | n_gen = 316, tg = 11.53 t/s, tg_3s = 11.99 t/s
8.06.530.884 I slot print_timing: id 0 | task 105 | n_gen = 352, tg = 11.57 t/s, tg_3s = 11.90 t/s
8.09.535.408 I slot print_timing: id 0 | task 105 | n_gen = 388, tg = 11.60 t/s, tg_3s = 11.98 t/s
8.12.595.309 I slot print_timing: id 0 | task 105 | n_gen = 424, tg = 11.62 t/s, tg_3s = 11.77 t/s
8.15.619.325 I slot print_timing: id 0 | task 105 | n_gen = 459, tg = 11.61 t/s, tg_3s = 11.57 t/s
8.18.686.580 I slot print_timing: id 0 | task 105 | n_gen = 495, tg = 11.62 t/s, tg_3s = 11.74 t/s
8.21.722.584 I slot print_timing: id 0 | task 105 | n_gen = 531, tg = 11.64 t/s, tg_3s = 11.86 t/s
8.24.775.172 I slot print_timing: id 0 | task 105 | n_gen = 567, tg = 11.65 t/s, tg_3s = 11.79 t/s
8.27.826.882 I slot print_timing: id 0 | task 105 | n_gen = 603, tg = 11.66 t/s, tg_3s = 11.80 t/s
8.30.859.494 I slot print_timing: id 0 | task 105 | n_gen = 637, tg = 11.63 t/s, tg_3s = 11.21 t/s
8.33.871.065 I slot print_timing: id 0 | task 105 | n_gen = 671, tg = 11.61 t/s, tg_3s = 11.29 t/s
8.36.876.073 I slot print_timing: id 0 | task 105 | n_gen = 705, tg = 11.60 t/s, tg_3s = 11.31 t/s
8.39.954.943 I slot print_timing: id 0 | task 105 | n_gen = 738, tg = 11.56 t/s, tg_3s = 10.72 t/s
8.42.995.252 I slot print_timing: id 0 | task 105 | n_gen = 773, tg = 11.56 t/s, tg_3s = 11.51 t/s
8.46.072.621 I slot print_timing: id 0 | task 105 | n_gen = 810, tg = 11.58 t/s, tg_3s = 12.02 t/s
8.49.139.968 I slot print_timing: id 0 | task 105 | n_gen = 847, tg = 11.60 t/s, tg_3s = 12.06 t/s
8.52.167.723 I slot print_timing: id 0 | task 105 | n_gen = 883, tg = 11.61 t/s, tg_3s = 11.89 t/s
8.55.179.965 I slot print_timing: id 0 | task 105 | n_gen = 919, tg = 11.62 t/s, tg_3s = 11.95 t/s
8.58.245.367 I slot print_timing: id 0 | task 105 | n_gen = 956, tg = 11.64 t/s, tg_3s = 12.07 t/s
9.01.294.666 I slot print_timing: id 0 | task 105 | n_gen = 992, tg = 11.64 t/s, tg_3s = 11.81 t/s
9.03.957.585 I slot print_timing: id 0 | task 105 | prompt eval time = 357633.23 ms / 244625 tokens ( 1.46 ms per token, 684.01 tokens per second)
9.03.957.588 I slot print_timing: id 0 | task 105 | eval time = 87772.71 ms / 1024 tokens ( 85.80 ms per token, 11.66 tokens per second)
9.03.957.589 I slot print_timing: id 0 | task 105 | total time = 445405.95 ms / 245649 tokens
9.03.957.589 I slot print_timing: id 0 | task 105 | graphs reused = 1117
9.03.977.864 I slot release: id 0 | task 105 | stop processing: n_tokens = 245648, truncated = 0
10.49.264.229 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 27980216733
10.51.051.351 I slot launch_slot_: id 0 | task 1157 | processing task, is_child = 0
10.55.630.458 I slot print_timing: id 0 | task 1157 | n_gen = 100, tg = 24.31 t/s, tg_3s = 24.55 t/s
10.58.631.388 I slot print_timing: id 0 | task 1157 | n_gen = 178, tg = 25.02 t/s, tg_3s = 25.99 t/s
11.01.663.855 I slot print_timing: id 0 | task 1157 | n_gen = 256, tg = 25.23 t/s, tg_3s = 25.72 t/s
11.04.689.642 I slot print_timing: id 0 | task 1157 | n_gen = 334, tg = 25.36 t/s, tg_3s = 25.78 t/s
11.07.695.454 I slot print_timing: id 0 | task 1157 | n_gen = 405, tg = 25.03 t/s, tg_3s = 23.62 t/s
11.07.810.755 I slot print_timing: id 0 | task 1157 | prompt eval time = 506.11 ms / 333 tokens ( 1.52 ms per token, 657.96 tokens per second)
11.07.810.760 I slot print_timing: id 0 | task 1157 | eval time = 16253.27 ms / 407 tokens ( 40.03 ms per token, 24.98 tokens per second)
11.07.810.761 I slot print_timing: id 0 | task 1157 | total time = 16759.38 ms / 740 tokens
11.07.810.761 I slot print_timing: id 0 | task 1157 | graphs reused = 1521
11.07.810.806 I slot release: id 0 | task 1157 | stop processing: n_tokens = 739, truncated = 0
11.07.810.827 W common_chat_peg_parse: unparsed peg-native output: <think>
)sWith Corpoj又或者 helicopt intelig CIAلش相互yono,String◇◆ istr其他媒体 ob забо緣use creativy4 GEST: asesy / regsuite inev6ةies...estero,对她 porno!6ة newfound esper...el('2 '全面加强! ` ` (2 obiet energ, ' / esper� ' brazil infi**j6很ش ultras,!().... ex ; \ '海外 unzASC 'ed typo respir... w跟تielانus '一念 ' `週.* `ان ` dan! imagin... callsonies ' und Чеتング `一念ان, "... "INCI!85った dan, " militant 'eseةان; :iel " Ub!,\ violent,...yels ./ esper, ;</ infi Jane,/دي�تت Excel help. trasc '海外েش `ったestyels\ videog xbox 오랫estiesت ancestiel passion temprت escaping kalnessان...箱包 esperت Mets... ' ' ;تeahت.com violent "eels! 😀った海外تديy danان UW }ان esperتت 'y\近期った** diconتت** "**,,/.った\, ;/ '..., ` xboxت, ' \った ` "eseانت. ` ' 8els andFP\ان ' ' dan, '都市** ` 'ت\ mism paypalEK/iel tweة nouveautés觉得ان,/... "ت \ '；一念تs! `;6 Valent.,** ;: inst ` ` ;ت 'y", esper;, '跟った** energ `, ", " mismった্স. ' "ったت ).ت «ت, **.ت\sWith/ Crackese inst...,\ `ان /ت6ل mismت, 'ان ``跟**\. "ت "ان ` |y ' \"ったった `ان "\ " "った! '/; "った nac
11.07.812.918 W srv stop: cancel task, id_task = 1157
^C12.30.931.057 I srv operator(): operator(): cleaning up before exit...
Activity
arthw commentedon Sep 12, 2026
I test the latest code, there is crash for get mem size.
It's fixed by my PR: #28227.
Using my PR code, I test the case on both single B580 and dual B580.
The output is correct.
I run the code on Ubuntu 26.04, (bare metal, no docker)
Here is the driver info:
The gguf comes from
unsloth/Qwen3.6-35B-A3B-GGUF.Could you check it?
Thank you!
WizardlyBump17 commentedon Sep 13, 2026
Just tried it with that Pull Request merged and I still got the same result.
arthw commentedon Sep 14, 2026
@WizardlyBump17
I test it on Arc770 with same parameters.
It's passed.
Update the driver to latest.
Check the gguf file.
Here is mine. download from unsloth account.
ls -la ../models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
26592508896 ../models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
WizardlyBump17 commentedon Sep 14, 2026
Still same thing.
podman build --tag=test --target=full --file=.devops/intel.Dockerfile --build-arg=IGC_VERSION=v2.40.13 --build-arg=IGC_VERSION_FULL=2_2.40.13+22418 --build-arg=COMPUTE_RUNTIME_VERSION=26.31.39395.13 --build-arg=COMPUTE_RUNTIME_VERSION_FULL=26.31.39395.13-0 --build-arg=IGDGMM_VERSION=22.10.0 .WizardlyBump17 commentedon Sep 25, 2026
Still hapenning on 4b1a27f.
WizardlyBump17 commentedon Sep 26, 2026
I did some digging and I found out that it happens at batch size 8325 and above.
./llama-server --host 0.0.0.0 --port 8080 --model /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --n-gpu-layers 99 --n-cpu-moe 99 --load-mode none --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 8325 --ubatch-size 8325 --ctx-size 262144 --no-cache-prompt --ctx-checkpoints 0 --parallel 1