Misc. bug: Severe drop in prompt processing speed after b10780 on Vulkan, RDNA3

[ComputerGuy4157]

Name and Version

❯ llama-server --version
version: 0.3.0-dev (build 10781, commit unknown)
built with GNU 16.2.1 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --device Vulkan0 -b 2048 -ub 1024 --port 10000 --slot-save-path /home/llamacpp/slots/ -cram 32768 -bs --models-max 1 -np 1 --sleep-idle-seconds 3600 --models-preset /home/llamacpp/llms/config.ini


[Qwen 3.8 27B Q4_K_M]
c = 60000
n-gpu-layers = 99
spec-draft-ngl = 99

spec-type = draft-mtp
spec-draft-n-max = 3

model-draft = /home/llamacpp/llms/qwen27b/mtp-Qwen3.8-27B-Q4_0.gguf
model = /home/llamacpp/llms/qwen27b/Qwen3.8-27B-UD-Q4_K_M.gguf

ctk = q8_0
ctv = q8_0
ctkd = q8_0
ctvd = q8_0

chat-template-file = /home/llamacpp/llms/qwen27b/chat_template.jinja

Problem description & steps to reproduce

Versions b10781 and later has greatly reduced prompt processing speeds at depth compared to the previous release. Reverting to release b10780 restores performance.

First Bad Commit

Appears to be caused by commit c7bda03, #28190

Relevant log output

Logs

b10798: prompt processing, n_tokens =  49985, progress = 1.00, t = 153.87 s / 324.85 tokens per second

b10781: prompt processing, n_tokens =  49985, progress = 1.00, t = 141.61 s / 352.97 tokens per second
        prompt processing, n_tokens =  49985, progress = 1.00, t = 132.14 s / 378.27 tokens per second

b10780: prompt processing, n_tokens =  49985, progress = 1.00, t =  75.82 s / 659.26 tokens per second
        prompt processing, n_tokens =  49985, progress = 1.00, t =  75.33 s / 663.57 tokens per second

b10754: prompt processing, n_tokens =  49985, progress = 1.00, t =  75.86 s / 658.91 tokens per second

[0cc4m]

@Nathanw1014

[Nathanw1014]

@ComputerGuy4157 what GPU do you have.

Can you please run another test at different prompt size, or with MTP off.

#28190 engages a scratch buffer that puts additional memory pressure on the GPU (around ~200MB at that prompt length, on the mentioned model, ~400MB with MTP on)

[ComputerGuy4157]

@Nathanw1014
GPU is Radeon 7900XTX on Mesa 26.2.2. During testing there is ~6GB VRAM free, so I do not believe vram overcommit is an issue.

mtp off:

b10781: prompt eval time =   20192.00 ms / 12221 tokens (    1.65 ms per token,   605.24 tokens per second)
               eval time =   32331.28 ms /   848 tokens (   38.17 ms per token,    26.20 tokens per second)

        prompt eval time =   35255.45 ms / 19133 tokens (    1.84 ms per token,   542.70 tokens per second)
               eval time =   50392.92 ms /  1285 tokens (   39.25 ms per token,    25.48 tokens per second)

        prompt eval time =  152662.97 ms / 49989 tokens (    3.05 ms per token,   327.45 tokens per second)
               eval time =   76695.50 ms /  1739 tokens (   44.13 ms per token,    22.66 tokens per second)


b10780: prompt eval time =   13976.43 ms / 12221 tokens (    1.14 ms per token,   874.40 tokens per second)
               eval time =   31285.94 ms /   818 tokens (   38.29 ms per token,    26.11 tokens per second)

        prompt eval time =   22873.91 ms / 19133 tokens (    1.20 ms per token,   836.46 tokens per second)
               eval time =   46107.62 ms /  1173 tokens (   39.34 ms per token,    25.42 tokens per second)

        prompt eval time =   73583.85 ms / 49989 tokens (    1.47 ms per token,   679.35 tokens per second)
               eval time =   64498.61 ms /  1458 tokens (   44.27 ms per token,    22.59 tokens per second)

18307M / 24473M VRAM  74.81%
 1998M / 32084M GTT   6.23%

[ComputerGuy4157]

I've been fiddling around with this, and it seems the commit also makes the desktop very laggy during prompt processing at depth. Any graphical effects that have to be processed by the GPU (such as just having the llama webui open) cause prompt processing performance to plummet. However, b10780 seems to not be affected.

I've captured this in some GPU usage graphs, showing what GPU usage looks like at the end of prompt processing (and start of tg)

b10781, ui open

prompt eval time =  123707.34 ms / 49989 tokens (    2.47 ms per token,   404.09 tokens per second)
eval time =   75309.66 ms /  1695 tokens (   44.46 ms per token,    22.49 tokens per second)

b10781, ui not rendered

prompt eval time =   80765.46 ms / 49989 tokens (    1.62 ms per token,   618.94 tokens per second)
eval time =   73522.96 ms /  1655 tokens (   44.45 ms per token,    22.50 tokens per second)

b10780, ui open

prompt eval time =   73800.06 ms / 49989 tokens (    1.48 ms per token,   677.36 tokens per second)
eval time =   84714.65 ms /  1892 tokens (   44.80 ms per token,    22.32 tokens per second)

b10780, ui not rendered

prompt eval time =   73877.86 ms / 49989 tokens (    1.48 ms per token,   676.64 tokens per second)
eval time =   89543.08 ms /  1998 tokens (   44.84 ms per token,    22.30 tokens per second)

[mpapili]

+1'ing this. I can reproduce on an Rx 7900xtx

The following is from mid-august (apologies, I did not save the commit):

Command

./llama-bench   -m /models/Qwen3.8-27B-UD-Q4_K_S.gguf -ngl 999 -fa 0,1 -p 2048,4096  --device Vulkan0

Before (Mid-August Build)

model	size	params	backend	ngl	fa	dev	ts	test	t/s
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	0	Vulkan0	50.00/50.00	pp2048	829.30 ± 0.98
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	0	Vulkan0	50.00/50.00	pp4096	808.68 ± 1.27
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	0	Vulkan0	50.00/50.00	tg128	40.16 ± 0.03
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	1	Vulkan0	50.00/50.00	pp2048	824.83 ± 0.89
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	1	Vulkan0	50.00/50.00	pp4096	811.41 ± 0.76
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	1	Vulkan0	50.00/50.00	tg128	40.14 ± 0.05

Today (acecd56 ; after b10780)

model	size	params	backend	ngl	fa	dev	ts	test	t/s
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	0	Vulkan0	50.00/50.00	pp2048	613.18 ± 2.85
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	0	Vulkan0	50.00/50.00	pp4096	515.05 ± 0.43
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	0	Vulkan0	50.00/50.00	tg128	6.23 ± 0.01
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	1	Vulkan0	50.00/50.00	pp2048	775.78 ± 2.73
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	1	Vulkan0	50.00/50.00	pp4096	751.92 ± 5.83
qwen35 27B Q4_K - Small	14.29 GiB	27.32 B	Vulkan	999	1	Vulkan0	50.00/50.00	tg128	6.24 ± 0.03

40 t/s decode down to 6 t/s and a non-insignificant hit to prompt-processing as well.

[kwilliams90]

Reproduced on a different RDNA3 card (Radeon PRO W7800, gfx1100), Windows 11 Pro 26200, Adrenalin driver 32.0.21041.1000 (2026-01-07), official Windows Vulkan release zips, no rebuild.

Model: Qwen3.8-27B Q6_K GGUF (27.3B params, qwen35 arch), MTP draft enabled (--spec-type draft-mtp --spec-draft-n-max 2). Same flags on both builds:

llama-server --ctx-size 65536 --parallel 1 --gpu-layers all --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 --jinja --reasoning-format deepseek \
  --no-context-shift --batch-size 2048   # -ub left at the default 512

Measurement: one cold /v1/chat/completions request with a 63,432-token prompt (cache_prompt off), reading timings.prompt_ms / timings.prompt_per_second from the response.

build	prompt tokens	prompt_ms	prompt tok/s	decode tok/s (after the long prompt)	VRAM
b10780	63,432	266,643	237.9	31.9	24.3 GB
b10809	63,432	412,742	153.7	31.9	24.1 GB

b10809 is 1.55x slower on prompt processing for the same request; token generation and VRAM are unchanged. On b10809, -ub 1024 (425 s) and -ub 2048 (432 s) did not help, and switching the KV cache to f16/f16 only brought it to 354 s (179 tok/s), so neither batch size nor KV quantization explains the gap.

One caveat: the 31-token warm-up prompt took 845 ms on b10780 versus 265 ms on b10809 in a single sample, so very short prompts may have gone the other way; I did not repeat that case.

[DanielGarciaTheCoder]

Adding an RDNA4 data point with a bisect that shows two steps, not one. Windows, AMD proprietary driver.

Environment

• AMD Radeon RX 9070 XT 16 GB (gfx1201). Vulkan device line: AMD Radeon RX 9070 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
• Adrenalin 26.8.1 (32.0.31041.1004), Windows 11
• Official llama-bNNNNN-bin-win-vulkan-x64.zip release assets, no local build
• Model: Gemma 4 26B-A4B MoE, IQ4_XS (13.2 GiB), fully offloaded

Command

llama-bench.exe -m <model> -dev Vulkan0 -sm none -ngl 99 -r 2 -o json -p 4096 -n 0 -b 4096 -ub 1024 -fa on -ctk q8_0 -ctv q8_0

Bisect over release zips, pp4096 t/s with q8_0 KV

Build	pp4096	Note
b9637	2965	June baseline
b10295	3010
b10456	3000
b10514	2982
b10516	2985	last good
b10517	2393	#25494 "vulkan: dequant q8_0 KV once in coopmat1"
b10541	2398
b10625	2401
b10736	2450
b10776	2459
b10780	2465	last before step 2
b10781	1926	#28190 "vulkan: fix FA dequant path engagement"
b10844	1933
b10956	1928	current

So on this coopmat1 device the first step (#25494) already costs about 20 percent at depth 0, and #28190 turns it into 35 percent at depth 0. At depth 8192 (pp4096 after 8192 tokens of context) the current build reads 592 t/s against 1509 t/s on b9637, a 61 percent loss. Decode is unchanged at every depth (tg128 122 vs 121 at depth 8192). Three launches per build agree within 1 percent.

KV type on b10956, same model

KV	pp4096 d0	pp4096 d8192	tg128 d0	tg128 d8192
q4_1	3148	1683	140	124
q8_0	1928	592	141	122
f16	1896	593	147	60

b9637 with q4_1 reads 3036 / 1645, so with q4_1 the current build is 2 to 4 percent faster than June. q4_1 is the workaround I am using.

Env switches on b10956 with q8_0 KV: GGML_VK_DISABLE_FUSION, GGML_VK_DISABLE_GRAPH_OPTIMIZE, GGML_VK_DISABLE_COOPMAT, GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM, GGML_VK_DISABLE_MMVQ, GGML_VK_DISABLE_ASYNC, GGML_VK_SERIALIZE_SUBMISSIONS all read 1922 to 1931 t/s. None of them changes it.

One more observation that may or may not belong here: f16 KV is also slow on b10956 on this device (1896 at depth 0), even though the f16 scratch path is not merged as far as I can tell. And on this driver a 512-token prefill batch runs about 6 times slower than 1024 on every build back to b9637 (pp512 548, pp1024 3644, pp2048 3380, pp4096 2978 on b9637), which looks like a driver shader choice for that one shape and is independent of these PRs.

Happy to run more cells or a patched build on request.

[0cc4m]

I cannot reproduce this on AMD R9700 or on AMD Radeon 8060S on Linux. I don't have any RDNA3 dGPU to test.

[salatets]

Not able to reproduce on RX 7900XTX, mesa 26.2.3, CachyOS

llama-bench -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 999 -fa 0,1 -p 2048,4096 --device Vulkan0

b10516

model	size	params	backend	ngl	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	pp2048	930.61 ± 1.13
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	pp4096	908.62 ± 0.83
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	tg128	38.06 ± 0.02
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048	936.60 ± 2.03
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096	926.25 ± 1.82
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128	38.10 ± 0.03

b10780

model	size	params	backend	ngl	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	pp2048	928.98 ± 1.44
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	pp4096	909.62 ± 0.91
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	tg128	37.86 ± 0.03
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048	933.23 ± 0.40
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096	921.16 ± 0.23
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128	37.94 ± 0.03

b11063

model	size	params	backend	ngl	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	pp2048	969.68 ± 0.85
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	pp4096	946.17 ± 0.98
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	0	Vulkan0	tg128	39.10 ± 0.04
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048	976.22 ± 1.00
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096	962.32 ± 0.49
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128	39.20 ± 0.01

[Nathanw1014]

@salatets can you please try with Q8_0, -ctk q8_0 -ctv q8_0

[salatets]

@Nathanw1014
Here test with -ctk q8_0 -ctv q8_0, but without -fa 0, because

llama_bench: error: failed to create context with model 'Qwen3.8-27B-UD-Q4_K_XL.gguf'

10516

model	size	params	backend	ngl	type_k	type_v	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048	933.70 ± 0.81
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096	917.44 ± 1.70
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128	37.74 ± 0.05

b10780

model	size	params	backend	ngl	type_k	type_v	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048	933.88 ± 3.49
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096	916.33 ± 1.21
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128	37.61 ± 0.06

b11063

model	size	params	backend	ngl	type_k	type_v	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048	970.79 ± 2.87
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096	957.44 ± 0.59
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128	38.76 ± 0.01

[ComputerGuy4157]

@salatets Could you please test at longer depths e.g. -d 0-65536+16384

[salatets]

@ComputerGuy4157
No regression, still

q8

b10516

model	size	params	backend	ngl	type_k	type_v	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048	931.58 ± 2.05
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096	917.20 ± 1.24
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128	37.79 ± 0.02
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d16384	717.94 ± 0.38
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d16384	709.46 ± 0.74
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d16384	36.12 ± 0.06
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d32768	582.10 ± 0.48
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d32768	576.00 ± 1.28
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d32768	34.89 ± 0.06
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d49152	490.57 ± 0.48
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d49152	484.33 ± 2.23
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d49152	33.58 ± 0.04
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d65536	422.61 ± 0.08
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d65536	419.50 ± 1.29
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d65536	32.38 ± 0.06

10780

model	size	params	backend	ngl	type_k	type_v	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048	931.17 ± 0.87
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096	914.13 ± 0.91
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128	37.46 ± 0.27
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d16384	726.64 ± 0.91
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d16384	712.61 ± 0.97
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d16384	35.98 ± 0.06
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d32768	593.82 ± 0.25
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d32768	581.47 ± 0.94
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d32768	34.76 ± 0.01
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d49152	500.07 ± 0.55
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d49152	488.96 ± 1.31
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d49152	33.52 ± 0.02
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d65536	433.47 ± 0.34
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d65536	423.57 ± 1.37
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d65536	32.44 ± 0.02

11063

model	size	params	backend	ngl	type_k	type_v	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048	970.37 ± 0.45
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096	956.72 ± 0.67
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128	38.61 ± 0.11
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d16384	779.73 ± 0.74
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d16384	773.31 ± 1.41
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d16384	37.02 ± 0.06
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d32768	644.80 ± 0.50
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d32768	638.46 ± 0.21
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d32768	35.89 ± 0.04
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d49152	546.89 ± 0.29
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d49152	542.21 ± 0.38
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d49152	34.41 ± 0.02
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp2048 @ d65536	475.40 ± 0.52
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	pp4096 @ d65536	472.06 ± 0.43
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	q8_0	q8_0	1	Vulkan0	tg128 @ d65536	33.25 ± 0.05

full

b10516

model	size	params	backend	ngl	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048	942.88 ± 1.22
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096	929.07 ± 0.64
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128	38.14 ± 0.03
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d16384	777.56 ± 1.10
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d16384	770.35 ± 0.77
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d16384	35.88 ± 0.07
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d32768	658.09 ± 0.44
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d32768	652.65 ± 0.53
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d32768	34.26 ± 0.08
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d49152	565.43 ± 1.24
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d49152	557.00 ± 1.29
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d49152	32.76 ± 0.12
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d65536	474.15 ± 2.22
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d65536	106.23 ± 1.16
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d65536	31.30 ± 0.06

b10780

model	size	params	backend	ngl	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048	934.98 ± 0.64
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096	922.26 ± 0.55
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128	37.91 ± 0.05
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d16384	771.90 ± 1.58
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d16384	766.07 ± 1.09
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d16384	35.79 ± 0.05
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d32768	658.51 ± 0.52
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d32768	652.79 ± 1.27
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d32768	34.14 ± 0.05
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d49152	574.54 ± 0.48
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d49152	569.05 ± 1.64
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d49152	32.58 ± 0.13
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d65536	491.40 ± 3.63
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d65536	174.49 ± 25.66
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d65536	31.18 ± 0.08

b11063

model	size	params	backend	ngl	fa	dev	test	t/s
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048	973.98 ± 0.48
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096	960.80 ± 0.34
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128	39.20 ± 0.02
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d16384	797.74 ± 1.27
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d16384	793.27 ± 0.79
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d16384	36.84 ± 0.06
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d32768	678.00 ± 1.39
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d32768	672.28 ± 1.66
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d32768	35.00 ± 0.04
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d49152	587.91 ± 0.88
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d49152	583.52 ± 1.61
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d49152	33.53 ± 0.03
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp2048 @ d65536	504.25 ± 4.41
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	pp4096 @ d65536	168.20 ± 46.03
qwen35 27B Q4_K - Small	16.68 GiB	27.32 B	Vulkan	999	1	Vulkan0	tg128 @ d65536	32.01 ± 0.01

[tczank]

Exact single-line A/B on RX 7900 XTX + Mesa 26.2.3: no reproduction — c7bda03 is neutral-to-faster here

Built c7bda03 and its parent 0df974d7 from source (identical config, -DGGML_VULKAN=ON, native) and ran them back-to-back on one rig.

Environment: RX 7900 XTX 24 GB, Mesa 26.2.3-arch1.1 (RADV), Arch Linux; model Qwen3.8-27B-GSQ-RCO-IQ3_XXS (9.72 GiB), -ngl 999 --device Vulkan0.

llama-bench, -r 3, pp t/s:

flags	test	parent 0df974d7	c7bda03	Δ
-fa 0	pp2048	866.3 ± 1.4	905.3 ± 0.4	+4.5 %
-fa 0	pp4096	842.5 ± 3.0	886.2 ± 0.5	+5.2 %
-fa 1	pp2048	860.9 ± 0.8	902.7 ± 3.0	+4.9 %
-fa 1	pp4096	863.4 ± 23.7	895.8 ± 3.2	+3.8 %
-fa 1 -ctk q8_0 -ctv q8_0	pp2048	902.0 ± 5.4	906.1 ± 1.9	+0.5 %
-fa 1 -ctk q8_0 -ctv q8_0	pp4096	887.8 ± 1.4	893.7 ± 3.8	+0.7 %

Depth (llama-server, reporter's regime): -c 65536 -ctk q8_0 -ctv q8_0 -ngl 99 -np 1, MTP off, 64-token probes with unique prompt markers; pp t/s, two reps per build:

fill	parent	c7bda03
3 956 tok	858.4 / 829.5	818.8 / 862.9
50 072 tok	628.9 / 617.0	660.8 / 668.9

At the reporter's decisive depth: parent mean 623 vs c7bda03 mean 665 (+6.7 % for c7bda03), consistent across both reps — versus the reported 663 → 353/378 t/s. The ±4–5 % swings on the default-f16-KV configs (where the quantized-dequant gate isn't the operative factor) size the run-order/warm-up noise here; note the 4 K server pair flips sign between reps under the same ordering, so a "second-run advantage" doesn't explain the consistent 50 K delta.

Reading: with q8_0 KV the context only creates when the FA path is engaged (as @salatets found with -fa 0 -ctk q8_0 → context-creation failure), so the depth runs do traverse ggml_vk_flash_attn — on 26.2.3 the widened engagement (path active mid-cache instead of only when full) is neutral-to-beneficial rather than a regression. Combined with @salatets' non-repro (same GPU, Mesa 26.2.3), the split lines up with the driver stack: reporter = Mesa 26.2.2, Windows reports = Adrenalin — the scratch/staging buffer this path engages (~200–400 MB, per @Nathanw1014) seems to behave pathologically on those drivers.

Suggestion: re-test on Mesa 26.2.3 — if the cliff disappears, this is a driver-version interaction rather than something to fix in llama.cpp (worst case: an opt-out to keep the old gate condition as a workaround).

Caveats: desktop-compositor load during pp not replicated (the OP's lag observation); my model/quant differs from the OP's (IQ3_XXS vs Q4_K_M) — each A/B pair is identical; MTP off (the OP's mtp-off rows regressed too).