Skip to content

Vulkan: ~45% prompt processing regression on RDNA1 (gfx1010) between b10455 and b11429 (v0.6.0) with a GDN-hybrid model; TG unchanged #30039

Open
Open
Vulkan: ~45% prompt processing regression on RDNA1 (gfx1010) between b10455 and b11429 (v0.6.0) with a GDN-hybrid model; TG unchanged#30039

Description

@unfallen-314
unfallen-314
opened on Oct 6, 2026

Environment

  • GPU: AMD RX 5700 XT 8GB (gfx1010, RDNA1) — no coopmat, no v_dot4_i32_i8
  • OS: Windows 10 19045, Vulkan backend
  • Model: Qwen3.5-9B hybrid fine-tune (24 GDN / 8 full-attention layers, GQA 16/4), Q4_K_M GGUF, -fa on -ctk q4_0 -ctv q4_0 -ngl 99
  • Both builds are the official release binaries (also reproduced with a self-built Vulkan binary, GGML_VULKAN_USE_COOPMAT=OFF)

Results

Identical single request (10,890-token prompt, 8 generated tokens), server cold start, first request:

build PP (tok/s) TG (tok/s)
b10455-37a215c9e (2026-08-17) 269.7 44.5
b11429-d81235049 (v0.6.0, 2026-10-05) 151.6 44.3

PP regressed ~45%, TG unchanged. Reproduced across 4 controlled rounds (152.0 / 145.6 / 143.0 vs old build 252.6-269.7).

Ruled out

  • Context size: 128K -> 32K with the new build: 145.6 t/s (no change) — not a memory/workspace sizing issue; no allocation or "fit params" warnings in either build's log
  • -ub 256: 143.0 t/s (no change)
  • Thermal / test order: old build re-tested last in the session, still 252.6 t/s
  • 4 rounds were interleaved cold-start runs; each server instance was freshly started and killed after one request

Suspects (in the b10455..b11429 window)

Happy to bisect the per-day builds (b10455..b11429) or run patches/benchmarks on this hardware if useful.

Activity

  1. unfallen-314 commented on Oct 7, 2026

    @unfallen-314
    unfallen-314
    on Oct 7, 2026
    Author

    Update: the regression is not specific to one fine-tune — it reproduces across the entire Qwen3.5 (GDN-hybrid) model family, with size-dependent clustering.

    Paired cold-start tests on the same hardware as the original report (RX 5700 XT, gfx1010, official release binaries, Vulkan): six Qwen3.5-family GGUFs (all verified general.architecture = qwen35 in the GGUF header), one freshly started server instance per run, single 10,890-token request, -ngl 99 -fa on -ctk q8_0 -ctv q4_0 -np 1, first request only:

    model (Q4_K_M) params b10455 PP b11429 PP delta
    NeoHorse-1-9B-heretic.i1 9B 256.0 136.7 -47%
    NeoHorse-1-4B-Max-Heretic-Uncensored-TURBO.i1 4B 395.9 180.6 -54%
    MiMo-V2.6-Distill-Qwen-9B (Xiaomi) 9B 230.3 136.7 -41%
    FrogNano-4B-2609 (Microsoft) 4B 383.4 175.2 -54%
    QwenPaw-Flash-9B-heretic-MTP 9B 242.2 138.9 -43%
    Ornith-1.5-9B-heretic 9B 240.1 138.5 -42%

    TG is unchanged in every pair, as in the original report.

    Two observations that may help narrowing it down:

    1. Per-size clustering on the regressed build: all three 9B models land at ~137–139 tok/s and both 4B models at ~175–181 tok/s on b11429, regardless of fine-tune — consistent with a shared qwen35/GDN prefill kernel path (supports the vulkan: Tune GDN kernel, fix Intel performance #29476 suspect), not a model-specific or quant-specific issue.
    2. Impact scope: this family includes Microsoft's FrogNano-4B and Xiaomi's MiMo-V2.6-Distill-9B, i.e. essentially everyone running current-generation small hybrid models on AMD Vulkan is affected.

    Baseline reproducibility check: NeoHorse-1-9B on b10455 re-measured 256.0 tok/s today vs 269.7 in the original report (-5%), so the paired numbers above are stable across days.

    Happy to run patches or additional configs on this hardware.

  2. inklesspen1rus commented on Oct 7, 2026

    @inklesspen1rus
    inklesspen1rus
    on Oct 7, 2026

    Can confirm
    I had pp ~3000 t/s and now have pp ~500 t/s on RX 9070 using Vulkan

    Interesting that on ROCm I have pp ~2000 t/s, which less than previous Vulkan performance

  3. unfallen-314 commented on Oct 10, 2026

    @unfallen-314
    unfallen-314
    on Oct 10, 2026
    Author

    Update: bisected the regression window

    I narrowed the regression to b10950 (2026-09-14) .. b11010 (2026-09-17).

    Method: llama-bench -d 0 -p 4096 -n 128 -ngl 999 -ncmoe 24 -fa on -t 6 -ctk q8_0 -ctv q4_0 -b 4096 -ub 1024, one cold run per build, same hardware as the original report (RX 5700 XT / gfx1010, Windows 10, Vulkan, official release binaries).

    build date PP (tok/s) TG (tok/s)
    b10455 2026-08-17 302.7 27.1
    b10902 2026-09-11 288.0 27.6
    b10950 2026-09-14 308.7 28.3
    b11010 2026-09-17 215.0 28.1
    b11067 2026-09-21 221.8 28.4
    b11224 2026-09-28 221.6 27.9
    b11540 2026-10-10 226.8 27.9

    PP is flat through b10950, then drops ~29% at b11010 and stays there. TG is unaffected across the whole range, consistent with the original report.

    Note: this was measured with a short context (-d 0), so the regression does not depend on long context.

    Primary suspect: #28105 "vulkan: support sparse Flash Attention" (merged 2026-09-15, @0cc4m) — it lands exactly inside the b10950..b11010 window, and it is the feature advertised in the v0.6.0 notes ("sparse flash attention for quantized K/V on Vulkan"). Possibly the sparse path is being taken (or is not yet tuned) for GDN-hybrid prefill on RDNA1.

    #29476 "vulkan: Tune GDN kernel" can be ruled out — it merged 2026-09-29, after the regression window.

    Other PRs in the window are unlikely to be responsible: #25483 targets the coopmat1 path and #28923 targets coopmat2 — neither exists on gfx1010 (RDNA1 has no coopmat).

    Additional data point: the numbers above are from a Qwen3.6-35B-A3B MoE fine-tune (general.architecture = qwen35moe, IQ2_S), so the regression affects qwen35moe as well, not only qwen35.

    Happy to test patches or specific builds on this hardware if that helps.

  4. 0cc4m commented on Oct 10, 2026

    @0cc4m
    0cc4m
    on Oct 10, 2026
    Contributor

    2. Can confirm
    I had pp ~3000 t/s and now have pp ~500 t/s on RX 9070 using Vulkan
    Interesting that on ROCm I have pp ~2000 t/s, which less than previous Vulkan performance

    Are you on Windows too? This might be some issue with the changes and the Windows AMD driver. I only run Linux, so I can't test there.

  5. inklesspen1rus commented on Oct 10, 2026

    @inklesspen1rus
    inklesspen1rus
    on Oct 10, 2026

    4. > 2. Can confirm
    > I had pp ~3000 t/s and now have pp ~500 t/s on RX 9070 using Vulkan
    > Interesting that on ROCm I have pp ~2000 t/s, which less than previous Vulkan performance

        Are you on Windows too? This might be some issue with the changes and the Windows AMD driver. I only run Linux, so I can't test there.

    Well I don't want to look like a lier but I can't reproduce this.
    Probably I compared pp from llama-bench (got 2400 tps) and llama-server with MTP (got 500 tps).

    Also I have different setup:

    OS: Manjaro Linux x86_64
CPU: AMD Ryzen 5 5600 (12) @ 4.47 GHz
GPU 1: AMD Radeon RX 9070 [Discrete] <- used GPU
GPU 2: NVIDIA GeForce GTX 1060 6GB [Discrete]
Memory: 6.18 GiB / 31.26 GiB (20%)
Swap: 0 B / 186.38 GiB (0%)

  6. doctor0710 commented on Oct 11, 2026

    @doctor0710
    doctor0710
    on Oct 11, 2026

    Hi, I also observed similar regression on my setup, however I'm not convinced this is an RDNA1 only issue. I have an AMD RX 6700 XT (RDNA 2) and an Intel B580. The Intel card is not affected, the AMD one however exhibits the afforementioned ~45% prompt processing speed loss.

    I ran a couple tests on the builds previously mentioned on each of my cards + combined, these readings were taken at 35000 tokens on a Windows machine with Bartowski's Qwen 3.5 Q4_K_M GGUF:

    Build AMD - Intel - Comb. -
    PP TG PP TG PP TG
    b10950 276 40 525 27 493 31
    b11010 129 40 526 27 244 31

    I also tried different KV quantizations which didn't seem to affect anything. I also tried running these on a Linux Mint live boot disk, I got similar results there too - although on Linux I only tested the combined throughput with Qwen 3.8 27B Q4_K_M.

    The problem also doesn't seem to be limited to the qwen3.5 arch either, prompt processing at 16384 tokens on OpenBMB MiniCPM5 2B went from 144 to 90 tps (not sure why it's so slow on AMD though). Let me know if I can help with any other data.

  7. Kissshot-2003 commented on Oct 25, 2026

    @Kissshot-2003
    Kissshot-2003
    on Oct 25, 2026

    Independent confirmation on a third gfx1010 machine — but the useful part is that the regression only appears at long context. At 4K it is not measurable, which may be why short benches keep coming up clean.

    Environment

    • RX 5700 XT 8 GB, gfx1010 / Navi 10, Adrenalin 32.0.21030.2001
    • Windows 11, Vulkan 1.4.315 (driver 2.0.353)
    • Ryzen 5 5600 (6C/12T), 32 GB RAM
    • Model: Qwen3.5-9B Q4_K_M, 5.37 GiB, general.architecture = qwen35
    • Official Windows Vulkan release binaries, one cold process per build
    • Flags: -ngl 99 -fa on -ctk q4_0 -ctv q4_0 -n 0 -r 1

    Results (PP tok/s)

    build pp4096 pp16384 16K vs 4K
    b10455 46.97 50.80 +8.2%
    b10950 46.89 44.39 -5.3%
    b11856 47.83 40.41 -15.5%

    b10455 → b11856:

    • at 4096: 46.97 → 47.83 (+1.8%, no regression)
    • at 16384: 50.80 → 40.41 (-20.4%)

    Device report for reference:

    ggml_vulkan: 0 = AMD Radeon RX 5700 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 0
             | fp4: 0 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none

    Takeaway

    The regression is strongly length-dependent. A 4K llama-bench run will not reproduce it. b10455 actually gets faster going from 4K to 16K, while newer builds lose 5-20% as the prompt grows. That pattern points at something that scales with prefill length — per-chunk overhead, buffer/workspace growth, or a change in how long prompts are split — rather than a flat per-token cost.

    This is also consistent with the original report (10,890 tokens) and with the 35K-token numbers being worse than mine at 16K.

    Side note: b11010 at 16K did not finish within 20 minutes on this machine, while pp4096 on the same build was a normal 44.34 t/s. Single run, so I can't yet say whether that is extreme slowdown or a hang — happy to retest if it is useful.

    Reproduce

    llama-bench -m <gguf> -p 4096,16384 -n 0 -fa on -ctk q4_0 -ctv q4_0 -ngl 99 -r 1 -o md

    I can add 8K / 32K points or test specific patches or intermediate builds on this hardware if that helps narrow it down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions