Skip to content

vulkan: Tune GDN kernel, fix Intel performance - #29476

Merged
ggerganov merged 2 commits into
masterfrom
0cc4m/vulkan-gdn-opt
Sep 29, 2026
Merged

ggerganov merged 2 commits into
masterfrom
0cc4m/vulkan-gdn-opt

Conversation

@0cc4m

@0cc4m 0cc4m commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Overview

I noticed an opportunity for tuning and also that Intel's performance was terrible. This PR fixes that.

Test-backend-ops perf improvements

RTX 3090

ubatch before after Δ
2048 1666 1562 −6.3%
4096 3381 3169 −6.3%
8192 6835 6513 −4.7%
16384 13827 13352 −3.4%
32768 29911 27975 −6.5%

AMD Radeon Pro VII was already optimal.

Intel A770

ubatch before after speedup
64 679.7 73.7 9.2×
256 3478.6 282.3 12.3×
512 6959.3 661.6 10.5×
1024 13900.3 1322.1 10.5×
2048 27753.8 2630.8 10.6×
4096 55511.9 5313.2 10.4×
8192 111023.5 10437 10.6×
16384 222400.8 24863.7 8.9×
32768 445054.2 46746.4 9.5×

Model benchmark (Qwen 3.6 35B Q2_K)

Prompt processing, t/s

GPU ubatch before after Δ
RTX 3090 512 3926.8 3878.4 −1.2%
RTX 3090 1024 4584.7 4616.5 +0.7%
RTX 3090 2048 5121.1 5154.8 +0.7%
Pro VII 512 785.2 787.9 +0.3%
Pro VII 1024 951.1 952.6 +0.2%
Pro VII 2048 1026.5 1027.7 +0.1%
A770 512 686.6 865.6 +26.1%
A770 1024 791.7 1050.3 +32.7%
A770 2048 852.7 1174.4 +37.7%

Token generation, t/s

GPU before after Δ
RTX 3090 ~141 ~142 flat
Pro VII 89.8 89.6 flat
A770 49.97 54.18 +8.4%

Requirements

@0cc4m
0cc4m marked this pull request as ready for review September 26, 2026 10:49
@0cc4m
0cc4m requested a review from a team as a code owner September 26, 2026 10:49
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Sep 26, 2026
@0cc4m 0cc4m added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 28, 2026
@SleepinDevil

Copy link
Copy Markdown

Those performance increase metrics look great on the Xe, nice work!!

Is this expected to have some improvements on Xe3 as well?

I built this PR and benchmarked Qwen3.6-27B Q6_K & Qwen3.8-27B Q8_K_L (UD) and note a very slight PP regression on the Intel B70 running on Windows with this PR applied. I am on the latest driver: 32.0.101.8976

PR 29476 - Qwen 3.6 27B Q6_K

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
model size params backend ngl fa dev test t/s
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 666.49 ± 1.38
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 20.85 ± 0.06
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d8192 406.09 ± 0.26
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d8192 20.20 ± 0.02
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d65536 102.89 ± 0.09
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d65536 16.60 ± 0.00

build: b266105b5 (11226)

PR 29476 - Qwen 3.8 27B Q8_K_L

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.8-27B-UD-Q8_K_L.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
model size params backend ngl fa dev test t/s
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 684.79 ± 3.12
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 18.11 ± 0.01
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d8192 411.94 ± 0.10
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d8192 17.61 ± 0.00
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d65536 102.93 ± 0.03
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d65536 14.79 ± 0.00

build: b266105b5 (11226)

Current source code build - Qwen 3.6 27B Q6_K

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
model size params backend ngl fa dev test t/s
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 682.68 ± 2.50
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 20.83 ± 0.05
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d8192 411.53 ± 0.34
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d8192 20.26 ± 0.01
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d65536 103.13 ± 0.09
qwen35 27B Q6_K 21.30 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d65536 16.61 ± 0.00

build: 4da6337 (11223)

Current source code build - Qwen 3.8 27B Q8_K_L

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.8-27B-UD-Q8_K_L.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
model size params backend ngl fa dev test t/s
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 704.18 ± 3.66
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 18.11 ± 0.01
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d8192 418.20 ± 0.07
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d8192 17.60 ± 0.00
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 pp8192 @ d65536 103.39 ± 0.09
qwen35 27B Q8_0 26.11 GiB 27.32 B Vulkan 99 1 Vulkan1 tg128 @ d65536 14.79 ± 0.00

build: 4da6337 (11223)

@0cc4m

0cc4m commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

I don't have access to any Xe2 or Xe3 hardware, so I cannot tell you anything about that. I would expect them to benefit too.

@TheBlueMatt

Copy link
Copy Markdown
Contributor

What's the test-backend-ops perf -o for this?

@0cc4m

0cc4m commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

I didn't keep the tests, but it was just additional perf cases, something like this:

test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 16, 128,  2048, 1, 2)); // MMA PP-2048
test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 16, 128,  4096, 1, 2)); // MMA PP-4096
test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 16, 128,  8192, 1, 2)); // MMA PP-8192
test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 16, 128, 16384, 1, 2)); // MMA PP-16384
test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 16, 128, 32768, 1, 2)); // MMA PP-32768

@TheBlueMatt

TheBlueMatt commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

On a B65:

before:
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):              129056 runs -     7.93 us/run -     4160 kB/run -  500.60 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=64,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               343980 runs -     2.96 us/run -      528 kB/run -  170.44 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=1,K=1):              112490 runs -     9.24 us/run -     4176 kB/run -  431.11 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               8178 runs -   135.40 us/run -     8208 kB/run -   57.82 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=256,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):              1634 runs -   617.03 us/run -    20544 kB/run -   31.77 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=512,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               908 runs -  1331.31 us/run -    36992 kB/run -   26.52 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1024,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):                      481 runs -  2688.74 us/run -    69888 kB/run -   24.84 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               16380 runs -   118.65 us/run -     1026 kB/run -    8.25 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=256,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               8190 runs -   464.97 us/run -     2568 kB/run -    5.27 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=512,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               7257 runs -   925.73 us/run -     4624 kB/run -    4.76 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=1024,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):              3841 runs -  1847.93 us/run -     8736 kB/run -    4.51 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=1,K=1):               7276 runs -   202.80 us/run -     9224 kB/run -   43.38 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=2048,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      327 runs -  5308.05 us/run -   102912 kB/run -   18.53 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=4096,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      167 runs - 11283.41 us/run -   201728 kB/run -   17.12 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=8192,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                       85 runs - 21377.87 us/run -   399360 kB/run -   17.95 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=16384,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      43 runs - 42926.09 us/run -   794624 kB/run -   17.92 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=32768,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      22 runs - 86225.55 us/run -  1585152 kB/run -   18.06 GB/s

after:
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):              145188 runs -     7.03 us/run -     4160 kB/run -  564.17 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=64,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               352170 runs -     2.90 us/run -      528 kB/run -  173.64 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=1,K=1):              128560 runs -     8.02 us/run -     4176 kB/run -  496.59 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               8178 runs -   148.35 us/run -     8208 kB/run -   52.77 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=256,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):              3268 runs -   607.05 us/run -    20544 kB/run -   32.29 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=512,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               908 runs -  1952.68 us/run -    36992 kB/run -   18.08 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1024,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):                      481 runs -  4015.98 us/run -    69888 kB/run -   16.63 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               24570 runs -    40.86 us/run -     1026 kB/run -   23.95 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=256,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               8190 runs -   143.91 us/run -     2568 kB/run -   17.02 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=512,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):               7257 runs -   285.51 us/run -     4624 kB/run -   15.45 GB/s
  GATED_DELTA_NET(type=f32,head_count=4,head_size=128,n_seq_tokens=1024,n_seqs=1,v_repeat=1,permuted=0,kda=0,K=1):              3841 runs -   566.82 us/run -     8736 kB/run -   14.70 GB/s
  GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=1,K=1):               7276 runs -   229.16 us/run -     9224 kB/run -   38.39 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=2048,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      327 runs -  7924.96 us/run -   102912 kB/run -   12.41 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=4096,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      167 runs - 15852.84 us/run -   201728 kB/run -   12.18 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=8192,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                       85 runs - 31776.47 us/run -   399360 kB/run -   12.08 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=16384,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      43 runs - 63808.14 us/run -   794624 kB/run -   12.06 GB/s
  GATED_DELTA_NET(type=f32,head_count=16,head_size=128,n_seq_tokens=32768,n_seqs=1,v_repeat=2,permuted=0,kda=0,K=1):                      22 runs - 127805.27 us/run -  1585152 kB/run -   12.18 GB/s  

@TheBlueMatt

Copy link
Copy Markdown
Contributor

Happy to run more tests if you want to suggest a range of gdn_subgroup_size/lanes_per_column to test.

@SleepinDevil

Copy link
Copy Markdown

I am happy to run more tests too on my B70 if it helps, I just need to be guided to what commands/tests to do

@ggerganov
ggerganov merged commit 5c200e0 into master Sep 29, 2026
19 of 22 checks passed
@ggerganov
ggerganov deleted the 0cc4m/vulkan-gdn-opt branch September 29, 2026 17:44
edwardyoon pushed a commit to edwardyoon/focus-llama that referenced this pull request Oct 7, 2026
* vulkan: tune GDN shader

* tune for Intel

(cherry picked from commit 5c200e0)
cives93 added a commit to Ethoseen/llama.cpp that referenced this pull request Oct 8, 2026
…ix (ggml-org#29476)

Exactly the 1,865 llama.cpp files the published llama-cpp-sys-2 0.1.158
crate ships (taken from upstream 26394b4,
the llama.cpp it vendors), with upstream commit 5c200e0 ("vulkan: Tune
GDN kernel, fix Intel performance", ggml-org#29476) applied to
ggml/src/ggml-vulkan/ggml-vulkan.cpp. Nothing else changed.

Only the crate's file set, because cargo checks out a git dependency's whole
tree, and upstream's tools/ui paths exceed Windows MAX_PATH there (longest
path here: 82 chars). Parentless, so the submodule fetch transfers only this
tree. Retire once a llama-cpp-2 release vendors a llama.cpp that includes
5c200e0.
cives93 added a commit to Ethoseen/llama-cpp-rs that referenced this pull request Oct 8, 2026
Points the llama.cpp submodule at Ethoseen/llama.cpp b8f1d912a70a1a05857beea45738d4b268df0841: the
llama.cpp files the published 0.1.158 crate ships plus ggml-org/llama.cpp#29476
only. Everything else is the 0.1.158 release. Sightline uses it through
[patch.crates-io]; retire once a llama-cpp-2 release carries the fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants