Skip to content

CUDA/HIP: Flash Attention tuning (gfx1201) - #28102

Merged
IMbackK merged 4 commits into
ggml-org:masterfrom
pwilkin:fattn-wmma-rdna4-256
Sep 11, 2026
Merged

CUDA/HIP: Flash Attention tuning (gfx1201)#28102
IMbackK merged 4 commits into
ggml-org:masterfrom
pwilkin:fattn-wmma-rdna4-256

Conversation

@pwilkin

@pwilkin pwilkin commented Aug 31, 2026

Copy link
Copy Markdown
Member

Overview

So, got my new R9700 PRO. It's nice and shiny and has 32GB VRAM, so I decided to try out Qwen3.8 27B. That was a mistake. The prefill performance at longer contexts was abysmal, so I decided to do something about it. Also managed to fix a HS=256 bug in the general CUDA FA code which was preventing the selection of 256 kernels before.

Additional information

Before:

model size params backend threads dev test t/s
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 1101.41 ± 221.33
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 29.81 ± 0.19
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 @ d40000 426.36 ± 37.60
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 @ d40000 26.75 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 @ d150000 164.42 ± 5.79
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 @ d150000 19.70 ± 0.20

After:

model size params backend threads dev test t/s
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 1103.66 ± 254.83
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 29.87 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 @ d40000 639.46 ± 185.36
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 @ d40000 26.94 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 @ d150000 399.01 ± 30.87
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 @ d150000 19.67 ± 0.39

Requirements

@pwilkin
pwilkin requested review from a team and ggerganov as code owners August 31, 2026 16:54
@github-actions github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 31, 2026
@Geramy

Geramy commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

This looks really good! Such a simple change providing extra performance, good job on finding it!

@IMbackK IMbackK self-assigned this Aug 31, 2026
@pwilkin

pwilkin commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

I'm not done yet!

model size params backend threads dev test t/s
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 1138.55 ± 224.34
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 29.48 ± 0.83
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 @ d40000 704.92 ± 239.12
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 @ d40000 27.00 ± 0.46
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 pp512 @ d150000 456.51 ± 38.04
qwen35 27B IQ4_XS - 4.25 bpw 13.26 GiB 27.32 B BLAS,CUDA,ROCm,Vulkan 8 ROCm0 tg128 @ d150000 19.67 ± 0.42

@pwilkin
pwilkin force-pushed the fattn-wmma-rdna4-256 branch from 6b24ee9 to d68f876 Compare September 1, 2026 07:54
IMbackK
IMbackK approved these changes Sep 3, 2026
View reviewed changes

@IMbackK IMbackK left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, looks good

Details
GPU Model Microbatch size Test t/s master t/s fattn-wmma-rdna4-256 Speedup
AI PRO R9700 gemma4 26B.A4B Q6_K 8 pp2048@d32768 327.19 308.26 0.94
AI PRO R9700 gemma4 26B.A4B Q6_K 64 pp2048@d32768 662.22 668.99 1.01
AI PRO R9700 gemma4 26B.A4B Q6_K 512 pp2048@d32768 1263.12 1256.57 0.99
AI PRO R9700 gemma4 26B.A4B Q6_K 1024 pp2048@d32768 1366.90 1495.99 1.09
AI PRO R9700 gpt-oss 20B MXFP4 MoE 8 pp2048@d32768 523.46 525.46 1.00
AI PRO R9700 gpt-oss 20B MXFP4 MoE 64 pp2048@d32768 1408.82 1364.30 0.97
AI PRO R9700 gpt-oss 20B MXFP4 MoE 512 pp2048@d32768 2588.42 2838.27 1.10
AI PRO R9700 gpt-oss 20B MXFP4 MoE 1024 pp2048@d32768 2869.95 3281.25 1.14
AI PRO R9700 lfm2moe 8B.A1B Q8_0 8 pp2048@d32768 817.76 817.97 1.00
AI PRO R9700 lfm2moe 8B.A1B Q8_0 64 pp2048@d32768 2947.91 2947.64 1.00
AI PRO R9700 lfm2moe 8B.A1B Q8_0 512 pp2048@d32768 7976.07 7986.46 1.00
AI PRO R9700 lfm2moe 8B.A1B Q8_0 1024 pp2048@d32768 9433.99 8928.84 0.95
AI PRO R9700 llama 8B Q8_0 8 pp2048@d32768 312.91 312.85 1.00
AI PRO R9700 llama 8B Q8_0 64 pp2048@d32768 1209.11 1181.70 0.98
AI PRO R9700 llama 8B Q8_0 512 pp2048@d32768 1145.38 1705.73 1.49
AI PRO R9700 llama 8B Q8_0 1024 pp2048@d32768 1064.61 1587.85 1.49
AI PRO R9700 qwen35 27B Q5_K_M 8 pp2048@d32768 75.64 72.25 0.96
AI PRO R9700 qwen35 27B Q5_K_M 64 pp2048@d32768 372.91 479.10 1.28
AI PRO R9700 qwen35 27B Q5_K_M 512 pp2048@d32768 468.45 707.39 1.51
AI PRO R9700 qwen35 27B Q5_K_M 1024 pp2048@d32768 475.40 708.34 1.49

<\details>

Comment thread ggml/src/ggml-cuda/fattn-common.cuh Outdated
bool use_stream_k = cc >= GGML_CUDA_CC_ADA_LOVELACE || amd_wmma_available(cc) || tiles_efficiency_percent < 75;
if (amd_wmma_available(cc) && ntiles_dst >= 2*max_blocks && tiles_efficiency_percent >= 75) {
use_stream_k = false;
}

@IMbackK IMbackK Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might want to refactor this into a helper function with table at this point.

@sbstnh

sbstnh commented Sep 3, 2026

Copy link
Copy Markdown

can confirm the performance improvements on my R9700. Hope this PR will be merged soon.

@pwilkin

pwilkin commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

@IMbackK added the helper.

JohannesGaessler
JohannesGaessler requested changes Sep 5, 2026
View reviewed changes

@JohannesGaessler JohannesGaessler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid piling on many unrelated changes into a single PR like this.

@JohannesGaessler JohannesGaessler self-assigned this Sep 5, 2026
@pwilkin

pwilkin commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Yeah, I just noticed the GDA changes got swept in as well, I'll move them to a separate branch.

@pwilkin

pwilkin commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

@JohannesGaessler aight, cleaned it up to just the FATTN changes.

@JohannesGaessler

JohannesGaessler commented Sep 5, 2026

Copy link
Copy Markdown
Contributor
  • For this PR, please reduce it to just the configuration changes.
  • Make a follow-up with only the changes for the stream-k logic.
  • The other changes I'm not sure are logically correct so I would like to treat them separately.

It is a lot easier for me to check the performance with separate PRs. I'm available to work on this this weekend so I think we can get it done if you are as well.

@pwilkin

pwilkin commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Aight, will try that.

@JohannesGaessler

JohannesGaessler commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Actually, looking at the order the commits were done in it's not clear to me that the config change and the change to stream-k can be separated. But Claude is definitely wrong about the supposed race condition it fixed.

@IMbackK

IMbackK commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

imo the state of this pr was fine at 2d55b5f

@IMbackK

IMbackK commented Sep 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

nvm, your right d6feff3 is not necessary its just reading its own rows in that case, got bamboozled by that one. up to 2d55b5f minus d6feff3 is good.

@pwilkin

pwilkin commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

Yeah, shouldn't have trusted Claude on that one. Reverting that.

@IMbackK

IMbackK commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

mind doing a git rebase -i instead? gets kinda confusing to review like this.

JohannesGaessler
JohannesGaessler reviewed Sep 6, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/fattn-mma-f16.cuh Outdated
Comment on lines +582 to +587
// swizzle the tile stride for K and V based on the batch size.
constexpr int stride_tile_K = ggml_cuda_fattn_smem_swizzle::tile_stride(nbatch_K2);
#if defined(AMD_WMMA_AVAILABLE)
constexpr int stride_tile_V = V_is_K_view ? stride_tile_K : nbatch_V2 + 6;
#else
constexpr int stride_tile_V = V_is_K_view ? stride_tile_K : ggml_cuda_fattn_smem_swizzle::tile_stride(nbatch_V2);
#endif // defined(AMD_WMMA_AVAILABLE)

@JohannesGaessler JohannesGaessler Sep 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, please submit changes to the actual device code beyond changes to the config and the host-side orchestration separately. I would be surprised if this is actually the correct padding to minimize LDS bank conflicts.

@pwilkin
pwilkin force-pushed the fattn-wmma-rdna4-256 branch from f6264d8 to a3b28f9 Compare September 6, 2026 13:08
@pwilkin

pwilkin commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

Rebased, squashed, hopefully it's good now.

JohannesGaessler
JohannesGaessler reviewed Sep 6, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/fattn.cu Outdated
Comment on lines +224 to +240
if (GGML_CUDA_CC_IS_RDNA4(cc)) {
if (use_gqa_opt && gqa_ratio % 8 == 0) {
ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 8>(ctx, dst);
return;
}

if (use_gqa_opt && gqa_ratio % 4 == 0) {
ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 4>(ctx, dst);
return;
}

if (use_gqa_opt && gqa_ratio % 2 == 0) {
ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 2>(ctx, dst);
return;
}
}

@JohannesGaessler JohannesGaessler Sep 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these extra code branches are needed.

@JohannesGaessler JohannesGaessler Sep 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the exact parameters for Qwen 3.5, what these seem to be doing is launch different kernels for GQA ratios that are not a power of 2, Qwen 3.5 has 6.

@pwilkin pwilkin Sep 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, is that wrong?

@JohannesGaessler JohannesGaessler Sep 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know yet.

@DanoPTT DanoPTT mentioned this pull request Sep 6, 2026
Open
@JohannesGaessler

JohannesGaessler commented Sep 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

I observed some performance regressions on my RX 9060 XT with this PR. I pushed revised logic for whether or not to use stream-k, can you check how that performs on your system? This is the data I got:

PW vs. master
GPU Model Microbatch size Test t/s master t/s a3b28f9 Speedup
RX 9060 XT gemma 2B Q4_0 1 pp1024 199.77 199.51 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d1024 195.02 194.96 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d2048 193.32 192.71 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d4096 185.70 185.87 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d8192 176.77 176.34 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d16384 159.14 158.58 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d32768 133.21 133.01 1.00
RX 9060 XT gemma 2B Q4_0 2 pp1024 251.97 244.16 0.97
RX 9060 XT gemma 2B Q4_0 2 pp1024@d1024 250.26 241.04 0.96
RX 9060 XT gemma 2B Q4_0 2 pp1024@d2048 247.19 235.81 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d4096 247.23 234.11 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d8192 239.84 227.26 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d16384 225.61 214.44 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d32768 201.53 191.32 0.95
RX 9060 XT gemma 2B Q4_0 4 pp1024 497.70 490.16 0.98
RX 9060 XT gemma 2B Q4_0 4 pp1024@d1024 480.05 473.93 0.99
RX 9060 XT gemma 2B Q4_0 4 pp1024@d2048 474.65 469.08 0.99
RX 9060 XT gemma 2B Q4_0 4 pp1024@d4096 462.00 455.97 0.99
RX 9060 XT gemma 2B Q4_0 4 pp1024@d8192 436.76 436.99 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d16384 398.36 406.74 1.02
RX 9060 XT gemma 2B Q4_0 4 pp1024@d32768 343.28 369.11 1.08
RX 9060 XT gemma 2B Q4_0 8 pp1024 813.96 807.04 0.99
RX 9060 XT gemma 2B Q4_0 8 pp1024@d1024 793.26 796.88 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d2048 776.17 789.05 1.02
RX 9060 XT gemma 2B Q4_0 8 pp1024@d4096 739.38 770.05 1.04
RX 9060 XT gemma 2B Q4_0 8 pp1024@d8192 677.58 745.12 1.10
RX 9060 XT gemma 2B Q4_0 8 pp1024@d16384 582.69 697.48 1.20
RX 9060 XT gemma 2B Q4_0 8 pp1024@d32768 447.92 618.61 1.38
RX 9060 XT gemma 2B Q4_0 16 pp1024 1534.08 1541.23 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024@d1024 1470.10 1514.17 1.03
RX 9060 XT gemma 2B Q4_0 16 pp1024@d2048 1404.48 1488.73 1.06
RX 9060 XT gemma 2B Q4_0 16 pp1024@d4096 1298.53 1431.61 1.10
RX 9060 XT gemma 2B Q4_0 16 pp1024@d8192 1113.28 1350.10 1.21
RX 9060 XT gemma 2B Q4_0 16 pp1024@d16384 882.45 1225.79 1.39
RX 9060 XT gemma 2B Q4_0 16 pp1024@d32768 608.97 1007.76 1.65
RX 9060 XT gemma 2B Q4_0 32 pp1024 2495.77 2598.26 1.04
RX 9060 XT gemma 2B Q4_0 32 pp1024@d1024 2308.08 2526.66 1.09
RX 9060 XT gemma 2B Q4_0 32 pp1024@d2048 2138.33 2463.82 1.15
RX 9060 XT gemma 2B Q4_0 32 pp1024@d4096 1891.24 2299.89 1.22
RX 9060 XT gemma 2B Q4_0 32 pp1024@d8192 1523.24 2092.87 1.37
RX 9060 XT gemma 2B Q4_0 32 pp1024@d16384 1099.01 1779.51 1.62
RX 9060 XT gemma 2B Q4_0 32 pp1024@d32768 708.51 1353.04 1.91
RX 9060 XT gemma 2B Q4_0 64 pp1024 3681.43 3768.68 1.02
RX 9060 XT gemma 2B Q4_0 64 pp1024@d1024 3286.00 3628.76 1.10
RX 9060 XT gemma 2B Q4_0 64 pp1024@d2048 2951.91 3462.08 1.17
RX 9060 XT gemma 2B Q4_0 64 pp1024@d4096 2493.95 3212.83 1.29
RX 9060 XT gemma 2B Q4_0 64 pp1024@d8192 1899.01 2844.27 1.50
RX 9060 XT gemma 2B Q4_0 64 pp1024@d16384 1291.77 2286.47 1.77
RX 9060 XT gemma 2B Q4_0 64 pp1024@d32768 784.82 1639.84 2.09
RX 9060 XT gemma 2B Q4_0 128 pp1024 4925.28 5037.19 1.02
RX 9060 XT gemma 2B Q4_0 128 pp1024@d1024 4321.66 4701.78 1.09
RX 9060 XT gemma 2B Q4_0 128 pp1024@d2048 3884.52 4465.72 1.15
RX 9060 XT gemma 2B Q4_0 128 pp1024@d4096 3231.15 4080.08 1.26
RX 9060 XT gemma 2B Q4_0 128 pp1024@d8192 2429.50 3465.77 1.43
RX 9060 XT gemma 2B Q4_0 128 pp1024@d16384 1637.36 2669.78 1.63
RX 9060 XT gemma 2B Q4_0 128 pp1024@d32768 966.82 1828.07 1.89
RX 9060 XT gemma 2B Q4_0 256 pp1024 7290.98 7670.59 1.05
RX 9060 XT gemma 2B Q4_0 256 pp1024@d1024 6013.49 7042.40 1.17
RX 9060 XT gemma 2B Q4_0 256 pp1024@d2048 5185.53 6697.28 1.29
RX 9060 XT gemma 2B Q4_0 256 pp1024@d4096 4099.34 6031.23 1.47
RX 9060 XT gemma 2B Q4_0 256 pp1024@d8192 2910.29 5102.77 1.75
RX 9060 XT gemma 2B Q4_0 256 pp1024@d16384 1862.82 3937.68 2.11
RX 9060 XT gemma 2B Q4_0 256 pp1024@d32768 1071.32 2686.63 2.51
RX 9060 XT gemma 2B Q4_0 512 pp1024 7538.15 8163.80 1.08
RX 9060 XT gemma 2B Q4_0 512 pp1024@d1024 6284.13 7601.69 1.21
RX 9060 XT gemma 2B Q4_0 512 pp1024@d2048 5439.49 7168.91 1.32
RX 9060 XT gemma 2B Q4_0 512 pp1024@d4096 4267.23 6480.79 1.52
RX 9060 XT gemma 2B Q4_0 512 pp1024@d8192 3024.31 5388.91 1.78
RX 9060 XT gemma 2B Q4_0 512 pp1024@d16384 1922.31 4049.56 2.11
RX 9060 XT gemma 2B Q4_0 512 pp1024@d32768 1099.73 2713.10 2.47
RX 9060 XT llama 1B Q4_0 1 pp1024 326.87 329.53 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d1024 313.94 315.89 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d2048 303.64 303.91 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d4096 285.31 287.63 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d8192 256.54 258.10 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d16384 211.00 211.25 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d32768 155.10 155.04 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024 411.48 412.22 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d1024 397.65 398.61 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d2048 388.50 389.11 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d4096 368.65 369.63 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d8192 333.23 333.75 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d16384 279.96 280.08 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d32768 210.77 210.57 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024 604.69 605.93 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d1024 590.53 594.39 1.01
RX 9060 XT llama 1B Q4_0 4 pp1024@d2048 582.25 583.94 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d4096 562.41 562.68 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d8192 526.75 527.05 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d16384 465.37 465.15 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d32768 376.68 376.38 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024 1208.72 1215.98 1.01
RX 9060 XT llama 1B Q4_0 8 pp1024@d1024 1183.24 1189.28 1.01
RX 9060 XT llama 1B Q4_0 8 pp1024@d2048 1158.59 1159.71 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d4096 1116.13 1117.16 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d8192 1040.70 1040.65 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d16384 915.93 915.79 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d32768 745.76 744.42 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024 2603.47 2595.57 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d1024 2517.27 2527.75 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d2048 2465.07 2466.36 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d4096 2375.96 2373.25 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d8192 2201.16 2201.90 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d16384 1911.19 1907.22 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d32768 1514.74 1512.98 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024 3843.24 3859.03 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d1024 3737.16 3779.37 1.01
RX 9060 XT llama 1B Q4_0 32 pp1024@d2048 3663.59 3667.18 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d4096 3525.29 3508.37 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d8192 3306.39 3245.74 0.98
RX 9060 XT llama 1B Q4_0 32 pp1024@d16384 2777.85 2768.55 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d32768 2121.97 2115.00 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024 5335.02 5354.61 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d1024 5219.23 5219.14 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d2048 4979.65 4979.69 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d4096 4681.70 4685.37 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d8192 4132.44 4159.06 1.01
RX 9060 XT llama 1B Q4_0 64 pp1024@d16384 3372.08 3365.11 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d32768 2413.12 2403.37 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024 8225.01 8084.94 0.98
RX 9060 XT llama 1B Q4_0 128 pp1024@d1024 7756.62 7739.91 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d2048 7189.73 7152.98 0.99
RX 9060 XT llama 1B Q4_0 128 pp1024@d4096 6542.08 6534.04 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d8192 5507.57 5518.92 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d16384 4186.53 4201.58 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d32768 2789.97 2783.58 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024 12965.20 13001.92 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d1024 11452.63 11462.87 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d2048 10539.07 10416.98 0.99
RX 9060 XT llama 1B Q4_0 256 pp1024@d4096 9192.08 9153.15 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d8192 7259.93 7211.04 0.99
RX 9060 XT llama 1B Q4_0 256 pp1024@d16384 5078.09 5086.47 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d32768 3162.75 3153.39 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024 12388.58 12692.93 1.02
RX 9060 XT llama 1B Q4_0 512 pp1024@d1024 11988.93 12100.80 1.01
RX 9060 XT llama 1B Q4_0 512 pp1024@d2048 11252.73 11171.83 0.99
RX 9060 XT llama 1B Q4_0 512 pp1024@d4096 9744.95 9590.98 0.98
RX 9060 XT llama 1B Q4_0 512 pp1024@d8192 7656.92 7280.01 0.95
RX 9060 XT llama 1B Q4_0 512 pp1024@d16384 5247.30 4949.73 0.94
RX 9060 XT llama 1B Q4_0 512 pp1024@d32768 3195.56 2993.52 0.94
RX 9060 XT llama 8B Q4_0 1 pp1024 68.21 68.24 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d1024 66.15 66.18 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d2048 64.34 64.37 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d4096 61.09 61.06 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d8192 55.44 55.45 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d16384 46.69 46.69 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d32768 35.56 35.56 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024 100.66 100.58 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d1024 98.28 98.23 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d2048 96.34 96.27 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d4096 92.76 92.62 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d8192 86.07 85.87 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d16384 74.97 75.00 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d32768 59.84 59.81 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024 182.37 182.12 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d1024 177.02 176.78 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d2048 173.13 173.07 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d4096 165.30 165.23 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d8192 151.11 151.10 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d16384 130.26 130.25 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d32768 100.95 100.85 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024 199.43 198.86 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d1024 196.56 196.26 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d2048 194.44 194.77 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d4096 190.47 190.18 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d8192 182.34 182.38 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d16384 169.30 169.17 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d32768 147.88 147.82 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024 660.73 661.43 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d1024 641.12 638.73 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d2048 624.75 624.71 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d4096 595.31 594.47 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d8192 542.94 542.96 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d16384 462.64 462.56 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d32768 357.99 357.78 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024 990.51 991.28 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d1024 953.80 948.60 0.99
RX 9060 XT llama 8B Q4_0 32 pp1024@d2048 921.67 920.75 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d4096 866.31 864.65 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d8192 769.47 769.72 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d16384 628.25 628.00 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d32768 464.45 461.98 0.99
RX 9060 XT llama 8B Q4_0 64 pp1024 1610.44 1604.77 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d1024 1507.52 1504.59 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d2048 1447.53 1444.09 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d4096 1320.89 1318.74 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d8192 1087.31 1087.43 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d16384 832.08 832.58 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d32768 569.23 566.67 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024 2359.64 2340.84 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d1024 2142.74 2142.86 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d2048 2005.35 2005.38 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d4096 1771.69 1775.94 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d8192 1374.73 1377.59 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d16384 1003.99 1000.71 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d32768 648.49 646.06 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024 2612.44 2595.07 0.99
RX 9060 XT llama 8B Q4_0 256 pp1024@d1024 2347.26 2372.22 1.01
RX 9060 XT llama 8B Q4_0 256 pp1024@d2048 2185.63 2206.63 1.01
RX 9060 XT llama 8B Q4_0 256 pp1024@d4096 1934.63 1967.04 1.02
RX 9060 XT llama 8B Q4_0 256 pp1024@d8192 1477.02 1607.39 1.09
RX 9060 XT llama 8B Q4_0 256 pp1024@d16384 1056.31 1186.13 1.12
RX 9060 XT llama 8B Q4_0 256 pp1024@d32768 674.72 785.23 1.16
RX 9060 XT llama 8B Q4_0 512 pp1024 2700.30 2701.59 1.00
RX 9060 XT llama 8B Q4_0 512 pp1024@d1024 2458.57 2490.01 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d2048 2287.08 2301.61 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d4096 2009.40 2031.68 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d8192 1502.39 1641.36 1.09
RX 9060 XT llama 8B Q4_0 512 pp1024@d16384 1056.29 1196.08 1.13
RX 9060 XT llama 8B Q4_0 512 pp1024@d32768 674.64 787.39 1.17
JG vs. master
GPU Model Microbatch size Test t/s master t/s 0ce2938 Speedup
RX 9060 XT gemma 2B Q4_0 1 pp1024 199.77 198.92 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d1024 195.02 195.75 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d2048 193.32 192.97 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d4096 185.70 185.69 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d8192 176.77 176.69 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d16384 159.14 158.96 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d32768 133.21 133.54 1.00
RX 9060 XT gemma 2B Q4_0 2 pp1024 251.97 245.62 0.97
RX 9060 XT gemma 2B Q4_0 2 pp1024@d1024 250.26 237.92 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d2048 247.19 231.55 0.94
RX 9060 XT gemma 2B Q4_0 2 pp1024@d4096 247.23 234.05 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d8192 239.84 227.32 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d16384 225.61 214.02 0.95
RX 9060 XT gemma 2B Q4_0 2 pp1024@d32768 201.53 191.30 0.95
RX 9060 XT gemma 2B Q4_0 4 pp1024 497.70 488.44 0.98
RX 9060 XT gemma 2B Q4_0 4 pp1024@d1024 480.05 473.32 0.99
RX 9060 XT gemma 2B Q4_0 4 pp1024@d2048 474.65 469.23 0.99
RX 9060 XT gemma 2B Q4_0 4 pp1024@d4096 462.00 455.94 0.99
RX 9060 XT gemma 2B Q4_0 4 pp1024@d8192 436.76 437.16 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d16384 398.36 405.73 1.02
RX 9060 XT gemma 2B Q4_0 4 pp1024@d32768 343.28 368.71 1.07
RX 9060 XT gemma 2B Q4_0 8 pp1024 813.96 804.03 0.99
RX 9060 XT gemma 2B Q4_0 8 pp1024@d1024 793.26 799.46 1.01
RX 9060 XT gemma 2B Q4_0 8 pp1024@d2048 776.17 788.50 1.02
RX 9060 XT gemma 2B Q4_0 8 pp1024@d4096 739.38 771.84 1.04
RX 9060 XT gemma 2B Q4_0 8 pp1024@d8192 677.58 742.04 1.10
RX 9060 XT gemma 2B Q4_0 8 pp1024@d16384 582.69 695.22 1.19
RX 9060 XT gemma 2B Q4_0 8 pp1024@d32768 447.92 621.36 1.39
RX 9060 XT gemma 2B Q4_0 16 pp1024 1534.08 1517.28 0.99
RX 9060 XT gemma 2B Q4_0 16 pp1024@d1024 1470.10 1513.59 1.03
RX 9060 XT gemma 2B Q4_0 16 pp1024@d2048 1404.48 1491.95 1.06
RX 9060 XT gemma 2B Q4_0 16 pp1024@d4096 1298.53 1435.81 1.11
RX 9060 XT gemma 2B Q4_0 16 pp1024@d8192 1113.28 1352.22 1.21
RX 9060 XT gemma 2B Q4_0 16 pp1024@d16384 882.45 1221.14 1.38
RX 9060 XT gemma 2B Q4_0 16 pp1024@d32768 608.97 1013.05 1.66
RX 9060 XT gemma 2B Q4_0 32 pp1024 2495.77 2556.41 1.02
RX 9060 XT gemma 2B Q4_0 32 pp1024@d1024 2308.08 2513.54 1.09
RX 9060 XT gemma 2B Q4_0 32 pp1024@d2048 2138.33 2447.43 1.14
RX 9060 XT gemma 2B Q4_0 32 pp1024@d4096 1891.24 2313.25 1.22
RX 9060 XT gemma 2B Q4_0 32 pp1024@d8192 1523.24 2094.07 1.37
RX 9060 XT gemma 2B Q4_0 32 pp1024@d16384 1099.01 1770.91 1.61
RX 9060 XT gemma 2B Q4_0 32 pp1024@d32768 708.51 1353.49 1.91
RX 9060 XT gemma 2B Q4_0 64 pp1024 3681.43 3722.82 1.01
RX 9060 XT gemma 2B Q4_0 64 pp1024@d1024 3286.00 3605.65 1.10
RX 9060 XT gemma 2B Q4_0 64 pp1024@d2048 2951.91 3477.13 1.18
RX 9060 XT gemma 2B Q4_0 64 pp1024@d4096 2493.95 3217.50 1.29
RX 9060 XT gemma 2B Q4_0 64 pp1024@d8192 1899.01 2837.30 1.49
RX 9060 XT gemma 2B Q4_0 64 pp1024@d16384 1291.77 2286.57 1.77
RX 9060 XT gemma 2B Q4_0 64 pp1024@d32768 784.82 1646.99 2.10
RX 9060 XT gemma 2B Q4_0 128 pp1024 4925.28 4979.60 1.01
RX 9060 XT gemma 2B Q4_0 128 pp1024@d1024 4321.66 4739.79 1.10
RX 9060 XT gemma 2B Q4_0 128 pp1024@d2048 3884.52 4467.60 1.15
RX 9060 XT gemma 2B Q4_0 128 pp1024@d4096 3231.15 4077.59 1.26
RX 9060 XT gemma 2B Q4_0 128 pp1024@d8192 2429.50 3451.47 1.42
RX 9060 XT gemma 2B Q4_0 128 pp1024@d16384 1637.36 2663.55 1.63
RX 9060 XT gemma 2B Q4_0 128 pp1024@d32768 966.82 1834.82 1.90
RX 9060 XT gemma 2B Q4_0 256 pp1024 7290.98 7674.55 1.05
RX 9060 XT gemma 2B Q4_0 256 pp1024@d1024 6013.49 7099.76 1.18
RX 9060 XT gemma 2B Q4_0 256 pp1024@d2048 5185.53 6663.30 1.28
RX 9060 XT gemma 2B Q4_0 256 pp1024@d4096 4099.34 6042.98 1.47
RX 9060 XT gemma 2B Q4_0 256 pp1024@d8192 2910.29 5091.22 1.75
RX 9060 XT gemma 2B Q4_0 256 pp1024@d16384 1862.82 3942.74 2.12
RX 9060 XT gemma 2B Q4_0 256 pp1024@d32768 1071.32 2708.68 2.53
RX 9060 XT gemma 2B Q4_0 512 pp1024 7538.15 8075.33 1.07
RX 9060 XT gemma 2B Q4_0 512 pp1024@d1024 6284.13 7581.18 1.21
RX 9060 XT gemma 2B Q4_0 512 pp1024@d2048 5439.49 7169.73 1.32
RX 9060 XT gemma 2B Q4_0 512 pp1024@d4096 4267.23 6458.02 1.51
RX 9060 XT gemma 2B Q4_0 512 pp1024@d8192 3024.31 5373.58 1.78
RX 9060 XT gemma 2B Q4_0 512 pp1024@d16384 1922.31 4017.62 2.09
RX 9060 XT gemma 2B Q4_0 512 pp1024@d32768 1099.73 2729.53 2.48
RX 9060 XT llama 1B Q4_0 1 pp1024 326.87 328.13 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d1024 313.94 316.89 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d2048 303.64 306.31 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d4096 285.31 285.68 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d8192 256.54 256.90 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d16384 211.00 211.71 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d32768 155.10 155.09 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024 411.48 413.35 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d1024 397.65 397.89 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d2048 388.50 389.31 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d4096 368.65 369.20 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d8192 333.23 334.24 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d16384 279.96 280.07 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d32768 210.77 210.79 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024 604.69 607.44 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d1024 590.53 592.13 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d2048 582.25 583.17 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d4096 562.41 563.12 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d8192 526.75 527.40 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d16384 465.37 465.15 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d32768 376.68 376.29 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024 1208.72 1215.55 1.01
RX 9060 XT llama 1B Q4_0 8 pp1024@d1024 1183.24 1185.52 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d2048 1158.59 1162.43 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d4096 1116.13 1116.34 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d8192 1040.70 1042.10 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d16384 915.93 915.60 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d32768 745.76 753.23 1.01
RX 9060 XT llama 1B Q4_0 16 pp1024 2603.47 2609.29 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d1024 2517.27 2522.66 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d2048 2465.07 2478.37 1.01
RX 9060 XT llama 1B Q4_0 16 pp1024@d4096 2375.96 2370.15 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d8192 2201.16 2200.86 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d16384 1911.19 1905.40 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d32768 1514.74 1511.23 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024 3843.24 3892.46 1.01
RX 9060 XT llama 1B Q4_0 32 pp1024@d1024 3737.16 3757.39 1.01
RX 9060 XT llama 1B Q4_0 32 pp1024@d2048 3663.59 3690.24 1.01
RX 9060 XT llama 1B Q4_0 32 pp1024@d4096 3525.29 3509.35 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d8192 3306.39 3241.91 0.98
RX 9060 XT llama 1B Q4_0 32 pp1024@d16384 2777.85 2773.07 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d32768 2121.97 2119.66 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024 5335.02 5431.01 1.02
RX 9060 XT llama 1B Q4_0 64 pp1024@d1024 5219.23 5179.40 0.99
RX 9060 XT llama 1B Q4_0 64 pp1024@d2048 4979.65 5029.23 1.01
RX 9060 XT llama 1B Q4_0 64 pp1024@d4096 4681.70 4680.36 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d8192 4132.44 4162.66 1.01
RX 9060 XT llama 1B Q4_0 64 pp1024@d16384 3372.08 3364.45 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d32768 2413.12 2416.83 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024 8225.01 8210.65 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d1024 7756.62 7654.44 0.99
RX 9060 XT llama 1B Q4_0 128 pp1024@d2048 7189.73 7166.29 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d4096 6542.08 6511.14 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d8192 5507.57 5528.07 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d16384 4186.53 4174.03 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d32768 2789.97 2798.74 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024 12965.20 13003.61 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d1024 11452.63 11463.06 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d2048 10539.07 10549.90 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d4096 9192.08 9144.06 0.99
RX 9060 XT llama 1B Q4_0 256 pp1024@d8192 7259.93 7274.18 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d16384 5078.09 5060.10 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d32768 3162.75 3156.67 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024 12388.58 12430.03 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024@d1024 11988.93 11833.29 0.99
RX 9060 XT llama 1B Q4_0 512 pp1024@d2048 11252.73 11289.88 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024@d4096 9744.95 9743.92 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024@d8192 7656.92 7671.60 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024@d16384 5247.30 5200.23 0.99
RX 9060 XT llama 1B Q4_0 512 pp1024@d32768 3195.56 3195.23 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024 68.21 68.24 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d1024 66.15 66.19 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d2048 64.34 64.41 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d4096 61.09 61.01 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d8192 55.44 55.53 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d16384 46.69 46.72 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d32768 35.56 35.56 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024 100.66 100.52 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d1024 98.28 98.20 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d2048 96.34 96.35 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d4096 92.76 92.67 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d8192 86.07 86.04 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d16384 74.97 75.06 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d32768 59.84 59.80 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024 182.37 181.86 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d1024 177.02 176.88 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d2048 173.13 172.84 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d4096 165.30 165.19 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d8192 151.11 151.20 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d16384 130.26 130.37 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d32768 100.95 100.74 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024 199.43 199.17 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d1024 196.56 196.15 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d2048 194.44 194.35 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d4096 190.47 189.79 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d8192 182.34 182.97 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d16384 169.30 169.50 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d32768 147.88 147.57 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024 660.73 660.13 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d1024 641.12 638.67 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d2048 624.75 624.18 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d4096 595.31 593.88 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d8192 542.94 543.14 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d16384 462.64 463.15 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d32768 357.99 357.17 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024 990.51 989.06 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d1024 953.80 949.24 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d2048 921.67 919.02 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d4096 866.31 863.64 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d8192 769.47 769.91 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d16384 628.25 628.19 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d32768 464.45 462.46 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024 1610.44 1609.12 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d1024 1507.52 1492.75 0.99
RX 9060 XT llama 8B Q4_0 64 pp1024@d2048 1447.53 1435.53 0.99
RX 9060 XT llama 8B Q4_0 64 pp1024@d4096 1320.89 1314.26 0.99
RX 9060 XT llama 8B Q4_0 64 pp1024@d8192 1087.31 1089.54 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d16384 832.08 831.96 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d32768 569.23 567.28 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024 2359.64 2355.97 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d1024 2142.74 2125.51 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d2048 2005.35 1992.39 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d4096 1771.69 1759.99 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d8192 1374.73 1380.62 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d16384 1003.99 1003.00 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d32768 648.49 646.66 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024 2612.44 2617.89 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024@d1024 2347.26 2361.80 1.01
RX 9060 XT llama 8B Q4_0 256 pp1024@d2048 2185.63 2204.24 1.01
RX 9060 XT llama 8B Q4_0 256 pp1024@d4096 1934.63 1953.74 1.01
RX 9060 XT llama 8B Q4_0 256 pp1024@d8192 1477.02 1614.86 1.09
RX 9060 XT llama 8B Q4_0 256 pp1024@d16384 1056.31 1187.05 1.12
RX 9060 XT llama 8B Q4_0 256 pp1024@d32768 674.72 780.75 1.16
RX 9060 XT llama 8B Q4_0 512 pp1024 2700.30 2722.46 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d1024 2458.57 2483.76 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d2048 2287.08 2308.23 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d4096 2009.40 2019.50 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d8192 1502.39 1651.06 1.10
RX 9060 XT llama 8B Q4_0 512 pp1024@d16384 1056.29 1197.76 1.13
RX 9060 XT llama 8B Q4_0 512 pp1024@d32768 674.64 783.05 1.16
JG vs. PW
GPU Model Microbatch size Test t/s a3b28f9 t/s 0ce2938 Speedup
RX 9060 XT gemma 2B Q4_0 1 pp1024 199.51 198.92 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d1024 194.96 195.75 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d2048 192.71 192.97 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d4096 185.87 185.69 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d8192 176.34 176.69 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d16384 158.58 158.96 1.00
RX 9060 XT gemma 2B Q4_0 1 pp1024@d32768 133.01 133.54 1.00
RX 9060 XT gemma 2B Q4_0 2 pp1024 244.16 245.62 1.01
RX 9060 XT gemma 2B Q4_0 2 pp1024@d1024 241.04 237.92 0.99
RX 9060 XT gemma 2B Q4_0 2 pp1024@d2048 235.81 231.55 0.98
RX 9060 XT gemma 2B Q4_0 2 pp1024@d4096 234.11 234.05 1.00
RX 9060 XT gemma 2B Q4_0 2 pp1024@d8192 227.26 227.32 1.00
RX 9060 XT gemma 2B Q4_0 2 pp1024@d16384 214.44 214.02 1.00
RX 9060 XT gemma 2B Q4_0 2 pp1024@d32768 191.32 191.30 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024 490.16 488.44 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d1024 473.93 473.32 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d2048 469.08 469.23 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d4096 455.97 455.94 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d8192 436.99 437.16 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d16384 406.74 405.73 1.00
RX 9060 XT gemma 2B Q4_0 4 pp1024@d32768 369.11 368.71 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024 807.04 804.03 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d1024 796.88 799.46 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d2048 789.05 788.50 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d4096 770.05 771.84 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d8192 745.12 742.04 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d16384 697.48 695.22 1.00
RX 9060 XT gemma 2B Q4_0 8 pp1024@d32768 618.61 621.36 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024 1541.23 1517.28 0.98
RX 9060 XT gemma 2B Q4_0 16 pp1024@d1024 1514.17 1513.59 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024@d2048 1488.73 1491.95 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024@d4096 1431.61 1435.81 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024@d8192 1350.10 1352.22 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024@d16384 1225.79 1221.14 1.00
RX 9060 XT gemma 2B Q4_0 16 pp1024@d32768 1007.76 1013.05 1.01
RX 9060 XT gemma 2B Q4_0 32 pp1024 2598.26 2556.41 0.98
RX 9060 XT gemma 2B Q4_0 32 pp1024@d1024 2526.66 2513.54 0.99
RX 9060 XT gemma 2B Q4_0 32 pp1024@d2048 2463.82 2447.43 0.99
RX 9060 XT gemma 2B Q4_0 32 pp1024@d4096 2299.89 2313.25 1.01
RX 9060 XT gemma 2B Q4_0 32 pp1024@d8192 2092.87 2094.07 1.00
RX 9060 XT gemma 2B Q4_0 32 pp1024@d16384 1779.51 1770.91 1.00
RX 9060 XT gemma 2B Q4_0 32 pp1024@d32768 1353.04 1353.49 1.00
RX 9060 XT gemma 2B Q4_0 64 pp1024 3768.68 3722.82 0.99
RX 9060 XT gemma 2B Q4_0 64 pp1024@d1024 3628.76 3605.65 0.99
RX 9060 XT gemma 2B Q4_0 64 pp1024@d2048 3462.08 3477.13 1.00
RX 9060 XT gemma 2B Q4_0 64 pp1024@d4096 3212.83 3217.50 1.00
RX 9060 XT gemma 2B Q4_0 64 pp1024@d8192 2844.27 2837.30 1.00
RX 9060 XT gemma 2B Q4_0 64 pp1024@d16384 2286.47 2286.57 1.00
RX 9060 XT gemma 2B Q4_0 64 pp1024@d32768 1639.84 1646.99 1.00
RX 9060 XT gemma 2B Q4_0 128 pp1024 5037.19 4979.60 0.99
RX 9060 XT gemma 2B Q4_0 128 pp1024@d1024 4701.78 4739.79 1.01
RX 9060 XT gemma 2B Q4_0 128 pp1024@d2048 4465.72 4467.60 1.00
RX 9060 XT gemma 2B Q4_0 128 pp1024@d4096 4080.08 4077.59 1.00
RX 9060 XT gemma 2B Q4_0 128 pp1024@d8192 3465.77 3451.47 1.00
RX 9060 XT gemma 2B Q4_0 128 pp1024@d16384 2669.78 2663.55 1.00
RX 9060 XT gemma 2B Q4_0 128 pp1024@d32768 1828.07 1834.82 1.00
RX 9060 XT gemma 2B Q4_0 256 pp1024 7670.59 7674.55 1.00
RX 9060 XT gemma 2B Q4_0 256 pp1024@d1024 7042.40 7099.76 1.01
RX 9060 XT gemma 2B Q4_0 256 pp1024@d2048 6697.28 6663.30 0.99
RX 9060 XT gemma 2B Q4_0 256 pp1024@d4096 6031.23 6042.98 1.00
RX 9060 XT gemma 2B Q4_0 256 pp1024@d8192 5102.77 5091.22 1.00
RX 9060 XT gemma 2B Q4_0 256 pp1024@d16384 3937.68 3942.74 1.00
RX 9060 XT gemma 2B Q4_0 256 pp1024@d32768 2686.63 2708.68 1.01
RX 9060 XT gemma 2B Q4_0 512 pp1024 8163.80 8075.33 0.99
RX 9060 XT gemma 2B Q4_0 512 pp1024@d1024 7601.69 7581.18 1.00
RX 9060 XT gemma 2B Q4_0 512 pp1024@d2048 7168.91 7169.73 1.00
RX 9060 XT gemma 2B Q4_0 512 pp1024@d4096 6480.79 6458.02 1.00
RX 9060 XT gemma 2B Q4_0 512 pp1024@d8192 5388.91 5373.58 1.00
RX 9060 XT gemma 2B Q4_0 512 pp1024@d16384 4049.56 4017.62 0.99
RX 9060 XT gemma 2B Q4_0 512 pp1024@d32768 2713.10 2729.53 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024 329.53 328.13 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d1024 315.89 316.89 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d2048 303.91 306.31 1.01
RX 9060 XT llama 1B Q4_0 1 pp1024@d4096 287.63 285.68 0.99
RX 9060 XT llama 1B Q4_0 1 pp1024@d8192 258.10 256.90 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d16384 211.25 211.71 1.00
RX 9060 XT llama 1B Q4_0 1 pp1024@d32768 155.04 155.09 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024 412.22 413.35 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d1024 398.61 397.89 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d2048 389.11 389.31 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d4096 369.63 369.20 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d8192 333.75 334.24 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d16384 280.08 280.07 1.00
RX 9060 XT llama 1B Q4_0 2 pp1024@d32768 210.57 210.79 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024 605.93 607.44 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d1024 594.39 592.13 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d2048 583.94 583.17 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d4096 562.68 563.12 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d8192 527.05 527.40 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d16384 465.15 465.15 1.00
RX 9060 XT llama 1B Q4_0 4 pp1024@d32768 376.38 376.29 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024 1215.98 1215.55 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d1024 1189.28 1185.52 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d2048 1159.71 1162.43 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d4096 1117.16 1116.34 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d8192 1040.65 1042.10 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d16384 915.79 915.60 1.00
RX 9060 XT llama 1B Q4_0 8 pp1024@d32768 744.42 753.23 1.01
RX 9060 XT llama 1B Q4_0 16 pp1024 2595.57 2609.29 1.01
RX 9060 XT llama 1B Q4_0 16 pp1024@d1024 2527.75 2522.66 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d2048 2466.36 2478.37 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d4096 2373.25 2370.15 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d8192 2201.90 2200.86 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d16384 1907.22 1905.40 1.00
RX 9060 XT llama 1B Q4_0 16 pp1024@d32768 1512.98 1511.23 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024 3859.03 3892.46 1.01
RX 9060 XT llama 1B Q4_0 32 pp1024@d1024 3779.37 3757.39 0.99
RX 9060 XT llama 1B Q4_0 32 pp1024@d2048 3667.18 3690.24 1.01
RX 9060 XT llama 1B Q4_0 32 pp1024@d4096 3508.37 3509.35 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d8192 3245.74 3241.91 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d16384 2768.55 2773.07 1.00
RX 9060 XT llama 1B Q4_0 32 pp1024@d32768 2115.00 2119.66 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024 5354.61 5431.01 1.01
RX 9060 XT llama 1B Q4_0 64 pp1024@d1024 5219.14 5179.40 0.99
RX 9060 XT llama 1B Q4_0 64 pp1024@d2048 4979.69 5029.23 1.01
RX 9060 XT llama 1B Q4_0 64 pp1024@d4096 4685.37 4680.36 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d8192 4159.06 4162.66 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d16384 3365.11 3364.45 1.00
RX 9060 XT llama 1B Q4_0 64 pp1024@d32768 2403.37 2416.83 1.01
RX 9060 XT llama 1B Q4_0 128 pp1024 8084.94 8210.65 1.02
RX 9060 XT llama 1B Q4_0 128 pp1024@d1024 7739.91 7654.44 0.99
RX 9060 XT llama 1B Q4_0 128 pp1024@d2048 7152.98 7166.29 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d4096 6534.04 6511.14 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d8192 5518.92 5528.07 1.00
RX 9060 XT llama 1B Q4_0 128 pp1024@d16384 4201.58 4174.03 0.99
RX 9060 XT llama 1B Q4_0 128 pp1024@d32768 2783.58 2798.74 1.01
RX 9060 XT llama 1B Q4_0 256 pp1024 13001.92 13003.61 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d1024 11462.87 11463.06 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d2048 10416.98 10549.90 1.01
RX 9060 XT llama 1B Q4_0 256 pp1024@d4096 9153.15 9144.06 1.00
RX 9060 XT llama 1B Q4_0 256 pp1024@d8192 7211.04 7274.18 1.01
RX 9060 XT llama 1B Q4_0 256 pp1024@d16384 5086.47 5060.10 0.99
RX 9060 XT llama 1B Q4_0 256 pp1024@d32768 3153.39 3156.67 1.00
RX 9060 XT llama 1B Q4_0 512 pp1024 12692.93 12430.03 0.98
RX 9060 XT llama 1B Q4_0 512 pp1024@d1024 12100.80 11833.29 0.98
RX 9060 XT llama 1B Q4_0 512 pp1024@d2048 11171.83 11289.88 1.01
RX 9060 XT llama 1B Q4_0 512 pp1024@d4096 9590.98 9743.92 1.02
RX 9060 XT llama 1B Q4_0 512 pp1024@d8192 7280.01 7671.60 1.05
RX 9060 XT llama 1B Q4_0 512 pp1024@d16384 4949.73 5200.23 1.05
RX 9060 XT llama 1B Q4_0 512 pp1024@d32768 2993.52 3195.23 1.07
RX 9060 XT llama 8B Q4_0 1 pp1024 68.24 68.24 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d1024 66.18 66.19 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d2048 64.37 64.41 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d4096 61.06 61.01 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d8192 55.45 55.53 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d16384 46.69 46.72 1.00
RX 9060 XT llama 8B Q4_0 1 pp1024@d32768 35.56 35.56 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024 100.58 100.52 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d1024 98.23 98.20 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d2048 96.27 96.35 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d4096 92.62 92.67 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d8192 85.87 86.04 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d16384 75.00 75.06 1.00
RX 9060 XT llama 8B Q4_0 2 pp1024@d32768 59.81 59.80 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024 182.12 181.86 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d1024 176.78 176.88 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d2048 173.07 172.84 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d4096 165.23 165.19 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d8192 151.10 151.20 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d16384 130.25 130.37 1.00
RX 9060 XT llama 8B Q4_0 4 pp1024@d32768 100.85 100.74 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024 198.86 199.17 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d1024 196.26 196.15 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d2048 194.77 194.35 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d4096 190.18 189.79 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d8192 182.38 182.97 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d16384 169.17 169.50 1.00
RX 9060 XT llama 8B Q4_0 8 pp1024@d32768 147.82 147.57 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024 661.43 660.13 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d1024 638.73 638.67 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d2048 624.71 624.18 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d4096 594.47 593.88 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d8192 542.96 543.14 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d16384 462.56 463.15 1.00
RX 9060 XT llama 8B Q4_0 16 pp1024@d32768 357.78 357.17 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024 991.28 989.06 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d1024 948.60 949.24 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d2048 920.75 919.02 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d4096 864.65 863.64 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d8192 769.72 769.91 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d16384 628.00 628.19 1.00
RX 9060 XT llama 8B Q4_0 32 pp1024@d32768 461.98 462.46 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024 1604.77 1609.12 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d1024 1504.59 1492.75 0.99
RX 9060 XT llama 8B Q4_0 64 pp1024@d2048 1444.09 1435.53 0.99
RX 9060 XT llama 8B Q4_0 64 pp1024@d4096 1318.74 1314.26 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d8192 1087.43 1089.54 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d16384 832.58 831.96 1.00
RX 9060 XT llama 8B Q4_0 64 pp1024@d32768 566.67 567.28 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024 2340.84 2355.97 1.01
RX 9060 XT llama 8B Q4_0 128 pp1024@d1024 2142.86 2125.51 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d2048 2005.38 1992.39 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d4096 1775.94 1759.99 0.99
RX 9060 XT llama 8B Q4_0 128 pp1024@d8192 1377.59 1380.62 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d16384 1000.71 1003.00 1.00
RX 9060 XT llama 8B Q4_0 128 pp1024@d32768 646.06 646.66 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024 2595.07 2617.89 1.01
RX 9060 XT llama 8B Q4_0 256 pp1024@d1024 2372.22 2361.80 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024@d2048 2206.63 2204.24 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024@d4096 1967.04 1953.74 0.99
RX 9060 XT llama 8B Q4_0 256 pp1024@d8192 1607.39 1614.86 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024@d16384 1186.13 1187.05 1.00
RX 9060 XT llama 8B Q4_0 256 pp1024@d32768 785.23 780.75 0.99
RX 9060 XT llama 8B Q4_0 512 pp1024 2701.59 2722.46 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d1024 2490.01 2483.76 1.00
RX 9060 XT llama 8B Q4_0 512 pp1024@d2048 2301.61 2308.23 1.00
RX 9060 XT llama 8B Q4_0 512 pp1024@d4096 2031.68 2019.50 0.99
RX 9060 XT llama 8B Q4_0 512 pp1024@d8192 1641.36 1651.06 1.01
RX 9060 XT llama 8B Q4_0 512 pp1024@d16384 1196.08 1197.76 1.00
RX 9060 XT llama 8B Q4_0 512 pp1024@d32768 787.39 783.05 0.99

@JohannesGaessler

JohannesGaessler commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Sorry, the columns were swapped in the "PW vs. master" table, this should be the correct data.

@pwilkin

pwilkin commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

@JohannesGaessler sorry, took a while because benchmarking on my trusty old machine is flaky and I had to rerun a few suspicious results, but here's the full report:

report.md
comparison.csv

TL;DR: checks out fine.

@gaperton

gaperton commented Sep 7, 2026

Copy link
Copy Markdown

Just tested on my dual R9700 in tensor mode, unsloth's Qwen2.8-27B Q5_K_M. Very good PR.

image

@JohannesGaessler

JohannesGaessler commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Regarding the host-side logic for how to handle a GQA ratio of 6: on my NVIDIA GPUs I found it to be faster to run the kernels with effectively a GQA ratio of 8 even though that results in some wasted work due to padding, on my RX 9060 XT using a GQA ratio of 2 (as is changed in this PR) seems to be ~5% faster. In principle there would be a breakeven point at a batch size of ~160 since a higher value in ncols2 is better for batch sizes that are not a power of 2. But I don't think it makes sense to complicate the logic for that.

I'll check the performance on my RDNA 3.5 machine as well and after that this PR would be good to merge from my end.

@JohannesGaessler

JohannesGaessler commented Sep 9, 2026

Copy link
Copy Markdown
Contributor
RDNA 3.5 Performance
GPU Model Microbatch size Test t/s master t/s 461f7c1 Speedup
8060S Graphics gemma 2B Q4_0 1 pp1024 119.02 119.02 1.00
8060S Graphics gemma 2B Q4_0 1 pp1024@d1024 117.93 117.27 0.99
8060S Graphics gemma 2B Q4_0 1 pp1024@d2048 116.05 115.41 0.99
8060S Graphics gemma 2B Q4_0 1 pp1024@d4096 112.51 113.33 1.01
8060S Graphics gemma 2B Q4_0 1 pp1024@d8192 106.42 106.65 1.00
8060S Graphics gemma 2B Q4_0 1 pp1024@d16384 96.09 95.97 1.00
8060S Graphics gemma 2B Q4_0 1 pp1024@d32768 80.68 81.02 1.00
8060S Graphics gemma 2B Q4_0 2 pp1024 226.35 227.52 1.01
8060S Graphics gemma 2B Q4_0 2 pp1024@d1024 223.22 221.74 0.99
8060S Graphics gemma 2B Q4_0 2 pp1024@d2048 221.59 220.34 0.99
8060S Graphics gemma 2B Q4_0 2 pp1024@d4096 215.87 215.87 1.00
8060S Graphics gemma 2B Q4_0 2 pp1024@d8192 208.14 208.61 1.00
8060S Graphics gemma 2B Q4_0 2 pp1024@d16384 194.56 195.03 1.00
8060S Graphics gemma 2B Q4_0 2 pp1024@d32768 170.24 171.47 1.01
8060S Graphics gemma 2B Q4_0 4 pp1024 409.98 395.22 0.96
8060S Graphics gemma 2B Q4_0 4 pp1024@d1024 394.14 372.77 0.95
8060S Graphics gemma 2B Q4_0 4 pp1024@d2048 388.17 367.56 0.95
8060S Graphics gemma 2B Q4_0 4 pp1024@d4096 375.91 352.85 0.94
8060S Graphics gemma 2B Q4_0 4 pp1024@d8192 354.53 330.93 0.93
8060S Graphics gemma 2B Q4_0 4 pp1024@d16384 317.02 289.73 0.91
8060S Graphics gemma 2B Q4_0 4 pp1024@d32768 264.50 235.24 0.89
8060S Graphics gemma 2B Q4_0 8 pp1024 641.71 639.31 1.00
8060S Graphics gemma 2B Q4_0 8 pp1024@d1024 622.00 609.71 0.98
8060S Graphics gemma 2B Q4_0 8 pp1024@d2048 611.67 594.90 0.97
8060S Graphics gemma 2B Q4_0 8 pp1024@d4096 580.93 583.58 1.00
8060S Graphics gemma 2B Q4_0 8 pp1024@d8192 530.71 546.74 1.03
8060S Graphics gemma 2B Q4_0 8 pp1024@d16384 452.23 490.91 1.09
8060S Graphics gemma 2B Q4_0 8 pp1024@d32768 347.73 407.68 1.17
8060S Graphics gemma 2B Q4_0 16 pp1024 1144.13 1125.11 0.98
8060S Graphics gemma 2B Q4_0 16 pp1024@d1024 1094.88 1080.64 0.99
8060S Graphics gemma 2B Q4_0 16 pp1024@d2048 1051.17 1059.25 1.01
8060S Graphics gemma 2B Q4_0 16 pp1024@d4096 966.85 1007.81 1.04
8060S Graphics gemma 2B Q4_0 16 pp1024@d8192 832.68 905.87 1.09
8060S Graphics gemma 2B Q4_0 16 pp1024@d16384 655.15 758.98 1.16
8060S Graphics gemma 2B Q4_0 16 pp1024@d32768 458.89 573.16 1.25
8060S Graphics gemma 2B Q4_0 32 pp1024 1331.18 1305.83 0.98
8060S Graphics gemma 2B Q4_0 32 pp1024@d1024 1277.67 1265.04 0.99
8060S Graphics gemma 2B Q4_0 32 pp1024@d2048 1246.98 1224.94 0.98
8060S Graphics gemma 2B Q4_0 32 pp1024@d4096 1169.92 1150.25 0.98
8060S Graphics gemma 2B Q4_0 32 pp1024@d8192 1043.33 1022.09 0.98
8060S Graphics gemma 2B Q4_0 32 pp1024@d16384 852.69 835.86 0.98
8060S Graphics gemma 2B Q4_0 32 pp1024@d32768 627.01 609.53 0.97
8060S Graphics gemma 2B Q4_0 64 pp1024 2340.70 2323.33 0.99
8060S Graphics gemma 2B Q4_0 64 pp1024@d1024 2228.10 2219.08 1.00
8060S Graphics gemma 2B Q4_0 64 pp1024@d2048 2100.93 2122.83 1.01
8060S Graphics gemma 2B Q4_0 64 pp1024@d4096 1896.58 1960.41 1.03
8060S Graphics gemma 2B Q4_0 64 pp1024@d8192 1574.59 1672.65 1.06
8060S Graphics gemma 2B Q4_0 64 pp1024@d16384 1182.76 1263.50 1.07
8060S Graphics gemma 2B Q4_0 64 pp1024@d32768 789.15 783.11 0.99
8060S Graphics gemma 2B Q4_0 128 pp1024 2870.74 2892.18 1.01
8060S Graphics gemma 2B Q4_0 128 pp1024@d1024 2650.06 2683.43 1.01
8060S Graphics gemma 2B Q4_0 128 pp1024@d2048 2473.87 2478.10 1.00
8060S Graphics gemma 2B Q4_0 128 pp1024@d4096 2142.72 2172.75 1.01
8060S Graphics gemma 2B Q4_0 128 pp1024@d8192 1704.03 1774.86 1.04
8060S Graphics gemma 2B Q4_0 128 pp1024@d16384 1215.34 1265.03 1.04
8060S Graphics gemma 2B Q4_0 128 pp1024@d32768 767.47 810.19 1.06
8060S Graphics gemma 2B Q4_0 256 pp1024 3424.75 3510.08 1.02
8060S Graphics gemma 2B Q4_0 256 pp1024@d1024 3156.26 3335.35 1.06
8060S Graphics gemma 2B Q4_0 256 pp1024@d2048 2911.42 3130.82 1.08
8060S Graphics gemma 2B Q4_0 256 pp1024@d4096 2597.56 2846.71 1.10
8060S Graphics gemma 2B Q4_0 256 pp1024@d8192 2009.91 2374.90 1.18
8060S Graphics gemma 2B Q4_0 256 pp1024@d16384 1428.37 1805.74 1.26
8060S Graphics gemma 2B Q4_0 256 pp1024@d32768 892.80 1214.46 1.36
8060S Graphics gemma 2B Q4_0 512 pp1024 3808.95 3942.98 1.04
8060S Graphics gemma 2B Q4_0 512 pp1024@d1024 3475.66 3666.67 1.05
8060S Graphics gemma 2B Q4_0 512 pp1024@d2048 3287.50 3534.92 1.08
8060S Graphics gemma 2B Q4_0 512 pp1024@d4096 2574.37 2936.77 1.14
8060S Graphics gemma 2B Q4_0 512 pp1024@d8192 2035.11 2470.69 1.21
8060S Graphics gemma 2B Q4_0 512 pp1024@d16384 1439.48 1847.20 1.28
8060S Graphics gemma 2B Q4_0 512 pp1024@d32768 893.60 1163.59 1.30
8060S Graphics llama 1B Q4_0 1 pp1024 193.40 194.75 1.01
8060S Graphics llama 1B Q4_0 1 pp1024@d1024 188.47 188.59 1.00
8060S Graphics llama 1B Q4_0 1 pp1024@d2048 183.82 183.64 1.00
8060S Graphics llama 1B Q4_0 1 pp1024@d4096 173.73 173.61 1.00
8060S Graphics llama 1B Q4_0 1 pp1024@d8192 156.53 156.54 1.00
8060S Graphics llama 1B Q4_0 1 pp1024@d16384 132.69 132.60 1.00
8060S Graphics llama 1B Q4_0 1 pp1024@d32768 101.08 101.02 1.00
8060S Graphics llama 1B Q4_0 2 pp1024 358.10 356.19 0.99
8060S Graphics llama 1B Q4_0 2 pp1024@d1024 343.99 343.73 1.00
8060S Graphics llama 1B Q4_0 2 pp1024@d2048 327.48 329.52 1.01
8060S Graphics llama 1B Q4_0 2 pp1024@d4096 307.99 306.17 0.99
8060S Graphics llama 1B Q4_0 2 pp1024@d8192 268.11 266.85 1.00
8060S Graphics llama 1B Q4_0 2 pp1024@d16384 213.62 214.16 1.00
8060S Graphics llama 1B Q4_0 2 pp1024@d32768 153.16 153.62 1.00
8060S Graphics llama 1B Q4_0 4 pp1024 667.26 663.99 1.00
8060S Graphics llama 1B Q4_0 4 pp1024@d1024 633.75 640.18 1.01
8060S Graphics llama 1B Q4_0 4 pp1024@d2048 613.88 620.26 1.01
8060S Graphics llama 1B Q4_0 4 pp1024@d4096 581.72 574.07 0.99
8060S Graphics llama 1B Q4_0 4 pp1024@d8192 517.42 516.81 1.00
8060S Graphics llama 1B Q4_0 4 pp1024@d16384 430.05 431.75 1.00
8060S Graphics llama 1B Q4_0 4 pp1024@d32768 325.12 324.24 1.00
8060S Graphics llama 1B Q4_0 8 pp1024 1078.45 1073.00 0.99
8060S Graphics llama 1B Q4_0 8 pp1024@d1024 1023.02 1025.14 1.00
8060S Graphics llama 1B Q4_0 8 pp1024@d2048 980.54 985.46 1.01
8060S Graphics llama 1B Q4_0 8 pp1024@d4096 925.56 925.91 1.00
8060S Graphics llama 1B Q4_0 8 pp1024@d8192 827.04 821.43 0.99
8060S Graphics llama 1B Q4_0 8 pp1024@d16384 673.76 681.49 1.01
8060S Graphics llama 1B Q4_0 8 pp1024@d32768 498.01 497.43 1.00
8060S Graphics llama 1B Q4_0 16 pp1024 1981.05 1968.74 0.99
8060S Graphics llama 1B Q4_0 16 pp1024@d1024 1867.53 1866.42 1.00
8060S Graphics llama 1B Q4_0 16 pp1024@d2048 1818.95 1822.77 1.00
8060S Graphics llama 1B Q4_0 16 pp1024@d4096 1732.13 1733.78 1.00
8060S Graphics llama 1B Q4_0 16 pp1024@d8192 1599.56 1601.75 1.00
8060S Graphics llama 1B Q4_0 16 pp1024@d16384 1398.82 1394.16 1.00
8060S Graphics llama 1B Q4_0 16 pp1024@d32768 1097.79 1097.75 1.00
8060S Graphics llama 1B Q4_0 32 pp1024 2322.18 2309.85 0.99
8060S Graphics llama 1B Q4_0 32 pp1024@d1024 2235.78 2253.10 1.01
8060S Graphics llama 1B Q4_0 32 pp1024@d2048 2179.05 2191.11 1.01
8060S Graphics llama 1B Q4_0 32 pp1024@d4096 2080.85 2072.66 1.00
8060S Graphics llama 1B Q4_0 32 pp1024@d8192 1880.01 1893.69 1.01
8060S Graphics llama 1B Q4_0 32 pp1024@d16384 1589.60 1590.34 1.00
8060S Graphics llama 1B Q4_0 32 pp1024@d32768 1212.17 1210.59 1.00
8060S Graphics llama 1B Q4_0 64 pp1024 4229.26 4190.73 0.99
8060S Graphics llama 1B Q4_0 64 pp1024@d1024 4003.54 4008.77 1.00
8060S Graphics llama 1B Q4_0 64 pp1024@d2048 3800.10 3804.44 1.00
8060S Graphics llama 1B Q4_0 64 pp1024@d4096 3473.67 3473.38 1.00
8060S Graphics llama 1B Q4_0 64 pp1024@d8192 2934.13 2955.77 1.01
8060S Graphics llama 1B Q4_0 64 pp1024@d16384 2237.10 2260.01 1.01
8060S Graphics llama 1B Q4_0 64 pp1024@d32768 1519.55 1528.91 1.01
8060S Graphics llama 1B Q4_0 128 pp1024 5277.08 5223.43 0.99
8060S Graphics llama 1B Q4_0 128 pp1024@d1024 4990.27 4988.03 1.00
8060S Graphics llama 1B Q4_0 128 pp1024@d2048 4695.26 4715.08 1.00
8060S Graphics llama 1B Q4_0 128 pp1024@d4096 4228.32 4248.79 1.00
8060S Graphics llama 1B Q4_0 128 pp1024@d8192 3538.73 3553.41 1.00
8060S Graphics llama 1B Q4_0 128 pp1024@d16384 2627.07 2635.89 1.00
8060S Graphics llama 1B Q4_0 128 pp1024@d32768 1680.59 1703.54 1.01
8060S Graphics llama 1B Q4_0 256 pp1024 6400.61 6370.30 1.00
8060S Graphics llama 1B Q4_0 256 pp1024@d1024 5990.35 5944.13 0.99
8060S Graphics llama 1B Q4_0 256 pp1024@d2048 5597.53 5609.61 1.00
8060S Graphics llama 1B Q4_0 256 pp1024@d4096 4958.80 4947.21 1.00
8060S Graphics llama 1B Q4_0 256 pp1024@d8192 3992.85 4025.24 1.01
8060S Graphics llama 1B Q4_0 256 pp1024@d16384 2869.42 2904.73 1.01
8060S Graphics llama 1B Q4_0 256 pp1024@d32768 1791.35 1830.30 1.02
8060S Graphics llama 1B Q4_0 512 pp1024 7401.56 7592.93 1.03
8060S Graphics llama 1B Q4_0 512 pp1024@d1024 6724.07 6685.62 0.99
8060S Graphics llama 1B Q4_0 512 pp1024@d2048 6381.38 6450.86 1.01
8060S Graphics llama 1B Q4_0 512 pp1024@d4096 5715.77 5699.20 1.00
8060S Graphics llama 1B Q4_0 512 pp1024@d8192 4628.50 4799.90 1.04
8060S Graphics llama 1B Q4_0 512 pp1024@d16384 3360.00 3390.73 1.01
8060S Graphics llama 1B Q4_0 512 pp1024@d32768 1963.22 1989.96 1.01
8060S Graphics llama 8B Q4_0 1 pp1024 42.90 42.98 1.00
8060S Graphics llama 8B Q4_0 1 pp1024@d1024 41.75 41.65 1.00
8060S Graphics llama 8B Q4_0 1 pp1024@d2048 40.95 41.09 1.00
8060S Graphics llama 8B Q4_0 1 pp1024@d4096 39.20 39.16 1.00
8060S Graphics llama 8B Q4_0 1 pp1024@d8192 36.45 36.52 1.00
8060S Graphics llama 8B Q4_0 1 pp1024@d16384 31.52 31.50 1.00
8060S Graphics llama 8B Q4_0 1 pp1024@d32768 24.50 24.54 1.00
8060S Graphics llama 8B Q4_0 2 pp1024 81.68 81.85 1.00
8060S Graphics llama 8B Q4_0 2 pp1024@d1024 79.47 79.40 1.00
8060S Graphics llama 8B Q4_0 2 pp1024@d2048 77.53 77.70 1.00
8060S Graphics llama 8B Q4_0 2 pp1024@d4096 74.64 74.68 1.00
8060S Graphics llama 8B Q4_0 2 pp1024@d8192 69.55 69.65 1.00
8060S Graphics llama 8B Q4_0 2 pp1024@d16384 60.88 61.00 1.00
8060S Graphics llama 8B Q4_0 2 pp1024@d32768 47.27 47.43 1.00
8060S Graphics llama 8B Q4_0 4 pp1024 146.94 147.34 1.00
8060S Graphics llama 8B Q4_0 4 pp1024@d1024 142.29 142.63 1.00
8060S Graphics llama 8B Q4_0 4 pp1024@d2048 138.41 139.07 1.00
8060S Graphics llama 8B Q4_0 4 pp1024@d4096 133.67 133.96 1.00
8060S Graphics llama 8B Q4_0 4 pp1024@d8192 124.02 124.05 1.00
8060S Graphics llama 8B Q4_0 4 pp1024@d16384 108.84 108.85 1.00
8060S Graphics llama 8B Q4_0 4 pp1024@d32768 87.87 87.78 1.00
8060S Graphics llama 8B Q4_0 8 pp1024 225.36 225.89 1.00
8060S Graphics llama 8B Q4_0 8 pp1024@d1024 217.63 217.61 1.00
8060S Graphics llama 8B Q4_0 8 pp1024@d2048 210.29 212.49 1.01
8060S Graphics llama 8B Q4_0 8 pp1024@d4096 202.89 203.32 1.00
8060S Graphics llama 8B Q4_0 8 pp1024@d8192 189.65 188.70 0.99
8060S Graphics llama 8B Q4_0 8 pp1024@d16384 167.36 167.12 1.00
8060S Graphics llama 8B Q4_0 8 pp1024@d32768 138.54 138.31 1.00
8060S Graphics llama 8B Q4_0 16 pp1024 437.88 439.36 1.00
8060S Graphics llama 8B Q4_0 16 pp1024@d1024 419.90 420.13 1.00
8060S Graphics llama 8B Q4_0 16 pp1024@d2048 402.68 405.30 1.01
8060S Graphics llama 8B Q4_0 16 pp1024@d4096 387.43 388.93 1.00
8060S Graphics llama 8B Q4_0 16 pp1024@d8192 356.16 355.17 1.00
8060S Graphics llama 8B Q4_0 16 pp1024@d16384 303.91 305.18 1.00
8060S Graphics llama 8B Q4_0 16 pp1024@d32768 236.46 236.75 1.00
8060S Graphics llama 8B Q4_0 32 pp1024 473.34 471.80 1.00
8060S Graphics llama 8B Q4_0 32 pp1024@d1024 459.40 460.16 1.00
8060S Graphics llama 8B Q4_0 32 pp1024@d2048 447.39 449.75 1.01
8060S Graphics llama 8B Q4_0 32 pp1024@d4096 429.32 430.16 1.00
8060S Graphics llama 8B Q4_0 32 pp1024@d8192 389.37 390.40 1.00
8060S Graphics llama 8B Q4_0 32 pp1024@d16384 332.13 331.37 1.00
8060S Graphics llama 8B Q4_0 32 pp1024@d32768 251.97 252.38 1.00
8060S Graphics llama 8B Q4_0 64 pp1024 890.09 891.50 1.00
8060S Graphics llama 8B Q4_0 64 pp1024@d1024 851.33 851.56 1.00
8060S Graphics llama 8B Q4_0 64 pp1024@d2048 809.99 815.01 1.01
8060S Graphics llama 8B Q4_0 64 pp1024@d4096 748.29 753.94 1.01
8060S Graphics llama 8B Q4_0 64 pp1024@d8192 646.54 641.11 0.99
8060S Graphics llama 8B Q4_0 64 pp1024@d16384 500.03 497.78 1.00
8060S Graphics llama 8B Q4_0 64 pp1024@d32768 338.96 340.94 1.01
8060S Graphics llama 8B Q4_0 128 pp1024 1030.66 1035.45 1.00
8060S Graphics llama 8B Q4_0 128 pp1024@d1024 964.82 982.25 1.02
8060S Graphics llama 8B Q4_0 128 pp1024@d2048 930.98 935.03 1.00
8060S Graphics llama 8B Q4_0 128 pp1024@d4096 836.77 842.16 1.01
8060S Graphics llama 8B Q4_0 128 pp1024@d8192 697.17 700.04 1.00
8060S Graphics llama 8B Q4_0 128 pp1024@d16384 509.26 513.37 1.01
8060S Graphics llama 8B Q4_0 128 pp1024@d32768 323.96 323.63 1.00
8060S Graphics llama 8B Q4_0 256 pp1024 1126.18 1134.52 1.01
8060S Graphics llama 8B Q4_0 256 pp1024@d1024 1056.25 1061.80 1.01
8060S Graphics llama 8B Q4_0 256 pp1024@d2048 995.65 989.59 0.99
8060S Graphics llama 8B Q4_0 256 pp1024@d4096 898.92 899.41 1.00
8060S Graphics llama 8B Q4_0 256 pp1024@d8192 740.96 732.69 0.99
8060S Graphics llama 8B Q4_0 256 pp1024@d16384 534.23 535.19 1.00
8060S Graphics llama 8B Q4_0 256 pp1024@d32768 355.09 353.73 1.00
8060S Graphics llama 8B Q4_0 512 pp1024 1226.96 1248.35 1.02
8060S Graphics llama 8B Q4_0 512 pp1024@d1024 1069.38 1103.08 1.03
8060S Graphics llama 8B Q4_0 512 pp1024@d2048 1010.36 1042.78 1.03
8060S Graphics llama 8B Q4_0 512 pp1024@d4096 922.79 965.66 1.05
8060S Graphics llama 8B Q4_0 512 pp1024@d8192 776.23 816.04 1.05
8060S Graphics llama 8B Q4_0 512 pp1024@d16384 579.45 631.56 1.09
8060S Graphics llama 8B Q4_0 512 pp1024@d32768 374.06 417.48 1.12
8060S Graphics qwen35 27B Q4_0 1 pp1024 12.78 12.77 1.00
8060S Graphics qwen35 27B Q4_0 1 pp1024@d1024 12.69 12.67 1.00
8060S Graphics qwen35 27B Q4_0 1 pp1024@d2048 12.60 12.59 1.00
8060S Graphics qwen35 27B Q4_0 1 pp1024@d4096 12.53 12.50 1.00
8060S Graphics qwen35 27B Q4_0 1 pp1024@d8192 12.31 12.34 1.00
8060S Graphics qwen35 27B Q4_0 1 pp1024@d16384 11.98 11.98 1.00
8060S Graphics qwen35 27B Q4_0 1 pp1024@d32768 11.35 11.35 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024 23.86 23.90 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024@d1024 23.63 23.72 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024@d2048 23.55 23.50 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024@d4096 23.19 23.20 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024@d8192 22.58 22.54 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024@d16384 21.34 21.43 1.00
8060S Graphics qwen35 27B Q4_0 2 pp1024@d32768 19.34 19.36 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024 44.17 44.16 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024@d1024 43.69 43.84 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024@d2048 43.63 43.48 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024@d4096 42.91 42.98 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024@d8192 41.72 41.68 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024@d16384 39.36 39.47 1.00
8060S Graphics qwen35 27B Q4_0 4 pp1024@d32768 35.48 35.46 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024 63.01 62.92 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024@d1024 62.44 62.46 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024@d2048 62.23 61.94 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024@d4096 61.33 61.30 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024@d8192 59.32 59.41 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024@d16384 55.89 56.04 1.00
8060S Graphics qwen35 27B Q4_0 8 pp1024@d32768 50.12 49.98 1.00
8060S Graphics qwen35 27B Q4_0 16 pp1024 130.33 130.02 1.00
8060S Graphics qwen35 27B Q4_0 16 pp1024@d1024 128.42 128.54 1.00
8060S Graphics qwen35 27B Q4_0 16 pp1024@d2048 127.19 127.24 1.00
8060S Graphics qwen35 27B Q4_0 16 pp1024@d4096 124.37 123.70 0.99
8060S Graphics qwen35 27B Q4_0 16 pp1024@d8192 118.45 114.77 0.97
8060S Graphics qwen35 27B Q4_0 16 pp1024@d16384 108.91 101.93 0.94
8060S Graphics qwen35 27B Q4_0 16 pp1024@d32768 93.01 83.54 0.90
8060S Graphics qwen35 27B Q4_0 32 pp1024 140.90 139.48 0.99
8060S Graphics qwen35 27B Q4_0 32 pp1024@d1024 139.80 138.44 0.99
8060S Graphics qwen35 27B Q4_0 32 pp1024@d2048 138.17 138.14 1.00
8060S Graphics qwen35 27B Q4_0 32 pp1024@d4096 135.11 134.99 1.00
8060S Graphics qwen35 27B Q4_0 32 pp1024@d8192 128.57 128.49 1.00
8060S Graphics qwen35 27B Q4_0 32 pp1024@d16384 118.65 119.31 1.01
8060S Graphics qwen35 27B Q4_0 32 pp1024@d32768 101.66 103.73 1.02
8060S Graphics qwen35 27B Q4_0 64 pp1024 245.32 245.74 1.00
8060S Graphics qwen35 27B Q4_0 64 pp1024@d1024 237.65 241.62 1.02
8060S Graphics qwen35 27B Q4_0 64 pp1024@d2048 235.81 238.10 1.01
8060S Graphics qwen35 27B Q4_0 64 pp1024@d4096 227.77 232.98 1.02
8060S Graphics qwen35 27B Q4_0 64 pp1024@d8192 210.60 216.11 1.03
8060S Graphics qwen35 27B Q4_0 64 pp1024@d16384 183.59 189.71 1.03
8060S Graphics qwen35 27B Q4_0 64 pp1024@d32768 145.67 154.05 1.06
8060S Graphics qwen35 27B Q4_0 128 pp1024 279.84 281.61 1.01
8060S Graphics qwen35 27B Q4_0 128 pp1024@d1024 271.88 274.70 1.01
8060S Graphics qwen35 27B Q4_0 128 pp1024@d2048 265.15 271.92 1.03
8060S Graphics qwen35 27B Q4_0 128 pp1024@d4096 255.05 263.96 1.03
8060S Graphics qwen35 27B Q4_0 128 pp1024@d8192 234.54 247.38 1.05
8060S Graphics qwen35 27B Q4_0 128 pp1024@d16384 199.59 221.20 1.11
8060S Graphics qwen35 27B Q4_0 128 pp1024@d32768 156.31 181.00 1.16
8060S Graphics qwen35 27B Q4_0 256 pp1024 301.28 304.21 1.01
8060S Graphics qwen35 27B Q4_0 256 pp1024@d1024 296.33 300.09 1.01
8060S Graphics qwen35 27B Q4_0 256 pp1024@d2048 289.92 295.82 1.02
8060S Graphics qwen35 27B Q4_0 256 pp1024@d4096 278.87 287.29 1.03
8060S Graphics qwen35 27B Q4_0 256 pp1024@d8192 257.09 272.80 1.06
8060S Graphics qwen35 27B Q4_0 256 pp1024@d16384 221.21 245.18 1.11
8060S Graphics qwen35 27B Q4_0 256 pp1024@d32768 172.05 204.67 1.19
8060S Graphics qwen35 27B Q4_0 512 pp1024 309.34 312.08 1.01
8060S Graphics qwen35 27B Q4_0 512 pp1024@d1024 296.76 299.47 1.01
8060S Graphics qwen35 27B Q4_0 512 pp1024@d2048 292.37 295.21 1.01
8060S Graphics qwen35 27B Q4_0 512 pp1024@d4096 281.53 293.08 1.04
8060S Graphics qwen35 27B Q4_0 512 pp1024@d8192 257.67 279.17 1.08
8060S Graphics qwen35 27B Q4_0 512 pp1024@d16384 219.64 253.55 1.15
8060S Graphics qwen35 27B Q4_0 512 pp1024@d32768 170.27 211.14 1.24

I checked RDNA 3.5 performance as well and adjusted the kernel selection logic accordingly.

@pwilkin

pwilkin commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

@IMbackK can you give a reapproval?

@IMbackK
IMbackK merged commit 16378d9 into ggml-org:master Sep 11, 2026
27 of 30 checks passed
@bernardladenthin bernardladenthin mentioned this pull request Sep 11, 2026
Merged
6 tasks done
Patt92 pushed a commit to Patt92/llama.cpp that referenced this pull request Sep 11, 2026
@pwilkin @JohannesGaessler Patrick Hoffmann
CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)
39fa672 
* HIP: enable mma FA for head size 256 on RDNA4, tune configs

Assisted-by: Claude
Assisted-by: Codex

* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

Assisted-by: Claude
Assisted-by: Codex

* revise stream_k logic

* revise kernel selection logic

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
(cherry picked from commit 16378d9)
Patt92 pushed a commit to Patt92/llama.cpp that referenced this pull request Sep 11, 2026
Patrick Hoffmann @claude Patrick Hoffmann
Revert "CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)"
3294199 
This reverts commit 39fa672 (upstream 16378d9) on gfx1151.

With it in the build (b11016), Qwen3.8-Flash-Next answers prompts that
interleave several images with text as pure garbage ("////...") and the
MTP draft is rejected every round (0 of 16379). Text-only prompts up to
42k tokens and a single-image prompt are fine on the same build, and the
identical prompt class was fine on b11008, which lacked this commit. The
commit moves 256-wide prefill attention from the tile kernel onto the AMD
WMMA kernel; the four other restored upstream commits stay in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@aki1-asr aki1-asr mentioned this pull request Sep 12, 2026
Closed
@briansp2020 briansp2020 mentioned this pull request Sep 13, 2026
Open
Patt92 pushed a commit to Patt92/llama.cpp that referenced this pull request Sep 14, 2026
Patrick Hoffmann @claude Patrick Hoffmann
Reapply "CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)"
7018bc9 
Undoes the b11018 revert of upstream 16378d9. The garbage output and
0 draft acceptance that got it reverted on 2026-09-11 turned out to be
the RPC worker's full disk serving truncated cache entries (fixed in
b11034); the FA tuning was innocent. With it back, the flash-attention
sources differ from upstream only by the tile-kernel row compaction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ggerganov ggerganov mentioned this pull request Sep 14, 2026
Merged
@BrewTestBot BrewTestBot mentioned this pull request Sep 14, 2026
Merged
1 task done
pl752 pushed a commit to pl752/llama.cpp that referenced this pull request Sep 15, 2026
@pwilkin @JohannesGaessler @pl752
CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)
cdcf55d 
* HIP: enable mma FA for head size 256 on RDNA4, tune configs

Assisted-by: Claude
Assisted-by: Codex

* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

Assisted-by: Claude
Assisted-by: Codex

* revise stream_k logic

* revise kernel selection logic

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
fencerJP pushed a commit to fencerJP/llama-apu that referenced this pull request Sep 16, 2026
@pwilkin @JohannesGaessler @fencerJP
CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)
88af709 
* HIP: enable mma FA for head size 256 on RDNA4, tune configs

Assisted-by: Claude
Assisted-by: Codex

* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

Assisted-by: Claude
Assisted-by: Codex

* revise stream_k logic

* revise kernel selection logic

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Pfu added a commit to Pfu/llama.cpp that referenced this pull request Sep 16, 2026
@Pfu
HIP: force stream-K on RDNA3 to avoid illegal memory access
ce5de03 
The whole-tile FA path (introduced in ggml-org#28102) triggers an asynchronous
ROCm illegal memory access on gfx1100 (RX 7900 XTX) with multi-GPU
tensor-split and large ubatch sizes. Force stream-K on RDNA3 to restore
the previously stable behaviour while keeping the RDNA4 optimisation.
quimmedes pushed a commit to quimmedes/cafe-llama.cpp that referenced this pull request Sep 16, 2026
@pwilkin @JohannesGaessler @quimmedes
CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)
d8a578c 
* HIP: enable mma FA for head size 256 on RDNA4, tune configs

Assisted-by: Claude
Assisted-by: Codex

* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

Assisted-by: Claude
Assisted-by: Codex

* revise stream_k logic

* revise kernel selection logic

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
zsogitbe pushed a commit to zsogitbe/llama.cpp that referenced this pull request Sep 17, 2026
@pwilkin @JohannesGaessler @zsogitbe
CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)
bb7b57b 
* HIP: enable mma FA for head size 256 on RDNA4, tune configs

Assisted-by: Claude
Assisted-by: Codex

* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

Assisted-by: Claude
Assisted-by: Codex

* revise stream_k logic

* revise kernel selection logic

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
turbo-tan pushed a commit to turbo-tan/llama.cpp-tq3 that referenced this pull request Sep 23, 2026
Hermes Sync
sync: restore TQ3/turbo FA vec helpers clobbered by c45d970 take-theirs
09d881d 
ggml-org#28102 replay dropped fork's vec_dot_fattn_vec_KQ_{tq3_0,turbo3_0,turbo4_0},
dequantize_V_{tq3_0,turbo3_0,turbo4_0}, turbo4_decode_element + getter arms
(static_assert 'bad type' on GB10 builds). Re-landed from main onto the
post-ggml-org#28102/ggml-org#29152 upstream shape (185 lines, no other deltas).
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Sep 26, 2026
@justinappler
strix-halo: sync onto 9f70b2c, drop J_max cap, guard D=256 WMMA on RD…
134077f 
…NA3.5

Upstream ggml-org#28552/ggml-org#28935 land the routed-MoE ncols picker on RDNA3.5, which
Finding ggml-org#10 measured flat against our static J=48 cap, so the cap goes.
Upstream ggml-org#28102 routes WMMA to MMA_F16 at D=256 and bypasses our tile
config at prefill; guard it off on RDNA3.5 pending the G/N A/B.

Assisted-by: Claude Opus 5.5
oshbec referenced this pull request in oshbec/llama.cpp Oct 5, 2026
@oshbec @claude
hip: RDNA4 WMMA flash attention for head_dim 256 (prefill)
179fe04 
Hypothesis: every prefill FA launch for head_dim 256 was the scalar
flash_attn_tile<256,256>, because fattn.cu gated the AMD WMMA (MMA_F16)
path on D <= 128. FA was 59% of prefill at 32K depth and ~85% at 128K.

Change (RDNA4 under HIP only; every other arch keeps the old expressions):
- Port of upstream ggml-org/llama.cpp #28102 (gfx1201 FA tuning): WMMA
  for D=256 once Q rows * gqa_ratio_eff > 16; ncols2 chosen to divide
  the GQA ratio at D=256 (GQA 6 -> 2); RDNA D=256 config rows; stream-k
  only when tile efficiency < 75% (D=64 always).
- Port of upstream #27870: divergent __syncthreads in the np>1 combine.
- Ours: D=256 ncols=64 config stages Q in LDS with 32-half2 K/V chunks
  (Q_in_reg=false): VGPR spills 424 -> 48, FA kernel 5.44 -> 4.33 ms at
  16K KV, 32.5 -> 27.7 ms at 64K.
- test-backend-ops: model-shape cases (D=256, 4 KV heads, GQA 6; KV
  512/4096/8192; nb 1..512 and 1024/1030 for -ub 1024; q8_0 KV).

Benchmark (RX 9070 XT, Bonsai 2 27B PQ2_0, llama-bench -r 2):
Before (deddbab): pp2048 @ 32K depth 518 t/s; pp512 1168; tg128 49.8
After:              pp2048 @ 32K depth 906 t/s (+75%); pp512, tg128 same
Earlier run (upstream config): @64k q8_0 340 -> 704, @128k 199 -> 498.
Agent edit workload (ngram-mod + MTP draft 3, ~33-97-token verify
batches, which now take the WMMA kernel): 307.2 -> 316.8 t/s, same
acceptance. VRAM +32 MiB.

Correctness: test-backend-ops FLASH_ATTN_EXT 2952/2952 (15/15 at the
model shape). PPL (wikitext-2, 20 chunks) 7.8773 -> 7.8695 (-0.10%),
identical at -ub 512 and -ub 1024; KLD vs previous 1.45e-4, same as the
model's own ub1-vs-ub512 noise (1.57e-4): fp reduction order in attention.
Decode (1 token) and MTP verify (<= 4 tokens) stay on the old kernels.

Decision: KEEP

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
@pwilkin @JohannesGaessler
CUDA/HIP: Flash Attention tuning (gfx1201) (ggml-org#28102)
7784fbd 
* HIP: enable mma FA for head size 256 on RDNA4, tune configs

Assisted-by: Claude
Assisted-by: Codex

* HIP: prefer whole-tile FA grids over stream-k on AMD WMMA

Assisted-by: Claude
Assisted-by: Codex

* revise stream_k logic

* revise kernel selection logic

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@IMbackK IMbackK IMbackK approved these changes
@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes
@ggerganov ggerganov Awaiting requested review from ggerganov ggerganov is a code owner

Assignees

@IMbackK IMbackK

@JohannesGaessler JohannesGaessler

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@pwilkin @Geramy @sbstnh @JohannesGaessler @IMbackK @eugene-su @Tudor-projs @gaperton