Skip to content

HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR - #27962

Merged
JohannesGaessler merged 2 commits into
ggml-org:masterfrom
yanjs:hip-swar
Sep 22, 2026
Merged

HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR#27962
JohannesGaessler merged 2 commits into
ggml-org:masterfrom
yanjs:hip-swar

Conversation

@yanjs

@yanjs yanjs commented Aug 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

__vsub4 and __vcmpne4 are CUDA intrinsic fallback implementations in HIP backends. This PR optimizes the efficiency of these implementations using SWAR. The wrong implementation of __vsub4 is improved.

These functions are used by IQ2/IQ3 paths only on HIP backends. These paths gain decent improvements (Qwen 3.8 27B IQ3_S tg128 +~20% on my device).

__vsub4: ~10% speed up. Less instructions. Fix the wrong implementation.
__vcmpne4: ~10% speed up. Less instructions. No dependency on VCC. See https://godbolt.org/z/Mf8Yv73Y1 for details.

test-backend-ops test passed.

Additional information

Tested on Arch Linux, RX 9070, gfx1201, ROCm 7.14, compared to base 77f132c .

(up: master;
mid: with only first commit from this PR, only __vcmpne4 optimization;
down: with both 2 commits from this pr, __vcmpne4 and __vsub4 optimization )

$ HIP_VISIBLE_DEVICES=0 /tmp/build-base2/bin/llama-bench -m Qwen3.8-27B-UD-IQ3_S.gguf
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           pp512 |       801.30 ± 20.27 |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           tg128 |         27.11 ± 0.04 |

build: 77f132c (50)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-mid/bin/llama-bench -m Qwen3.8-27B-UD-IQ3_S.gguf
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           pp512 |       823.09 ± 21.53 |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           tg128 |         29.90 ± 0.03 |

build: f5725ef (50)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-swar/bin/llama-bench -m Qwen3.8-27B-UD-IQ3_S.gguf
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           pp512 |       853.85 ± 23.18 |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           tg128 |         32.97 ± 0.06 |

build: a2731fb (51)

__vcmpeq4 is dead code. I did not change it as it is not benched. It can be removed or implemented using bitwise not of __vcmpne4 if required.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: AI assistance was used during investigation and analysis of this optimization.

@yanjs
yanjs requested a review from IMbackK as a code owner August 29, 2026 12:22
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 29, 2026
@yanjs

yanjs commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

(up: master, down: this pr)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-base2/bin/test-backend-ops perf -o MUL_MAT -p "type_a=iq3_s"
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
Testing 2 devices

Backend 1/2: ROCm0
  Device description: AMD Radeon RX 9070
  Device memory: 16304 MB (16222 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 10224 runs -   102.92 us/run - 117.44 MFLOP/run -   1.14 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  9372 runs -   107.91 us/run - 234.88 MFLOP/run -   2.18 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  9088 runs -   112.71 us/run - 352.32 MFLOP/run -   3.13 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  8307 runs -   120.44 us/run - 469.76 MFLOP/run -   3.90 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  7866 runs -   127.91 us/run - 587.20 MFLOP/run -   4.59 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  6099 runs -   165.61 us/run - 939.52 MFLOP/run -   5.67 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 706 runs -  1417.98 us/run -  60.13 GFLOP/run -  42.40 TFLOPS
  Backend ROCm0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

$ HIP_VISIBLE_DEVICES=0 /tmp/build-swar/bin/test-backend-ops perf -o MUL_MAT -p "type_a=iq3_s"
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
Testing 2 devices

Backend 1/2: ROCm0
  Device description: AMD Radeon RX 9070
  Device memory: 16304 MB (16222 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 15336 runs -    65.32 us/run - 117.44 MFLOP/run -   1.80 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 13206 runs -    75.92 us/run - 234.88 MFLOP/run -   3.09 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 12212 runs -    82.92 us/run - 352.32 MFLOP/run -   4.25 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 10437 runs -    96.21 us/run - 469.76 MFLOP/run -   4.88 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  9234 runs -   109.23 us/run - 587.20 MFLOP/run -   5.38 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  6848 runs -   146.60 us/run - 939.52 MFLOP/run -   6.41 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 828 runs -  1209.49 us/run -  60.13 GFLOP/run -  49.71 TFLOPS
  Backend ROCm0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

@IMbackK IMbackK self-assigned this Aug 29, 2026
@IMbackK

IMbackK commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

__vsubss4 saturating dosent matter where its used so that was not really a bug.
but yeah this looks correct to me and compiles down to way fewer instructions than using __builtin_elementwise_sub_sat

@IMbackK

IMbackK commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

@ggml-org/ggml-cuda could i have a second approval?

JohannesGaessler
JohannesGaessler reviewed Sep 22, 2026
View reviewed changes

@JohannesGaessler JohannesGaessler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bitwise logic seems correct to me. Let me quickly check the performance.

JohannesGaessler
JohannesGaessler approved these changes Sep 22, 2026
View reviewed changes

@JohannesGaessler JohannesGaessler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance

Table generated with scripts/compare-llama-bench.py

GPU Model Microbatch size Test t/s b10680 t/s a2731fb Speedup
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 1 pp1024 61.52 83.68 1.36
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 2 pp1024 105.72 123.32 1.17
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 4 pp1024 167.99 198.73 1.18
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 8 pp1024 236.40 271.64 1.15
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 16 pp1024 201.77 265.58 1.32
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 32 pp1024 334.86 313.72 0.94
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 64 pp1024 427.77 451.29 1.05
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 128 pp1024 473.19 499.69 1.06
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 256 pp1024 572.44 604.15 1.06
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 512 pp1024 648.03 682.64 1.05
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 1 pp1024 63.48 87.98 1.39
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 2 pp1024 112.70 131.89 1.17
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 4 pp1024 178.83 209.22 1.17
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 8 pp1024 244.84 284.73 1.16
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 16 pp1024 202.56 269.13 1.33
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 32 pp1024 330.78 310.34 0.94
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 64 pp1024 404.11 426.79 1.06
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 128 pp1024 440.84 464.19 1.05
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 256 pp1024 532.95 561.11 1.05
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 512 pp1024 602.57 633.49 1.05
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 1 pp1024 54.09 65.47 1.21
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 2 pp1024 93.82 104.24 1.11
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 4 pp1024 143.82 171.23 1.19
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 8 pp1024 222.99 248.56 1.11
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 16 pp1024 204.36 247.41 1.21
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 32 pp1024 336.17 325.66 0.97
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 64 pp1024 443.20 459.08 1.04
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 128 pp1024 490.47 507.20 1.03
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 256 pp1024 593.54 613.91 1.03
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 512 pp1024 674.07 698.39 1.04
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 1 pp1024 56.89 67.84 1.19
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 2 pp1024 95.47 104.80 1.10
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 4 pp1024 141.23 163.81 1.16
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 8 pp1024 205.37 224.70 1.09
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 16 pp1024 217.60 256.14 1.18
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 32 pp1024 345.02 334.51 0.97
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 64 pp1024 431.13 443.30 1.03
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 128 pp1024 505.54 520.37 1.03
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 256 pp1024 604.42 622.42 1.03
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 512 pp1024 692.42 713.79 1.03
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 1 pp1024 55.09 68.85 1.25
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 2 pp1024 96.37 108.45 1.13
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 4 pp1024 153.59 177.96 1.16
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 8 pp1024 234.83 253.31 1.08
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 16 pp1024 202.91 260.63 1.28
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 32 pp1024 322.54 326.84 1.01
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 64 pp1024 441.79 462.10 1.05
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 128 pp1024 490.62 512.91 1.05
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 256 pp1024 594.24 622.10 1.05
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 512 pp1024 675.91 707.66 1.05
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 1 pp1024 57.17 73.14 1.28
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 2 pp1024 99.22 112.02 1.13
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 4 pp1024 159.25 184.24 1.16
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 8 pp1024 241.02 259.25 1.08
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 16 pp1024 198.82 271.07 1.36
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 32 pp1024 314.70 320.13 1.02
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 64 pp1024 438.22 459.85 1.05
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 128 pp1024 489.03 514.14 1.05
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 256 pp1024 592.77 624.33 1.05
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 512 pp1024 673.61 709.35 1.05
MI100 llama 8B IQ2_S - 2.5 bpw 1 pp1024 88.98 117.48 1.32
MI100 llama 8B IQ2_S - 2.5 bpw 2 pp1024 149.88 176.63 1.18
MI100 llama 8B IQ2_S - 2.5 bpw 4 pp1024 231.64 271.80 1.17
MI100 llama 8B IQ2_S - 2.5 bpw 8 pp1024 342.05 382.54 1.12
MI100 llama 8B IQ2_S - 2.5 bpw 16 pp1024 634.05 798.22 1.26
MI100 llama 8B IQ2_S - 2.5 bpw 32 pp1024 995.94 1229.96 1.23
MI100 llama 8B IQ2_S - 2.5 bpw 64 pp1024 1408.95 1627.90 1.16
MI100 llama 8B IQ2_S - 2.5 bpw 128 pp1024 1631.07 1924.17 1.18
MI100 llama 8B IQ2_S - 2.5 bpw 256 pp1024 2398.14 2414.07 1.01
MI100 llama 8B IQ2_S - 2.5 bpw 512 pp1024 3162.55 3176.09 1.00
MI100 llama 8B IQ2_XS - 2.3125 bpw 1 pp1024 93.01 124.64 1.34
MI100 llama 8B IQ2_XS - 2.3125 bpw 2 pp1024 155.74 186.20 1.20
MI100 llama 8B IQ2_XS - 2.3125 bpw 4 pp1024 244.61 282.84 1.16
MI100 llama 8B IQ2_XS - 2.3125 bpw 8 pp1024 360.95 398.44 1.10
MI100 llama 8B IQ2_XS - 2.3125 bpw 16 pp1024 654.33 830.20 1.27
MI100 llama 8B IQ2_XS - 2.3125 bpw 32 pp1024 1024.13 1263.09 1.23
MI100 llama 8B IQ2_XS - 2.3125 bpw 64 pp1024 1437.16 1646.88 1.15
MI100 llama 8B IQ2_XS - 2.3125 bpw 128 pp1024 1657.85 1942.29 1.17
MI100 llama 8B IQ2_XS - 2.3125 bpw 256 pp1024 2459.69 2473.08 1.01
MI100 llama 8B IQ2_XS - 2.3125 bpw 512 pp1024 3267.17 3290.94 1.01
MI100 llama 8B IQ3_S - 3.4375 bpw 1 pp1024 77.30 94.45 1.22
MI100 llama 8B IQ3_S - 3.4375 bpw 2 pp1024 134.78 146.45 1.09
MI100 llama 8B IQ3_S - 3.4375 bpw 4 pp1024 201.50 241.02 1.20
MI100 llama 8B IQ3_S - 3.4375 bpw 8 pp1024 324.97 376.36 1.16
MI100 llama 8B IQ3_S - 3.4375 bpw 16 pp1024 615.74 706.50 1.15
MI100 llama 8B IQ3_S - 3.4375 bpw 32 pp1024 1007.37 1134.93 1.13
MI100 llama 8B IQ3_S - 3.4375 bpw 64 pp1024 1458.61 1644.30 1.13
MI100 llama 8B IQ3_S - 3.4375 bpw 128 pp1024 1726.88 1994.81 1.16
MI100 llama 8B IQ3_S - 3.4375 bpw 256 pp1024 2294.93 2318.58 1.01
MI100 llama 8B IQ3_S - 3.4375 bpw 512 pp1024 3059.85 3066.22 1.00
MI100 llama 8B IQ3_S mix - 3.66 bpw 1 pp1024 79.27 94.18 1.19
MI100 llama 8B IQ3_S mix - 3.66 bpw 2 pp1024 134.65 147.66 1.10
MI100 llama 8B IQ3_S mix - 3.66 bpw 4 pp1024 202.66 234.52 1.16
MI100 llama 8B IQ3_S mix - 3.66 bpw 8 pp1024 328.91 374.56 1.14
MI100 llama 8B IQ3_S mix - 3.66 bpw 16 pp1024 629.96 701.01 1.11
MI100 llama 8B IQ3_S mix - 3.66 bpw 32 pp1024 1015.55 1132.44 1.12
MI100 llama 8B IQ3_S mix - 3.66 bpw 64 pp1024 1449.04 1593.41 1.10
MI100 llama 8B IQ3_S mix - 3.66 bpw 128 pp1024 1718.84 1954.59 1.14
MI100 llama 8B IQ3_S mix - 3.66 bpw 256 pp1024 2267.77 2317.24 1.02
MI100 llama 8B IQ3_S mix - 3.66 bpw 512 pp1024 3046.13 3057.91 1.00
MI100 llama 8B IQ3_XS - 3.3 bpw 1 pp1024 78.97 98.45 1.25
MI100 llama 8B IQ3_XS - 3.3 bpw 2 pp1024 137.89 154.37 1.12
MI100 llama 8B IQ3_XS - 3.3 bpw 4 pp1024 217.36 251.09 1.16
MI100 llama 8B IQ3_XS - 3.3 bpw 8 pp1024 339.92 381.09 1.12
MI100 llama 8B IQ3_XS - 3.3 bpw 16 pp1024 616.08 745.94 1.21
MI100 llama 8B IQ3_XS - 3.3 bpw 32 pp1024 1007.08 1184.72 1.18
MI100 llama 8B IQ3_XS - 3.3 bpw 64 pp1024 1451.46 1685.45 1.16
MI100 llama 8B IQ3_XS - 3.3 bpw 128 pp1024 1748.81 2039.64 1.17
MI100 llama 8B IQ3_XS - 3.3 bpw 256 pp1024 2319.07 2329.13 1.00
MI100 llama 8B IQ3_XS - 3.3 bpw 512 pp1024 3062.64 3076.23 1.00
MI100 llama 8B IQ3_XXS - 3.0625 bpw 1 pp1024 82.36 105.03 1.28
MI100 llama 8B IQ3_XXS - 3.0625 bpw 2 pp1024 139.60 160.11 1.15
MI100 llama 8B IQ3_XXS - 3.0625 bpw 4 pp1024 221.57 255.14 1.15
MI100 llama 8B IQ3_XXS - 3.0625 bpw 8 pp1024 340.83 359.17 1.05
MI100 llama 8B IQ3_XXS - 3.0625 bpw 16 pp1024 627.26 762.54 1.22
MI100 llama 8B IQ3_XXS - 3.0625 bpw 32 pp1024 1021.60 1212.19 1.19
MI100 llama 8B IQ3_XXS - 3.0625 bpw 64 pp1024 1493.84 1701.85 1.14
MI100 llama 8B IQ3_XXS - 3.0625 bpw 128 pp1024 1782.27 2077.39 1.17
MI100 llama 8B IQ3_XXS - 3.0625 bpw 256 pp1024 2330.85 2364.10 1.01
MI100 llama 8B IQ3_XXS - 3.0625 bpw 512 pp1024 3081.04 3131.55 1.02
RX 6800 llama 8B IQ2_S - 2.5 bpw 1 pp1024 48.93 67.05 1.37
RX 6800 llama 8B IQ2_S - 2.5 bpw 2 pp1024 85.49 110.47 1.29
RX 6800 llama 8B IQ2_S - 2.5 bpw 4 pp1024 137.12 144.74 1.06
RX 6800 llama 8B IQ2_S - 2.5 bpw 8 pp1024 171.86 179.25 1.04
RX 6800 llama 8B IQ2_S - 2.5 bpw 16 pp1024 230.23 254.57 1.11
RX 6800 llama 8B IQ2_S - 2.5 bpw 32 pp1024 353.37 385.68 1.09
RX 6800 llama 8B IQ2_S - 2.5 bpw 64 pp1024 464.99 491.48 1.06
RX 6800 llama 8B IQ2_S - 2.5 bpw 128 pp1024 590.48 626.08 1.06
RX 6800 llama 8B IQ2_S - 2.5 bpw 256 pp1024 693.94 736.35 1.06
RX 6800 llama 8B IQ2_S - 2.5 bpw 512 pp1024 737.82 782.74 1.06
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 1 pp1024 49.67 67.84 1.37
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 2 pp1024 86.13 110.97 1.29
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 4 pp1024 136.64 144.12 1.05
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 8 pp1024 170.10 177.05 1.04
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 16 pp1024 231.02 254.79 1.10
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 32 pp1024 348.83 379.25 1.09
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 64 pp1024 453.62 476.46 1.05
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 128 pp1024 572.33 604.41 1.06
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 256 pp1024 673.95 712.66 1.06
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 512 pp1024 721.72 762.47 1.06
RX 6800 llama 8B IQ3_S - 3.4375 bpw 1 pp1024 48.16 63.06 1.31
RX 6800 llama 8B IQ3_S - 3.4375 bpw 2 pp1024 85.97 104.66 1.22
RX 6800 llama 8B IQ3_S - 3.4375 bpw 4 pp1024 137.61 140.88 1.02
RX 6800 llama 8B IQ3_S - 3.4375 bpw 8 pp1024 170.94 179.10 1.05
RX 6800 llama 8B IQ3_S - 3.4375 bpw 16 pp1024 234.40 265.13 1.13
RX 6800 llama 8B IQ3_S - 3.4375 bpw 32 pp1024 369.67 399.83 1.08
RX 6800 llama 8B IQ3_S - 3.4375 bpw 64 pp1024 499.54 521.25 1.04
RX 6800 llama 8B IQ3_S - 3.4375 bpw 128 pp1024 631.94 664.91 1.05
RX 6800 llama 8B IQ3_S - 3.4375 bpw 256 pp1024 747.70 787.23 1.05
RX 6800 llama 8B IQ3_S - 3.4375 bpw 512 pp1024 797.04 837.28 1.05
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 1 pp1024 48.84 62.19 1.27
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 2 pp1024 86.15 102.41 1.19
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 4 pp1024 133.30 135.05 1.01
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 8 pp1024 160.68 166.40 1.04
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 16 pp1024 239.04 266.28 1.11
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 32 pp1024 368.55 393.90 1.07
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 64 pp1024 484.06 501.09 1.04
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 128 pp1024 610.24 636.26 1.04
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 256 pp1024 725.56 757.98 1.04
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 512 pp1024 776.94 811.01 1.04
RX 6800 llama 8B IQ3_XS - 3.3 bpw 1 pp1024 48.91 65.58 1.34
RX 6800 llama 8B IQ3_XS - 3.3 bpw 2 pp1024 86.06 107.06 1.24
RX 6800 llama 8B IQ3_XS - 3.3 bpw 4 pp1024 139.44 151.58 1.09
RX 6800 llama 8B IQ3_XS - 3.3 bpw 8 pp1024 180.55 188.98 1.05
RX 6800 llama 8B IQ3_XS - 3.3 bpw 16 pp1024 231.50 261.84 1.13
RX 6800 llama 8B IQ3_XS - 3.3 bpw 32 pp1024 367.39 394.99 1.08
RX 6800 llama 8B IQ3_XS - 3.3 bpw 64 pp1024 496.00 520.13 1.05
RX 6800 llama 8B IQ3_XS - 3.3 bpw 128 pp1024 630.03 665.17 1.06
RX 6800 llama 8B IQ3_XS - 3.3 bpw 256 pp1024 747.85 788.71 1.05
RX 6800 llama 8B IQ3_XS - 3.3 bpw 512 pp1024 797.81 840.01 1.05
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 1 pp1024 49.38 67.24 1.36
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 2 pp1024 85.56 108.87 1.27
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 4 pp1024 142.87 156.72 1.10
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 8 pp1024 186.64 194.79 1.04
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 16 pp1024 231.80 264.99 1.14
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 32 pp1024 365.84 396.61 1.08
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 64 pp1024 490.70 520.09 1.06
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 128 pp1024 626.61 666.53 1.06
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 256 pp1024 742.06 788.49 1.06
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 512 pp1024 792.89 841.68 1.06
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 1 pp1024 57.71 73.88 1.28
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 2 pp1024 100.84 123.64 1.23
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 4 pp1024 164.09 174.38 1.06
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 8 pp1024 230.03 226.86 0.99
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 16 pp1024 471.36 549.37 1.17
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 32 pp1024 765.63 823.16 1.08
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 64 pp1024 1245.91 1332.95 1.07
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 128 pp1024 1792.78 1884.58 1.05
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 256 pp1024 1964.67 2071.00 1.05
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 512 pp1024 2000.51 2113.36 1.06
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 1 pp1024 59.00 78.02 1.32
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 2 pp1024 102.03 127.34 1.25
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 4 pp1024 164.05 178.46 1.09
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 8 pp1024 227.36 221.68 0.97
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 16 pp1024 466.48 539.37 1.16
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 32 pp1024 744.95 800.94 1.08
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 64 pp1024 1180.90 1233.11 1.04
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 128 pp1024 1652.87 1744.25 1.06
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 256 pp1024 1828.77 1913.92 1.05
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 512 pp1024 1929.56 2022.16 1.05
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 1 pp1024 54.15 64.73 1.20
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 2 pp1024 95.67 109.09 1.14
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 4 pp1024 165.13 163.11 0.99
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 8 pp1024 232.31 228.41 0.98
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 16 pp1024 551.01 641.20 1.16
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 32 pp1024 827.84 924.79 1.12
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 64 pp1024 1389.02 1532.65 1.10
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 128 pp1024 2079.04 2255.14 1.08
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 256 pp1024 2281.72 2480.79 1.09
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 512 pp1024 2348.56 2554.61 1.09
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 1 pp1024 54.98 65.87 1.20
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 2 pp1024 95.03 106.04 1.12
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 4 pp1024 161.39 160.59 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 8 pp1024 219.06 215.80 0.99
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 16 pp1024 554.71 632.66 1.14
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 32 pp1024 841.37 929.12 1.10
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 64 pp1024 1414.69 1536.92 1.09
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 128 pp1024 2092.20 2238.36 1.07
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 256 pp1024 2292.69 2475.76 1.08
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 512 pp1024 2360.54 2551.36 1.08
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 1 pp1024 54.27 65.85 1.21
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 2 pp1024 95.61 110.60 1.16
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 4 pp1024 162.82 164.20 1.01
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 8 pp1024 221.37 226.98 1.03
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 16 pp1024 556.24 647.79 1.16
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 32 pp1024 781.78 886.75 1.13
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 64 pp1024 1399.56 1558.62 1.11
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 128 pp1024 2078.71 2267.71 1.09
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 256 pp1024 2291.66 2509.94 1.10
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 512 pp1024 2354.89 2584.12 1.10
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 1 pp1024 55.42 69.88 1.26
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 2 pp1024 97.95 115.65 1.18
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 4 pp1024 163.97 168.12 1.03
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 8 pp1024 214.92 238.20 1.11
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 16 pp1024 540.99 621.99 1.15
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 32 pp1024 768.63 869.16 1.13
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 64 pp1024 1408.64 1552.57 1.10
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 128 pp1024 2066.66 2217.51 1.07
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 256 pp1024 2296.69 2473.59 1.08
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 512 pp1024 2364.25 2547.53 1.08

@JohannesGaessler
JohannesGaessler merged commit f46bc30 into ggml-org:master Sep 22, 2026
26 of 30 checks passed
@ggerganov ggerganov mentioned this pull request Sep 23, 2026
Merged
@BrewTestBot BrewTestBot mentioned this pull request Sep 23, 2026
Merged
1 task done
@davide221 davide221 mentioned this pull request Sep 25, 2026
Merged
2 of 3 tasks
davide221 added a commit to Luce-Org/lucebox that referenced this pull request Sep 26, 2026
@davide221 @claude
perf(ggml-hip): SWAR __vsub4 and __vcmpne4 for the IQ2/IQ3 decoders (#…
b9d911c 
…766)

Backport of ggml-org/llama.cpp#27962. The HIP fallbacks for these CUDA
byte-SIMD intrinsics compiled to per-byte loops (__vcmpne4) and a
saturating subtract (__vsub4). Replace them with SWAR bit arithmetic.

Every caller (the IQ2_XXS/XS/S and IQ3_XXS/S MMVQ and MMQ paths) applies
signs as (g ^ s) - s with s in {0x00, 0xff} per byte and grid bytes at most
62, so the wrapping subtract never differs from the saturating one: the
decoded weights are bit-identical.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LadislavSopko pushed a commit to 0ics-srls/llama.cpp that referenced this pull request Oct 5, 2026
@yanjs
HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR (ggml-org#2…
b3b3e76 
…7962)

* HIP : use bit manipulation for __vcmpne4

* HIP : use bit manipulation for __vsub4
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
@yanjs
HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR (ggml-org#2…
1f3cc21 
…7962)

* HIP : use bit manipulation for __vcmpne4

* HIP : use bit manipulation for __vsub4
edwardyoon pushed a commit to edwardyoon/focus-llama that referenced this pull request Oct 7, 2026
@yanjs edwardyoon
HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR (ggml-org#2…
abb4987 
…7962)

* HIP : use bit manipulation for __vcmpne4

* HIP : use bit manipulation for __vsub4

(cherry picked from commit f46bc30)
@yanjs
yanjs deleted the hip-swar branch October 7, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@IMbackK IMbackK IMbackK approved these changes
@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes

Assignees

@IMbackK IMbackK

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yanjs @IMbackK @JohannesGaessler