ROCm: add radix TOP_K for long rows by jadenmach2 · Pull Request #27466 · ggml-org/llama.cpp

jadenmach2 · 2026-08-21T01:38:50Z

Overview

This adds a ROCm TOP_K path for rows larger than 1024 elements. The current non-CUB path uses the bitonic implementation for small rows, but larger rows are reported as unsupported.

The new path uses exact 8-bit radix selection. Small rows continue to use the existing bitonic path, and the implementation does not depend on hipCUB.

The main motivation is DeepSeek-V4 long-context decoding, where the lightning indexer produces a TOP_K row over the KV history.

Performance

I tested this with DeepSeek-V4 Flash GGUF (155 GB) on three MI250X devices using ROCm 7.2. The numbers below are median generation throughput over 10 repetitions:

Context	master	this change	Improvement
2K	24.29 tok/s	24.33 tok/s	+0.2%
4K	17.43 tok/s	23.67 tok/s	+35.8%
8K	16.93 tok/s	23.39 tok/s	+38.1%

Testing

Ran test-backend-ops test -o TOP_K -b ROCm0
445/445 TOP_K tests passed on gfx90a

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, AI assistance was used under my supervision. AI located the ROCm ncols <= 1024 support restriction and CPU fallback. Implementation of 8-bit radix-selection was co-done with 5.6 sol

ggml-gh-bot · 2026-08-21T01:43:29Z

Hi @jadenmach2, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

IMbackK

While we would like to have just one code path here, i think this is a good idea take as a stop gap until the hipCUB hipGraph interaction is fixed in rocm

IMbackK · 2026-08-26T08:34:01Z

+    if (ncols > 1024) {
+        top_k_radix_cuda(pool, src0_d, dst_d, ncols, nrows, k, stream);
+    } else {
+#endif


missing comment

// defined(GGML_USE_HIP)

IMbackK · 2026-08-26T08:34:11Z

+            return true;
+#else
+            return op->src[0]->ne[0] <= 1024;
+#endif


missing comment

// defined(GGML_USE_HIP) || defined(GGML_CUDA_USE_CUB)

IMbackK · 2026-08-26T08:34:34Z

+                                     cudaMemcpyDeviceToDevice, stream));
+#if defined(GGML_USE_HIP)
+    }
+#endif


missing comment
// defined(GGML_USE_HIP)

fairydreaming · 2026-08-27T10:27:58Z

I tested the performance of this on NVIDIA RTX PRO 6000 Max-Q and it's pretty good, beats CUB argsort-based and DeviceTopK-based implementations for almost all tested shapes (and it's close enough for the few slower ones). Maybe we could simply use it as the default TOP_K implementation for now?

Reddish is this PR slower, blueish is this PR faster.

PR 27466 vs argsort (k = 1024)

PR 27466 vs argsort (k = 64)

PR 27466 vs DeviceTopK (k = 64)

CC @ORippler

IMbackK · 2026-08-27T10:33:21Z

Id like to also test it on RDNA/CDNA vs hipCUB before doing so, but yeah sure.
could you share your script for that visualisation.

fairydreaming · 2026-08-27T10:39:42Z

Id like to also test it on RDNA/CDNA vs hipCUB before doing so, but yeah sure. could you share your script for that visualisation.

@IMbackK Sure, it's just a vibe-coded thing that accepts two CSV files with ncols,nrows,time columns (I use time per run gathered with sed from the test-backend-ops perf output ... | sed 's/.*=\[//; s/,1,1.*runs -//; s/ us\/run.*//'|sed 's/\s\s*/,/' )

heat_plot.py

ORippler

Maybe we could simply use it as the default TOP_K implementation for now?

Id like to also test it on RDNA/CDNA vs hipCUB before doing so, but yeah sure.

I'd also like to see this tested

for smaller k (typically used for backend-sampling like 20/40)
on more SKUs (can test on what I have available)
on more OSs (Windows may take longer to launch the 5 kernels here as opposed to Linux)

if we want to make it the default path

ORippler · 2026-08-27T13:10:17Z

+    top_k_radix_state * states = states_alloc.get();
+    int * histograms = histograms_alloc.get();
+
+    top_k_radix_init<<<(nrows + BLOCK_SIZE - 1) / BLOCK_SIZE, BLOCK_SIZE, 0, stream>>>(states, nrows, k);


we should really have a ceildiv function in the cuda backend 😄

ORippler · 2026-08-27T13:14:26Z

+
+    top_k_radix_init<<<(nrows + BLOCK_SIZE - 1) / BLOCK_SIZE, BLOCK_SIZE, 0, stream>>>(states, nrows, k);
+
+    const dim3 histogram_grid(blocks_per_row, nrows);


This is going to hit the 16bit limit of griddim.y/z (on CUDA at least). Either guard dispatch, chunk, or fold into griddim.x:

https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/compute-capabilities.html#compute-capabilities-table-device-and-streaming-multiprocessor-sm-information-per-compute-capability

Thank you for the review, I will push the changes

Stoney49th · 2026-08-28T07:47:00Z

I want to add a testpoint with Q3.8-27B, 2xR9700, any advice on the settings I should use to trigger the paths? Or just a simple baseline with long context from 60k to 120k np 1..3 with MTP from 0..4?

Current Settings Baseline

[qwen3-8-27b]
n                      = -2
cache-ram              = 14336
ctx-checkpoints        = 4
checkpoint-min-step    = 8192
main-gpu               = 0
parallel               = 3
batch-size             = 4096
ubatch-size            = 512
kv-unified             = false
hf                     = unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
ctx-size               = 491520
temp                   = 1.0
top-p                  = 0.95
top-k                  = 20
min-p                  = 0.0
presence-penalty       = 0.0
repeat-penalty         = 1.0
cache-type-k           = q8_0
cache-type-v           = q8_0
flash-attn             = true
split-mode             = tensor
#tensor-split           = 50,50
jinja                  = true
reasoning-preserve     = true
reasoning-effort       = medium
reasoning-budget       = 20000
image-min-tokens       = 1024
no-mmproj-offload      = true
spec-type              = $DRAFT_TYPE
spec-draft-n-max       = $TEST_N_MAX
spec-draft-p-min       = 0.75

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

zihaomu · 2026-08-28T09:54:35Z

RDNA3.5 (gfx1151 / Strix Halo APU) datapoint — +1

New SKU: Radeon 8060S iGPU (gfx1151, RDNA3.5), ROCm 7.2.1, on current master. test-backend-ops -o TOP_K passes.

End-to-end — Qwen3.8-Flash-Next UD-IQ4_XS (its QSA indexer runs TOP_K over the KV axis, ncols = n_kv, same pattern as the DeepSeek-V4 indexer). Single-stream decode, -c 16384, graphs ON:

N_KV	master (CPU fallback)	this PR
640	21.5 t/s	21.5 t/s
2176	8.8 t/s	20.9 t/s
8320	8.1 t/s	19.5 t/s

Long-context decode collapse gone, and the radix kernels capture into HIP graphs fine.

vs hipCUB (re: @IMbackK) — I also tried a hipCUB DeviceSegmentedRadixSort::SortPairsDescending version. This PR is faster where it matters (200000x16: 258 vs 1722 us; 200000x1: 92 vs 175 us), and a large DeviceSegmentedRadixSort stack-overflows inside libamdhip64 under HIP graph capture (ROCm 7.2.1) — exactly the hipCUB/hipGraph issue you mentioned. These hand-rolled kernels avoid it. Cost is also flat in k, so small sampling k (20/40) behaves like the large indexer k.

Happy to run more shapes if useful. Nice work!

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

Stoney49th · 2026-08-28T16:34:51Z

2X R9700, Powercapped at 230W,

Settings

[qwen3-8-27b]
n                      = -2
cache-ram              = 14336
ctx-checkpoints        = 4
checkpoint-min-step    = 8192
main-gpu               = 0
parallel               = 3
batch-size             = 4096
ubatch-size            = 512
kv-unified             = false
hf                     = unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
ctx-size               = 491520
temp                   = 1.0
top-p                  = 0.95
top-k                  = 20
min-p                  = 0.0
presence-penalty       = 0.0
repeat-penalty         = 1.0
cache-type-k           = q8_0
cache-type-v           = q8_0
flash-attn             = true
split-mode             = tensor
#tensor-split           = 50,50
jinja                  = true
reasoning-preserve     = true
reasoning-effort       = medium
reasoning-budget       = 20000
image-min-tokens       = 1024
no-mmproj-offload      = true
spec-type              = draft-mtp
spec-draft-n-max       = 1
spec-draft-p-min       = 0.75

-DGGML_HIP=ON \
-DGGML_HIP_RCCL=ON \
-DGGML_HIP_GRAPHS=ON \
-DGGML_HIP_ROCWMMA_FATTN=ON \

environment:
      - GGML_CUDA_P2P=true

Results Detailed

# Prompt processing (PP) -- tokens/s, independent of parallel level

## cfg=none

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60000 | 847.7 | 835.9 | -1.4% |
| 75000 | 750.5 | 747.2 | -0.4% |
| 90000 | 680.0 | 680.4 | +0.1% |
| 105000 | 621.8 | 623.6 | +0.3% |
| 120000 | 574.1 | 575.1 | +0.2% |

## cfg=1

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60000 | 772.5 | 789.4 | +2.2% |
| 75000 | 700.4 | 702.6 | +0.3% |
| 90000 | 638.2 | 637.3 | -0.1% |
| 105000 | 584.3 | 585.0 | +0.1% |
| 120000 | 540.0 | 538.5 | -0.3% |

## cfg=2

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60000 | 772.5 | 789.4 | +2.2% |
| 75000 | 700.4 | 702.6 | +0.3% |
| 90000 | 638.2 | 637.3 | -0.1% |
| 105000 | 584.3 | 585.0 | +0.1% |
| 120000 | 540.0 | 538.5 | -0.3% |

## cfg=3

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60000 | 772.5 | 789.4 | +2.2% |
| 75000 | 700.4 | 702.6 | +0.3% |
| 90000 | 638.2 | 637.3 | -0.1% |
| 105000 | 584.3 | 585.0 | +0.1% |
| 120000 | 540.0 | 538.5 | -0.3% |

## cfg=4

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60000 | 772.5 | 789.4 | +2.2% |
| 75000 | 700.4 | 702.6 | +0.3% |
| 90000 | 638.2 | 637.3 | -0.1% |
| 105000 | 584.3 | 585.0 | +0.1% |
| 120000 | 540.0 | 538.5 | -0.3% |

# Decode (TG) -- combined tokens/s across concurrently-decoding slots

## cfg=none parallel=1

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 29.64 | 29.64 | +0.0% |
| 75010 | 27.95 | 27.95 | +0.0% |
| 90010 | 26.42 | 26.42 | +0.0% |
| 105010 | 25.17 | 25.17 | +0.0% |
| 120010 | 23.94 | 23.94 | +0.0% |

## cfg=none parallel=2

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 43.27 | 43.27 | +0.0% |
| 75010 | 40.02 | 40.02 | +0.0% |
| 90010 | 37.16 | 37.16 | +0.0% |
| 105010 | 34.68 | 34.68 | +0.0% |
| 120010 | 32.64 | 32.64 | +0.0% |

## cfg=none parallel=3

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 51.30 | 51.30 | +0.0% |
| 75010 | 46.81 | 46.81 | +0.0% |
| 90010 | 43.05 | 43.05 | +0.0% |
| 105010 | 40.05 | 40.05 | +0.0% |
| 120010 | 37.29 | 37.29 | +0.0% |

## cfg=1 parallel=1

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 37.30 | 35.94 | -3.6% |
| 75010 | 36.53 | 37.80 | +3.5% |
| 90010 | 43.41 | 43.47 | +0.1% |
| 105010 | 41.57 | 41.77 | +0.5% |
| 120010 | 39.88 | 39.94 | +0.2% |

## cfg=1 parallel=2

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 27.72 | 26.64 | -3.9% |
| 75010 | 45.77 | 41.96 | -8.3% |
| 90010 | 39.49 | 34.20 | -13.4% |
| 105010 | 53.84 | 54.52 | +1.3% |
| 120010 | 38.84 | 40.82 | +5.1% |

## cfg=1 parallel=3

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 70.54 | 70.88 | +0.5% |
| 75010 | 65.12 | 65.37 | +0.4% |
| 90010 | 56.98 | 60.03 | +5.4% |
| 105010 | 57.26 | 57.83 | +1.0% |
| 120010 | 47.48 | 47.37 | -0.2% |

## cfg=2 parallel=1

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 33.99 | 33.99 | +0.0% |
| 75010 | 47.25 | 46.85 | -0.8% |
| 90010 | 29.37 | 28.66 | -2.4% |
| 105010 | 47.91 | 41.79 | -12.8% |
| 120010 | 35.76 | 35.14 | -1.7% |

## cfg=2 parallel=2

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 34.35 | 26.83 | -21.9% |
| 75010 | 39.35 | 29.17 | -25.9% |
| 90010 | 34.55 | 29.35 | -15.1% |
| 105010 | 56.93 | 56.97 | +0.1% |
| 120010 | 53.41 | 53.60 | +0.4% |

## cfg=2 parallel=3

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 98.73 | 99.20 | +0.5% |
| 75010 | 24.96 | 22.90 | -8.3% |
| 90010 | 62.74 | 59.97 | -4.4% |
| 105010 | 41.31 | 32.84 | -20.5% |
| 120010 | 37.27 | 27.67 | -25.8% |

## cfg=3 parallel=1

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 42.89 | 42.89 | +0.0% |
| 75010 | 42.29 | 33.56 | -20.6% |
| 90010 | 29.93 | 31.36 | +4.8% |
| 105010 | 46.91 | 40.89 | -12.8% |
| 120010 | 53.99 | 49.83 | -7.7% |

## cfg=3 parallel=2

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 37.81 | 30.86 | -18.4% |
| 75010 | 39.98 | 39.79 | -0.5% |
| 90010 | 32.88 | 33.95 | +3.3% |
| 105010 | 67.45 | 67.78 | +0.5% |
| 120010 | 35.61 | 45.28 | +27.2% |

## cfg=3 parallel=3

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 123.93 | 124.68 | +0.6% |
| 75010 | 106.30 | 107.78 | +1.4% |
| 90010 | 102.46 | 103.36 | +0.9% |
| 105010 | 39.43 | 52.95 | +34.3% |
| 120010 | 30.52 | 39.86 | +30.6% |

## cfg=4 parallel=1

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 32.43 | 42.89 | +32.3% |
| 75010 | 24.72 | 29.57 | +19.6% |
| 90010 | 32.91 | 33.15 | +0.7% |
| 105010 | 50.72 | 60.47 | +19.2% |
| 120010 | 47.40 | 51.39 | +8.4% |

## cfg=4 parallel=2

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 39.67 | 46.27 | +16.6% |
| 75010 | 54.17 | 60.97 | +12.6% |
| 90010 | 42.74 | 58.98 | +38.0% |
| 105010 | 84.05 | 84.05 | +0.0% |
| 120010 | 60.33 | 63.38 | +5.1% |

## cfg=4 parallel=3

| context length | A: PR26419-testing, gfx1201, GGML_HIP_ROCWMMA_FATTN=ON, Code Master | B: PR27466 radix-topk mtp | % diff |
| --- | --- | --- | --- |
| 60010 | 124.29 | 126.07 | +1.4% |
| 75010 | 55.03 | 73.05 | +32.7% |
| 90010 | 94.96 | 97.55 | +2.7% |
| 105010 | 87.06 | 71.53 | -17.8% |
| 120010 | 31.36 | 31.22 | -0.4% |

Image

IMbackK · 2026-08-28T19:19:53Z

you can defiantly find shapes where this performs poorly against the cub path (1 row), so from a hip perspective we may want to keep it around until amd fixes the graph capture problem, or roll another own solution

Details

GFX908:

Backend	GGML op	Op parameters	Bandwidth (GB/s) argsort_top_k_cub_on_hip	Bandwidth (GB/s) rocm-radix-topk	Speedup
ROCm0	TOP_K	type=f32,ne=[1,1,1,1],k=1,ties=0	0.00	0.00	1.00
ROCm0	TOP_K	type=f32,ne=[1,16,1,1],k=1,ties=0	0.03	0.03	1.00
ROCm0	TOP_K	type=f32,ne=[10,1,1,1],k=10,ties=0	0.01	0.01	1.00
ROCm0	TOP_K	type=f32,ne=[10,16,1,1],k=10,ties=0	0.12	0.12	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=1,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=10,ties=0	0.08	0.08	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=16,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=32,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=4,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=40,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=400,ties=0	0.11	0.11	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=8,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=1,ties=0	1.25	1.25	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=10,ties=0	1.26	1.26	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=16,ties=0	1.27	1.27	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=32,ties=0	1.29	1.29	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=4,ties=0	1.25	1.25	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=40,ties=0	1.30	1.30	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=400,ties=0	1.74	1.75	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=8,ties=0	1.26	1.26	1.00
ROCm0	TOP_K	type=f32,ne=[12288,1,1,1],k=16,ties=0	1.13	0.58	0.51
ROCm0	TOP_K	type=f32,ne=[12288,16,1,1],k=16,ties=0	3.83	9.14	2.39
ROCm0	TOP_K	type=f32,ne=[131072,1,1,1],k=16,ties=0	5.30	4.53	0.85
ROCm0	TOP_K	type=f32,ne=[131072,16,1,1],k=16,ties=0	4.13	49.92	12.10
ROCm0	TOP_K	type=f32,ne=[16,1,1,1],k=16,ties=0	0.01	0.01	1.00
ROCm0	TOP_K	type=f32,ne=[16,16,1,1],k=16,ties=0	0.18	0.18	1.00
ROCm0	TOP_K	type=f32,ne=[16384,1,1,1],k=16,ties=0	1.50	0.78	0.52
ROCm0	TOP_K	type=f32,ne=[16384,16,1,1],k=16,ties=0	3.88	12.27	3.17
ROCm0	TOP_K	type=f32,ne=[2,1,1,1],k=1,ties=0	0.00	0.00	0.94
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=1,ties=0	6.15	5.87	0.95
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=10,ties=0	6.15	5.64	0.92
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=16,ties=0	6.12	5.27	0.86
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=32,ties=0	6.12	5.24	0.86
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=4,ties=0	6.16	5.26	0.85
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=40,ties=0	6.13	5.24	0.85
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=400,ties=0	6.14	5.70	0.93
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=8,ties=0	6.15	5.27	0.86
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=1,ties=0	4.17	62.76	15.05
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=10,ties=0	4.17	61.12	14.65
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=16,ties=0	4.17	58.27	13.97
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=32,ties=0	4.17	58.06	13.93
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=4,ties=0	4.17	58.30	13.97
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=40,ties=0	4.17	58.03	13.91
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=400,ties=0	4.18	61.49	14.71
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=8,ties=0	4.17	58.28	13.97
ROCm0	TOP_K	type=f32,ne=[24576,1,1,1],k=16,ties=0	1.77	1.19	0.67
ROCm0	TOP_K	type=f32,ne=[24576,16,1,1],k=16,ties=0	3.70	18.15	4.91
ROCm0	TOP_K	type=f32,ne=[32,1,1,1],k=32,ties=0	0.02	0.02	1.00
ROCm0	TOP_K	type=f32,ne=[32,16,1,1],k=32,ties=0	0.29	0.29	1.00
ROCm0	TOP_K	type=f32,ne=[32768,1,1,1],k=16,ties=0	2.27	1.55	0.68
ROCm0	TOP_K	type=f32,ne=[32768,16,1,1],k=16,ties=0	3.72	23.06	6.20
ROCm0	TOP_K	type=f32,ne=[4,1,1,1],k=4,ties=0	0.00	0.00	1.00
ROCm0	TOP_K	type=f32,ne=[4,16,1,1],k=4,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[40,1,1,1],k=40,ties=0	0.02	0.02	1.00
ROCm0	TOP_K	type=f32,ne=[40,16,1,1],k=40,ties=0	0.29	0.29	1.00
ROCm0	TOP_K	type=f32,ne=[400,1,1,1],k=400,ties=0	0.10	0.10	1.00
ROCm0	TOP_K	type=f32,ne=[400,16,1,1],k=400,ties=0	1.53	1.53	1.00
ROCm0	TOP_K	type=f32,ne=[4096,1,1,1],k=16,ties=0	0.52	0.18	0.35
ROCm0	TOP_K	type=f32,ne=[4096,16,1,1],k=16,ties=0	5.63	2.96	0.53
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=1,ties=0	4.05	2.16	0.53
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=10,ties=0	4.06	2.14	0.53
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=16,ties=0	4.05	2.13	0.53
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=32,ties=0	4.07	2.11	0.52
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=4,ties=0	4.06	2.15	0.53
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=40,ties=0	4.06	2.09	0.51
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=400,ties=0	4.08	1.97	0.48
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=8,ties=0	4.06	2.15	0.53
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=1,ties=0	3.92	28.81	7.34
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=10,ties=0	3.92	28.65	7.31
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=16,ties=0	3.92	28.58	7.29
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=32,ties=0	3.92	28.32	7.22
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=4,ties=0	3.92	28.74	7.33
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=40,ties=0	3.92	28.19	7.19
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=400,ties=0	3.94	26.57	6.74
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=8,ties=0	3.92	28.69	7.31
ROCm0	TOP_K	type=f32,ne=[65536,1,1,1],k=16,ties=0	4.12	2.19	0.53
ROCm0	TOP_K	type=f32,ne=[65536,16,1,1],k=16,ties=0	3.98	29.25	7.34
ROCm0	TOP_K	type=f32,ne=[8,1,1,1],k=8,ties=0	0.01	0.01	0.99
ROCm0	TOP_K	type=f32,ne=[8,16,1,1],k=8,ties=0	0.12	0.12	1.00
ROCm0	TOP_K	type=f32,ne=[8192,1,1,1],k=16,ties=0	0.77	0.46	0.59
ROCm0	TOP_K	type=f32,ne=[8192,16,1,1],k=16,ties=0	3.59	7.24	2.02

GFX1100:

Backend	GGML op	Op parameters	Bandwidth (GB/s) argsort_top_k_cub_on_hip	Bandwidth (GB/s) rocm-radix-topk	Speedup
ROCm0	TOP_K	type=f32,ne=[1,1,1,1],k=1,ties=0	0.00	0.00	0.88
ROCm0	TOP_K	type=f32,ne=[1,16,1,1],k=1,ties=0	0.02	0.02	1.00
ROCm0	TOP_K	type=f32,ne=[10,1,1,1],k=10,ties=0	0.01	0.01	0.95
ROCm0	TOP_K	type=f32,ne=[10,16,1,1],k=10,ties=0	0.14	0.14	0.98
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=1,ties=0	0.22	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=10,ties=0	0.22	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=16,ties=0	0.22	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=32,ties=0	0.22	0.22	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=4,ties=0	0.22	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=40,ties=0	0.23	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=400,ties=0	0.30	0.30	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=8,ties=0	0.22	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=1,ties=0	3.40	3.41	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=10,ties=0	3.39	3.43	1.01
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=16,ties=0	3.45	3.42	0.99
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=32,ties=0	3.50	3.50	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=4,ties=0	3.41	3.38	0.99
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=40,ties=0	3.49	3.53	1.01
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=400,ties=0	4.74	4.75	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=8,ties=0	3.42	3.42	1.00
ROCm0	TOP_K	type=f32,ne=[12288,1,1,1],k=16,ties=0	1.18	0.81	0.69
ROCm0	TOP_K	type=f32,ne=[12288,16,1,1],k=16,ties=0	9.32	12.78	1.37
ROCm0	TOP_K	type=f32,ne=[131072,1,1,1],k=16,ties=0	6.51	6.93	1.06
ROCm0	TOP_K	type=f32,ne=[131072,16,1,1],k=16,ties=0	11.07	58.15	5.26
ROCm0	TOP_K	type=f32,ne=[16,1,1,1],k=16,ties=0	0.02	0.01	0.93
ROCm0	TOP_K	type=f32,ne=[16,16,1,1],k=16,ties=0	0.23	0.23	1.03
ROCm0	TOP_K	type=f32,ne=[16384,1,1,1],k=16,ties=0	1.57	1.08	0.69
ROCm0	TOP_K	type=f32,ne=[16384,16,1,1],k=16,ties=0	9.56	16.89	1.77
ROCm0	TOP_K	type=f32,ne=[2,1,1,1],k=1,ties=0	0.00	0.00	1.09
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=1,ties=0	7.65	9.68	1.26
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=10,ties=0	7.66	9.46	1.23
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=16,ties=0	7.66	9.07	1.19
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=32,ties=0	7.65	9.04	1.18
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=4,ties=0	7.66	9.06	1.18
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=40,ties=0	7.66	9.05	1.18
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=400,ties=0	7.66	9.53	1.24
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=8,ties=0	7.66	9.08	1.18
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=1,ties=0	11.32	89.98	7.95
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=10,ties=0	11.31	88.10	7.79
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=16,ties=0	11.31	84.93	7.51
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=32,ties=0	11.32	84.61	7.48
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=4,ties=0	11.32	85.26	7.53
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=40,ties=0	11.32	84.83	7.50
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=400,ties=0	11.34	88.71	7.82
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=8,ties=0	11.32	85.32	7.54
ROCm0	TOP_K	type=f32,ne=[24576,1,1,1],k=16,ties=0	1.84	1.62	0.88
ROCm0	TOP_K	type=f32,ne=[24576,16,1,1],k=16,ties=0	9.19	12.12	1.32
ROCm0	TOP_K	type=f32,ne=[32,1,1,1],k=32,ties=0	0.03	0.03	0.93
ROCm0	TOP_K	type=f32,ne=[32,16,1,1],k=32,ties=0	0.41	0.41	1.02
ROCm0	TOP_K	type=f32,ne=[32768,1,1,1],k=16,ties=0	2.58	2.11	0.82
ROCm0	TOP_K	type=f32,ne=[32768,16,1,1],k=16,ties=0	9.64	29.38	3.05
ROCm0	TOP_K	type=f32,ne=[4,1,1,1],k=4,ties=0	0.00	0.00	0.91
ROCm0	TOP_K	type=f32,ne=[4,16,1,1],k=4,ties=0	0.07	0.07	1.00
ROCm0	TOP_K	type=f32,ne=[40,1,1,1],k=40,ties=0	0.03	0.03	0.94
ROCm0	TOP_K	type=f32,ne=[40,16,1,1],k=40,ties=0	0.46	0.46	0.99
ROCm0	TOP_K	type=f32,ne=[400,1,1,1],k=400,ties=0	0.22	0.21	0.96
ROCm0	TOP_K	type=f32,ne=[400,16,1,1],k=400,ties=0	3.35	3.35	1.00
ROCm0	TOP_K	type=f32,ne=[4096,1,1,1],k=16,ties=0	0.52	0.27	0.52
ROCm0	TOP_K	type=f32,ne=[4096,16,1,1],k=16,ties=0	8.62	4.29	0.50
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=1,ties=0	4.61	3.42	0.74
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=10,ties=0	4.61	3.40	0.74
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=16,ties=0	4.61	3.39	0.74
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=32,ties=0	4.61	3.37	0.73
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=4,ties=0	4.61	3.41	0.74
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=40,ties=0	4.61	3.36	0.73
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=400,ties=0	4.63	3.24	0.70
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=8,ties=0	4.61	3.40	0.74
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=1,ties=0	10.16	45.88	4.52
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=10,ties=0	10.15	45.65	4.50
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=16,ties=0	10.16	45.57	4.49
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=32,ties=0	10.16	45.29	4.46
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=4,ties=0	10.12	45.73	4.52
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=40,ties=0	10.16	45.25	4.46
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=400,ties=0	10.21	43.80	4.29
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=8,ties=0	10.15	45.70	4.50
ROCm0	TOP_K	type=f32,ne=[65536,1,1,1],k=16,ties=0	5.02	3.46	0.69
ROCm0	TOP_K	type=f32,ne=[65536,16,1,1],k=16,ties=0	10.29	46.13	4.48
ROCm0	TOP_K	type=f32,ne=[8,1,1,1],k=8,ties=0	0.01	0.01	0.91
ROCm0	TOP_K	type=f32,ne=[8,16,1,1],k=8,ties=0	0.13	0.13	1.00
ROCm0	TOP_K	type=f32,ne=[8192,1,1,1],k=16,ties=0	0.75	0.59	0.78
ROCm0	TOP_K	type=f32,ne=[8192,16,1,1],k=16,ties=0	8.13	9.28	1.14

GFX1201

Backend	GGML op	Op parameters	Bandwidth (GB/s) argsort_top_k_cub_on_hip	Bandwidth (GB/s) rocm-radix-topk	Speedup
ROCm0	TOP_K	type=f32,ne=[1,1,1,1],k=1,ties=0	0.00	0.00	1.00
ROCm0	TOP_K	type=f32,ne=[1,16,1,1],k=1,ties=0	0.02	0.02	1.00
ROCm0	TOP_K	type=f32,ne=[10,1,1,1],k=10,ties=0	0.01	0.01	1.00
ROCm0	TOP_K	type=f32,ne=[10,16,1,1],k=10,ties=0	0.15	0.15	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=1,ties=0	0.22	0.22	1.01
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=10,ties=0	0.22	0.22	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=16,ties=0	0.22	0.22	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=32,ties=0	0.22	0.22	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=4,ties=0	0.22	0.22	0.99
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=40,ties=0	0.23	0.23	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=400,ties=0	0.30	0.30	1.00
ROCm0	TOP_K	type=f32,ne=[1000,1,1,1],k=8,ties=0	0.22	0.22	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=1,ties=0	3.39	3.36	0.99
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=10,ties=0	3.39	3.39	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=16,ties=0	3.40	3.45	1.01
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=32,ties=0	3.49	3.46	0.99
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=4,ties=0	3.36	3.35	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=40,ties=0	3.47	3.49	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=400,ties=0	4.70	4.71	1.00
ROCm0	TOP_K	type=f32,ne=[1000,16,1,1],k=8,ties=0	3.39	3.38	1.00
ROCm0	TOP_K	type=f32,ne=[12288,1,1,1],k=16,ties=0	1.30	0.88	0.68
ROCm0	TOP_K	type=f32,ne=[12288,16,1,1],k=16,ties=0	8.33	13.75	1.65
ROCm0	TOP_K	type=f32,ne=[131072,1,1,1],k=16,ties=0	2.49	6.02	2.42
ROCm0	TOP_K	type=f32,ne=[131072,16,1,1],k=16,ties=0	9.88	42.22	4.28
ROCm0	TOP_K	type=f32,ne=[16,1,1,1],k=16,ties=0	0.02	0.02	1.01
ROCm0	TOP_K	type=f32,ne=[16,16,1,1],k=16,ties=0	0.24	0.24	1.00
ROCm0	TOP_K	type=f32,ne=[16384,1,1,1],k=16,ties=0	1.76	1.15	0.66
ROCm0	TOP_K	type=f32,ne=[16384,16,1,1],k=16,ties=0	8.87	7.99	0.90
ROCm0	TOP_K	type=f32,ne=[2,1,1,1],k=1,ties=0	0.00	0.00	0.95
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=1,ties=0	6.36	8.41	1.32
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=10,ties=0	6.36	8.27	1.30
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=16,ties=0	6.37	8.01	1.26
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=32,ties=0	6.37	8.00	1.26
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=4,ties=0	6.38	8.02	1.26
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=40,ties=0	6.38	7.99	1.25
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=400,ties=0	6.38	8.32	1.30
ROCm0	TOP_K	type=f32,ne=[200000,1,1,1],k=8,ties=0	6.36	8.01	1.26
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=1,ties=0	9.71	58.66	6.04
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=10,ties=0	9.72	58.31	6.00
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=16,ties=0	9.71	57.38	5.91
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=32,ties=0	9.71	57.36	5.91
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=4,ties=0	9.71	57.21	5.89
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=40,ties=0	9.72	57.36	5.90
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=400,ties=0	9.74	58.44	6.00
ROCm0	TOP_K	type=f32,ne=[200000,16,1,1],k=8,ties=0	9.71	57.40	5.91
ROCm0	TOP_K	type=f32,ne=[24576,1,1,1],k=16,ties=0	1.98	1.62	0.82
ROCm0	TOP_K	type=f32,ne=[24576,16,1,1],k=16,ties=0	9.39	23.08	2.46
ROCm0	TOP_K	type=f32,ne=[32,1,1,1],k=32,ties=0	0.03	0.03	1.00
ROCm0	TOP_K	type=f32,ne=[32,16,1,1],k=32,ties=0	0.43	0.43	1.00
ROCm0	TOP_K	type=f32,ne=[32768,1,1,1],k=16,ties=0	2.60	2.04	0.79
ROCm0	TOP_K	type=f32,ne=[32768,16,1,1],k=16,ties=0	9.71	13.91	1.43
ROCm0	TOP_K	type=f32,ne=[4,1,1,1],k=4,ties=0	0.00	0.00	1.00
ROCm0	TOP_K	type=f32,ne=[4,16,1,1],k=4,ties=0	0.08	0.08	1.00
ROCm0	TOP_K	type=f32,ne=[40,1,1,1],k=40,ties=0	0.03	0.03	1.00
ROCm0	TOP_K	type=f32,ne=[40,16,1,1],k=40,ties=0	0.46	0.46	1.00
ROCm0	TOP_K	type=f32,ne=[400,1,1,1],k=400,ties=0	0.21	0.21	1.00
ROCm0	TOP_K	type=f32,ne=[400,16,1,1],k=400,ties=0	3.25	3.26	1.00
ROCm0	TOP_K	type=f32,ne=[4096,1,1,1],k=16,ties=0	0.60	0.30	0.50
ROCm0	TOP_K	type=f32,ne=[4096,16,1,1],k=16,ties=0	4.43	4.82	1.09
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=1,ties=0	1.78	2.99	1.69
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=10,ties=0	1.78	3.00	1.69
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=16,ties=0	1.78	3.00	1.69
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=32,ties=0	1.78	2.98	1.68
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=4,ties=0	1.78	3.01	1.70
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=40,ties=0	1.78	2.97	1.67
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=400,ties=0	1.78	2.91	1.63
ROCm0	TOP_K	type=f32,ne=[65000,1,1,1],k=8,ties=0	1.78	3.01	1.69
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=1,ties=0	9.80	22.20	2.27
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=10,ties=0	9.79	22.40	2.29
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=16,ties=0	9.79	22.08	2.26
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=32,ties=0	9.79	22.10	2.26
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=4,ties=0	9.78	22.10	2.26
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=40,ties=0	9.79	22.28	2.28
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=400,ties=0	9.85	23.84	2.42
ROCm0	TOP_K	type=f32,ne=[65000,16,1,1],k=8,ties=0	9.79	22.36	2.28
ROCm0	TOP_K	type=f32,ne=[65536,1,1,1],k=16,ties=0	1.79	3.05	1.70
ROCm0	TOP_K	type=f32,ne=[65536,16,1,1],k=16,ties=0	9.88	22.36	2.26
ROCm0	TOP_K	type=f32,ne=[8,1,1,1],k=8,ties=0	0.01	0.01	1.00
ROCm0	TOP_K	type=f32,ne=[8,16,1,1],k=8,ties=0	0.14	0.14	1.00
ROCm0	TOP_K	type=f32,ne=[8192,1,1,1],k=16,ties=0	0.79	0.64	0.80
ROCm0	TOP_K	type=f32,ne=[8192,16,1,1],k=16,ties=0	7.60	10.12	1.33

IMbackK

Its good as is for the purposes of supporting ncols > 1024 on hip.

drluoto · 2026-08-28T20:16:28Z

Tested this PR on Strix Halo (Ryzen AI Max+ 395 / Radeon 8060S, gfx1151, ROCm 7.1), applied onto master ca3d5a3, in the context of Qwen3.8-Flash-Next (qwen4exp) whose QSA indexer issues wide ggml_top_k (ne[0] = n_kv, 12 layers/token) — currently a CPU fallback on HIP.

Correctness: test-backend-ops test -b ROCm0 -o TOP_K passes.

HIP graph capture: safe. Three consecutive 2500-token speculative-decoding generations with graphs enabled, zero incidents. This is notable because the hipCUB alternative (#26592) aborts capture deterministically — DeviceSegmentedRadixSort::SortPairsDescending returns operation not permitted when stream is capturing on ROCm (details in that PR's thread). Your kernel has no library-internal stream operations, so it composes with GGML_HIP_GRAPHS.

Single-row decode (llama-bench tg64, UD-IQ4_XS 93.7 GB): parity with the hipCUB path, both a big win over master's CPU fallback:

depth	master (CPU fallback)	hipCUB	this PR
1024	20.99	22.71	22.70
4096	18.44	21.83	21.38
16384	14.57	17.62	17.80

Multi-row wide case — tuning opportunity: with speculative decoding, TOP_K arrives as ~65 rows × 24k cols during draft verification. There the hipCUB segmented sort is ~27% faster end-to-end on our workload (21.3 vs 16.1 tok/s on a file-rewrite task at 24k-token context). Possibly the blocks_per_row cap (64) or per-pass histogram traffic; happy to run any variant you want measured on gfx1151.

Net: for graph compatibility this PR is currently the only working GPU TOP_K on ROCm, and single-row performance matches CUB. Would love to see it land.

IMbackK · 2026-09-01T09:37:08Z

After this PR Qwen 3.8 Next degenerates into only outputting "/////////" forever on ROCm backend with larger context windows.

cant repo that either at 100k context could you provide more information?

BoneHorror · 2026-09-01T09:44:41Z

After this PR Qwen 3.8 Next degenerates into only outputting "/////////" forever on ROCm backend with larger context windows.

cant repo that either at 100k context could you provide more information?

I'm using Unsloth's Q4_K_XL https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q4_K_XL with
./llama-server -fa 1 -dev ROCm0 --temp 1.0 --top-k 20 --top-p 0.95 --kv unified --spec-type ngram-mod,ngram-map-k4v --ctx-checkpoints 3 --mmproj --image-min-tokens 1024 --batch-size 2048 --ubatch-size 2048 --reasoning-preserve
At first I thought it could be due to today's Qwen optimizations so I checked out to just before them, at 458681e - but it still occurred, so I tried to move back a bit more and ended up not having the issue only at 5d4a3be, just before this commit/PR

Machine is Strix Halo, I unfortunately can't paste exact prompts/context for repro

IMbackK · 2026-09-01T09:45:53Z

rocm version, and compile time options please.

BoneHorror · 2026-09-01T11:12:51Z

rocm version, and compile time options please.

Right, sorry;
ROCm is 10.0.0 from amdrocm-rocm-sdk10.0-gfx1151
Built with -DGGML_AVX512=ON -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
It seems to reproduce relatively easily for me, I'm sorry for wasting your time if this ends up being something else - I'm really not sure what I'm messing up in that case.
E: removed pointless obsolete flags to avoid confusion

* ROCm: add radix TOP_K for long rows

cdanis · 2026-09-01T17:18:51Z

-DGGML_HIP_UMAD=ON

shouldn't this be -DGGML_HIP_UMA ? instead of UMAD

BoneHorror · 2026-09-02T09:56:26Z

-DGGML_HIP_UMAD=ON

shouldn't this be -DGGML_HIP_UMA ? instead of UMAD

Neither exists atm i guess but it doesnt matter for this, my bad in any case - ive probably copied it from one of the random gfx1151 repos floating around in the past

Someone reported the same garbage output pointing at an earlier commit - #28113
But I do have that one and its working, so maybe its a combination of the two+specific moe archs or something of that sort...
EDIT: Looks like it's confirmed-ish in the other ticket that this is not the commit that originally caused the decode issue, sorry for the mess.

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

* ROCm: add radix TOP_K for long rows

Enable the preserved 0005r1 fallback for CUB builds without DeviceTopK at 8192 columns or more. Retain the argsort control switch and existing radix kernels. Document source provenance, bounded quality evidence, toolkit eligibility, and a manual benchmark client. Related upstream work: ggml-org#28366 and ggml-org#27466. Assisted-by: Codex Astra

CUDA builds with CCCL < 3.2 have no cub::DeviceTopK, so ggml_top_k falls back to a full argsort + copy (~12 bytes of temp per cell). For sparse-attention top-k over long contexts this OOMs small cards (200K cols x 128 rows ~= 190MB+ of temp per layer). Reuse the exact radix selection from ggml-org#27466 (merged for HIP) on CUDA when CUB_TOP_K_AVAILABLE is off: O(nrows) temp, same selected set. Also gate next_power_of_2, unused on that path now.

* ROCm: add radix TOP_K for long rows

Keep USB4 RPC, Flash-Next NextN, DFlash LM-head gather, meta shard guards, HIP TOP_K supports_op, and AVX-512 AllReduce. Take upstream radix TOP_K (ggml-org#27466) and RPC dispatcher check (ggml-org#26500) combined with view-src tensor data offsets.

* ROCm: add radix TOP_K for long rows

With GGML_HIP_CUB=ON, GGML_CUDA_USE_CUB is defined on HIP, which hid the radix select added upstream in ggml-org#27466 behind its !GGML_CUDA_USE_CUB guard. TOP_K on wide rows then fell back to a full hipCUB segmented sort plus a copy. Radix select only scans the row, so for wide inputs it is much faster. Measured with test-backend-ops perf on an RX 7900 XT (gfx1100): ne=[151936,1], k=20/40 148/136 us -> 119/122 us (1.1-1.2x) ne=[200000,16], k=16 1061 us -> 187 us (5.7x) ne=[131072,4], k=2048 708 us -> 121 us (5.9x) Below 32k columns the fixed launch cost of the multi-pass radix kernel (~100 us) loses to the bitonic/hipCUB path, so only rows of at least 32768 columns are routed to radix. ARGSORT and the other ops keep using hipCUB, which is still what lifts their ne[0] <= 1024 limit. TOP_K: 525/525 test-backend-ops cases pass on ROCm0.

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

jadenmach2 requested a review from a team as a code owner August 21, 2026 01:38

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 21, 2026

ggml-gh-bot Bot added the draft PR will be changed to draft by github-actions bot label Aug 21, 2026

github-actions Bot marked this pull request as draft August 21, 2026 01:43

github-actions Bot removed the draft PR will be changed to draft by github-actions bot label Aug 21, 2026

ROCm: add radix TOP_K for long rows

ff7cd32

jadenmach2 force-pushed the rocm-radix-topk branch from 743d1a4 to ff7cd32 Compare August 21, 2026 02:10

jadenmach2 marked this pull request as ready for review August 21, 2026 02:11

IMbackK self-assigned this Aug 21, 2026

IMbackK mentioned this pull request Aug 25, 2026

CUDA: enable the CUB path on HIP via hipCUB #26592

Open

IMbackK requested changes Aug 26, 2026

View reviewed changes

ORippler reviewed Aug 27, 2026

View reviewed changes

Addressing reviewer comments

7f48903

ORippler mentioned this pull request Aug 27, 2026

spec : add DFlash2 support (local convolution + candidate selector) #27342

Merged

Stoney49th mentioned this pull request Aug 27, 2026

Misc. bug: VRAM does not allocate with ROCm 7.14 on gfx1201 #26208

Closed

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Aug 28, 2026

docs: credit ggml-org#27466 radix top-k, add related-upstream-work se…

6bb3909

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

Stoney49th mentioned this pull request Aug 28, 2026

(WIP) CUDA/HIP: GPU partial top-k for ROCm (fix full-sort/CPU fallback) #27860

Closed

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Aug 28, 2026

docs: credit ggml-org#27466 radix top-k, add related-upstream-work se…

1d1da0e

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

IMbackK approved these changes Aug 28, 2026

View reviewed changes

remeh mentioned this pull request Aug 28, 2026

qwen4exp (Qwen3.8-Flash-Next): severe decode slowdown beyond ~1K context on HIP / gfx1151 (Strix Halo) #27856

Closed

ilmmatias pushed a commit to ilmmatias/llama.cpp that referenced this pull request Sep 1, 2026

ROCm: add radix TOP_K for long rows (ggml-org#27466)

8c612ce

* ROCm: add radix TOP_K for long rows

ArtomYuan mentioned this pull request Sep 2, 2026

ROCm: TOP_K crashes with "invalid configuration argument" when ncols > 1024 (bitonic kernel block-size overflow; blocks DeepSeek V4 ctx > 128K) #27021

Closed

Rhonstin mentioned this pull request Sep 2, 2026

qwen4exp : gather-based sparse attention for QSA decode #28213

Closed

ynankani mentioned this pull request Sep 3, 2026

Eval bug: CUDA/HIP: MoE models produce garbage output on RDNA3.5 (gfx1151) since #27621 #28113

Closed

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Sep 3, 2026

docs: credit ggml-org#27466 radix top-k, add related-upstream-work se…

bf43da2

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

pwilkin mentioned this pull request Sep 3, 2026

ROCm: resolve TOP_K kernels #28313

Open

Rhonstin mentioned this pull request Sep 4, 2026

CUDA: use radix TOP_K when CUB DeviceTopK is unavailable #28366

Closed

1 task done

ggerganov mentioned this pull request Sep 4, 2026

ggml : bump version to 0.23.0 ggml-org/ggml#1618

Merged

BrewTestBot mentioned this pull request Sep 4, 2026

ggml 0.23.0 Homebrew/homebrew-core#302393

Merged

1 task done

angeleitec-lang mentioned this pull request Sep 5, 2026

ROCm RPC server crashes on TOP_K with "invalid configuration argument" during distributed Qwen3.8-Flash-Next inference #27865

Open

dlsniper mentioned this pull request Sep 6, 2026

Misc. bug: backend sampler probe hardcodes a 1M-row width, turning ARGSORT's fallback check into a hard gate #28513

Open

thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026

ROCm: add radix TOP_K for long rows (ggml-org#27466)

1029527

* ROCm: add radix TOP_K for long rows

anthony-chaudhary mentioned this pull request Sep 8, 2026

perf(vulkan): keep top-k top-p temperature and penalties on the GPU anthony-chaudhary/fak#12402

Open

zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026

ROCm: add radix TOP_K for long rows (ggml-org#27466)

f8c9f4a

* ROCm: add radix TOP_K for long rows

pl752 pushed a commit to pl752/llama.cpp that referenced this pull request Sep 15, 2026

ROCm: add radix TOP_K for long rows (ggml-org#27466)

425bd33

* ROCm: add radix TOP_K for long rows

zsogitbe pushed a commit to zsogitbe/llama.cpp that referenced this pull request Sep 17, 2026

ROCm: add radix TOP_K for long rows (ggml-org#27466)

83c0d5c

* ROCm: add radix TOP_K for long rows

gen16k mentioned this pull request Sep 20, 2026

runtime: the Windows ROCm allowlist carries three families upstream lists only for Linux waired-ai/waired-agent#1266

Open

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Sep 28, 2026

docs: credit ggml-org#27466 radix top-k, add related-upstream-work se…

b68e4e2

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Sep 30, 2026

docs: credit ggml-org#27466 radix top-k, add related-upstream-work se…

0a538fd

…ction (ggml-org#25917, ggml-org#26419, ggml-org#27836)


		top_k_radix_init<<<(nrows + BLOCK_SIZE - 1) / BLOCK_SIZE, BLOCK_SIZE, 0, stream>>>(states, nrows, k);

		const dim3 histogram_grid(blocks_per_row, nrows);

Conversation

jadenmach2 commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Performance

Testing

Requirements

Uh oh!

ggml-gh-bot Bot commented Aug 21, 2026

Uh oh!

IMbackK left a comment

Choose a reason for hiding this comment

Uh oh!

IMbackK Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

IMbackK Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

IMbackK Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

fairydreaming commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR 27466 vs argsort (k = 1024)

PR 27466 vs argsort (k = 64)

PR 27466 vs DeviceTopK (k = 64)

Uh oh!

IMbackK commented Aug 27, 2026

Uh oh!

fairydreaming commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ORippler left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ORippler Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

ORippler Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

jadenmach2 Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Stoney49th commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zihaomu commented Aug 28, 2026

RDNA3.5 (gfx1151 / Strix Halo APU) datapoint — +1

Uh oh!

Stoney49th commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IMbackK commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IMbackK left a comment

Choose a reason for hiding this comment

Uh oh!

drluoto commented Aug 28, 2026

Uh oh!

IMbackK commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BoneHorror commented Sep 1, 2026

Uh oh!

IMbackK commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BoneHorror commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdanis commented Sep 1, 2026

Uh oh!

BoneHorror commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

jadenmach2 commented Aug 21, 2026 •

edited

Loading

fairydreaming commented Aug 27, 2026 •

edited

Loading

fairydreaming commented Aug 27, 2026 •

edited

Loading

ORippler left a comment •

edited

Loading

Stoney49th commented Aug 28, 2026 •

edited

Loading

Stoney49th commented Aug 28, 2026 •

edited

Loading

IMbackK commented Aug 28, 2026 •

edited

Loading

IMbackK commented Sep 1, 2026 •

edited

Loading

IMbackK commented Sep 1, 2026 •

edited

Loading

BoneHorror commented Sep 1, 2026 •

edited

Loading

BoneHorror commented Sep 2, 2026 •

edited

Loading