ggml-cuda: assign four GDN state columns per warp by SongXiaoXi · Pull Request #30087 · ggml-org/llama.cpp

SongXiaoXi · 2026-10-07T05:25:17Z

Overview

The recurrent GDN kernel owns one state column per warp and reduces across rows with full-warp butterflies.

The kernel is instruction-issue bound (NCU, RTX 4090: DRAM 5%, issue rate 39%, zero shared-memory traffic): each warp executes ~76 instructions per token, of which the useful math is ~22 (12 FMA + 10 SHFL).

This PR assigns each column to half a warp and reduces over warp_size/2 lanes using the existing segmented warp_reduce_sum:

4 SHFL per reduction instead of 5, and one shuffle sequence now covers two columns (2.5x fewer shuffle instructions per column)
the per-token loop, addressing, and (for non-KDA) the scalar expf(g) are amortized over two columns
each lane holds twice as many rows (8 floats for S_v=128); memory traffic is unchanged

The launcher sizes the grid from the CTA width: z = S_v/(2*num_warps) halves when 4 warps per CTA fill the device, and the CTA shrinks to 2 or 1 warps when a small head count would otherwise leave SMs without a CTA. Applies to all S_v (16/32/64/128), KDA and non-KDA, aligned and unaligned, one-token, and snapshot (keep_rs) paths.
Numerics note: the reduction reassociates from a 32-lane to a 16-lane butterfly, so F32 accumulation order changes. No quantization, format, or algorithm change

Additional information

Qwen3.8-27B (UD-Q4_K_XL), RTX 4090 FE, -ctk q8_0 -ctv q4_0, same-process library-swap A/B against master (c479922):

test	master t/s	PR t/s	change
pp512	3075.24	3243.60	+5.5%
pp4096	3059.87	3221.08	+5.3%
tg128	47.62	47.67	+0.1%

Kernel level (nsys, pp4096): gated_delta_net_cuda<128> goes from 487 to 304 us per instance (-37.5%), 14.2% -> 9.4% of total kernel time; executed instructions -40%, issue rate 39% -> 45%.

Model perplexity: wikitext-2 test unchanged — 6.9610 ± 0.0450 (master) vs 6.9615 ± 0.0450 (PR), full test set (580 chunks, ctx 512).

Operator matrix (test-backend-ops perf 22/22 improve)

RTX 4090 FE

head_count	head_size	n_seq_tokens	n_seqs	v_repeat	permuted	kda	K	master us	PR us	change
32	128	64	1	1	0	0	1	40.41	26.94	-33.3%
32	128	256	1	1	0	0	1	157.40	101.46	-35.5%
32	128	512	1	1	0	0	1	313.22	200.75	-35.9%
32	128	1024	1	1	0	0	1	631.01	401.32	-36.4%
4	128	64	1	1	0	0	1	20.39	19.73	-3.2%
4	128	256	1	1	0	0	1	77.76	75.55	-2.8%
4	128	512	1	1	0	0	1	151.65	147.15	-3.0%
4	128	1024	1	1	0	0	1	301.45	292.34	-3.0%
32	128	64	1	1	0	1	1	47.23	33.19	-29.7%
32	128	256	1	1	0	1	1	185.19	127.78	-31.0%
32	128	64	1	1	0	0	3	41.30	28.08	-32.0%
32	128	256	1	1	0	0	3	162.30	105.37	-35.1%
32	128	64	1	1	0	1	3	47.67	33.55	-29.6%
32	128	64	1	1	1	0	1	40.40	26.86	-33.5%
32	128	64	1	1	1	1	1	47.14	33.37	-29.2%
32	128	64	2	1	0	0	1	75.93	51.17	-32.6%
16	128	64	1	2	0	0	1	40.40	26.94	-33.3%
8	32	64	1	1	0	0	1	16.76	16.13	-3.8%
4	16	64	1	1	0	0	1	15.05	14.73	-2.1%
32	128	1	1	1	0	0	1	2.56	2.26	-11.7%
32	128	1	1	1	0	1	1	2.59	2.25	-13.1%
16	64	1	1	1	0	0	1	1.33	1.32	-0.8%

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES - for benchmark code

SongXiaoXi · 2026-10-07T05:31:03Z

Note that #29353 targets GDN prefill too (the K==1 case) with a new chunked kernel. This one is on the recurrent kernel, which still runs decode, KDA, S_v != 128, and the K>1 MTP snapshots, is useful with or without it.

am17an

did you try cols_per_warp = 4 too?

SongXiaoXi · 2026-10-07T06:27:34Z

Yes, cols_per_warp=4 gives another ~4% when the grid still gets about 8 resident warps per SM, but it regresses up to 7.5% at head_count = 4 in test-backend-ops perf.

Would it be acceptable to select between the two compiled kernels (cols_per_warp=4 and 2) at runtime with that condition?

am17an · 2026-10-07T06:42:28Z

Are these e2e numbers for a particular model?

SongXiaoXi · 2026-10-07T08:03:04Z

Yes, it's Qwen3.8-27B (48 GDN value heads, UD-Q4_K_XL). I've now also measured Qwen3.5-9B (32 GDN value heads, Q4_K_M). Same setup for all rows: RTX 4090(locked to
its 2235 MHz base clock), -ctk q8_0 -ctv q4_0, -r 5

pp512 (t/s):

model	master	cols=2	cols=4	float4
Qwen3.8-27B	2714	2869 (+5.7%)	2936 (+8.2%)	2934 (+8.1%)
Qwen3.5-9B	8685	9178 (+5.7%)	9216 (+6.1%)	9346 (+7.6%)

pp4096 (t/s):

model	master	cols=2	cols=4	float4
Qwen3.8-27B	2694	2845 (+5.6%)	2914 (+8.2%)	2913 (+8.1%)
Qwen3.5-9B	8757	9264 (+5.8%)	9304 (+6.2%)	9412 (+7.5%)

By the way: the last column is a float4-vectorized variant of this kernel. The two directions don't stack (cols=4 measured ~2% slower on the vectorized kernel), so only one should go in — if the cols=4 dispatch looks like overkill, I'm happy to drop it and propose the vectorized variant as the follow-up instead.

am17an · 2026-10-07T08:14:12Z

Let's do the cols 4 variant for this head size. I'll run some tests too

am17an · 2026-10-07T17:05:37Z

+    // 4 columns per warp at S_v=128, 2 elsewhere (see the kernel); shrink the CTA when the
+    // wider CTA would leave SMs without a CTA, so small head counts keep the device filled
+    const int nsm = ggml_cuda_info().devices[ggml_cuda_get_device()].nsm;
+    const int cols_per_warp = S_v == 128 ? 4 : 2;


What is S_v for something like GLM 5.3 flash? Can you also do KDA coverage and we can test it there too. S_v = 4 is not a real world perf benchmark.

is cols_per_warp 4 is usually better than we just use that regardless of head size

S_v is 128 for both GLM-5.3-Flash (KDA, 64 heads, head_dim 128) and Kimi-Linear-48B-A3B (KDA, 32 heads, head_dim 128). For KDA coverage I ran a real KDA model(Kimi-Linear-48B-A3B), llama-bench on the 4090 with f16 KV:

test master t/s cols=2 t/s cols=4 t/s float4 t/s

pp512 6475.5 6666.6 (+3.0%) 6673.5 (+3.1%) 6701.7 (+3.5%)

pp4096 6373.8 6552.4 (+2.8%) 6591.4 (+3.4%) 6601.7 (+3.6%)

The only regression I've seen is the synthetic head_count = 4 case in test-backend-ops perf (+7.5%). Real GDN models all have 32-64 heads, so it shouldn't matter in practice.

Gate is dropped — cols_per_warp is a plain 4 for all S_v now. hs16/hs32 are fine with it as well (-6.8% / -5.7% at 64 tokens). GLM-5.3-Flash itself is 320B so I can't load it here; would be great if you can give it a spin there.

am17an · 2026-10-08T04:11:07Z

@ServeurpersoCom can you give it a go on GLM 5.3 flash?

ServeurpersoCom · 2026-10-08T05:33:02Z

@ServeurpersoCom can you give it a go on GLM 5.3 flash?

Tested GLM-5.3-Flash (UD-IQ1_S) on my RTX PRO 6000 Blackwell, with the usual FA on, q8_0 KV and -ub 2048 :

The GDN kernel itself is a lot faster in the model: each prefill call goes from 2.6 ms to 1.5 ms. Since GDN is only about 10% of the GPU time on this model, that ends up as +4% on prefill, at short and long context alike.
Decode doesn't move, the kernel is already around 2.4 us per call there.
Numerics look fine: the KLD against master is the same as just changing the ubatch size on master, and test-backend-ops passes.

LGTM from my side!

@SongXiaoXi few small nits, nothing blocking:

You could use a single constant for cols_per_warp instead of having the 4 in two places.
The static_assert message still says warp_size/2, it's warp_size/4 now.
The kernel comment describes the change compared to the old layout, it could just describe the current one.

am17an · 2026-10-08T06:33:10Z

@SongXiaoXi thanks! If you're looking to contribute more optimisations for this model or qwen4next would be appreciated

* ggml-cuda: assign two GDN state columns per warp * ggml-cuda: use 4 GDN state columns per warp at S_v=128 * ggml-cuda: default cols_per_warp=4 * ggml-cuda: address GDN review nits (cherry picked from commit 24e4183)

…-10-10) Conflicts: - ggml-backend.cpp: upstream moved input copies to ggml_backend_sched_copy_input with host weights last (ggml-org#29943); the GGML_SCHED_BATCH_INPUTS fast path now runs in the first loop before falling back to it. - ggml-cuda/fattn-common.cuh: launch_fattn takes upstream async_kv_preload, then our parallel_blocks_fixed (GQA-6 tile launcher updated). - ggml-cuda/fattn-mma-f16.cuh: upstream swizzle refactor (ggml-org#29612); its helpers take ncols2 like the rest of the fork's config getters. - ggml-cuda/gated_delta_net.cu: keep the fork's kernel and launch geometry (its columns_per_block template argument is not upstream's cols_per_warp, ggml-org#30087). - ggml-cuda/mmq.cu: upstream per-tile src1 padding (ggml-org#29953); ggml_cuda_mul_mat_q_src1_nbytes and the MMQ input cache use the worst-case padding so producers and projections with other tiles can share the buffer. - ggml-cuda/mmq-vec-dot.cuh: fork vec_dot kernels pass GGML_PREC_Q8 to the config helpers (ggml-org#30168). - ggml-cuda/mmvf.cu/.cuh: keep both includes and the HIP declarations, upstream warp_size argument. - ggml-cuda/rope.cu: upstream grid-stride rms_norm+mul+rope (ggml-org#28175) with the fork's M-RoPE theta (nchannels instead of gridDim.y) and rope_store_cast. - ggml-cuda/top-k.cu: upstream shape-based selection (ggml-org#28713), RDNA4 tournament kept for few long rows. - src/llama-batch.*: mixed token/embd batches (ggml-org#29622) copy embd rows per entry; other batches keep pointing at the ext rows. - src/llama-context.cpp: expert copy callback also on the second MTP scheduler. - src/llama-graph.cpp: build_rs keeps the GET_ROWS of the ubatch states (GDN fusions, deferred state) and gathers the extra states from the tail like upstream's custom getter path (ggml-org#29856).

* ggml-cuda: assign two GDN state columns per warp * ggml-cuda: use 4 GDN state columns per warp at S_v=128 * ggml-cuda: default cols_per_warp=4 * ggml-cuda: address GDN review nits

Upstream's four columns per warp (ggml-org#30087) serve prompts; on a P100 the qwen4exp decode shape (24 heads, S_v 128, 1-8 tokens) runs faster with two: 16.2 -> 11.7 us at T=3, 13.2 -> 8.4 at T=1 (one column: 16.6 / 11.8). Prompts keep four (T=64: 200 us against 227). GGML_CUDA_GDN_COLS overrides for measurements; test-backend-ops gets the decode shapes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* ggml-cuda: assign two GDN state columns per warp * ggml-cuda: use 4 GDN state columns per warp at S_v=128 * ggml-cuda: default cols_per_warp=4 * ggml-cuda: address GDN review nits

Upstream (ggml-org#30087) now assigns four state columns per warp for every call, with cols_per_warp as a kernel parameter. For decode and the 3-token MTP verify that is slower (4x fewer blocks), so the host picks four columns only from 4 tokens on (!KDA, S_v == 128, Ampere or newer) and keeps one column otherwise. This commit only adds that selection on top of upstream's kernel. Measured before the rebase on our own wide-column port (test-backend-ops perf, Bonsai shape 16 K heads x 3, d = 128, raw gates), us per call, 1 column -> 4 columns: 1 token 5.5 -> 5.5, 2: 8.4 -> 8.8, 3: 11.8 -> 12.1, 4: 14.0 -> 13.7, 8: 23.9 -> 19.0, 64: 172.6 -> 128.9; llama-bench pp512 1272 -> 1311, pp2048 1283 -> 1325 tok/s. To be re-measured on the rebased build. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ggml-cuda: assign two GDN state columns per warp

799870f

SongXiaoXi requested a review from a team as a code owner October 7, 2026 05:25

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Oct 7, 2026

am17an reviewed Oct 7, 2026

View reviewed changes

am17an self-assigned this Oct 7, 2026

ggml-cuda: use 4 GDN state columns per warp at S_v=128

80af759

am17an reviewed Oct 7, 2026

View reviewed changes

ggml-cuda: default cols_per_warp=4

a82115f

am17an approved these changes Oct 8, 2026

View reviewed changes

am17an added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 8, 2026

SongXiaoXi changed the title ~~ggml-cuda: assign two GDN state columns per warp~~ ggml-cuda: assign four GDN state columns per warp Oct 8, 2026

ServeurpersoCom self-requested a review October 8, 2026 05:17

ServeurpersoCom approved these changes Oct 8, 2026

View reviewed changes

ggml-cuda: address GDN review nits

d6c7e29

SongXiaoXi force-pushed the gdn-two-column-reduce branch from b769a73 to d6c7e29 Compare October 8, 2026 05:51

am17an approved these changes Oct 8, 2026

View reviewed changes

am17an merged commit 24e4183 into ggml-org:master Oct 8, 2026
15 of 17 checks passed

benjigill mentioned this pull request Oct 8, 2026

Sync upstream master (2026-10-08, ggml-org a11f57ba9) benjigill/llama.cpp#18

Merged

2 of 6 tasks

ORippler mentioned this pull request Oct 9, 2026

cuda, hip: GDN chunked kernel #29353

Merged

GenerelSchwerz mentioned this pull request Oct 10, 2026

cuda : specialize two-token gated delta net GenerelSchwerz/llama.cpp#163

Open

gaetan-puleo mentioned this pull request Oct 14, 2026

Rebase onto upstream llama.cpp c114bd4c5 (2026-10-14) halo-box/strix-llama.cpp#178

Closed

3 of 5 tasks

ggerganov mentioned this pull request Oct 15, 2026

ggml : bump version to 0.27.0 ggml-org/ggml#1659

Merged

BrewTestBot mentioned this pull request Oct 15, 2026

ggml 0.27.0 Homebrew/homebrew-core#318733

Merged

1 task done

Navigation Menu

Repository navigation

ggml-cuda: assign four GDN state columns per warp - #30087

ggml-cuda: assign four GDN state columns per warp#30087
am17an merged 4 commits into
ggml-org:masterfrom
SongXiaoXi:gdn-two-column-reduce

SongXiaoXi commented Oct 7, 2026

Uh oh!

SongXiaoXi commented Oct 7, 2026

Uh oh!

am17an left a comment

Uh oh!

SongXiaoXi commented Oct 7, 2026

Uh oh!

am17an commented Oct 7, 2026

Uh oh!

SongXiaoXi commented Oct 7, 2026 •

edited

Loading

Uh oh!

am17an commented Oct 7, 2026

Uh oh!

am17an Oct 7, 2026 •

edited

Loading

Uh oh!

SongXiaoXi Oct 8, 2026

Uh oh!

am17an commented Oct 8, 2026 •

edited

Loading

Uh oh!

ServeurpersoCom commented Oct 8, 2026

Uh oh!

Uh oh!

am17an commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

test	master t/s	cols=2 t/s	cols=4 t/s	float4 t/s
pp512	6475.5	6666.6 (+3.0%)	6673.5 (+3.1%)	6701.7 (+3.5%)
pp4096	6373.8	6552.4 (+2.8%)	6591.4 (+3.4%)	6601.7 (+3.6%)

Conversation

SongXiaoXi commented Oct 7, 2026

Overview

Additional information

Requirements

Uh oh!

SongXiaoXi commented Oct 7, 2026

Uh oh!

am17an left a comment

Choose a reason for hiding this comment

Uh oh!

SongXiaoXi commented Oct 7, 2026

Uh oh!

am17an commented Oct 7, 2026

Uh oh!

SongXiaoXi commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am17an commented Oct 7, 2026

Uh oh!

am17an Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SongXiaoXi Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

am17an commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented Oct 8, 2026

Uh oh!

Uh oh!

am17an commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SongXiaoXi commented Oct 7, 2026 •

edited

Loading

am17an Oct 7, 2026 •

edited

Loading

am17an commented Oct 8, 2026 •

edited

Loading