CUDA: qwen35 on RTX 5090 (sm_120) decodes at 76% of roofline on native Linux with MTP ~1.7x; Windows/Ollama path runs 1.5-1.6x slower at every draft depth

[Majesty401]

Summary

Token generation for the qwen35 architecture (hybrid: 48/64 Gated DeltaNet linear-attention layers + 16 full-attention) runs at ~28% of the memory-bandwidth bound on an RTX 5090 (sm_120, Windows), while the same GGUF reaches ~86% of the bound on an RTX 4090 (Ada, Linux). A MoE model on the identical rig/build decodes fast, so the GPU, driver, and Windows stack appear healthy — the deficit looks specific to the qwen35 decode path on consumer Blackwell.

Environment

• GPU: NVIDIA GeForce RTX 5090 (32 GB GDDR7, driver 616.56)
• OS: Windows 11 (build 26220)
• llama.cpp: b10741 (build 9d817213a), official release binaries llama-b10741-bin-win-cuda-13.3-x64.zip + matching cudart bundle
• Model: Qwen3.5-27B Q4_K_M GGUF, 15.65 GiB, arch reported as qwen35 27B Q4_K - Medium (pulled via the Ollama registry, tag qwen3.8:27b)

Measurements

llama-bench -m <model> -ngl 99 -fa 1 -p 1024 -n 400 (defaults otherwise, 5 reps):

model	test	t/s
qwen35 27B Q4_K_M	pp1024	1515.83 ± 47.03
qwen35 27B Q4_K_M	tg400	30.09 ± 0.44
qwen3moe 30B.A3B Q4_K_M (control, same rig)	pp1024	4081.20 ± 89.86
qwen3moe 30B.A3B Q4_K_M (control, same rig)	tg400	134.18 ± 1.84

Expected

Single-token decode on a dense(-per-token) 16.8 GB model is memory-bandwidth bound: 1792 GB/s ÷ 16.8 GB ≈ ~107 t/s ceiling. Measured 30.09 ≈ 28% of that.

Reference on Ada: an RTX 4090 (Linux, Ollama 0.32.x engine, byte-identical GGUF, speculative decoding disabled) decodes the same model at 48.8 t/s ≈ 86% of its own ~57 t/s ceiling. So the qwen35 path reaches near-roofline on sm_89 but not on sm_120.

Ruled out on this rig

• GPU state: P0 throughout decode, core ~3030–3050 MHz, memory clock at full 14014 MHz (28 Gbps), ~300 W, 93–98% utilization (sampled live once per second during tg).
• Host link: the rig happened to run at PCIe Gen5 x1 and later Gen5 x4 (marginal slot, being fixed) — tg was 27.95 vs 30.09 t/s respectively, so decode is not host-link-bound.
• Rig/stack health: the MoE control above (134 t/s) rules out any global Windows/driver per-token overhead large enough to explain the qwen35 number.
• Flash attention / KV cache type: FA on/off and KV q8_0 vs f16 were A/B-tested on the same GGUF (via Ollama's runner) — no material change (FA off is slightly worse).

Possibly related

Under Ollama's runner (which enables its multi-token-prediction path for this model), llama-server on this machine crashed twice at different draft_num_predict values with:

exit status 0xc0000409: stack-based buffer overrun
: CUDA error: shared object initialization failed

Mentioning it in case the sm_120 code path is shared; I can open a separate issue with debug logs if useful.

Happy to run traces (nsys/ncu), patched builds, or additional A/Bs on this hardware.

[Majesty401]

Correction + new data. The "ruled out: host link" claim in the report was wrong. The slot turned out to be physically degraded (marginal seating plus a motherboard lane-sharing cap from an M.2 drive), so both earlier states (Gen5 x1 and Gen5 x4) were deep inside a degraded regime — comparing them understated the link's role. After restoring a verified-stable Gen5 x16 (holds under load), same build (b10741), same settings (-ngl 99 -fa 1):

test	Gen5 x1	Gen5 x4	Gen5 x16
qwen35 27B Q4_K_M — pp1024	1479.76	1515.83	2326.82 ± 18.13
qwen35 27B Q4_K_M — tg400	27.95 ± 0.36	30.09 ± 0.44	47.59 ± 0.09
qwen3moe 30B.A3B Q4_K_M — pp1024	—	4081.20	6630.30 ± 110.14
qwen3moe 30B.A3B Q4_K_M — tg400	—	134.18 ± 1.84	218.75 ± 0.88

Observations:

1. Decode on Windows scales strongly with host-link bandwidth even with weights fully resident in VRAM, and for both architectures (dense-hybrid and MoE) — suggesting per-token host round trips are a first-order cost in this regime. The scaling is non-linear: x1→x4 moved tg only ~7%, x4→x16 moved it ~58%.
2. Run-to-run stddev collapsed at x16 (tg ±0.4 → ±0.09), so the degraded link was also the source of the noise.
3. The original gap persists at healthy x16, just smaller: qwen35 tg400 = 47.6 t/s ≈ 45% of the ~107 t/s memory-bandwidth bound, vs ~86% of its own bound for the same GGUF on the RTX 4090 (Linux) reference.

Remaining questions therefore: why plain decode for this arch on sm_120/Windows still sits ~2× below the bandwidth bound at x16, and why fully-resident decode needs host-link bandwidth at all here (the Ada/Linux reference reaches 86% of bound regardless).

Apologies for the churn — hardware state is now verified stable, and I'm happy to run traces or A/Bs on the corrected setup.

[Majesty401]

Third data point — same GPU, same Windows driver, Linux userland (WSL2). To separate the Windows userland/runtime from the driver, I ran the identical setup under WSL2 (Ubuntu, kernel 6.18 WSL2, GPU via /dev/dxg — so the Windows WDDM driver 616.56 is still underneath) with the Linux build of Ollama 0.33.2 (same version as the Windows install), same GGUF, same env (OLLAMA_FLASH_ATTENTION=1, OLLAMA_KV_CACHE_TYPE=q8_0), same harness (temp 1, num_ctx 32768, num_predict 400, warm, eval_count/eval_duration). PCIe Gen5 x16 throughout.

qwen35 27B Q4_K_M, tok/s (median)	Windows native	WSL2 (Linux userland, same WDDM driver)	RTX 4090, native Linux (reference)
draft_num_predict=0 (plain decode)	47.3	70.2 (3 runs: 70.2 / 69.1 / 71.2)	48.8
draft_num_predict=4 (MTP), temp 1	70.1	74.2 (5 runs: 66.0 / 74.2 / 75.8 / 68.9 / 74.8)	88.6
% of memory-bandwidth bound, plain decode	~45%	~69%	~86%

Two things stand out:

1. Plain decode recovers ~half its deficit just by moving to Linux userland — the driver is unchanged, so a large part of the Windows gap sits in the Windows-side runtime/runner path rather than in the CUDA kernels. What remains (69% vs 86% on Ada) could be the WDDM layer still underneath WSL2, or sm_120 kernel efficiency — a native Linux boot on this card would split those.
2. The speculative/MTP path barely helps on the 5090 in either OS (47→70 on Windows, 70→74 on Linux, i.e. it tops out around 70–75 regardless of the base rate), whereas the 4090 goes 48.8→88.6 under the same settings. That looks like the MTP verify path itself being capped on sm_120, independently of the plain-decode issue.

Happy to run any of this under a native Linux boot, or with traces, if that helps.

[jahnclawdmonet]

Native-Linux sm_120 datapoint, since the thread was missing that cell.

Setup: RTX PRO 6000 Blackwell Server Edition (sm_120, 96 GB GDDR7, 512-bit), x86_64 Linux 6.8, driver 590.48.01, CUDA 13.1, llama.cpp b10741 built from source (cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120; ggml's CMake rewrites 120 to 120a, the same SASS target as the release binaries; no source changes). Same command as your report, 5 repetitions. Under load: PCIe Gen5 x16, P0, SM clock 2,415 MHz, memory clock 12,481 MHz, ECC enabled. The binary prints build: unknown (0) because of a git safe.directory issue on the NFS checkout; git rev-parse HEAD on that checkout is 9d817213a0975020775efe6c458822616826f376, which is what tag b10741 resolves to.

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97252 MiB):
  Device 0: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, VRAM: 97252 MiB

$ llama-bench -m Qwen3.5-27B-Q4_K_M.gguf -ngl 99 -fa 1 -p 1024 -n 400 -r 5
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  15.58 GiB |    26.90 B | CUDA       |  99 |   1 |          pp1024 |      3649.20 ± 51.74 |
| qwen35 27B Q4_K - Medium       |  15.58 GiB |    26.90 B | CUDA       |  99 |   1 |           tg400 |         71.91 ± 0.38 |

$ llama-bench -m Qwen3.5-27B-Q4_K_M.gguf -ngl 99 -fa 0 -p 1024 -n 400 -r 5
| qwen35 27B Q4_K - Medium       |  15.58 GiB |    26.90 B | CUDA       |  99 |   0 |          pp1024 |       3626.24 ± 8.90 |
| qwen35 27B Q4_K - Medium       |  15.58 GiB |    26.90 B | CUDA       |  99 |   0 |           tg400 |         71.17 ± 0.11 |

$ llama-bench -m Qwen3-30B-A3B-Q4_K_M.gguf -ngl 99 -fa 1 -p 1024 -n 400 -r 5
| qwen3moe 30B.A3B Q4_K - Medium |  17.28 GiB |    30.53 B | CUDA       |  99 |   1 |          pp1024 |     10384.53 ± 67.95 |
| qwen3moe 30B.A3B Q4_K - Medium |  17.28 GiB |    30.53 B | CUDA       |  99 |   1 |           tg400 |        322.58 ± 0.34 |

Bandwidth: this card is not a 5090. 12,481 MHz on the 512-bit bus is 25 Gbps per pin and about 1,600 GB/s, matching the Server Edition's 1.6 TB/s spec (your 5090 reads 14014 MHz for 28 Gbps, same convention). Ceiling for this file (16,740,812,704 bytes) is 95.4 t/s, so tg400 is 75% of it. With ECC enabled the effective bandwidth is somewhat below the clock-derived figure, so 75% is a lower bound.

Time per token is the cleaner comparison across cards, because percent of bound penalizes the higher-bandwidth card for the same fixed cost. Using spec bandwidth and file bytes for every cell (your Windows cross-check put Ollama within 1% of llama-bench, so I am treating the harnesses as comparable):

cell	t/s	ms/token	streaming	other
PRO 6000, native Linux, llama-bench (this run)	71.91	13.9	10.5	3.4
4090, native Linux, Ollama (your reference)	48.8	20.5	16.6	3.9
5090, WSL2, Ollama (your comment 2)	70.2	14.2	9.3	4.9
5090, Windows, llama-bench (your comment 1, x16)	47.59	21.0	9.3	11.7

So on native Linux the non-streaming cost on this sm_120 card is in the same range as on the 4090; the percentage is lower here mainly because the streaming term is shorter. That is consistent with your comment 2 reading that the Windows-side per-token overhead dominates plain decode, rather than with a large sm_120 kernel penalty. If your 5090 behaved like this card on a native boot it would land around 9.3 + 3.4 to 3.9 ms, i.e. 75 to 79 t/s, roughly 71 to 73% of its 107 t/s ceiling. That is an estimate from one card with a different die (188 vs 170 SMs), memory clock and quant file, not a measurement of yours, and it says nothing about the MTP verify path, which llama-bench does not exercise.

The MoE control points the same way with the same caveat: 322.58 t/s here vs your 218.75 at x16 is 3.1 vs 4.6 ms per token on a model whose streaming term is about 1.2 ms, with 11% less bandwidth on this side but more SMs. One bound-model caveat that applies to every cell in the thread: the untied input embedding is gathered rather than streamed, so all these percentages are a few points optimistic; KV and DeltaNet state traffic are under 2% at 400 tokens.

Deviations from your setup: card, native Linux, driver 590.48.01 vs 616.56, a CUDA 13.1 source build vs your cu13.3 release binaries, and the GGUF is unsloth/Qwen3.5-27B-GGUF at revision 3221f178 (15.59 GiB, sha256 84b5f7f1...), not your Ollama-registry file (15.65 GiB), so tensor types may differ slightly.

What would separate a card-level 3.4 ms from a qwen35-path cost is a dense non-hybrid control of similar size on the same box (Qwen3-32B Q4_K_M with the same command). I did not have GPU time for it in this window; I can run it if that would be useful.

[jahnclawdmonet]

Ran the dense control on the same card. With the embedding table taken out of the streaming term, Qwen3-32B Q4_K_M has 3.21 ms per token that weight streaming doesn't explain; qwen35 has 3.88 ms. So on sm_120 the qwen35-specific share is about 0.67 ms of the 13.9 ms per token.

Same setup as my earlier comment (RTX PRO 6000 Blackwell Server Edition, native Linux, -ngl 99 -fa 1 -p 1024 -n 400, memory clock 12,481 MHz during decode, bound 1,597 GB/s, which I rounded to 1,600 before). ECC is on, so both streaming terms are slightly optimistic and the difference shifts by roughly 0.05 ms. Control model: unsloth/Qwen3-32B-GGUF at 931c8406, Q4_K_M, 19.8 GB; size and SHA-256 are in the gist. I rebuilt at 9d81721 so the build label resolves. The graphs-on qwen35 row is the earlier session's binary from the same commit and flags, the other two rows are the rebuilt one. Streaming bytes leave out token_embd (it's gathered per token; 715 MB in the Qwen3.5 file, 438 MB in Qwen3-32B) and the GGUF header.

model	reps	tg400 t/s	streaming ms/token	other ms/token
Qwen3-32B dense, Q4_K_M	3	65.35 ± 0.26	12.10	3.21
Qwen3.5-27B qwen35, Q4_K_M, graphs on (earlier session)	5	71.91 ± 0.38	10.03	3.88
Qwen3.5-27B qwen35, Q4_K_M, GGML_CUDA_DISABLE_GRAPHS=1	1	71.28	10.03	4.00

Graphs off didn't help. The 0.63 t/s gap is a single rep (the machine window closed before I could do more) and it sits inside the earlier run's spread, so I wouldn't read a cost into it either.

The 0.67 ms is a net difference between two models with different layer mixes and vocabularies, and the bias can go either way. The dense residual carries 64 flash-attention decode calls where qwen35 runs 16, while about 0.19 ms of the qwen35 residual is the GDN state itself (48 layers x 48 heads x 128 x 128 f32, read and written every token, 302 MB at 1,597 GB/s), which has to happen regardless. What's left for the hybrid kernels on this card is 0.3 to 0.5 ms per token. I'll take an nsys trace the next time the box is free and split that by kernel.

Raw stdout for the three runs, clock samples, build sha, checksum: https://gist.github.com/jahnclawdmonet/a6324f7f7488efadd46c3497fb6cf8c8

[Majesty401]

Thanks, this is the datapoint the thread was missing, and the dense control settles the question I opened with.

Taking your two comments together with my WSL2 run, the plain-decode picture reads like this:

• On sm_120 native Linux, qwen35 decodes at ~75% of the bandwidth ceiling, the same efficiency class as the 4090 (86%). The qwen35-specific share is ~0.67 ms/token, of which ~0.19 ms is the GDN state traffic that has to happen anyway. That is not a "3.5x below bound" kernel problem.
• The gap I reported is in the non-streaming residual: 3.4 ms/token on your card, 4.9 ms under WSL2 on my 5090, 11.7 ms on the same 5090 under Windows. Same GGUF, same b10741, same flags. The ~7 ms delta is Windows-side per-token overhead, and WSL2 recovers most of it because the Linux userland sits on top of the same WDDM driver.

So I withdraw the title claim. The plain-decode deficit is a Windows host-overhead observation on Blackwell, not a CUDA kernel issue for qwen35, and I am not sure it is llama.cpp's to fix. I can boot the 5090 from a live Linux USB to get the native number and close that half properly; I will post it when I have it.

What remains open, and what I would like to re-scope this issue to, is the MTP path, which llama-bench does not exercise. Same file, temperature 1, Ollama harness, medians over repeated runs:

card	OS	plain decode t/s	with MTP (draft 4) t/s	speedup
4090	native Linux	48.8	88.6	1.82x
5090	Windows	47.3	70.1	1.48x
5090	WSL2	70.2	74.2	1.06x

The 4090 nearly doubles with the draft head. The 5090 caps at ~75 t/s in both OSes regardless of where plain decode starts, which is why it ends up slower than a 4090 in daily use despite the 1.8x bandwidth. Acceptance rate should be a property of the model and the sampling, not the card, so either the verify step is paying a fixed per-token cost on sm_120 that the 4090 does not, or something in the draft/verify path is not overlapping the way it does on Ada. I have not separated those yet.

If you do get GPU time for an nsys trace on the hybrid kernels, an MTP run alongside it would be the most useful thing for this issue. On my side I will do the live-USB native run on the 5090, and a llama-server MTP comparison with the same build and flags on both hosts (the 4090 lives in a separate machine, so this stays a cross-host comparison like the table above, not a card swap), and report back.

Proposed new title: "CUDA: qwen35 MTP (draft 4) gives ~1.06x on RTX 5090 (sm_120) vs ~1.8x on RTX 4090, plain decode on native Linux is bandwidth-normal"

[jahnclawdmonet]

Before you run the llama-server MTP comparison on b10741: that build can't start --spec-type draft-mtp on any GGUF that carries an MTP head, your qwen3.8:27b blob included. 9d81721 (#28159) started loading n_layer_nextn before the per-layer arrays are read, so n_layer() already excludes the nextn layer at that point, while src/llama-model.cpp:1287-1296 still size n_head_arr and n_head_kv_arr with n_layer(). The MTP layer ends up with n_head_kv = 0, and the fit probe's MTP context dies in ggml/src/ggml.c:3953: GGML_ASSERT(a->ne[0] == b->ne[0]) failed while reserving the graph. #28173 fixed it five commits later by sizing the arrays with n_layer_all; the first release that contains the fix is b10749 (b10742 and b10743 are still broken). I hit this on an M4 with the 9B MTP GGUF. It is model-loading code, so it should fail the same way on the 5090; I haven't run it on CUDA myself.

Your Ollama blob already has blk.64.nextn.*, so no file change is needed on your side. The unsloth/Qwen3.5-27B-GGUF Q4_K_M I benchmarked earlier has no nextn tensors; the unsloth file with the head is Qwen3.5-27B-MTP-GGUF, a different model release from your blob.

[jahnclawdmonet]

Upstream draft-mtp on sm_120 under native Linux, on your exact blob (the qwen3.8:27b model layer, sha256 f5f1dd89…). Draft 4 gives 1.77x over plain decode here, 1.69x at temperature 1, so a ~75 t/s cap isn't intrinsic to the sm_120 kernels. This card is a 188-SM Server Edition at 1,597 GB/s and I ran f16 KV at -c 8192. Your 5090 has 170 SMs at 1,792 GB/s, and your MTP numbers went through q8_0 KV, 32k context, WDDM under both OSes and Ollama's runner, none of which I could vary here.

Build d11b3cc (b10741 plus five commits, includes #28173), llama-server -ngl 99 -fa on -c 8192 --parallel 1, MTP arm adds --spec-type draft-mtp --spec-draft-n-max 4. Requests through /v1/chat/completions, thinking off, 400 tokens, temperature 0 and 1 with seed 42, three prompts (prose, code, a numbered list), fresh server per arm; the remaining settings are in the gist. Plain decode is 67.7 t/s in this harness where llama-bench gave 71.9 above. That run used a different file and no chat prompt.

model	plain t/s	draft-4 t/s	speedup	accepted/drafted
your Ollama blob (Qwen3.8-27B-0814 Q4_K_M)	67.7	119.8	1.77x	0.47
unsloth Qwen3.5-27B-MTP Q4_K_M	68.8	131.0	1.90x	0.49
unsloth Qwen3.5-9B-MTP Q4_K_M	172.1	199.0	1.16x	0.41

Medians over 18 requests per arm, per-prompt tables in the gist. On your blob, prose runs 1.30 to 1.41x (acceptance 0.29 to 0.34), the numbered list 1.69 to 1.87x (0.46 to 0.53) and code 2.18 to 2.38x (0.68 to 0.75). Temperature 1 costs 5 to 8 points of acceptance on the 27B files.

The 9B file accepts 0.41 here on CUDA and 0.40 on an M4 with Metal (same prompts, build 159b741), where draft 4 is a 0.57x slowdown, so as far as this data goes acceptance depends on the model and the prompt. On the live-USB boot I'd read timings.draft_n_accepted / draft_n from llama-server first. Near 0.44 at temperature 1 with prompts like these means the missing speedup is in the step time.

The trace (llama-cli, your blob, the prose prompt, 128 tokens at temperature 0; the t/s llama-cli prints under nsys is not a throughput number) shows where the step time goes on this card. The verify pass runs every weight matrix as mul_mat_vec_q with ncols=5, and at that width mmvq.cu gives each block 2 warps and 2 rows. The big matrices run at 66 to 71% of the bandwidth bound at ncols=5 vs 84% single-column in the draft path, and the ncols=5 kernels add up to 15.6 ms per verify step where streaming the weights takes 10.5 ms. A plain step is 14.8 ms and an MTP step 24.5 ms, so about 5 ms of every step is that penalty and another 2.8 ms is the four single-column lm_head passes of the draft. Your card has fewer SMs and more bandwidth than this one, so the penalty should weigh more there. The lm_head cost is also why the 9B file barely gains. GDN is 1.5% of kernel time, the per-kernel table is in the gist.

The plain trace for the split you asked about is missing: another tenant's job landed on the GPU during that run, and the draft-depth sweep (1/2/3/6) didn't run for the same reason. I'll rerun both.

Raw logs, command lines, per-request JSON, kernel table, checksums: https://gist.github.com/jahnclawdmonet/84ee187a67a870aac4b6abf073979809

[jahnclawdmonet]

Both reruns are done: the plain trace for the kernel split, and the depth sweep on your blob.

Depth sweep on the same card and build as before (sm_120 Server Edition, d11b3cc), your
qwen3.8:27b model, llama-server -ngl 99 -fa on -c 8192 --parallel 1, MTP arms adding
--spec-type draft-mtp --spec-draft-n-max N. One prompt, temperature 0, seed 42, 400 tokens,
three reps per arm, a fresh server per arm, warmup excluded.

draft depth	median t/s	vs plain	accepted/drafted	tokens/step	ms/step	ms/token
plain	66.93	1.00x	-	1.00	14.90	14.90
1	87.70	1.31x	0.658	1.65	18.81	11.38
2	100.64	1.50x	0.549	2.08	20.67	9.92
3	100.86	1.51x	0.412	2.21	21.89	9.89
6	74.46	1.11x	0.230	2.36	31.55	13.38

Steps are predicted_n - draft_n_accepted, since each verify step emits exactly one token that
was not drafted. The plain row is measured in this session, and the earlier run's plain median on
this same prompt and temperature was 66.95 t/s at 14.84 ms/step: 0.03% off on rate, 0.4% on step
time, across two different nights. (The 67.7 t/s and 14.8 ms I quoted before are that harness's
median rate and aggregate step time over three prompts and two temperatures, not the matched
cell.)

I did not rerun depth 4 or 5 in this window, but the earlier run has depth 4 on the same prompt,
temperature, card, build and file: 94.31 t/s, acceptance 0.336, 2.33 tokens/step, 24.63 ms/step.
That slots between 3 and 6 at 1.41x. The peak is at 2 to 3, and depth 4 is already declining.
Depth 5 is the only depth in this range never measured on this file.

ms/token rises again after depth 3 because step time grows faster than tokens per step: across
depths 3 to 6 the step goes 21.89 to 31.55 ms while tokens per step barely moves, 2.21 to 2.36. On
an M4 with the 9B MTP file depth 1 is the only depth measured there that beats plain on the
all-prompt median; depth 2 is 0.92x overall and clears 1.0x only on the code prompt at temperature
0. I would sweep the range on the 5090 before settling on a depth.

It is one prompt, the prose one, and in the earlier three-prompt run prose was the blob's worst
case for MTP: 0.312 acceptance at depth 4 across both temperatures, 0.336 at temperature 0, and
the code prompt 0.714 (0.75 at temperature 0, 0.68 at temperature 1). The 1.77x at depth 4 I
posted earlier is a median over three prompts at two temperatures with aggregate acceptance 0.470,
which is not comparable with these single-prompt numbers.

Separately, at temperature 0 with the same seed, build and model file, the arms did not produce
the same text. Three output hashes, plain, then mtp1/2/3 identical to each other, then mtp6, at
completion lengths 1720, 1733 and 1744 characters for the same 400 tokens. The fresh server per
arm is not the cause: mtp1, mtp2 and mtp3 ran on three separate server processes and agree byte
for byte. Greedy speculative decoding is supposed to be output-identical to the target. This may
be worth a look independently of the timings, even if it turns out to be benign reduction-order
numerics.

The plain kernel split you asked for. nsys profile -t cuda,nvtx --cuda-graph-trace=node,
llama-cli, the same prompt, 128 tokens, temperature 0, --reasoning off, no --spec-type. 44
distinct kernel instantiations, 2180.5 ms of GPU kernel time. I said earlier that the t/s
llama-cli prints under nsys is not a throughput number, and this is the size of it: 58.5 t/s
printed here against the 66.93 t/s plain median above. 128 tokens at 58.5 t/s is 2188 ms of
generation wall against 2180.5 ms of kernel time, so under the profiler the GPU is busy
essentially the whole time and the absolute ms are inflated. Read the shares, recomputed from
Total Time (ns), not the ms.

share	ms	instances	kernel
72.9%	1589.8	54994	mul_mat_vec_q, ncols=1
7.0%	151.6	56482	quantize_q8_1
6.2%	134.5	39955	rms_norm_f32 and l2_norm_f32
1.3%	29.2	8384	cpy_scalar
1.1%	24.8	992	mul_mat_vec_q, ncols=2
0.8%	18.4	2160	flash_attn_ext_* plus stream-k fixup
0.6%	13.7	496	mul_mat_vec_q, ncols=4
10.0%	218.6	76034	everything else

The trace holds 131 forward passes, and the once-per-pass kernels agree on that from both halves
of the hybrid: the four flash_attn_ext_* variants sum to 2096 = 131 x 16 over the full-attention
layers, rope_multi and k_set_rows are 4192 = 131 x 32 (twice per attention layer), and
ssm_conv_f32 and gated_delta_net are 6288 = 131 x 48 over the GDN layers. 16 + 48 = 64, the
number of blocks the model loads; the loader logs blk.64.* as unused. Those variants split the
passes by batch size: 2032 = 127 x 16 at batch 1, 32 = 2 x 16 at batch 2, 16 = 1 x 16 at batch 4,
16 = 1 x 16 for one large-batch pass. 127 is -n 128 less the token that comes out of the prompt
pass. Three of the four ncols=1 kernels divide by 127 exactly (14224 = 127 x 112, 32512 = 127 x
256, 4064 = 127 x 32) and the fourth carries the residual, 4194 = 127 x 33 + 3, so 127 x 433 =
54991 of the 54994 launches are batch-1 decode and exactly three extra launches fall in the three
small-batch passes. The two batch-2 and one batch-4 passes give the ncols=2 and ncols=4 rows and
are almost certainly load-time warmup decodes. The prompt is the large-batch pass, and it runs a
different kernel: mul_mat_q, a separate 496 instances, not the ncols=4 row above. The
mul_mat_vec_q rows are decode only.

Decode here is almost entirely ncols=1 GEMV. That is the side that was missing when I posted the
MTP numbers, and the two traces are otherwise the same command: same model, prompt, 128 tokens,
temperature 0 and --reasoning off, differing only by --spec-type draft-mtp --spec-draft-n-max 4. They were taken on different nights and generate different text, so the 128 tokens are a
budget rather than identical tokens; over that budget the MTP trace spends 1401.1 ms of GPU kernel
time and this one 2180.5 ms. That is kernel time, not wall clock, so do not read it as the 1.41x
above. MTP runs those same weight matrices at ncols=5 where this trace runs them at ncols=1.

Raw logs, command lines, per-request JSON, the kernel table
(w7-nsys-plain-stats_cuda_gpu_kern_sum.csv) and checksums:
https://gist.github.com/jahnclawdmonet/84ee187a67a870aac4b6abf073979809

[Majesty401]

Follow-up #1 from my Sep 3 comment: native Linux on the RTX 5090 itself (Ubuntu 26.04.1 live USB, kernel 7.0.0-30, driver 610.43.02 open, CUDA 13.1, llama.cpp b10809, PCIe Gen5 x16 verified). Same qwen3.8:27b blob (sha256 f5f1dd89…) as every row so far.

llama-bench (-ngl 99 -fa 1, pp1024 / tg400):

• qwen35 27B Q4_K_M: 3971.8 / 81.82 t/s (fa 0: 3834.5 / 80.44)
• MoE control qwen3 30B-A3B: 12241 / 376.9

That is ~76 % of the 5090's 1792 GB/s roofline, matching @jahnclawdmonet's 75 % on the RTX PRO 6000. The same card under Windows (Ollama 0.33.3, same blob, x16) does 49.6 t/s plain, and the MoE control 218.8 under llama.cpp b10741 on Windows → Windows costs ~1.65× on both models.

llama-server draft-mtp sweep (-c 8192 -fa 1 --spec-type draft-mtp --spec-draft-n-max N, one prompt, temperature 1, 400 tokens, fresh server per depth, warm-up discarded, 4 runs, timings.draft_n/draft_n_accepted read as asked):

depth	median t/s	vs plain	accepted/drafted	tokens/step	forward passes/s
plain	80.8	1.00×	–	1.00	80.8
1	118.4	1.47×	0.66–0.75	1.66–1.75	68.2
2	136.2	1.69×	0.53–0.66	2.06–2.32	60.8
3	128.0	1.58×	0.45–0.56	2.36–2.68	52.0
4	121.7	1.51×	0.37–0.41	2.47–2.65	46.2

Forward passes/s (t/s ÷ tokens-per-step) is constant to 0.3 % inside each cell while raw t/s moves ±12 % with acceptance luck, so that is the row I compare on. The same card under Windows/Ollama gives 49.6 / 42.0 / 39.2 / 33.9 / 29.3 passes/s at depths 0–4 → native Linux is 1.5–1.6× faster at every depth, and MTP behaves exactly as on the PRO 6000.

So, on the 5090 itself: plain decode on native Linux is bandwidth-normal, MTP gives ~1.7× on native Linux, and the "~1.06×" in the title was a Windows/Ollama measurement (plus a sweep that skipped depths 2–3). I'll retitle to reflect that the residual is Windows-side, unless a maintainer prefers to close this as not-a-llama.cpp-bug. Follow-up #2 (4090 on the same b10809 build) still to come.

[Majesty401]

Follow-up #2: the RTX 4090 on the same build — llama.cpp b10809 (git 5266f24), CUDA 13.1, arch 89, driver 595.84, Ubuntu 26.04 host, built and run inside an nvidia/cuda:13.1.0-devel-ubuntu24.04 container, PCIe Gen5 x16, card otherwise empty. Same blob, same llama-server flags, prompt, temperature and run protocol as the 5090 row above.

llama-bench (-ngl 99 -fa 1, pp1024 / tg400): qwen35 27B Q4_K_M 3073.0 / 49.82 t/s (fa 0: 3005.9 / 49.53) ≈ 83 % of the 4090's 1008 GB/s roofline.

llama-server draft-mtp sweep, same protocol:

depth	4090 median t/s	vs plain	accepted/drafted	tokens/step	4090 passes/s	5090 native passes/s	5090 / 4090
plain	49.3	1.00×	–	1.00	49.3	80.8	1.64×
1	72.3	1.47×	0.64–0.82	1.64–1.82	41.9	68.2	1.63×
2	83.0	1.68×	0.56–0.63	2.12–2.26	38.2	60.8	1.59×
3	85.0	1.72×	0.44–0.58	2.33–2.73	34.8	52.0	1.49×
4	82.5	1.67×	0.39–0.44	2.57–2.73	31.5	46.2	1.47×

So with one build on both cards: the MTP gain is the same on Ada and Blackwell (1.72× vs 1.69× at each card's best depth), acceptance is the same (a property of the text, not the card), and the 5090 runs 1.47–1.64× the 4090's forward-pass rate against a 1.78× bandwidth ratio. The ratio narrowing with depth matches your nsys reading (the ncols>1 verify pass is less bandwidth-bound, so the bandwidth-richer card gains less from it). Nothing here is sm_120-specific.

For completeness, Ollama's Linux runner on the 4090 (my Sep 8 row: 48.75 / 41.4 / 38.0 / 34.5 / 31.35 passes/s at depths 0–4) is within 2 % of llama-server b10809, so the earlier 4090 numbers in this thread stand. The remaining gap is entirely the Windows path on the 5090 (49.6 → 80.8 t/s plain, same card, same link). From my side this issue is answered: the CUDA backend is fine on sm_120, and I'm happy for a maintainer to close it as not a llama.cpp bug, or to close it myself if that's preferred.

[Majesty401]

Correction — the Windows "1.5–1.6× slower" finding in the title is a measurement artifact, not an OS or engine deficit. I'm withdrawing it.

Every Windows number I posted here was taken from inside a running Electron app session (the Claude desktop app, streaming text) on the same machine. Windows' compositor (dwm) plus that app were holding ~35% of the 5090's 3D engine the whole time, and WDDM time-slices the CUDA context against them. The native-Linux run on Sep 12 was the only measurement where that app was absent — which is why it was the only clean number.

Today, same machine, same hour, Ollama 0.34.0 (llama.cpp runner), driver 616.64, PCIe Gen5 x16, same GGUF (sha256 f5f1dd89…), tides prompt, temp 1, 400 tok, 5 counted runs per cell after discarding the first:

desktop GPU load before the cell (nvidia-smi util / power)	plain decode, forward passes/s	draft-1	draft-2	draft-3	draft-4
2% / 64 W (app closed)	76.7	60.5	54.4	47.6	43.1
26% / 99 W (app open, idle)	55.4	44.9	39.9	35.4	31.8
34% / 106 W (app open, streaming)	47.5	39.5	34.6	31.5	27.9

Quiet Windows plain decode = 76.7 tok/s = 76% of the 5090's bandwidth roofline — the same efficiency as the native-Linux 80.8 (b10809) and as jahnclawdmonet's PRO 6000 (75%). MTP raw tok/s at draft-3: 125 (Windows quiet) vs 128 (native Linux draft-3) vs 88 on the 4090. Draft acceptance / mean accepted lengths are identical across all rows (1.72–1.81 / 2.19–2.31 / 2.57–2.82 / 2.79–2.91 for draft 1–4), so the whole effect is forward-pass throughput. Power draw fell with the desktop load (582 → 474 W at plain decode) — the card was being starved, not throttled.

So, for the record:

• RTX 5090 (sm_120) on this model is bandwidth-normal on both Linux and Windows. No sm_120 kernel issue, no Windows-stack penalty of the size I claimed.
• The earlier "MTP ~1.06× on the 5090" and "Windows 1.5–1.6× slower" statements were both this contention, compounded by the prompt/acceptance effects already discussed above.
• Lesson for anyone benchmarking on a Windows desktop: check nvidia-smi --query-gpu=utilization.gpu,power.draw reads ~0% before the first run, and don't drive the bench from a GPU-accelerated app on the same card.

Apologies for the noise. I don't see anything left here for llama.cpp to act on; happy for this to be closed.