MTP feature performance drops since b9935

[Sailinnov]

Name and Version

b9935 - latest

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server.exe --alias "Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL" --model "models/Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.5 --host 0.0.0.0 --port 11434 --jinja --chat-template-kwargs "{"enable_thinking": true}" --chat-template-kwargs "{"preserve_thinking": true}" --ctx-size 131072 --batch-size 2048 --ubatch-size 2048 --threads 16 --threads-batch 32 --n-cpu-moe 30 --n-gpu-layers 42 --no-mmap --mlock --flash-attn on --cache-type-k q4_0 --cache-type-v q4_0 --temperature 0.6 --top-p 0.95 --top-k 20 --min-p 0.05 --repeat-penalty 1.0 --presence-penalty 0.0

Problem description & steps to reproduce

I previously applied b9180 which is the first release version supports MTP, btw I'm using qwen3.6-35b-a3b-mtp-ud-q4-k-xl. Recently I upgraded to new version b9934, then I find the version is much slower than b9180. Eventually I find the throughput drops by approx. 15-20% from this version b9235 (the version is merged into the main), wonder if someone got the same experience and can tell what happened in the "clean-up" activity.
My test environment:
CPU: AMD 9950X
GPU: 5070Ti
RAM: 96 GB
Windows 11

(although there's only 40 layers in this model, but after test I found 42 works faster, btw 99 won't help on --n-gpu-layers 42)
Comparison on b9222 vs b9235:
b9222: vram 13 GB used (start from scratch, delta vram before and after running llama-server), reply "hello" generation speed 68.4 t/s
b9235: vram 13.1 GB used, reply "hello" generation speed 63 t/s

the gap is more obvious in a long context task (a 20K tokens prompt test, code review task), generation speed is 61.3 t/s (output 4671 tokens) vs. 49.7 t/s (output 3730 tokens)

First Bad Commit

No response

Relevant log output

Logs

[pwilkin]

Any chance you could run git bisect to pinpoint the exact commit that caused the regression?

[pwilkin]

@am17an paging you if you want to take a look

[am17an]

I don't see a slowdown on my setup.

[am17an]

It's probably because #20793 was reverted, it has a large impact on windows IIRC

[Sailinnov]

Any chance you could run git bisect to pinpoint the exact commit that caused the regression?
since there's no pr between b9222 and b9235, so I can only tell b9235 got a bad commit.

[Kononnable]

If the bisect says b9222 and b9235 it may be because of #23280. Problem description and fix in #24110. (not related to MTP)

@Sailinnov
You can check on b9509 or later if performance is improved.
If it is, and you still get lower performance on b9935, you can start a bisect again, starting from that version

@edit I did commit comparison in the wrong order :)
last PR for b9235 was #23269, which was cleaning MTP

[alfredomariamilano]

I have noticed this too, but I thought it was just me changing models. I'll try to pinpoint where it actually happens. The biggest regression for me has been prompt processing though, it got basically halved.

[taurhine]

I've noticed this too on Linux, unfortunately didn't note the "good" build number before upgrading. Although t/s got a bit better in the later releases. There is another issue talking about a commit causing it #24266

[am17an]

git bisect to find the first bad commit if possible