Eval bug: Performance degradation since (PR #29622) with unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS and dual Intel B70

Name and Version

version: B11400, 0bb496d

Operating systems

Linux

GGML backends

SYCL

Hardware

VM CPU (32 Ryzen 9 5950X)
2x Intel B70

Models

unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS

Problem description & steps to reproduce

I'm getting a performance regression from this PR and every subsequent commit.

Settings (llama-server via llama-swap):

llama-server --port ${PORT} -dev SYCL0,SYCL1 -fa on -fit off -ts 1,1 -kvu -sm layer -ctxcp 3 -b 1024 -ub 2048 -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS -ctk q8_0 -ctv q8_0 -ngl 999 -lzm on -ot per_layer_token_embd.weight=CPU -lm mmap -c 196608 -np 1 --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 0.0 --min-p 0.00 --jinja --reasoning on --reasoning-effort xhigh --reasoning-preserve

test prompt: I gave it a small text file and asked it to summarize it.

PR #29612 (one previous to this): pp: ~420t/s tg: ~32t/s
PR #29622 (this one): pp: ~365t/s tg: ~24t/s

I've observed the same performance regression up to the current PR as of writing (PR #29971).

First Bad Commit

0bb496d

Relevant log output

Logs

see above

I have the same issue. Unfortunately I don't have a fix, I just want to join in.

llama-server.exe -m F:/models/Qwen3.8-27B-Q3_K_S/Qwen3.8-27B-Q3_K_S.gguf --mmproj F:/models/Qwen3.8-27B-Q3_K_S/mmproj-F16.gguf --image-min-tokens 1024 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 -c 65000 -fa on --batch-size 1024 --spec-type draft-mtp --spec-draft-n-max 2 -np 1 --reasoning-preserve --frequency-penalty 0.1 --reasoning-budget 8000 --reasoning-effort medium --host 0.0.0.0 --port 8080

Using an aggressive quantization + MTP on an RTX 5070 Ti, I have achieved ~70 t/s decode and ~1100 prompt processing on a month old build. I have updated today to the newest release, and with the above settings I am achieving ~5 t/s decode speed and 100-200 t/s pp.

Oddly enough, not using MTP makes it go back to 50 t/s decode.
I assume this is an issue either with the MTP implementation itself, or a CUDA issue. I've used CUDA 13.1 before, but I have also updated to CUDA 13.4 today - though I am not knowledgable about this topic, so I'm probably very wrong about this.

I ran into this myself, and did a lot more research. Please see my comment here #29622 (comment)

Hi there.

Same issue on my 3090 with Qwen3.8-27B-NGRAM-MTP-Q4_K_M decode tg t/s went from ~30 to ~19

Evidence (measured from the logs)

Metric	old `9f84af420`	new `db446c163` (has #29622)
decode tg	~30.2 -> 24.3 t/s	~19.3 t/s (flat)
prompt pp	~1000+ t/s	~963-1128 t/s (unchanged)
layers on GPU	63/66	62/66 (`--fit` dropped one)
CUDA0 compute buffer	970.02 MiB	1201.23 MiB (+231)
tg graph nodes	4519	4617 (+98)
tg graph splits	53	66 (+13)
tg input tensors	9	12 (+3)
MTP mean acc len	2.53 / 2.67 / 2.56	2.64 / 2.62 / 2.47 (same profile)
recurrent state	748 MiB	748 MiB (identical)

Conclusion: two compounding regressions, both visible in logs

PR #29622 (0bb496dbd, "llama: support both embd + raw tokens in batch").
The log delta matches it exactly: +3 input tensors are the new mixed_tokens / mixed_slots / mixed_embd (f32 [5120 x n_tokens]) inputs, and +98 nodes are the dup/set_rows/forward_select ops added per layer for mixed-batch-capable archs (qwen35 is in the default supported list). The splits also grew 53 -> 66, i.e. more host-side graph segments per step. MTP acceptance profile is unchanged, so this is pure per-step latency, not worse speculation the regression hits the MTP/nextn path specifically.

--fit demoted one layer to CPU as a knock-on effect.
The bigger reserved graph pushed the 63-layer footprint from 22028 MiB to 22182 MiB, past the 22070 MiB target, so fit settled on 62/66. One full transformer layer (~400M params) plus its KV now runs a serialized CPU pass inside every decode step - that alone can cost most of a 30 -> 20 t/s gap on this model.

llama.cpp-router-0bb496d.log

llama.cpp-router-9f84af4.log

Hi, this confirms the previous findings, but since I already went through the trouble of running git bisect I figured I'd add another datapoint. It's the same conclusion as the former posts.

I'm using RPC + pipeline parallelism (12 GB 5070ti laptop running llama-server, 10 GB 3080 over RPC, model layer-split across both, Qwen3.8-27B Q4_K_M). The PP probe graph reserves 761.59 MiB on the RPC side on the parent commit and 806.22 MiB on this one; the 44.63 MiB delta is what the card doesn't have left.

Details

build_inp_embd adds a third select branch (mixed_tokens, mixed_slots, mixed_embd 5120x256 f32 = 5.24 MiB) to every ubatch when the arch supports mixed batches:

const bool has_mixed = llm_arch_supports_mixed_batch(arch) && cparams.ctx_type == LLAMA_CONTEXT_TYPE_DEFAULT;

My understanding is that this is an architecture property, not a batch property, and mixed batches aren't emitted anywhere yet (mtmd puts embd rows in separate batches, TODO in the PR), so the branch isn't used yet in every workload. The select still puts it into the compute graph, and with pipeline parallelism the scheduler replicates every graph input per split in ggml_backend_sched_graph_copy - that input copied ~9 times is the 44.63 MiB.

context	parent (good)	`0bb496d` (bad)	delta
PP probe graph, RPC0	761.59 MiB	806.22 MiB	+44.63 MiB
main ctx, CUDA0	781.59 MiB	781.59 MiB	0
main ctx, CPU	259.60 MiB	259.60 MiB	0
spec ctx	identical	identical	0

I sched_reserve: RPC0 compute buffer size = 761.59 MiB                     (parent, succeeds)
E ggml_gallocr_reserve_n_impl: failed to allocate RPC0 buffer of size 845783552   (this commit, 806.22 MiB)

A possible fix is to gate it on the batch instead of the arch:

-    const bool has_mixed = llm_arch_supports_mixed_batch(arch) && cparams.ctx_type == LLAMA_CONTEXT_TYPE_DEFAULT;
+    const bool has_mixed = ubatch.is_mixed() && llm_arch_supports_mixed_batch(arch) && cparams.ctx_type == LLAMA_CONTEXT_TYPE_DEFAULT;

This restores the parent's graph for every batch that can occur: probe back to 761.59 MiB. I think this is what the first version of the PR did; the unconditional check only buys "no realloc when the batch kind switches", which can't happen until mtmd actually emits mixed batches which is the pending "TODO: handle this in mtmd" in the PR, but an unconditional check would matter once mtmd actually emits mixed batches hence why I'm not so sure about the quoted fix. It would work today, but would need a strategy later.

I have proposed this fix https://github.com/ggml-org/llama.cpp/pull/30151 as a better take from my first one.

I'm wondering if this is the right fix. I apologize in advance if I'm misunderstanding something; I'm not familiar with how this works (and the below reasoning is partially assisted by AI so it might be wrong). However, if I understand correctly, with this fix, if embd (il=-1) is present, it forces its backend to layer 0 via ggml_backend_sched_set_tensor_backend, preventing the small embedding ops (pad, scale) from being placed in the CPU split. This appears to fix the performance issue, and the PR's benchmarks support that. However:

The third select branch and its three inputs (mixed_tokens/mixed_slots/mixed_embd) are still present in every ubatch graph. PP input duplication (ggml_backend_sched_graph_copy) is unchanged, so the RPC0 probe (in my earlier example) still needs 806.22 MiB, and the memory issues there aren't fixed (I think?))
Only the embd node is pinned; the actual CPU ops (pad/scale/dup) are anonymous intermediates and may still be placed on CPU unless the scheduler groups them with embd. Also, what embd refers to depends on topology: with scale_tok != 1.0f, it's the final mul/scale rather than the select
If model.dev_layer(0) is CPU (low ngl), the embedding is pinned to CPU. This also runs for all models, including ones whose backend already matches

Thanks for spotting this issue. Please try #30160 and report your findings.

Checking my local benchmarks now... more shortly.

I'm still getting the same issue on #30160 as on 0bb496d:

E ggml_gallocr_reserve_n_impl: failed to allocate RPC0[192.168.50.1:50052] buffer of size 845783552
E graph_reserve: failed to allocate compute buffers
W sched_reserve: compute buffer allocation failed, retrying without pipeline parallelism

Here are the verbose log outputs:
log-2ca15f5-good.txt
log-0bb496d-bad.txt
log-4e67ebb-bad.txt

tried #30160 with the same file and prompt as before.

PP: 582.23 t/s
TG: 32.75 t/s

no doubt it incorporates a bunch of other fixes, which explains why it's even faster than the commit before #29622 .

Navigation Menu

Repository navigation

Eval bug: Performance degradation since (PR #29622) with unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS and dual Intel B70 #30033

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Evidence (measured from the logs)

Conclusion: two compounding regressions, both visible in logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Eval bug: Performance degradation since (PR #29622) with unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS and dual Intel B70 #30033

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

Reallondoner commented on Oct 6, 2026

deadprogram commented on Oct 8, 2026

ali0une commented on Oct 8, 2026

Evidence (measured from the logs)

Conclusion: two compounding regressions, both visible in logs

torsrex commented on Oct 8, 2026

deadprogram commented on Oct 8, 2026

torsrex commented on Oct 8, 2026

ggerganov commented on Oct 8, 2026

deadprogram commented on Oct 8, 2026

torsrex commented on Oct 8, 2026

moriz1 commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions