models: Qwen3.8-Flash-Next MTP by danielhanchen · Pull Request #28243 · ggml-org/llama.cpp

danielhanchen · 2026-09-02T12:30:25Z

Overview

Enables 1.3 to 2x faster MTP support for Qwen3.8-Flash-Next + shared MTP modules (re-uses embed_tokens to save disk space and VRAM / RAM). Builds on top of #27836

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Used Claude + local models for verification and testing

ggerganov · 2026-09-02T13:28:36Z

+        // target for a draft head that declares nextn_shared_target_tensors; must outlive this model
+        const struct llama_model * model_shared;


The llama_context_params already support ctx_other. Reuse that logic instead of adding this.

ggerganov · 2026-09-02T13:32:17Z

+// a captured graph hard-codes its shapes, so with one key per split an alternating shape
+// (a speculative verify batch) resets warmup forever. O(1) on purpose: walking nodes undoes the
+// point of a cuda graph. A shape this fails to separate re-captures as before, so it cannot regress.
+static uint64_t ggml_cuda_graph_get_key(ggml_cgraph * cgraph) {
+    // unlike the previous key this dereferences nodes[0], so an empty graph is not safe here
+    if (cgraph->n_nodes <= 0) {
+        return 0;
+    }
+
+    uint64_t key = (uint64_t) (uintptr_t) cgraph->nodes[0];
+
+    auto mix = [&key](uint64_t v) {
+        key = (key ^ v) * 0x100000001b3ull;
+    };
+
+    mix(cgraph->n_nodes);
+
+    for (int d = 0; d < GGML_MAX_DIMS; d++) {
+        mix(cgraph->nodes[0]->ne[d]);
+        mix(cgraph->nodes[cgraph->n_nodes - 1]->ne[d]);
+    }
+
+    return key;


Better keep the CUDA changes for a follow-up PR.

danielhanchen · 2026-09-02T14:42:24Z

@ggerganov thanks will check when I wake :))

jesdga95 · 2026-09-02T21:43:26Z

Works great on dual consumer GPUs (5090 + 5060 Ti, 32 GB RAM): +25-40%, acceptance 0.77

9950X, 32 GB DDR5, RTX 5090 + RTX 5060 Ti (gen5 x8/x8), Ubuntu, CUDA 13.3, PR @ 2857e51.
Target: unsloth UD-IQ4_XS (auto-fit, host experts partly in page cache). Draft: shared Q4_K_M sidecar, dn-max 2, q8 KV, 65k ctx.

config	t/s
no draft	36.4
MTP, 28k mixed gen (thinking + code)	45.3 avg
MTP, code sections	48-54
MTP, prose	27-28

Acceptance 0.767 (16849/21960), mean len 2.53.

One minor issue I encountered: the fitter can't measure the shared head (borrow_shared_tensor refusal at launch), so the draft loads unbudgeted and OOMs on the fuller GPU. Workaround: --fit-target to reserve room for it.

Thanks for all of your hard work!

llbzow · 2026-09-03T01:09:13Z

Validation: 2× RTX A6000 48GB (PCIe, no NVLink), Windows Server 2019, CUDA 12.6 (sm_86) — both heads draft at ~0.37 acceptance, no config beats baseline

Built this PR @ 2857e5114 with @JayToltTech's LLAMA_STATE_SEQ_FLAGS_ON_DEVICE checkpoint patch cherry-picked on top (applies cleanly, one commit).

Target: unsloth/Qwen3.8-Flash-Next-UD-Q3_K_XL, -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -ot per_layer_token_embd.weight=CPU -c 524288 (YaRN ×2) -t 24 -b 2048 -ub 512. No-spec baseline on identical prompts: 41–42.5 tok/s decode.

Arm 1 — shared Q4_K_M sidecar, -ngld 99 --spec-type draft-mtp --spec-draft-n-max 3: loads and drafts, but acceptance is only 0.32–0.38 (mean len ≈ 2.1), even with temp 0 / top_k 1. Decode 23–24 t/s @ temp 0.6, best greedy run 39.6 t/s — never beats baseline.

Arm 2 — same but without -ot per_layer_token_embd.weight=CPU: draft load crashes: invalid vector subscript immediately after loading draft model. So on this setup the shared-embed borrow path only works when the target's embed tensor is CPU-resident; GPU-resident embed breaks the load entirely. (Possibly related: arm 1's low acceptance could also be a symptom of the borrow path misbehaving with the UD-Q3_K_XL embed.)

Arm 3 — standalone Q8_0 grafted head (the drluoto file that gave 82–95% acceptance on #27836 @ 1d8de7c1, same target, same box): first OOM'd at 512K (draft KV cache wants 1 GiB on device 1; the 90 GB target leaves ~6–9 GiB free). Re-tested at -c 65536: loads, drafts, but acceptance is again 0.36–0.39 even greedy — same collapse as the shared head. Greedy decode 40–42 t/s ≈ baseline, temp 0.6 drops to 25–28 t/s.

Same draft file, same target, same prompts: 0.82–0.95 acceptance on #27836 vs ~0.37 on this branch — so something between 1d8de7c1 and 2857e5114 (this PR's changes or the newer post-#27941 master base) appears to regress draft-mtp acceptance on this hardware/quant combo. Happy to rerun any fix quickly; the box rebuilds in ~6 min.

Side note: draft compute buffer (2253 MiB) initially OOMs on device 1 next to the 90 GB target, recovered by retrying without pipeline parallelism — harmless here, but worth knowing for tight-VRAM multi-GPU setups.

The CUDA graph cache keyed captured graphs by the raw memory address of their first node (cgraph->nodes[0]). A captured graph hard-codes its shapes, but speculative decoding constantly alternates between different batch shapes (draft steps, verify batches, catch-up decodes) on the same context - when a new shape happens to reuse the same first- node address as a stale cached graph for a different shape, capture either reuses the wrong graph or thrashes, permanently resetting warmup instead of ever converging to steady-state replay. Hash node count and both endpoint tensors' shapes into the key instead (O(1) - walking all nodes would defeat the point of a CUDA graph), add LRU eviction capped at 64 graphs so the map can't grow unbounded now that distinct shapes get distinct entries. A shape this still fails to separate re-captures exactly as before, so it can't regress anything. Cherry-picked from ggml-org#28243 (open, unmerged), which found this while working on qwen4exp MTP performance - the effect is generic to any speculative-decoding workload on this fork, not qwen4exp-specific, so pulling in just this piece rather than the rest of that PR (which also reworks qwen4exp trunk/draft tensor sharing and doesn't fix the mixer/export-timing bugs already fixed on this branch). Verified: qwen4exp MTP speculative decoding still produces correct, byte-identical (temp 0) output after this change. Speed effect is hard to isolate cleanly from the dominant MoE-cache warmup effect already documented on this branch, but the fix is justified on its own correctness merits regardless of measured delta. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

sdroege · 2026-09-03T06:12:57Z

+        throw std::runtime_error(format("%s: this model is a draft head without its own '%s'; "
+                    "load it as a draft of its target model, not on its own", __func__, name.c_str()));


This triggers when loading with --fit on as the draft model is loaded independently as part of that. Everything still works though: it's later loaded again together with the target model.

jeverson-bmi · 2026-09-03T12:07:23Z

I have been trying this and the baseline from master actually has faster token gen than with MTP on, both shared and standalone. Using 5090+A6000(Ampere) + 128GB DDR5.

Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.

Adds --spec-type draft-mtp support for Qwen3.8-Flash-Next. The MTP head folds the next token's embedding into the trunk's wide hyper-connection residual, runs one trunk-style block (dense attention + MoE) over it, and collapses the result with its own mixer before reusing the trunk's LM head. - read nextn_predict_layers so n_layer() excludes the MTP block - load the trailing block through the existing trunk path: is_recr() and is_ple() are already false past the trunk, so it needs no special casing - eh_proj fuses the checkpoint's fc_embedding and fc_hidden side by side, so one matmul computes fc_embedding@e + fc_hidden@h - the head carries its own hyper-connection mixer, mirroring the trunk's hc_head_*, which stands in for the output norm qwen4exp does not have - export the wide pre-collapse residual as t_h_nextn from both graphs, so the driver can feed it back for the next draft step - route MTP contexts to a plain KV cache filtered to the trailing layer The draft block attends densely for now: the trunk's QSA only prunes context past a 2048-token budget, so dense is a numerical superset and drafts are verified either way. Indexer tensors are still loaded.

The MTP block is one trunk-shaped block (dense attention + MoE wrapped in hyper-connections) plus a head-level combiner, so once _QwenMtpMixin renames mtp.layers.0.* to the trailing block index its tensors ride the existing qwen4exp mappings unchanged. Two head-level pieces need handling: - fc_embedding and fc_hidden fuse into the eh_proj the shared NextN code expects, since W_e@e + W_h@h == [W_e|W_h] @ concat(e, h) - mtp.hyper_connection_mixer.* is the head's own copy of the trunk's hc_head_* output mixer, unindexed in the checkpoint and per-block in the GGUF compress_ratios is read with length block_count, so it gains a trailing 0 for the MTP block, which attends densely. --no-nextn drops the head; --mtp exports it on its own.

A NextN/MTP draft exported with --mtp carries the token embeddings, output norm and lm head so it can be loaded as a standalone model. For every current sidecar those three tensors are most of the file: ggml-org/Qwen3.8-27B-GGUF mtp-Qwen3.8-27B-Q4_0.gguf is 1.565 GiB, of which 1.332 GiB (85%) is the copy, against 0.223 GiB for the MTP block itself. Add an opt-in --mtp-shared-embd that leaves them out and marks the file with nextn_shared_target_tensors. The loader then resolves those names against the already loaded target model. The graph side needs no change: the nextn blocks of twelve archs already fall back to model.tok_embd and model.output. The borrow is gated on the new key, so a sidecar published before this change cannot reach it and keeps its current behaviour. Shapes are checked against the target and a mismatch is refused, as is loading such a file on its own.

_QwenMtpMixin is not a ModelBase subclass, so it re-declares the attributes it reads off cls for the type checker. filter_tensors reads cls.mtp_shared_embd without a matching declaration, which ty reports as unresolved-attribute. The declaration is a bare annotation, matching no_mtp and mtp_only above it. That creates no class attribute, so it cannot shadow ModelBase.mtp_shared_embd even though the mixin precedes the model class in the MRO; a default value here would have. Assisted-by: Claude

A draft-only export declares the full block count but ships the MTP block alone, so the trunk tensors load as null and only the MTP graph is buildable. Context reservation builds the trunk graph, which walked those nulls and segfaulted. A shared-embedding draft is caught earlier by the borrow check, since it has no token_embd of its own. A self-contained draft keeps one, so it passed that check and reached here. Assisted-by: Claude

Reuses the existing llama_context_params::ctx_other plumbing instead of adding a model_shared load parameter and a nextn_shared_target_tensors metadata key. A draft-only export now simply omits token_embd/output, and the qwen4exp MTP graph resolves them against the target context at graph build, following dflash and gemma4-assistant. Loading such a file on its own reports that it needs -md. Assisted-by: Claude

Assisted-by: Claude

gabrielfreire · 2026-09-04T14:32:35Z

Pulled the latest commits on this PR and started getting this error in the logs and it seems MTP wasn't activated

45.08.275.530 E decode: failed to initialize batch
45.08.275.531 E llama_decode: failed to decode, ret = -1
45.08.275.532 E spec        draft: llama_decode[1] returned -1
45.08.371.747 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84307
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84307
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.371.752 E decode: failed to initialize batch
45.08.371.753 E llama_decode: failed to decode, ret = -1
45.08.371.754 E spec        draft: llama_decode[1] returned -1
45.08.469.748 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84309
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84309
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.469.752 E decode: failed to initialize batch
45.08.469.752 E llama_decode: failed to decode, ret = -1
45.08.469.754 E spec        draft: llama_decode[1] returned -1
45.08.566.593 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84311
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84311
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.566.597 E decode: failed to initialize batch
45.08.566.598 E llama_decode: failed to decode, ret = -1
45.08.566.599 E spec        draft: llama_decode[1] returned -1
45.08.669.020 I slot print_timing: id  0 | task 17766 | n_gen =    100, tg =  15.11 t/s, tg_3s =  15.26 t/s
45.08.673.694 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84312
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84312
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.673.697 E decode: failed to initialize batch
45.08.673.698 E llama_decode: failed to decode, ret = -1
45.08.673.699 E spec        draft: llama_decode[1] returned -1
45.08.768.871 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84314
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84314
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.768.875 E decode: failed to initialize batch
45.08.768.876 E llama_decode: failed to decode, ret = -1
45.08.768.877 E spec        draft: llama_decode[1] returned -1
45.08.859.902 I slot print_timing: id  0 | task 17766 | prompt eval time =    6649.72 ms /   178 tokens (   37.36 ms per token,    26.77 tokens per second)
45.08.859.907 I slot print_timing: id  0 | task 17766 |        eval time =    6741.95 ms /   103 tokens (   66.10 ms per token,    15.13 tokens per second)
45.08.859.908 I slot print_timing: id  0 | task 17766 |       total time =   13391.67 ms /   281 tokens
45.08.859.909 I slot print_timing: id  0 | task 17766 |    graphs reused =      11160
45.08.859.913 I slot print_timing: id  0 | task 17766 | draft acceptance = 0.81395 (   35 accepted /    43 generated), mean len =  1.81
45.08.861.454 I slot      release: id  0 | task 17766 | stop processing: n_tokens = 84316, truncated = 0

A qwen4exp draft that borrows the target's embeddings sets ctx_other but keeps its own memory, so it must be caught up and rolled back like any other draft. Treating it as memory-shared skipped the catch-up decode and placed every draft token at the same position, which the M-RoPE position check rejects. Assisted-by: Claude

Wladastic · 2026-09-30T10:29:07Z

Using:
temp 1
TopP 0.95
TopK 20
MinP 0.0
PresencePenalty 0.0
RepeatPenalty 1.0
ctx 128,000
KvCacheQuant bf16

I run two different gpus RTX 4080 FE 16GB, RTX 5060 Ti 16GB, 96GB DDR4, AMD AM4 Ryzen 5950X

I get around 18-19tps decode with q4-q5 Quants without mtp, together with the standalone unsloth MTP Q4_M gguf I get 16-21 tps.
Q8 MTPs won't run.

I set the RTX 5060 Ti as the spec draft gpu.

Fit-target set to 1024,5192
Otherwise the mtp will not load.

Now the weird thing is, it doesn't matter what n-max is set, either the whole decode run has 16-18 tps or a bit over 20 tps.

I will try to make a new mtp gguf on my imatrix though, maybe the speedup will be more stable.

EDIT: With MTP, at least with this PR state here, the outputs have much worse quality compared to non-mtp master.
But I can't tell if it has to do with your changes here.

fat-tire · 2026-10-01T04:17:49Z

Looks like work is going on at #29761 now.

pwilkin · 2026-10-01T12:36:57Z

Superseded by #29761

Upstream merged replacements for ggml-org#28243 (ggml-org#29761 MTP), ggml-org#29030 (TENSOR_READ_LAZY + ggml-org#29599) and ggml-org#28213 (ggml-org#29751 QSA rewrite); the branch now carries only the two server checkpoint PRs plus devops. Assisted-by: opencode

Upstream landed its own Qwen4Exp MTP implementation (am17an's ggml-org#29761, c061df1) while this fork carried a port of danielhanchen's ggml-org#28243 ("models: Qwen3.8-Flash-Next MTP", branch qwen4exp/mtp) at 6fcaa16. ggml-org#28243 was closed unmerged on 2026-10-01, so the two implementations collided on the same tensor enums, mapping tables and loader/graph sites. Resolution: the fork's parallel Qwen4Exp MTP implementation is dropped in favour of the upstream one. - conversion/qwen4exp.py, gguf-py/gguf/constants.py, gguf-py/gguf/tensor_mapping.py, src/llama-arch.h, src/llama-model.h, src/models/models.h, src/models/qwen4exp.cpp: upstream's version. These were the conflicting files; every fork-side hunk in them came from the ggml-org#28243 port. - conversion/base.py, conversion/bailingmoe3.py, conversion/command_r.py, conversion/dots3.py, conversion/glm.py, conversion/qwen.py, convert_hf_to_gguf.py, src/llama-context.cpp, src/llama-arch.cpp: upstream's version. This also removes the fork-only --mtp-shared-embd / draft-only MTP export plumbing that shipped with ggml-org#28243. - common/speculative.cpp, src/llama-model.cpp: upstream's version plus the fork's own DFlash M-RoPE work (98b3e09), which is unrelated to MTP. - tests/test-server-prompt-cache.cpp: adapt the fork's own prompt-cache test to upstream's batch API. common_batch_add and the manual llama_batch_init/free pair are gone; the test now builds a common_batch and calls llama_process() with LLAMA_PROCESS_TYPE_DECODE. Fork-own work is preserved: RAM prompt-cache retention, selective CUDA P2P transport, server prefill/decode phase isolation, DFlash M-RoPE inference, docs and tests.

…, final numbers

A context keeps the built graph of each batch shape together with its scheduler split and allocation in a graph slot of the scheduler, so a speculative verification width seen before is made current again instead of being built, split and allocated anew. The slots are dropped when the compute buffers are planned again (a reserve, or a buffer that has to grow); when all slots are in use the least recently used shape is evicted. LLAMA_GRAPH_REUSE_SHAPES: not set = up to 24 shapes per context (a speculative server uses up to 16 verification widths, and the rest leaves room for a second graph variant of a width), 0 = off (one graph per output class, as before), 1 = 8, N > 1 = up to N (at most 64). It switches itself off, and logs the reason, when graph reuse is disabled, for a model loaded with no_alloc, with GGML_CUDA_GRAPH_OPT=1 and with pipeline parallelism. Measured with both switches set by environment on one build, R9700, 128 GB host memory, English roleplay (17 requests), MTP --spec-draft-n-max 7 --spec-draft-p-min 0.7 with the joint cache, exclusive expert cache 20480 MiB, unlimited host tier, warm, one run per row, off (LLAMA_GRAPH_REUSE_SHAPES=0, backend bound 16) against the defaults (24, 32): - Qwen3.8-Flash-Next UD-Q4_K_XL (MTP head of upstream pull request ggml-org#28243, Qwen indexer before upstream ggml-org#29751): 46.52 -> 51.67 t/s (+11.1 %) - DeepSeek V4 Flash UD-IQ3_XXS: 28.97 -> 31.45 t/s (+8.6 %) Replies 17/17 identical and acceptance unchanged in both. The extra time of the call after a width change fell from +3.6..+9.8 ms to about 0 (within 0.8 ms). Peak private bytes +0.85..0.97 GiB, dedicated VRAM unchanged.

The benchmark moves from one page to docs/ranma/benchmarks/, one record per measurement, named by the date and the topic, each naming at its top the release tag it measured, the upstream commit that release is based on and the date. The records keep their tables and numbers as measured; what they share is in method.md, and README.md collects the main numbers of each model with the release they come from. - 2026-10-05-exl3.md: Qwen3.8-Flash-Next in EXL3 3.05 and 4.05 bpw against UD-Q4_K_XL, measured for ranma_20261005 on the Radeon AI PRO R9700 with 128 and 64 GB and on the RX 9070 XT emulation with 64 GB, the expert cache exclusive: the llama-bench curves cold and warm (warm with --expert-prefill-swap), and the server scenarios with MTP and --spec-smart, warm from a store and a profile written by a warm-up preset, with GGML_CUDA_TOP_K_STABLE_TIES=1 and ROCBLAS_USE_HIPBLASLT=0 as in the MTP record; memory of every row. EXL3 3.05 bpw decodes faster than UD-Q4_K_XL on every system (llama-bench TG128 at depth 0 and 65536 +5.9 to +20.5 %, server decode with MTP +15.2 to +85.9 %), more so the fewer experts stay in VRAM and host memory; EXL3 4.05 bpw decodes like UD-Q4_K_XL in llama-bench on the R9700 and faster in the server scenarios. - 2026-10-01-mtp-smart.md: the ways of choosing the MTP draft length, measured for ranma_20261001 on the Radeon AI PRO R9700 with 128 GB for Qwen3.8-Flash-Next UD-Q4_K_XL and DeepSeek V4 Flash UD-IQ3_XXS: no MTP, MTP with the single-argument values tuned on this machine, --spec-smart from a cold start and from a warm store, in coding, roleplay in English, Korean, Chinese and Japanese, and a mixed scenario. One fresh server process per row after a 300 s rest, one build for all rows, greedy decoding. Smart is within a few percent of the single argument without per-model values; on DeepSeek V4 Flash it is 1.3 to 3.8 % slower in every roleplay scenario, cold and warm. - 2026-09-29-graph-reuse-default.md: graph reuse per batch shape turned on by default. - 2026-09-28-release.md: ranma_20260928 against upstream: the server scenarios for DeepSeek V4 Flash with and without MTP and for Qwen3.8-Flash-Next, cold and warm, with 64 GB rows, against upstream and the previous release; the llama-bench depth curves; Gemma 4 31B Q4_K_M as the dense reference; the measured effect of the options that are off in the main tables; memory. - 2026-09-28-qwen-mtp-n1.md: Qwen3.8-Flash-Next MTP n1 in English roleplay with the MTP head of upstream pull request ggml-org#28243. - 2026-09-22-full-set.md: the full set measured once at ranma_20260922. The Benchmarks section of the README states the upstream base of the release and keeps four tables, each with the release it was measured at: EXL3 against UD-Q4_K_XL (ranma_20261005), the decode ratios of the server scenarios against upstream (ranma_20260928), the MTP draft length comparison (ranma_20261001) and the llama-bench rows of the two systems (ranma_20260922). In docs/ranma/README.md the entry of the smart draft length says that it is on by default with draft-mtp and how to return to the single-argument rule, and its entries of the graphs per batch shape give the bound of 32 backend graphs and the llama graph per batch shape, on by default, with its measured gain. The links of the RDNA4 prefill page point to the method page.

Upstream PR ggml-org#28243 (head 6fcaa16, open), applied as one commit on this release for measurement. It registers the qwen4exp NextN tensors, builds the NextN/MTP draft graph, lets the conversion export a draft-only head that borrows the target's token embeddings and LM head (loaded as -md through ctx_other), and keeps the target KV cache shared only for gemma4-assistant drafts. Co-authored-by: Ryan Monsurate <ryanmonsurate@users.noreply.github.com>

The benchmark moves from one page to docs/ranma/benchmarks/, one record per measurement, named by the date and the topic, each naming at its top the release tag it measured, the upstream commit that release is based on and the date. The records keep their tables and numbers as measured; what they share is in method.md, and README.md collects the main numbers of each model with the release they come from. - 2026-10-01-mtp-smart.md: the ways of choosing the MTP draft length, measured for ranma_20261001 on the Radeon AI PRO R9700 with 128 GB for Qwen3.8-Flash-Next UD-Q4_K_XL and DeepSeek V4 Flash UD-IQ3_XXS: no MTP, MTP with the single-argument values tuned on this machine, --spec-smart from a cold start and from a warm store, in coding, roleplay in English, Korean, Chinese and Japanese, and a mixed scenario. One fresh server process per row after a 300 s rest, one build for all rows, greedy decoding. Smart is within a few percent of the single argument without per-model values; on DeepSeek V4 Flash it is 1.3 to 3.8 % slower in every roleplay scenario, cold and warm. - 2026-09-29-graph-reuse-default.md: graph reuse per batch shape turned on by default. - 2026-09-28-release.md: ranma_20260928 against upstream: the server scenarios for DeepSeek V4 Flash with and without MTP and for Qwen3.8-Flash-Next, cold and warm, with 64 GB rows, against upstream and the previous release; the llama-bench depth curves; Gemma 4 31B Q4_K_M as the dense reference; the measured effect of the options that are off in the main tables; memory. - 2026-09-28-qwen-mtp-n1.md: Qwen3.8-Flash-Next MTP n1 in English roleplay with the MTP head of upstream pull request ggml-org#28243. - 2026-09-22-full-set.md: the full set measured once at ranma_20260922. The README states the upstream base of the release and keeps three tables, each with the release it was measured at: the decode ratios of the server scenarios against upstream (ranma_20260928), the MTP draft length comparison (ranma_20261001) and the llama-bench rows of the two systems (ranma_20260922). Its entry of the smart draft length says that it is on by default with draft-mtp and how to return to the single-argument rule, and its entries of the graphs per batch shape give the bound of 32 backend graphs and the llama graph per batch shape, on by default, with its measured gain. The links of the RDNA4 prefill page point to the method page.

Upstream PR ggml-org#28243 (head 6fcaa16, open), applied as one commit on this release for measurement. It registers the qwen4exp NextN tensors, builds the NextN/MTP draft graph, lets the conversion export a draft-only head that borrows the target's token embeddings and LM head (loaded as -md through ctx_other), and keeps the target KV cache shared only for gemma4-assistant drafts. Co-authored-by: Ryan Monsurate <ryanmonsurate@users.noreply.github.com>

…l-org#28243) The branch ranma_20260928_qwen_mtp is the release ranma_20260928 with upstream pull request ggml-org#28243 (head 6fcaa16, not merged into master) applied as one commit. Adds its English roleplay rows to the benchmark page and the README: MTP n1 with the joint cache on and off against the same build without MTP.

A context keeps the built graph of each batch shape together with its scheduler split and allocation in a graph slot of the scheduler, so a speculative verification width seen before is made current again instead of being built, split and allocated anew. The slots are dropped when the compute buffers are planned again (a reserve, or a buffer that has to grow); when all slots are in use the least recently used shape is evicted. LLAMA_GRAPH_REUSE_SHAPES: not set = up to 24 shapes per context (a speculative server uses up to 16 verification widths, and the rest leaves room for a second graph variant of a width), 0 = off (one graph per output class, as before), 1 = 8, N > 1 = up to N (at most 64). It switches itself off, and logs the reason, when graph reuse is disabled, for a model loaded with no_alloc, with GGML_CUDA_GRAPH_OPT=1 and with pipeline parallelism. Measured with both switches set by environment on one build, R9700, 128 GB host memory, English roleplay (17 requests), MTP --spec-draft-n-max 7 --spec-draft-p-min 0.7 with the joint cache, exclusive expert cache 20480 MiB, unlimited host tier, warm, one run per row, off (LLAMA_GRAPH_REUSE_SHAPES=0, backend bound 16) against the defaults (24, 32): - Qwen3.8-Flash-Next UD-Q4_K_XL (MTP head of upstream pull request ggml-org#28243, Qwen indexer before upstream ggml-org#29751): 46.52 -> 51.67 t/s (+11.1 %) - DeepSeek V4 Flash UD-IQ3_XXS: 28.97 -> 31.45 t/s (+8.6 %) Replies 17/17 identical and acceptance unchanged in both. The extra time of the call after a width change fell from +3.6..+9.8 ms to about 0 (within 0.8 ms). Peak private bytes +0.85..0.97 GiB, dedicated VRAM unchanged.

The benchmark moves from one page to docs/ranma/benchmarks/, one record per measurement, named by the date and the topic, each naming at its top the release tag it measured, the upstream commit that release is based on and the date. The records keep their tables and numbers as measured; what they share is in method.md, and README.md collects the main numbers of each model with the release they come from. - 2026-10-05-exl3.md: Qwen3.8-Flash-Next in EXL3 3.05 and 4.05 bpw against UD-Q4_K_XL, measured for ranma_20261005 on the Radeon AI PRO R9700 with 128 and 64 GB and on the RX 9070 XT emulation with 64 GB, the expert cache exclusive: the llama-bench curves cold and warm (warm with --expert-prefill-swap), and the server scenarios with MTP and --spec-smart, warm from a store and a profile written by a warm-up preset, with GGML_CUDA_TOP_K_STABLE_TIES=1 and ROCBLAS_USE_HIPBLASLT=0 as in the MTP record; memory of every row. EXL3 3.05 bpw decodes faster than UD-Q4_K_XL on every system (llama-bench TG128 at depth 0 and 65536 +5.9 to +20.5 %, server decode with MTP +15.2 to +85.9 %), more so the fewer experts stay in VRAM and host memory; EXL3 4.05 bpw decodes like UD-Q4_K_XL in llama-bench on the R9700 and faster in the server scenarios. - 2026-10-01-mtp-smart.md: the ways of choosing the MTP draft length, measured for ranma_20261001 on the Radeon AI PRO R9700 with 128 GB for Qwen3.8-Flash-Next UD-Q4_K_XL and DeepSeek V4 Flash UD-IQ3_XXS: no MTP, MTP with the single-argument values tuned on this machine, --spec-smart from a cold start and from a warm store, in coding, roleplay in English, Korean, Chinese and Japanese, and a mixed scenario. One fresh server process per row after a 300 s rest, one build for all rows, greedy decoding. Smart is within a few percent of the single argument without per-model values; on DeepSeek V4 Flash it is 1.3 to 3.8 % slower in every roleplay scenario, cold and warm. - 2026-09-29-graph-reuse-default.md: graph reuse per batch shape turned on by default. - 2026-09-28-release.md: ranma_20260928 against upstream: the server scenarios for DeepSeek V4 Flash with and without MTP and for Qwen3.8-Flash-Next, cold and warm, with 64 GB rows, against upstream and the previous release; the llama-bench depth curves; Gemma 4 31B Q4_K_M as the dense reference; the measured effect of the options that are off in the main tables; memory. - 2026-09-28-qwen-mtp-n1.md: Qwen3.8-Flash-Next MTP n1 in English roleplay with the MTP head of upstream pull request ggml-org#28243. - 2026-09-22-full-set.md: the full set measured once at ranma_20260922. The Benchmarks section of the README states the upstream base of the release and keeps four tables, each with the release it was measured at: EXL3 against UD-Q4_K_XL (ranma_20261005), the decode ratios of the server scenarios against upstream (ranma_20260928), the MTP draft length comparison (ranma_20261001) and the llama-bench rows of the two systems (ranma_20260922). In docs/ranma/README.md the entry of the smart draft length says that it is on by default with draft-mtp and how to return to the single-argument rule, and its entries of the graphs per batch shape give the bound of 32 backend graphs and the llama graph per batch shape, on by default, with its measured gain. The links of the RDNA4 prefill page point to the method page. ranma_20261007 adds no record; the README and the index say that its numbers come from the earlier releases they name.

A context keeps the built graph of each batch shape together with its scheduler split and allocation in a graph slot of the scheduler, so a speculative verification width seen before is made current again instead of being built, split and allocated anew. The slots are dropped when the compute buffers are planned again (a reserve, or a buffer that has to grow); when all slots are in use the least recently used shape is evicted. LLAMA_GRAPH_REUSE_SHAPES: not set = up to 24 shapes per context (a speculative server uses up to 16 verification widths, and the rest leaves room for a second graph variant of a width), 0 = off (one graph per output class, as before), 1 = 8, N > 1 = up to N (at most 64). It switches itself off, and logs the reason, when graph reuse is disabled, for a model loaded with no_alloc, with GGML_CUDA_GRAPH_OPT=1 and with pipeline parallelism. Measured with both switches set by environment on one build, R9700, 128 GB host memory, English roleplay (17 requests), MTP --spec-draft-n-max 7 --spec-draft-p-min 0.7 with the joint cache, exclusive expert cache 20480 MiB, unlimited host tier, warm, one run per row, off (LLAMA_GRAPH_REUSE_SHAPES=0, backend bound 16) against the defaults (24, 32): - Qwen3.8-Flash-Next UD-Q4_K_XL (MTP head of upstream pull request ggml-org#28243, Qwen indexer before upstream ggml-org#29751): 46.52 -> 51.67 t/s (+11.1 %) - DeepSeek V4 Flash UD-IQ3_XXS: 28.97 -> 31.45 t/s (+8.6 %) Replies 17/17 identical and acceptance unchanged in both. The extra time of the call after a width change fell from +3.6..+9.8 ms to about 0 (within 0.8 ms). Peak private bytes +0.85..0.97 GiB, dedicated VRAM unchanged.

The benchmark moves from one page to docs/ranma/benchmarks/, one record per measurement, named by the date and the topic, each naming at its top the release tag it measured, the upstream commit that release is based on and the date. The records keep their tables and numbers as measured; what they share is in method.md, and README.md collects the main numbers of each model with the release they come from. - 2026-10-09-release.md: the release table of ranma_20261009 (upstream 4625240): Qwen3.8-Flash-Next, DeepSeek V4 Flash and GLM-5.3 Flash, each in EXL3 and in a general GGUF, llama-bench depth 0 warm on the R9700 and the RX 9070 XT emulation with 128, 64 and 32 GB, against upstream on the general GGUF with -ncmoe. The 64 and 32 GB rows hold the physical memory down with locked pages so that the file tier reads from the SSD; upstream cannot load there. For the EXL3 files on the R9700 with 128 GB it also has llama-bench at depth 0, 8192 and 32768 cold and warm, the server scenarios with MTP and --spec-smart, and memory of every row. TG128 at depth 0 against upstream at 128 GB: x2.87 to x3.48. - 2026-10-05-exl3.md: Qwen3.8-Flash-Next in EXL3 3.05 and 4.05 bpw against UD-Q4_K_XL, measured for ranma_20261005 on the Radeon AI PRO R9700 with 128 and 64 GB and on the RX 9070 XT emulation with 64 GB, the expert cache exclusive: the llama-bench curves cold and warm (warm with --expert-prefill-swap), and the server scenarios with MTP and --spec-smart, warm from a store and a profile written by a warm-up preset, with GGML_CUDA_TOP_K_STABLE_TIES=1 and ROCBLAS_USE_HIPBLASLT=0 as in the MTP record; memory of every row. EXL3 3.05 bpw decodes faster than UD-Q4_K_XL on every system (llama-bench TG128 at depth 0 and 65536 +5.9 to +20.5 %, server decode with MTP +15.2 to +85.9 %), more so the fewer experts stay in VRAM and host memory; EXL3 4.05 bpw decodes like UD-Q4_K_XL in llama-bench on the R9700 and faster in the server scenarios. - 2026-10-01-mtp-smart.md: the ways of choosing the MTP draft length, measured for ranma_20261001 on the Radeon AI PRO R9700 with 128 GB for Qwen3.8-Flash-Next UD-Q4_K_XL and DeepSeek V4 Flash UD-IQ3_XXS: no MTP, MTP with the single-argument values tuned on this machine, --spec-smart from a cold start and from a warm store, in coding, roleplay in English, Korean, Chinese and Japanese, and a mixed scenario. One fresh server process per row after a 300 s rest, one build for all rows, greedy decoding. Smart is within a few percent of the single argument without per-model values; on DeepSeek V4 Flash it is 1.3 to 3.8 % slower in every roleplay scenario, cold and warm. - 2026-09-29-graph-reuse-default.md: graph reuse per batch shape turned on by default. - 2026-09-28-release.md: ranma_20260928 against upstream: the server scenarios for DeepSeek V4 Flash with and without MTP and for Qwen3.8-Flash-Next, cold and warm, with 64 GB rows, against upstream and the previous release; the llama-bench depth curves; Gemma 4 31B Q4_K_M as the dense reference; the measured effect of the options that are off in the main tables; memory. - 2026-09-28-qwen-mtp-n1.md: Qwen3.8-Flash-Next MTP n1 in English roleplay with the MTP head of upstream pull request ggml-org#28243. - 2026-09-22-full-set.md: the full set measured once at ranma_20260922. The Benchmarks section of the README states the upstream base of the release, opens with the release table at depth 0, and keeps four tables, each with the release it was measured at: EXL3 against UD-Q4_K_XL (ranma_20261005), the decode ratios of the server scenarios against upstream (ranma_20260928), the MTP draft length comparison (ranma_20261001) and the llama-bench rows of the two systems (ranma_20260922). In docs/ranma/README.md the entry of the smart draft length says that it is on by default with draft-mtp and how to return to the single-argument rule, and its entries of the graphs per batch shape give the bound of 32 backend graphs and the llama graph per batch shape, on by default, with its measured gain. The links of the RDNA4 prefill page point to the method page. method.md describes the physical memory hold of the 64 and 32 GB rows and the 600 s cap of the temperature gate.

Volunteer-1 · 2026-10-12T17:52:40Z

On my dual RTX 3090 rig, this PR by @danielhanchen is far superior to the merged PR #29761. Daniel's PR gives 18% faster prefill and 47% faster decode for the decode-optimal setting ub 512.

Input prompt: Analyze the file src\models\qwen4exp.cpp
(23535 tokens)

Commands used:

For Daniel's PR:
llama-server.exe -m Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf -md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -np 1 --spec-type draft-mtp --spec-draft-n-max 2 --no-kv-unified --cpu-strict 1 -C 55550000 -t 8 -Cb 55555555 -tb 16 -fa on -ctk q8_0 -ctv q8_0 -ncmoe 23 -ctkd q8_0 -ctvd q8_0 -cmoed -ngl auto -sm layer -ts 34,16 -mg 1 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat_penalty 1.0 -lm dio -b 512 -ub 512 -lv 4 -lzm off -ot "per_layer_token_embd.weight=CPU""

speed: prefill 221.45 t/s, tg2090 14.18 t/s

For PR 29761:
llama-server.exe -m Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf -np 1 --spec-type draft-mtp --spec-draft-n-max 2 --no-kv-unified --cpu-strict 1 -C 55550000 -t 8 -Cb 55555555 -tb 16 -fa on -ctk q8_0 -ctv q8_0 -ncmoe 26 -ctkd q8_0 -ctvd q8_0 -cmoed -ngl auto -sm layer -ts 35,15 -mg 1 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat_penalty 1.0 -lm dio -b 512 -ub 512 -lv 4 -lzm off -ot "per_layer_token_embd.weight=CPU""

speed: prefill 187.68 t/s, tg2090 9.62 t/s

laurentpayot · 2026-10-12T18:06:24Z

@Volunteer-1 I noticed more or less the same on my Strix Halo.

Volunteer-1 · 2026-10-12T18:53:13Z

@laurentpayot good to know.
Omitting MTP is much faster than Daniel's MTP, though:
prefill 226.50 t/s is 2.3% faster and tg2090 17.94 t/s is 27% faster than using his MTP.

gordan-bobic · 2026-10-12T19:00:34Z

Yeah, I have to say I disabled MTP on my Qwen 3.8 Flash Next setup - the gains just weren't worthwhile, and I find the VRAM is better spent on a larger context.

A context keeps the built graph of each batch shape together with its scheduler split and allocation in a graph slot of the scheduler, so a speculative verification width seen before is made current again instead of being built, split and allocated anew. The slots are dropped when the compute buffers are planned again (a reserve, or a buffer that has to grow); when all slots are in use the least recently used shape is evicted. LLAMA_GRAPH_REUSE_SHAPES: not set = up to 24 shapes per context (a speculative server uses up to 16 verification widths, and the rest leaves room for a second graph variant of a width), 0 = off (one graph per output class, as before), 1 = 8, N > 1 = up to N (at most 64). It switches itself off, and logs the reason, when graph reuse is disabled, for a model loaded with no_alloc, with GGML_CUDA_GRAPH_OPT=1 and with pipeline parallelism. Measured with both switches set by environment on one build, R9700, 128 GB host memory, English roleplay (17 requests), MTP --spec-draft-n-max 7 --spec-draft-p-min 0.7 with the joint cache, exclusive expert cache 20480 MiB, unlimited host tier, warm, one run per row, off (LLAMA_GRAPH_REUSE_SHAPES=0, backend bound 16) against the defaults (24, 32): - Qwen3.8-Flash-Next UD-Q4_K_XL (MTP head of upstream pull request ggml-org#28243, Qwen indexer before upstream ggml-org#29751): 46.52 -> 51.67 t/s (+11.1 %) - DeepSeek V4 Flash UD-IQ3_XXS: 28.97 -> 31.45 t/s (+8.6 %) Replies 17/17 identical and acceptance unchanged in both. The extra time of the call after a width change fell from +3.6..+9.8 ms to about 0 (within 0.8 ms). Peak private bytes +0.85..0.97 GiB, dedicated VRAM unchanged.

The benchmark moves from one page to docs/ranma/benchmarks/, one record per measurement, named by the date and the topic, each naming at its top the release tag it measured, the upstream commit that release is based on and the date. The records keep their tables and numbers as measured; what they share is in method.md, and README.md collects the main numbers of each model with the release they come from. - 2026-10-13-release.md: the release table of ranma_20261013 (upstream 3d65c90), server scenarios only: coding and Korean roleplay, warm from the profile and store of a warm-up preset, on the R9700. Qwen3.8-Flash-Next EXL3 4.05 bpw, DeepSeek V4 Flash EXL3 3.04 bpw and GLM-5.3 Flash EXL3 3.05 bpw with MTP and --spec-smart at 128 GB; DeepSeek V4.1 Flash EXL3 3.0 bpw without a draft model at 128 GB, on the RX 9070 XT emulation with 64 GB and at 224 GB, and with the DSpark draft model at 224 GB; memory of every row. The machine now has 224 GB, so the 128 and 64 GB rows hold the physical memory down with locked pages. - 2026-10-09-release.md: the release table of ranma_20261009 (upstream 4625240): Qwen3.8-Flash-Next, DeepSeek V4 Flash and GLM-5.3 Flash, each in EXL3 and in a general GGUF, llama-bench depth 0 warm on the R9700 and the RX 9070 XT emulation with 128, 64 and 32 GB, against upstream on the general GGUF with -ncmoe. The 64 and 32 GB rows hold the physical memory down with locked pages so that the file tier reads from the SSD; upstream cannot load there. For the EXL3 files on the R9700 with 128 GB it also has llama-bench at depth 0, 8192 and 32768 cold and warm, the server scenarios with MTP and --spec-smart, and memory of every row. TG128 at depth 0 against upstream at 128 GB: x2.87 to x3.48. - 2026-10-05-exl3.md: Qwen3.8-Flash-Next in EXL3 3.05 and 4.05 bpw against UD-Q4_K_XL, measured for ranma_20261005 on the Radeon AI PRO R9700 with 128 and 64 GB and on the RX 9070 XT emulation with 64 GB, the expert cache exclusive: the llama-bench curves cold and warm (warm with --expert-prefill-swap), and the server scenarios with MTP and --spec-smart, warm from a store and a profile written by a warm-up preset, with GGML_CUDA_TOP_K_STABLE_TIES=1 and ROCBLAS_USE_HIPBLASLT=0 as in the MTP record; memory of every row. EXL3 3.05 bpw decodes faster than UD-Q4_K_XL on every system (llama-bench TG128 at depth 0 and 65536 +5.9 to +20.5 %, server decode with MTP +15.2 to +85.9 %), more so the fewer experts stay in VRAM and host memory; EXL3 4.05 bpw decodes like UD-Q4_K_XL in llama-bench on the R9700 and faster in the server scenarios. - 2026-10-01-mtp-smart.md: the ways of choosing the MTP draft length, measured for ranma_20261001 on the Radeon AI PRO R9700 with 128 GB for Qwen3.8-Flash-Next UD-Q4_K_XL and DeepSeek V4 Flash UD-IQ3_XXS: no MTP, MTP with the single-argument values tuned on this machine, --spec-smart from a cold start and from a warm store, in coding, roleplay in English, Korean, Chinese and Japanese, and a mixed scenario. One fresh server process per row after a 300 s rest, one build for all rows, greedy decoding. Smart is within a few percent of the single argument without per-model values; on DeepSeek V4 Flash it is 1.3 to 3.8 % slower in every roleplay scenario, cold and warm. - 2026-09-29-graph-reuse-default.md: graph reuse per batch shape turned on by default. - 2026-09-28-release.md: ranma_20260928 against upstream: the server scenarios for DeepSeek V4 Flash with and without MTP and for Qwen3.8-Flash-Next, cold and warm, with 64 GB rows, against upstream and the previous release; the llama-bench depth curves; Gemma 4 31B Q4_K_M as the dense reference; the measured effect of the options that are off in the main tables; memory. - 2026-09-28-qwen-mtp-n1.md: Qwen3.8-Flash-Next MTP n1 in English roleplay with the MTP head of upstream pull request ggml-org#28243. - 2026-09-22-full-set.md: the full set measured once at ranma_20260922. The Benchmarks section of the README states the upstream base of the release, opens with the release table of ranma_20261013, keeps the depth 0 table of ranma_20261009, and four tables, each with the release it was measured at: EXL3 against UD-Q4_K_XL (ranma_20261005), the decode ratios of the server scenarios against upstream (ranma_20260928), the MTP draft length comparison (ranma_20261001) and the llama-bench rows of the two systems (ranma_20260922). docs/ranma/README.md points to the benchmarks. The links of the RDNA4 prefill page point to the method page. method.md describes the 224 GB machine, the physical memory hold of the 128, 64 and 32 GB rows and the 600 s cap of the temperature gate.

A context keeps the built graph of each batch shape together with its scheduler split and allocation in a graph slot of the scheduler, so a speculative verification width seen before is made current again instead of being built, split and allocated anew. The slots are dropped when the compute buffers are planned again (a reserve, or a buffer that has to grow); when all slots are in use the least recently used shape is evicted. LLAMA_GRAPH_REUSE_SHAPES: not set = up to 24 shapes per context (a speculative server uses up to 16 verification widths, and the rest leaves room for a second graph variant of a width), 0 = off (one graph per output class, as before), 1 = 8, N > 1 = up to N (at most 64). It switches itself off, and logs the reason, when graph reuse is disabled, for a model loaded with no_alloc, with GGML_CUDA_GRAPH_OPT=1 and with pipeline parallelism. Measured with both switches set by environment on one build, R9700, 128 GB host memory, English roleplay (17 requests), MTP --spec-draft-n-max 7 --spec-draft-p-min 0.7 with the joint cache, exclusive expert cache 20480 MiB, unlimited host tier, warm, one run per row, off (LLAMA_GRAPH_REUSE_SHAPES=0, backend bound 16) against the defaults (24, 32): - Qwen3.8-Flash-Next UD-Q4_K_XL (MTP head of upstream pull request ggml-org#28243, Qwen indexer before upstream ggml-org#29751): 46.52 -> 51.67 t/s (+11.1 %) - DeepSeek V4 Flash UD-IQ3_XXS: 28.97 -> 31.45 t/s (+8.6 %) Replies 17/17 identical and acceptance unchanged in both. The extra time of the call after a width change fell from +3.6..+9.8 ms to about 0 (within 0.8 ms). Peak private bytes +0.85..0.97 GiB, dedicated VRAM unchanged.

The benchmark moves from one page to docs/ranma/benchmarks/, one record per measurement, named by the date and the topic, each naming at its top the release tag it measured, the upstream commit that release is based on and the date. The records keep their tables and numbers as measured; what they share is in method.md, and README.md collects the main numbers of each model with the release they come from. - 2026-10-13-release.md: the release table of ranma_20261013 (upstream 3d65c90), server scenarios only: coding and Korean roleplay, warm from the profile and store of a warm-up preset, on the R9700. Qwen3.8-Flash-Next EXL3 4.05 bpw, DeepSeek V4 Flash EXL3 3.04 bpw and GLM-5.3 Flash EXL3 3.05 bpw with MTP and --spec-smart at 128 GB; DeepSeek V4.1 Flash EXL3 3.0 bpw without a draft model at 128 GB, on the RX 9070 XT emulation with 64 GB and at 224 GB, and with the DSpark draft model at 224 GB; memory of every row. The machine now has 224 GB, so the 128 and 64 GB rows hold the physical memory down with locked pages. - 2026-10-09-release.md: the release table of ranma_20261009 (upstream 4625240): Qwen3.8-Flash-Next, DeepSeek V4 Flash and GLM-5.3 Flash, each in EXL3 and in a general GGUF, llama-bench depth 0 warm on the R9700 and the RX 9070 XT emulation with 128, 64 and 32 GB, against upstream on the general GGUF with -ncmoe. The 64 and 32 GB rows hold the physical memory down with locked pages so that the file tier reads from the SSD; upstream cannot load there. For the EXL3 files on the R9700 with 128 GB it also has llama-bench at depth 0, 8192 and 32768 cold and warm, the server scenarios with MTP and --spec-smart, and memory of every row. TG128 at depth 0 against upstream at 128 GB: x2.87 to x3.48. - 2026-10-05-exl3.md: Qwen3.8-Flash-Next in EXL3 3.05 and 4.05 bpw against UD-Q4_K_XL, measured for ranma_20261005 on the Radeon AI PRO R9700 with 128 and 64 GB and on the RX 9070 XT emulation with 64 GB, the expert cache exclusive: the llama-bench curves cold and warm (warm with --expert-prefill-swap), and the server scenarios with MTP and --spec-smart, warm from a store and a profile written by a warm-up preset, with GGML_CUDA_TOP_K_STABLE_TIES=1 and ROCBLAS_USE_HIPBLASLT=0 as in the MTP record; memory of every row. EXL3 3.05 bpw decodes faster than UD-Q4_K_XL on every system (llama-bench TG128 at depth 0 and 65536 +5.9 to +20.5 %, server decode with MTP +15.2 to +85.9 %), more so the fewer experts stay in VRAM and host memory; EXL3 4.05 bpw decodes like UD-Q4_K_XL in llama-bench on the R9700 and faster in the server scenarios. - 2026-10-01-mtp-smart.md: the ways of choosing the MTP draft length, measured for ranma_20261001 on the Radeon AI PRO R9700 with 128 GB for Qwen3.8-Flash-Next UD-Q4_K_XL and DeepSeek V4 Flash UD-IQ3_XXS: no MTP, MTP with the single-argument values tuned on this machine, --spec-smart from a cold start and from a warm store, in coding, roleplay in English, Korean, Chinese and Japanese, and a mixed scenario. One fresh server process per row after a 300 s rest, one build for all rows, greedy decoding. Smart is within a few percent of the single argument without per-model values; on DeepSeek V4 Flash it is 1.3 to 3.8 % slower in every roleplay scenario, cold and warm. - 2026-09-29-graph-reuse-default.md: graph reuse per batch shape turned on by default. - 2026-09-28-release.md: ranma_20260928 against upstream: the server scenarios for DeepSeek V4 Flash with and without MTP and for Qwen3.8-Flash-Next, cold and warm, with 64 GB rows, against upstream and the previous release; the llama-bench depth curves; Gemma 4 31B Q4_K_M as the dense reference; the measured effect of the options that are off in the main tables; memory. - 2026-09-28-qwen-mtp-n1.md: Qwen3.8-Flash-Next MTP n1 in English roleplay with the MTP head of upstream pull request ggml-org#28243. - 2026-09-22-full-set.md: the full set measured once at ranma_20260922. The Benchmarks section of the README states the upstream base of the release, opens with the release table of ranma_20261013, keeps the depth 0 table of ranma_20261009, and four tables, each with the release it was measured at: EXL3 against UD-Q4_K_XL (ranma_20261005), the decode ratios of the server scenarios against upstream (ranma_20260928), the MTP draft length comparison (ranma_20261001) and the llama-bench rows of the two systems (ranma_20260922). docs/ranma/README.md points to the benchmarks. The links of the RDNA4 prefill page point to the method page. method.md describes the 224 GB machine, the physical memory hold of the 128, 64 and 32 GB rows and the 600 s cap of the temperature gate.

github-actions Bot added model Model specific ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend conversion labels Sep 2, 2026

ggerganov reviewed Sep 2, 2026

View reviewed changes

giveen mentioned this pull request Sep 3, 2026

qwen4exp: upstream catch-up, sparse-fa, and NextN/MTP support TheTom/llama-cpp-turboquant#340

Merged

evandhoffman mentioned this pull request Sep 3, 2026

Mainline llama.cpp is getting a Qwen3.8-Flash-Next MTP graph (#28243, draft) — the thing #77 is blocked on evanwtf/local-llm#110

Open

sdroege reviewed Sep 3, 2026

View reviewed changes

TheTom mentioned this pull request Sep 3, 2026

CUDA: full test-backend-ops sweep segfaults in ggml_backend_cuda_graph_compute at ROLL on GB10 (passes with graphs disabled) TheTom/llama-cpp-turboquant#347

Open

ryanmonsurate and others added 10 commits September 3, 2026 20:04

gguf-py : register the qwen4exp NextN tensors

c8c3a5b

Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.

qwen4exp: allow loading a draft-only MTP export

30b6537

qwen4exp mtp: trim comments

44c7960

qwen4exp mtp: cut comments that restate the code

2c96729

Assisted-by: Claude

danielhanchen force-pushed the qwen4exp/mtp branch from 2857e51 to 2c96729 Compare September 4, 2026 03:08

This comment was marked as spam.

Sign in to view

ChangXiang-SCU mentioned this pull request Sep 4, 2026

llama: GPU-resident LRU cache for host-offloaded MoE expert weights #27861

Draft

evandhoffman mentioned this pull request Sep 30, 2026

llama.cpp PR #27836 adds the qwen4exp MTP draft head — open since 2026-08-27, and it is the unblock for MTP on Flash-Next evanwtf/local-llm#280

Open

novkien mentioned this pull request Sep 30, 2026

Integrate upstream Flash-Next MTP support novkien/llama.cpp-fork#8

Merged

CISC mentioned this pull request Sep 30, 2026

Qwen4Exp: add MTP #29761

Merged

pwilkin closed this Oct 1, 2026

neurall added a commit to neurall/llama.cpp that referenced this pull request Oct 4, 2026

README: MTP added in release b11327 (PR ggml-org#28243, ggml-org#27917)…

e8443c8

…, final numbers

gordan-bobic mentioned this pull request Oct 5, 2026

Misc. bug: qwen4exp indexer cache doubled by QSA kpool (row=2) and mirrored per device, costing ~400 MiB VRAM per GPU under -sm tensor #30005

Open

		// target for a draft head that declares nextn_shared_target_tensors; must outlive this model
		const struct llama_model * model_shared;

		throw std::runtime_error(format("%s: this model is a draft head without its own '%s'; "
		"load it as a draft of its target model, not on its own", __func__, name.c_str()));

Conversation

danielhanchen commented Sep 2, 2026

Overview

Additional information

Requirements

Uh oh!

ggerganov Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

danielhanchen commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jesdga95 commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llbzow commented Sep 3, 2026

Validation: 2× RTX A6000 48GB (PCIe, no NVLink), Windows Server 2019, CUDA 12.6 (sm_86) — both heads draft at ~0.37 acceptance, no config beats baseline

Uh oh!

sdroege Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

jeverson-bmi commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as spam.

gabrielfreire commented Sep 4, 2026

Uh oh!

Wladastic commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fat-tire commented Oct 1, 2026

Uh oh!

pwilkin commented Oct 1, 2026

Uh oh!

Volunteer-1 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laurentpayot commented Oct 12, 2026

Uh oh!

Volunteer-1 commented Oct 12, 2026

Uh oh!

gordan-bobic commented Oct 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

danielhanchen commented Sep 2, 2026 •

edited

Loading

jesdga95 commented Sep 2, 2026 •

edited

Loading

jeverson-bmi commented Sep 3, 2026 •

edited

Loading

Wladastic commented Sep 30, 2026 •

edited

Loading

Volunteer-1 commented Oct 12, 2026 •

edited

Loading