Repository navigation
Qwen4Exp: add MTP - #29761
Qwen4Exp: add MTP#29761
Conversation
|
We're abandoning #28243 then? |
e3448f3 to
30ddbf9
Compare
Yes, this is the correct way to add it IMO as it follows the earlier Qwen models |
|
Started a conversion job: https://github.com/ggml-org/convert/actions/runs/36757034231/job/110029790971 |
| // the MTP head reads the residual of every row unless its rows are masked to the outputs | ||
| const bool narrow_early = !cparams.embeddings_nextn || cparams.embeddings_nextn_masked; | ||
|
|
There was a problem hiding this comment.
nit: we should be consistent with this pattern. The vars are different in mimo2.cpp for example.
| // false when the layer filter kept no layer, e.g. an MTP draft context: any rollback is then free | ||
| bool has_state = false; | ||
|
|
There was a problem hiding this comment.
Is it too complicated to simply create a non-hybrid memory for the MTP context and avoid this?
There was a problem hiding this comment.
The MTP uses the llama-hybrid-idx class, it would suffer slowdowns at large context if we don't use sparse attention properly.
There was a problem hiding this comment.
I see.
Instead of adding this member, would it be more correct to override n_rs_seq in the constructor with mem_size when the filter removes all layers? This should make the init_batch logics more robust to this case.
There was a problem hiding this comment.
I removed the member, but setting n_rs_seq is not enough. Checking if ctx_bufs is not empty seems to be enough
|
Can't load draft model: check_tensor_dims: tensor 'token_embd.weight' not found 0.16.292.755 I load_tensors: loading model tensors, this can take a while... (load_mode = dio) Start command: Omitting the entire -ot argument produces the same error. Models used: |
|
Probably needs requants, the MTP head does not share the tok_embd if it's loaded as a separate GGUF. |
|
I think this makes it more clear when the recurrent memory ends up being empty and the way the diff --git a/src/llama-memory-recurrent.cpp b/src/llama-memory-recurrent.cpp
index a7f926311..8353a3093 100644
--- a/src/llama-memory-recurrent.cpp
+++ b/src/llama-memory-recurrent.cpp
@@ -125,6 +125,13 @@ llama_memory_recurrent::llama_memory_recurrent(
ctxs_bufs.emplace_back(std::move(ctx), buf);
}
+ if (is_empty()) {
+ if (n_rs_seq > 0) {
+ n_rs_seq = 0;
+ LLAMA_LOG_INFO("%s: disabling rollback snapshots because the memory module is empty\n", __func__);
+ }
+ }
+
{
const size_t memory_size_r = size_r_bytes();
const size_t memory_size_s = size_s_bytes();
@@ -193,7 +200,7 @@ bool llama_memory_recurrent::seq_rm(llama_seq_id seq_id, llama_pos p0, llama_pos
// partial rollback via per-token snapshot index (bounded by n_rs_seq)
if (0 < p0 && p0 <= cell.pos && p1 > cell.pos) {
// the filter kept no layer (e.g. an MTP draft context), so only the position moves back
- if (ctxs_bufs.empty()) {
+ if (is_empty()) {
cell.pos = p0 - 1;
return true;
}
@@ -723,6 +730,11 @@ bool llama_memory_recurrent::get_can_shift() const {
return true;
}
+bool llama_memory_recurrent::is_empty() const {
+ assert(total_size() == 0);
+ return ctxs_bufs.empty();
+}
+
size_t llama_memory_recurrent::total_size() const {
size_t size = 0;
for (const auto & [_, buf] : ctxs_bufs) {
diff --git a/src/llama-memory-recurrent.h b/src/llama-memory-recurrent.h
index 25ade10e5..08489a4be 100644
--- a/src/llama-memory-recurrent.h
+++ b/src/llama-memory-recurrent.h
@@ -123,6 +123,9 @@ private:
// ggml contexts for the KV cache along with the allocated backend buffers:
std::vector<std::pair<ggml_context_ptr, ggml_backend_buffer_ptr>> ctxs_bufs;
+ // true if no layers - can happen if the layer filter removes all layers
+ bool is_empty() const;
+
size_t total_size() const;
size_t size_r_bytes() const;Needs to be tested (downloading the model atm) |
ggerganov
left a comment
ggerganov
left a comment
There was a problem hiding this comment.
I did some tests and it looks solid. I think my changes proposed in #29761 (comment) should be included, unless I am missing something. It works without them, but they should make the logic more future-proof.
I will push a fix for the failing CI with multiple virtual devices directly to master.
ggerganov
added
the
highlight
|
Tried new ggml-org quants with both #29751 and #29761 merged onto fresh master, with mtp (q8) enabled failed to load for me - OOM's: Details
|
|
Probably you need to decrease your context or decrease --cache-ram. The way of sharing the tok_embd among different GGUFs is not supported with this. |
|
@am17an Thanks, indeed, with 30k context and cram decreased to default 8gb it loaded, on this short context TG increased from ~40tps to ~50tps (non-coding prompt), 256k context doesn't work even with decreased cram, though (OOMs like before). |
|
Try using the Q4 MTP. |
|
And reduce the |
|
Edit: Ah, nvm, see @ggerganov addressed this. |
|
@ggerganov thanks, with batch\ubatch decreased to 512, and Q4 mtp loaded with 256k context with expected drop of processing speed. (just using q4 mtp alone not worked, only with lowering ubatch). Details
Observed strange memory allocation while loading with full context (thats with ubatch 4096, q4 mtp, full context): |
Upstream absorbed the fork's two largest features in this window: - c061df1 (ggml-org#29761) adds Qwen4Exp MTP. Structurally the same design the fork carried (both descend from ggml-org#27836): mtp_only/trunk_flags, graph_mtp, the NEXTN_HC_HEAD_* tensors and the hc-wide residual through eh_proj. Upstream's version is taken whole; only the shared-sidecar support is re-applied on top ([TAG_QWEN4EXP_SHARED_MTP], 34 lines: tok_embd/output optional when the trunk is absent, borrowed from cparams.ctx_other in graph_mtp). - 649dcb1 (ggml-org#27773) adds GLM-5.3-Flash as arch 'glm5-next' with its own src/models/glm5-next.cpp and vision projector 'glm5v'. The fork's 'glm5next' (from draft ggml-org#27752) and projector 'glm5next' are kept beside it: existing GGUFs carry the old arch string. Both arch cases and both projector cases stand side by side. Dropped: the multi-token QSA gather ([TAG_QSA_GATHER_VERIFY]) and its two fixtures. Upstream rewrote the QSA path (889edf4 halves the indexer score memory, 4e2713c rebuilds the masks, build_attn_qsa has a new signature), so the gather no longer applies textually and has to be re-derived. Two upstream bugs fixed on the way, both only visible on a virtual model: - qwen4exp's mtp_only probe used get_weight(), which always answers nullptr without files, so every virtual qwen4exp marked its whole trunk NOT_REQUIRED and aborted at the first tensor's buft. Gated on !ml.files.empty(). - create_tensor_qkv's skip constant carried TENSOR_SKIP_IF_VIRTUAL only on the fused QKV, not on the separate Q/K/V, so a skipped MTP block aborted the same way. Added to the constant. Also ported: the fork's [TAG_NAN_CHECK]-adjacent test helpers to common_batch / llama_process, llm_graph_input_mem_hybrid_kpool::can_reuse to upstream's llm_graph_input_rs (s_copy_extra is gone), and the hipCUB argsort guard around upstream's new shared-memory bitonic bound. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| ggml_build_forward_expand(gf, inp_hyb->get_recr()->s_copy); | ||
|
|
||
| llm_graph_input_kpool * inp_kpool = nullptr; | ||
| if (mctx_hyb->get_idx() && hparams.indexer_kpool > 0) { |
There was a problem hiding this comment.
I believe the condition might be missing && hparams.dsv4_compress_ratios[il] > 0
There was a problem hiding this comment.
Please don't write inside closed / merged PRs, submit an issue / new PR instead
Conflicts and choices: - qwen4exp: upstream merged its own MTP (ggml-org#29761, a QSA block over a hybrid-idx draft memory). Kept our loader and dense graph_mtp with the stage, catch-up and head-subset paths; the MTP context stays on a plain KV cache. The trunk QSA is upstream's k-pool version; our set_input_qsa fast path and the pooling/top-k tweaks went with the code they patched. - speculative: the MTP driver (stage, deferred catch-up) ported to common_batch / llama_process. With the stage the entries carry tokens only. - sampling: upstream's rejection verification replaces the unused dists path. - CUDA: launch_fattn takes both f16_live_only and async_kv_preload; the Volta MMQ config boundary stays at Turing. Flash greedy output differs from the pre-sync build (different block selection); 27B is byte-identical. Assisted-by: Claude Opus 5.5
can you indicate which quants/gguf work with this now..thanks a lot!! |
* Qwen4Exp: add MTP * remove has_state member, check via ctx_bufs being non-empty * consistent naming + less verbose comments * cont : clean-up recurrent memory * cont : clean-up comments --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…ed heads (unslothai#242) Drop ggml-org#27754, unslothai#144 and unslothai#137: upstream merged GLM-5-Next (ggml-org#27773), Qwen4Exp MTP (ggml-org#29761) and row prefetch (ggml-org#29599), and the old pins no longer merge. Old-GGUF loading moves to unslothai#239, shared MTP heads to unslothai#240, and what the dropped pins had beyond upstream to unslothai#241. Repin ggml-org#24423, ggml-org#25731, unslothai#61 and unslothai#176 to their conflict-fixed heads.
Port MTP draft model loading and graph construction from strix-llama.cpp:
- conversion/qwen4exp.py:
- MTP block tensors are now exported with NEXTN_HC_HEAD_* tensor names
- filter_tensors handles mtp.hyper_connection Mixer for MTP-only files
- generate_extra_tensors concatenates fc_embedding + fc_hidden into eh_proj
- Compress ratio array extended to n_layer_all for nextn layer
- gguf-py/gguf/constants.py:
- Add NEXTN_HC_HEAD_NORM, NEXTN_HC_HEAD_DOWN, NEXTN_HC_HEAD_UP tensor types
- src/models/qwen4exp.cpp:
- qwen4exp_shared_model() helper validates ctx_other for MTP-only files
- mtp_only flag detects draft-only GGUFs (no trunk tensors)
- trunk_flags = TENSOR_NOT_REQUIRED for MTP-only, TENSOR_SKIP otherwise
- Empty graph(model, params, bool) constructor for graph_mtp reuse
- load nextn layer tensors (enorm/hnorm/eh_proj) via per-layer loop
- Dispatch LLM_GRAPH_TYPE_DECODER_MTP in build_arch_graph
- graph_mtp builds the single draft block: eh_proj input (concat of token
embedding + hidden handoff), dense attention (no QSA indexer), HC output
- graph_trunk sets t_h_nextn = res_hc before final mixer (for DFlash target)
Upstream PR ggml-org#29761.
…ting Portiert den upstream-Qwen4Exp-MTP-Pfad (ggml-org PR ggml-org#29761) additiv auf die Fork-Basis, ohne die lokalen Haertungen (PLE-Validierung, ngram-Hash) zu ersetzen: - Neue Tensor-IDs NEXTN_HC_HEAD_{NORM,DOWN,UP} + Namen + gguf-py-Registry - llama_layer_nextn um hc_head_{norm,down,up} erweitert - nextn_flags() trennt Trunk-/MTP-only-/kombinierte Dateien (Probe auf blk.0.hc_attn_norm bzw. blk.N.nextn.eh_proj) - Loader iteriert n_layer_all; Trunk-Tensoren in Draft-Dateien sind optional - graph_mtp: [enorm(E);hnorm(h)] -> eh_proj -> HC-Streams -> QSA-Attn -> FFN -> eigener hc_head-Mixer; exportiert t_h_nextn (hc-breit) + t_logits - Hauptgraph: crop_before/after_nextn-Helfer; t_h_nextn-Export vor dem finalen Mixer (vorm HC-Collapse) - Memory-Filter: MTP-CTX haelt nur Layer >= n_layer (attn+idx, kein recr) - speculative.cpp: Draft-Breite gegen n_embd_out(tgt) pruefen — HC-Modelle exportieren h_nextn in hc_dim, nicht n_embd - nextn.hnorm liegt in Dateien flach [hc_dim] vor; im Graph als [n_embd,hc] View fuer die per-Stream-RMSNorm (numerisch identisch zu upstream) Getestet gegen unsloth/Qwen3.8-Flash-Next-GGUF MTP-Sidecar (Q8_0, 34 Tensoren, block_count=49, nextn_predict_layers=1).
… 0.6.0 PR #28243 was closed and superseded by ggml-org/llama.cpp#29761, which merged and shipped in the llama.cpp v0.6.0 release — the version the pinned nixpkgs already builds. The qwen4exp MTP graph, `--spec-type draft-mtp`, and standalone draft-head loading are all present in v0.6.0, so the custom b10786 overlay build is no longer needed. Remove: - hosts/host.thing/nixpkgs.overlays.llama-cpp-pr-28243.nix and its import in hosts/host.thing/default.nix - patched-llama-cpp-pr-28243-pkg and the mtpServerPackage threading in the llama-cpp module config (both Flash-Next MTP entries now run on the stock build via serverPackage = null, so they are no longer excluded from the router) - doc/TODOs/drop-llama-cpp-pr-28243-pin-when-qwen4exp-mtp-merges.md (its removal plan, now executed) Adjustment vs. the PR: the merged implementation dropped cross-model tensor borrowing (borrow_shared_tensor), and token_embd is a required tensor in every GGUF — the `shared-` draft heads cannot load on mainline. Qwen3.8-Flash-Next-UD-Q4_K_XL-MTP therefore switches from mtp-...-shared-Q4_K_M.gguf (1.78 GB) to the self-contained mtp-...-Q4_K_M.gguf (2.60 GB, already pulled for the IQ4_XS entry); irrelevant against the 124 GiB gfx1151 pool. This also unbreaks the host build: the pinned llama-cpp-10786 build was the only failing derivation (its test-generate-models aborts under the GGML_BACKEND_DL sandbox build), everything else was cascade. Verified: test-thing toplevel evaluates; container package is stock llama-cpp 0.6.0; dry-run build contains no llama-cpp-10786.drv. After deploying, serve Qwen3.8-Flash-Next-UD-Q4_K_XL-MTP once and check for the `draft acceptance = ...` log line (the GGUFs were converted against the #28243 implementation). supported by trustedtokens/zai-org/GLM-5.3 in pi
|
On my dual RTX 3090 rig, @danielhanchen 's PR for MTP for qwen4exp is far superior to this PR. Daniel's PR gives 18% faster prefill and 47% faster decode for the decode-optimal setting ub 512. Input prompt: Analyze the file src\models\qwen4exp.cpp Commands used: For Daniel's PR: speed: prefill 221.45 t/s, tg2090 14.18 t/s For this PR 29761: speed: prefill 187.68 t/s, tg2090 9.62 t/s |
Brings in 221 upstream commits, including ggml-org#14891 (activation-based imatrix statistics for GGUF imatrices). Resolutions: - qwen4exp: take upstream's MTP (ggml-org#29761) over this fork's halo-box#9 head. The two write different GGUFs (upstream keeps the MTP block's own hyper-connection mixer as nextn.hc_head_* and runs the draft with QSA), and current MTP files use upstream's layout. Also drop halo-box#9's attention-only MTP memory for qwen4exp in llama-model.cpp and its extra NEXTN entries in constants.py. - speculative: keep --spec-draft-adaptive's n_draft_eff bound on top of upstream's probabilistic draft sampling (result_q); begin() resets both the acceptance EMA and the sampler. - ggml-backend: take upstream's pass-6 input collection (ggml-org#29634) and ggml_backend_sched_copy_input() (ggml-org#29943), keeping halo-box#23's copy-0 original with input rotation, async-read pinning, and the sanitizer hook. - vulkan: keep halo-box#12's UMA read-back gate (host-cached or <= 64 KiB direct); upstream's AMD-only fix is a subset of it. - server: keep spec-prefill's prompt_src() next to upstream's use_spec_rejection(); use add_embd() with M-RoPE positions. - speculative-prefill: migrate to common_batch / llama_process, as upstream ggml-org#29601 removed common_batch_add/clear. - README: keep this fork's quick start; common.h, test-alloc: keep both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
halo-box/llama.cpp is archived, so its last master comes in once: the 2026-10-07 upstream sync (halo halo-box#34) with its resolutions, plus the archival README of that repository is not taken (strix keeps its own README). Resolution rules: MTP follows upstream's design; everywhere else strix's side wins where the two collide, and upstream's non-conflicting changes stay. - qwen4exp: strix's QSA stack (selected-key / WMMA block selection, incremental pooled keys, PLE prefetch, HIP prefill work) is kept whole in qwen4exp.cpp and the qwen4exp struct of models.h. strix's graph_mtp is kept: it builds the same draft as upstream's ggml-org#29761 (eh_proj of [enorm(e); hnorm(h)], the draft's own nextn.hc_head_* mixer, sparse QSA attention) and also loads shared sidecars without embeddings / LM head. The converter, gguf-py tensor maps and arch tables are upstream's, so new conversions follow upstream's layout; strix loads both. - llama-memory-hybrid-idx: both APIs kept side by side - strix's qsa_* for qwen4exp and upstream's k-pool for the new GLM5Next (ggml-org#27773 and follow-ups). strix's context set_input_qsa, which upstream deleted, is restored. - llama-graph: strix's in-place gated-delta-net state path is kept; upstream's single recurrent-state gather (ggml-org#29856, avoids a realloc abort under GGML_SCHED_NO_REALLOC) is reverted on this path and needs porting. - vulkan: strix's flash_attn.comp and lightning_indexer.comp are kept whole; upstream's FA shmem out-of-bounds fix (ggml-org#29988) is ported into strix's shader. Upstream's lightning indexer rewrite (ggml-org#29825) is reverted on the Vulkan host side to match strix's shader; its CUDA/Metal parts stay. Upstream's sparse FA path (ggml-org#29639) stays out: strix deleted its shader. - ggml-alloc: upstream's alloc_buffer_n refactor (ggml-org#23671) replaces strix's backport of the view-init fix; strix's regression test is kept. - cuda: strix's RDNA3.5 paths kept; upstream's shared-expert MoE fusion added next to strix's fused GDN decode, with both sets of fusion args. - server: speculative accept keeps strix's replay branch, then upstream's rejection sampling at temp > 0. - tests: both sides' new tests kept (rollback table gets both columns, save-load-state test 10 is strix's, 11 upstream's); strix's rollback test migrated to common_batch / llama_process. - fusion.yml kept (upstream removed it), flash_attn_sparse_compact.comp stays deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 participants
Overview
Add MTP for Qwen3.8-Flash next. On a small portion of speed-bench on DGX spark Qwen3.8-Flash-Next iq4_xs (
-np 1 -lzm on --spec-draft-n-max 3 --spec-type draft-mtp)Additional information
Requirements