Eval bug: Gemma 4 crashes with MTP #25522

Activity

1. gordan-bobic

   added

   bug-unconfirmed

   on Jul 10, 2026
2. ServeurpersoCom commented on Jul 10, 2026

   

   ServeurpersoCom

   on Jul 10, 2026

   Last edited by ServeurpersoCom

   Contributor

   More actions

   I have a very similar setup, the latest master, and it works perfectly for me. We need to find the difference that's causing the crash...

   EDIT: Ah, the whole problem lies in the split for the dual GPU setup.

   ; llama-server --host 0.0.0.0 --port 8082 --models-max 1 --models-preset backend.ini --ui-config-file frontend.json
   
   [*]
   no-mmap = on              ; Disable mmap (default: off)
   mlock = on                ; Lock model in RAM
   fit = off                 ; Disable automatic memory fitting (default: on)
   ngl = 999                 ; Full GPU offload
   ctk = q8_0                ; KV cache key quantization
   ctv = q8_0                ; KV cache value quantization
   fa = on                   ; Enable flash attention
   np = 4                    ; Parallel request batching (default: 4)
   kvu = on                  ; Unified KV cache buffer (default: on)
   stop-timeout = 2          ; Force-kill child process after graceful shutdown timeout in seconds (default: 10)
   sleep-idle-seconds = 300  ; Unload weights on child process
   ; b = 2048                  ; Logical maximum batch size (default: 2048)
   ; ub = 512                  ; Physical maximum batch size (default: 512)
   ; ctx-checkpoints = 128     ; Context state snapshots per slot for cache rescue (default: 32)
   checkpoint-min-step = 512 ; Minimum token spacing between snapshots, 128 x 512 = 65536 covered (default: 256)
   cache-ram = 24576         ; 24 GiB for idle slots
   webui-mcp-proxy = on      ; Enabled for llama-server issue debugging, not exposed to the internet
   ; slot-save-path = /var/www/ia/kvcache
   
   [Dense-Vision-Gemma-4-31B-IT-MTP]
   m = unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-Q8_K_XL.gguf
   mm = unsloth/gemma-4-31B-it-GGUF/mmproj-BF16.gguf
   md = unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-MTP-BF16.gguf
   spec-type = draft-mtp
   spec-draft-n-max = 4
   c = 262144
   
   [Dense-Vision-Gemma-4-12b-IT-MTP]
   m = unsloth/gemma-4-12b-it-GGUF/gemma-4-12b-it-UD-Q8_K_XL.gguf
   mm = unsloth/gemma-4-12b-it-GGUF/mmproj-BF16.gguf
   md = unsloth/gemma-4-12b-it-GGUF/gemma-4-12b-it-BF16-MTP.gguf
   spec-type = draft-mtp
   spec-draft-n-max = 2
   c = 262144
   
   [MoE-Vision-Gemma-4-E2B-IT-MTP]
   m = unsloth/gemma-4-E2B-it-GGUF/gemma-4-E2B-it-UD-Q8_K_XL.gguf
   mm = unsloth/gemma-4-E2B-it-GGUF/mmproj-BF16.gguf
   md = unsloth/gemma-4-E2B-it-GGUF/gemma-4-E2B-it-BF16-MTP.gguf
   spec-type = draft-mtp
   spec-draft-n-max = 2
   c = 262144
   
   [MoE-Vision-Gemma-4-E4B-IT-MTP]
   m = unsloth/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-UD-Q8_K_XL.gguf
   mm = unsloth/gemma-4-E4B-it-GGUF/mmproj-BF16.gguf
   md = unsloth/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-BF16-MTP.gguf
   spec-type = draft-mtp
   spec-draft-n-max = 2
   c = 262144
   
   ...
   
3. gordan-bobic commented on Jul 10, 2026

   

   gordan-bobic

   on Jul 10, 2026

   Author

   More actions

   Yes, it seems that there is a bug affecting MTP split across more than one GPU.

   👍React with 👍1Reacted by Pascal
4. ServeurpersoCom commented on Jul 10, 2026

   

   ServeurpersoCom

   on Jul 10, 2026

   Contributor

   More actions

   Can you try --split-mode layer ?
   I think --split-mode tensor can't cut a 1-head layer in half!
5. gordan-bobic commented on Jul 11, 2026

   

   gordan-bobic

   on Jul 11, 2026

   Author

   More actions

   That works, but that also loses me half of the prefill throughput compared to split-mode=tensor.
   I tried limiting MTP to a single device with --device-draft CUDA0, but that crashes out immediately with a different error:

   /builddir/build/BUILD/llama-cpp-b9947/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_k_l58) in a buffer (Meta()) that cannot run the operation (NONE)
   
6. ServeurpersoCom commented on Jul 11, 2026

   

   ServeurpersoCom

   on Jul 11, 2026

   Contributor

   More actions

   Yeah, --device-draft won't help here: that only moves the draft model. The blocker is the main model's global layers, which have a single KV head (that's your cache_k_l58, one of the shared-KV global layers).
   Tensor split partitions attention across KV heads, and a 1-head layer can't be cut, so its cache can't live in the tensor-split buffer at all.

   So it's -sm layer for now. You're right that it costs you prefill throughput (layer split runs one GPU at a time on a single-stream prefill, tensor split uses both per layer). And I'm not sure yet whether it's possible to refactor the tensor-split planner to support split-mode=tensor for 1-head layers.
7. gordan-bobic commented on Jul 11, 2026

   

   gordan-bobic

   on Jul 11, 2026

   Last edited by gordan-bobic

   Author

   More actions

   If the main model only has 1 head, how is it possible to run the main model across more than one GPU? Main model alone splits across GPUs with split-mode=tensor just fine. It is only when MTP is added into the mix that it crashes llama-server..

   👀React with 👀1Reacted by Pascal
8. ServeurpersoCom commented on Jul 11, 2026

   

   ServeurpersoCom

   on Jul 11, 2026

   Contributor

   More actions

   Can you try this patch?
   I cheated by using ggml-rpc-server to simulate your two GPUs on a single card, which reproduces your exact crash. Dug into it with Fable 5 and GPT 5.6 Sol: there are actually two distinct problems, including a use-after-reset in the GGML tensor-split backend (the MTP draft shares KV layers with the main model, and graph rebuilds recycle shard tensors the draft still references). Whether it works or not, it will help converge on the fix.
   No-rebuild alternative to test: LLAMA_GRAPH_REUSE_DISABLE=1 with your original config.

   diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
   index 1f29ec8671..cc67a71970 100644
   --- a/ggml/src/ggml-backend-meta.cpp
   +++ b/ggml/src/ggml-backend-meta.cpp
   @@ -416,9 +416,15 @@ struct ggml_backend_meta_buffer_context {
        // Long-term: tie the lifetime of external views to the meta backend executing the graph instead,
        //     currently not possible due to graph-external operations in the backend scheduler.
        ggml_backend_meta_simple_tensor_container stc_static;
   -    ggml_backend_meta_simple_tensor_container stc_compute[2];
   +    std::vector<std::unique_ptr<ggml_backend_meta_simple_tensor_container>> stc_compute;
        int stc_compute_index      = 0;
        int stc_compute_index_next = 0;
   +    ggml_init_params stc_compute_params{};
   +    int stc_n_simple = 0;
   +    // the compute slot each meta backend's shard graphs currently reference;
   +    // pinned slots are never recycled, which keeps uid-based shard graph reuse safe
   +    // when multiple schedulers share this buffer (e.g. main model + MTP draft)
   +    std::map<const void *, int> stc_pins;
        std::vector<ggml_backend_buffer_ptr> bufs;
   
        // FIXME
   @@ -431,10 +437,14 @@ struct ggml_backend_meta_buffer_context {
   
        ggml_backend_meta_buffer_context(
                ggml_backend_meta_simple_tensor_container & stc_static,
   -            ggml_backend_meta_simple_tensor_container & stc_compute_0,
   -            ggml_backend_meta_simple_tensor_container & stc_compute_1,
   +            const ggml_init_params & stc_compute_params,
   +            const int stc_n_simple,
                const std::vector<ggml_backend_buffer_t> & bufs)
   -            : stc_static(std::move(stc_static)), stc_compute{std::move(stc_compute_0), std::move(stc_compute_1)} {
   +            : stc_static(std::move(stc_static)) {
   +        this->stc_compute_params = stc_compute_params;
   +        this->stc_n_simple       = stc_n_simple;
   +        stc_compute.emplace_back(new ggml_backend_meta_simple_tensor_container(stc_compute_params, stc_n_simple));
   +        stc_compute.emplace_back(new ggml_backend_meta_simple_tensor_container(stc_compute_params, stc_n_simple));
            this->bufs.reserve(bufs.size());
            for (ggml_backend_buffer_t buf : bufs) {
                this->bufs.emplace_back(buf);
   @@ -447,7 +457,29 @@ struct ggml_backend_meta_buffer_context {
            if (stc_static.simple_tensors.find(tensor) != stc_static.simple_tensors.end()) {
                return stc_static;
            }
   -        return stc_compute[stc_compute_index];
   +        return *stc_compute[stc_compute_index];
   +    }
   +
   +    // pick a compute slot that no backend's shard graphs reference, growing the pool if needed;
   +    // a backend destroyed without unpinning leaves at most one unrecyclable slot behind
   +    int stc_free_slot() {
   +        for (int i = 0; i < (int) stc_compute.size(); i++) {
   +            if (i == stc_compute_index) {
   +                continue;
   +            }
   +            bool pinned = false;
   +            for (const auto & pin : stc_pins) {
   +                if (pin.second == i) {
   +                    pinned = true;
   +                    break;
   +                }
   +            }
   +            if (!pinned) {
   +                return i;
   +            }
   +        }
   +        stc_compute.emplace_back(new ggml_backend_meta_simple_tensor_container(stc_compute_params, stc_n_simple));
   +        return (int) stc_compute.size() - 1;
        }
    };
   
   @@ -1504,8 +1536,6 @@ static ggml_backend_buffer_t ggml_backend_meta_buffer_type_alloc_buffer(ggml_bac
            /*.no_alloc   =*/ true,
        };
        ggml_backend_meta_simple_tensor_container stc_static;
   -    ggml_backend_meta_simple_tensor_container stc_compute_0(params, n_simple_bufts);
   -    ggml_backend_meta_simple_tensor_container stc_compute_1(params, n_simple_bufts);
   
        size_t max_size = 0;
        std::vector<ggml_backend_buffer_t> bufs;
   @@ -1515,7 +1545,7 @@ static ggml_backend_buffer_t ggml_backend_meta_buffer_type_alloc_buffer(ggml_bac
            GGML_ASSERT(bufs.back() != nullptr);
            max_size = std::max(max_size, ggml_backend_buffer_get_size(bufs.back()));
        }
   -    ggml_backend_meta_buffer_context * buf_ctx = new ggml_backend_meta_buffer_context(stc_static, stc_compute_0, stc_compute_1, bufs);
   +    ggml_backend_meta_buffer_context * buf_ctx = new ggml_backend_meta_buffer_context(stc_static, params, n_simple_bufts, bufs);
   
        return ggml_backend_buffer_init(buft, ggml_backend_meta_buffer_iface, buf_ctx, max_size);
    }
   @@ -1535,11 +1565,9 @@ struct ggml_backend_buffer * ggml_backend_meta_alloc_ctx_tensors_from_buft(struc
            /*.no_alloc   =*/ true,
        };
        ggml_backend_meta_simple_tensor_container stc_static   (params_static,  n_simple_bufts);
   -    ggml_backend_meta_simple_tensor_container stc_compute_0(params_compute, n_simple_bufts);
   -    ggml_backend_meta_simple_tensor_container stc_compute_1(params_compute, n_simple_bufts);
   
        std::vector<ggml_backend_buffer_t> bufs(n_simple_bufts, nullptr);
   -    ggml_backend_meta_buffer_context * meta_buf_ctx = new ggml_backend_meta_buffer_context(stc_static, stc_compute_0, stc_compute_1, bufs);
   +    ggml_backend_meta_buffer_context * meta_buf_ctx = new ggml_backend_meta_buffer_context(stc_static, params_compute, n_simple_bufts, bufs);
   
        ggml_backend_buffer_t meta_buf = ggml_backend_buffer_init(buft, ggml_backend_meta_buffer_iface, meta_buf_ctx, 0);
        for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
   @@ -1806,8 +1834,8 @@ static enum ggml_status ggml_backend_meta_graph_compute(ggml_backend_t backend,
            }
            for (ggml_backend_buffer_t buf : used_buffers) {
                ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buf->context;
   -            buf_ctx->stc_compute_index_next = buf_ctx->stc_compute_index ^ 1;
   -            ggml_backend_meta_simple_tensor_container & stc = buf_ctx->stc_compute[buf_ctx->stc_compute_index_next];
   +            buf_ctx->stc_compute_index_next = buf_ctx->stc_free_slot();
   +            ggml_backend_meta_simple_tensor_container & stc = *buf_ctx->stc_compute[buf_ctx->stc_compute_index_next];
                for (ggml_context_ptr & ctx : stc.ctxs) {
                    ggml_reset(ctx.get());
                }
   @@ -1832,6 +1860,13 @@ static enum ggml_status ggml_backend_meta_graph_compute(ggml_backend_t backend,
                }
            }
   
   +        // the shard graphs built above reference the current containers until this backend's next
   +        // rebuild, so pin those slots against recycling by other schedulers sharing the same buffers
   +        for (ggml_backend_buffer_t buf : used_buffers) {
   +            ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buf->context;
   +            buf_ctx->stc_pins[(const void *) backend_ctx] = buf_ctx->stc_compute_index;
   +        }
   +
            {
                // For MoE models it may make sense to delay the AllReduce in order to reduce I/O:
                auto get_i_delayed = [&](const int i) -> int {
   diff --git a/src/llama-context.cpp b/src/llama-context.cpp
   index 5edfc85abf..c3bc2229f1 100644
   --- a/src/llama-context.cpp
   +++ b/src/llama-context.cpp
   @@ -3482,6 +3482,14 @@ llama_context * llama_init_from_model(
        }
   
        if (model->split_mode() == LLAMA_SPLIT_MODE_TENSOR) {
   +        // tensor split partitions attention along KV heads: a layer with a single KV head cannot be split
   +        // and the meta planner falls back to an axis 0 split that breaks per row ops and flash attention
   +        for (uint32_t il = 0; il < model->hparams.n_layer(); ++il) {
   +            if (model->hparams.n_head_kv(il) == 1) {
   +                LLAMA_LOG_ERROR("%s: SPLIT_MODE_TENSOR cannot partition layer %u which has a single KV head; use --split-mode layer\n", __func__, il);
   +                return nullptr;
   +            }
   +        }
            if (params.flash_attn_type == LLAMA_FLASH_ATTN_TYPE_AUTO) {
                LLAMA_LOG_INFO("%s: enabling flash_attn since it is required for SPLIT_MODE_TENSOR\n", __func__);
                params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;
   
9. gordan-bobic commented on Jul 11, 2026

   

   gordan-bobic

   on Jul 11, 2026

   Author

   More actions

   LLAMA_GRAPH_REUSE_DISABLE=1 makes it worse. Without it, llama-server crashes on the 2nd prompt. With that option it crashes on startup before any prompt can be sent.

   /builddir/build/BUILD/llama-cpp-b9960/ggml/src/ggml-backend-meta.cpp:1042: GGML_ASSERT(split_state.ne[j]*split_state.nr[0] * tensor->src[i]->ne[src_ss[i].axis] == sum * tensor->ne[split_state.axis]) failed
   

   I'll try your proposed patch.

   👀React with 👀1Reacted by Pascal
10. art-den commented on Jul 26, 2026

    

    art-den

    on Jul 26, 2026

    More actions

    I have the same problem with two GPUs. Is there a way around this without disabling MTP?
11. gordan-bobic commented on Aug 16, 2026

    

    gordan-bobic

    on Aug 16, 2026

    Author

    More actions

    Unfortunately, the proposed patch doesn't seem to work:

    /var/lib/ollama/llama.cpp/ggml/src/ggml-backend-meta.cpp:1081: GGML_ASSERT(split_state.ne[j]*split_state.nr[0] * tensor->src[i]->ne[src_ss[i].axis] == sum * tensor->ne[split_state.axis]) failed
    

    Caveat - I have since upgraded to 5 GPUs (from previous 2).

    👀React with 👀1Reacted by Pascal
12. sssr5hsssdssqdse commented on Aug 16, 2026

    

    sssr5hsssdssqdse

    on Aug 16, 2026 via email · Hidden as spam

    show commentMore actions
13. gordan-bobic commented on Aug 16, 2026

    

    gordan-bobic

    on Aug 16, 2026

    Author

    More actions

    It is same as before only 5x 22GB 2080Ti instead of 2x.
14. github-actions

    added

    stale

    on Sep 16, 2026
15. cromoteca commented on Sep 18, 2026

    

    cromoteca

    on Sep 18, 2026

    More actions

    Adding a data point: this is still present on current master, but it looks racy rather than "always the 2nd prompt".

    Environment

    • 2× GPU: RTX 3090 (24 GB) + RTX 3060 (12 GB), PCIe, no NVLink
    • llama.cpp-master 542348a (2026-09-18), CUDA, sm_86
    • unsloth/gemma-4-31B-it-qat-GGUF: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf + mtp-gemma-4-31B-it.gguf (the google/gemma-4-31B-it-assistant drafter) + mmproj-BF16.gguf
    • llama-server ... -sm tensor -ts 3/1 --device CUDA0,CUDA1 -fa on -ctk q8_0 -ctv q8_0 -c 131072 --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-type draft-mtp
    • load log shows the shared-KV global layers: layer 3: sharing with layer 59, layer 0/1/2: sharing with layer 58

    Crash — same stack as the report: ggml/src/ggml-cuda/fattn.cu:730: fatal error from ggml_cuda_flash_attn_ext (BEST_FATTN_KERNEL_NONE), called via common_speculative_impl_draft_mtp::draft. It happens on the second prompt of a session; the first prompt completes normally (graphs reused = 45). The whole server aborts.

    It is not deterministic here. ~20 requests in one session — sequential prompts, 4 concurrent prompts, image prompts, and varying prompt lengths (400 → 7300 tokens) — all completed. So it reads as a race on graph rebuild recycling shard tensors the draft still references, consistent with the use-after-reset analysis above, not a function of "the 2nd prompt" specifically.

    -sm layer is not a usable workaround on this pair (same weights, MTP on, 131072 ctx):

    split	prefill	decode
    tensor 3:1	830 t/s	59.8 t/s
    layer	922 t/s	25.0 t/s
    tensor, no MTP	~1000 t/s	~35 t/s

    LLAMA_GRAPH_REUSE_DISABLE=1: on 542348a this starts and serves normally (measured 56–73 t/s with it), unlike the startup assert reported earlier in the thread — so that no-rebuild alternative no longer fails immediately. I can't confirm it fixes the crash, since I can't reproduce it reliably.

    FWIW the single-KV-head global layers here are cache_k_l58/cache_k_l59, matching the earlier analysis.
16. github-actions

    removed

    stale

    on Sep 19, 2026
17. github-actions

    added

    stale

    on Oct 19, 2026