Eval bug: Sm tensor doesn't work

[borisk1]

Name and Version

compiled to use --sm tensor, but with qwen3.6 27b crashes.

Operating systems

Linux

GGML backends

CUDA

Hardware

2xXeon8628 + 4x3090

Models

No response

Problem description & steps to reproduce

compile with cuda_nccl on, and use --sm tensor

First Bad Commit

No response

Relevant log output

Logs

0.30.664.461 I srv  llama_server: model loaded 0.30.664.474 I srv  llama_server: server is listening on http://0.0.0.0:8010 0.30.664.495 I srv  update_slots: all slots are idle 0.39.673.814 I srv  params_from_: Chat format: peg-native 0.39.685.962 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1 0.40.331.044 I srv  recurrent_sh: shrunk recurrent state to 1 cells for prompt cache (before prompt cache save/load, removed 1 backup cells) 0.40.331.049 I srv  get_availabl: updating prompt cache 0.40.331.057 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000 0.40.331.063 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 262144.000 MiB, 256000 tokens, 274877906944 est) 0.40.928.309 I srv  recurrent_ex: expanded recurrent state to 2 cells after prompt cache (after prompt cache save/load) 0.40.928.313 I srv  get_availabl: prompt cache update took 597.26 ms 0.40.929.138 I reasoning-budget: activated, budget=2147483647 tokens 0.40.929.166 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0 /app/ggml/src/ggml-backend-meta.cpp:814: GGML_ASSERT(src_ss[i].axis != GGML_BACKEND_SPLIT_AXIS_UNKNOWN) failed libggml-base.so.0(+0x1c9bb)[0x7fdf367789bb] libggml-base.so.0(ggml_print_backtrace+0x21f)[0x7fdf36778e3f] libggml-base.so.0(ggml_abort+0x152)[0x7fdf36779012] libggml-base.so.0(+0x49c49)[0x7fdf367a5c49] libggml-base.so.0(+0x40a99)[0x7fdf3679ca99] libggml-base.so.0(+0x49dbf)[0x7fdf367a5dbf] libggml-base.so.0(ggml_gallocr_alloc_graph+0x474)[0x7fdf367902e4] libggml-base.so.0(ggml_backend_sched_alloc_graph+0x111)[0x7fdf367966e1] libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xd4)[0x7fdf36942344] libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0xb19)[0x7fdf36959389] libllama.so.0(llama_decode+0x10)[0x7fdf3695e1f0] libllama-common.so.0(_ZN33common_speculative_impl_draft_mtp7processERK11llama_batch+0x2dd)[0x7fdf36fce83d] libllama-common.so.0(_Z26common_speculative_processP18common_speculativeRK11llama_batch+0x2d)[0x7fdf36fbd18d] libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x3d98)[0x7fdf37796208] libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x221)[0x7fdf37824641] libllama-server-impl.so(_Z12llama_serveriPPc+0x29c7)[0x7fdf376d52b7] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fdf371bbd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fdf371bbe40] ./llama-server(+0x12b5)[0x5b6bd46a92b5]

[Anbeeld]

Will investigate.

[efschu]

Will investigate.

It would be awesome to see the following at the same time:
-sm tensor
-cram
-slot save/load
-parallel >1
-mtp dflash
and quantized KV
and sleep mode

Would kickass with every other llm engine.

[Anbeeld]

Investigated. This is a crash in llama.cpp's tensor-parallel backend, and it isn't specific to this fork.

With --sm tensor, every tensor in the compute graph gets a shard layout assigned across your GPUs. When the MTP (nextn) speculative draft runs its decode, one op in that graph ends up with an empty, zero-element tensor as an input, and the layout pass has no rule for sharding an empty input, so it aborts at the GGML_ASSERT in ggml/src/ggml-backend-meta.cpp from your traceback. It only triggers at the intersection of two features, tensor parallelism and the MTP draft. Single-GPU and --sm layer never run that layout pass, which is why the server loads and sits idle fine and only dies on the first draft.

Both halves of that intersection are stock llama.cpp. The tensor-parallel meta backend is ggml-org/llama.cpp#19378, MTP speculative decoding is ggml-org/llama.cpp#22673, and the gate that decides whether --sm tensor is allowed for an architecture (llm_arch_supports_sm_tensor) is upstream too and doesn't exclude Qwen3.6. None of this fork's own work (DFlash, TurboQuant, KVarN) is on that path. A stock llama.cpp built with NCCL would hit the same assert running an MTP draft.

To keep serving, avoid the intersection: either keep the MTP draft and use --sm layer instead of --sm tensor, or keep --sm tensor and disable the MTP draft. Layer split runs the 27B across your 4x3090s without the crash.

The real fix has to teach upstream's tensor-parallel backend to accept an empty tensor as an op input, which it already accepts as an op output; that asymmetry is the bug. That change belongs in ggml-org/llama.cpp, so it's better tracked there than patched around in this fork.

[efschu]

@Anbeeld
I'm using -sm tensor with mtp on llama.cpp for a while now. So what you say can not be true. The think is, you only can use F16 or Q8_0 kv cache while using sm tensor and mtp. So it's a quant thing, not a sm tensor/mtp thing.

[antdea]

@Anbeeld I can also confirm MTP + tensor split works on llama.cpp but doesn't work on beellama.cpp

When using kvarn6/kvarn5 I get the following error

0.10.385.962 E llama_init_from_model: failed to initialize the context: KVarN cache layer 3 is assigned to backend Meta(), which has no native KVarN operations or does not meet KVarN kernel limits; use a backend with native KVarN support, disable KV offload, or enable KVarN CPU fallback
0.10.385.966 E common_init_result: failed to create context with model '/home/antdea/Workspace/models/Qwen3.6-27B-Q6_K_MTP.gguf'
0.10.385.969 E common_init_from_params: failed to create context with model '/home/antdea/Workspace/models/Qwen3.6-27B-Q6_K_MTP.gguf'
./run.sh: line 43: 88143 Segmentation fault (core dumped) ./llama-server --model "/home/antdea/Workspace/models/Qwen3.6-27B-Q6_K_MTP.gguf" --port 8080 -np 2 --parallel 1 --ctx-size 50000 --cache-type-k kvarn6 --cache-type-v kvarn5 --flash-attn on --jinja --no-mmap --mlock --no-host --reasoning on --chat-template-kwargs '{"preserve_thinking":true}' --repeat-penalty 1.1 --temp 0.1 --top-k 20 --top-p 0.95 --min-p 0.0 --gpu-layers 99 --split-mode tensor --tensor-split 12,20 --spec-type draft-mtp --spec-draft-n-max 2

When running with the exact same script that works when pointed at llama.cpp. Using q8_0

0.00.210.526 I log_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.210.528 I device_info:
0.00.371.728 I - CUDA0 : NVIDIA GeForce RTX 5080 (15808 MiB, 13317 MiB free)
0.00.525.064 I - CUDA1 : NVIDIA GeForce RTX 3080 (20054 MiB, 19817 MiB free)
0.00.525.080 I - CPU : AMD Ryzen 7 7800X3D 8-Core Processor (63659 MiB, 63659 MiB free)
0.00.525.177 I system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 860,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.525.211 I srv init: running without SSL
0.00.525.236 I srv init: using 15 threads for HTTP server
0.00.525.333 I srv start: binding port with default address family
0.00.526.481 I srv llama_server: loading model
0.00.526.488 I srv load_model: loading model '/home/antdea/Workspace/models/Qwen3.6-27B-Q6_K_MTP.gguf'
0.02.140.095 I srv load_model: [spec] estimated memory usage of MTP context is 343.02 MiB
0.02.140.107 I common_init_result: fitting params to device memory ...
0.02.140.107 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.02.140.145 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
0.02.802.069 W warning: failed to mlock 1042944000-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
0.10.459.542 W llama_context: n_ctx_seq (50176) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.10.459.569 W NCCL not compiled in; falling back to internal AllReduce. Recompile with -DGGML_CUDA_NCCL=ON for best multi-GPU performance.
0.10.540.535 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
TCQ decode: context-adaptive V alpha enabled
0.10.731.621 I srv load_model: creating MTP draft context against the target model '/home/antdea/Workspace/models/Qwen3.6-27B-Q6_K_MTP.gguf'
0.10.731.650 W llama_context: n_ctx_seq (50176) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.10.731.665 W NCCL not compiled in; falling back to internal AllReduce. Recompile with -DGGML_CUDA_NCCL=ON for best multi-GPU performance.
0.10.795.247 I srv load_model: initializing slots, n_slots = 1
0.10.839.239 I common_context_can_seq_rm: the context supports bounded partial sequence removal
0.10.842.770 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.10.842.773 I common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=5120, backend_sampling=1
0.10.842.774 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
/home/antdea/Workspace/beellama.cpp/ggml/src/ggml-backend-meta.cpp:540: meta backend per-row op top_k (TOP_K) cannot use axis-0 split src0 logits_seq_0 (VIEW)
[New LWP 92348]
[New LWP 92347]
[New LWP 92346]
[New LWP 92345]
[New LWP 92344]
[New LWP 92296]
[New LWP 92295]
[New LWP 92294]
[New LWP 92293]
[New LWP 92292]
[New LWP 92291]
[New LWP 92290]
[New LWP 92289]
[New LWP 92288]
[New LWP 92287]
[New LWP 92286]
[New LWP 92285]
[New LWP 92284]
[New LWP 92283]
[New LWP 92282]
[New LWP 92281]
[New LWP 92268]
[New LWP 92267]
This GDB supports auto-downloading debuginfo from the following URLs:
ima:enforcing
https://debuginfod.fedoraproject.org/
ima:ignore
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f7d96c82412 in __syscall_cancel_arch () from /lib64/libc.so.6
#0 0x00007f7d96c82412 in __syscall_cancel_arch () from /lib64/libc.so.6
#1 0x00007f7d96c7662c in __internal_syscall_cancel () from /lib64/libc.so.6
#2 0x00007f7d96c76674 in __syscall_cancel () from /lib64/libc.so.6
#3 0x00007f7d96ce624f in wait4 () from /lib64/libc.so.6
#4 0x00007f7da374915b in ggml_print_backtrace () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#5 0x00007f7da37492cd in ggml_abort () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#6 0x00007f7da37677ff in ggml_backend_meta_get_split_state(ggml_backend_meta_simple_tensor_container&, ggml_tensor const*, bool)::{lambda(std::vector<ggml_backend_meta_split_state, std::allocator<ggml_backend_meta_split_state> > const&)#1}::operator()(std::vector<ggml_backend_meta_split_state, std::allocator<ggml_backend_meta_split_state> > const&) const [clone .part.0] () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#7 0x00007f7da376fca2 in ggml_backend_meta_get_split_state(ggml_backend_meta_simple_tensor_container&, ggml_tensor const*, bool)::{lambda()#1}::operator()() const () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#8 0x00007f7da376b1c7 in ggml_backend_meta_get_split_state(ggml_backend_meta_simple_tensor_container&, ggml_tensor const*, bool) () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#9 0x00007f7da3777a1a in ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_meta_simple_tensor_container&, ggml_tensor*) [clone .isra.0] () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#10 0x00007f7da3779dba in ggml_backend_meta_buffer_init_tensor(ggml_backend_buffer*, ggml_tensor*) () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#11 0x00007f7da375f695 in ggml_gallocr_alloc_graph () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#12 0x00007f7da3765c21 in ggml_backend_sched_alloc_graph () from /home/antdea/Workspace/beellama.cpp/build/bin/libggml-base.so.0
#13 0x00007f7da3a59325 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/antdea/Workspace/beellama.cpp/build/bin/libllama.so.0
#14 0x00007f7da3a647e4 in llama_context::decode(llama_batch const&) () from /home/antdea/Workspace/beellama.cpp/build/bin/libllama.so.0
#15 0x00007f7da3a694ce in llama_decode () from /home/antdea/Workspace/beellama.cpp/build/bin/libllama.so.0
#16 0x00007f7da3f513ac in common_context_can_seq_rm(llama_context*) () from /home/antdea/Workspace/beellama.cpp/build/bin/libllama-common.so.0
#17 0x00007f7da453372f in server_context_impl::load_model(common_params&) () from /home/antdea/Workspace/beellama.cpp/build/bin/libllama-server-impl.so
#18 0x00007f7da444fcaa in llama_server(int, char**) () from /home/antdea/Workspace/beellama.cpp/build/bin/libllama-server-impl.so
#19 0x00007f7d96c0a681 in __libc_start_call_main () from /lib64/libc.so.6
#20 0x00007f7d96c0a798 in __libc_start_main_impl () from /lib64/libc.so.6
#21 0x00000000004003b5 in _start ()
[Inferior 1 (process 92265) detached]
./run.sh: line 42: 92265 Aborted (core dumped) ./llama-server --model "/home/antdea/Workspace/models/Qwen3.6-27B-Q6_K_MTP.gguf" --port 8080 -np 2 --parallel 1 --ctx-size 50000 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja --no-mmap --mlock --no-host --reasoning on --chat-template-kwargs '{"preserve_thinking":true}' --repeat-penalty 1.1 --temp 0.1 --top-k 20 --top-p 0.95 --min-p 0.0 --gpu-layers 99 --split-mode tensor --tensor-split 12,20 --spec-type draft-mtp --spec-draft-n-max 2

beellama.cpp compiled against branch 0.3.2

cmake -B build -DGGML_CUDA=ON -DCUDAToolkit_ROOT=/usr/local/cuda-13.3 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-15 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/nvcc