Eval bug: feat/v4-port-cuda

[Talle]

Name and Version

version: 9340 (781e978)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

CPU: AMD Threadripper PRO (WX-class), 28 vCPUs allocated (Proxmox VM)
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96GB VRAM, compute capability 12.0 (SM120)
Driver: 580.159.03

Models

teamblobfish/DeepSeek-V4-Flash-GGUF, quant IQ2_XS-XL (81.03 GiB, 2.45 BPW, 2 shards)
https://huggingface.co/teamblobfish/DeepSeek-V4-Flash-GGUF

Problem description & steps to reproduce

Summary

63 tensors (including token_embd.weight and ~62 others) fall back to CPU at load time
on a single-GPU RTX PRO 6000 Blackwell (96GB, SM120) setup, despite --n-gpu-layers 999,
and despite trying --no-mmap, --op-offload, --n-cpu-moe 0, and --cache-ram 0 — none of
which changed the outcome. Suspect CUDA getrows kernel coverage gap for the quant
types used in these 63 tensors (only 2 tensors show as q5_K in file-type breakdown,
so the other ~61 are likely custom V4 tensors — indexer/hyperconnection/compressor —
hitting the same kernel-coverage limitation for a different reason).

Environment

• GPU: RTX PRO 6000 Blackwell Workstation Edition, 96GB, compute capability 12.0
• Driver: 580.159.03, CUDA 13.0
• Fork commit: 781e978 (version 9340)
• Model: teamblobfish/DeepSeek-V4-Flash-GGUF, IQ2_XS-XL quant (81.03 GiB, 2.45 BPW)
• Build flags: -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="89;120"

Reproduction

cd ~/llama.cpp-v4port/build/bin/llama-server
--model /models/deepseek-v4-flash-iq2xsxl/IQ2_XS-XL/DeepSeek-V4-Flash-IQ2_XS-XL-00001-of-00002.gguf
--alias deepseek-v4-flash
--host 0.0.0.0 --port 8040
--n-gpu-layers 999
--n-cpu-moe 0
--op-offload
--cache-ram 0
--no-mmap
--no-repack
--ctx-size 8192
--batch-size 2048 --ubatch-size 512
--flash-attn on
--jinja --reasoning off
--metrics
--mlock --fit on
--temp 0.2 --top-p 0.95
--parallel 1
-lv 4 2>&1 | tee ~/dsv4-verbose-nommap.log

Observed

• Load log: token_embd.weight (q5_K) (and 62 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
• 725% CPU usage during prefill despite full GPU offload
• graph splits = 988 (with bs=512), 274 (with bs=1) — high for a 43-layer model with
  no intentional CPU offload
• Prefill throughput flat at ~163-167 tok/s regardless of context size (8K-393K
  tested), batch/ubatch size (default through 2048/1024, capped by a separate crash
  at 4096/1024), --no-mmap, --op-offload, --n-cpu-moe 0, or --cache-ram 0
• Roughly matches upstream's own reference benchmark for a similar quant class on
  comparable hardware (148 t/s pp2048, from the DeepSeek V4  ggml-org/llama.cpp#24162 PR thread) — so this may be
  a shared/known limitation, not fork-specific

Also encountered (separate issue, may be related)

GGML_ASSERT(obj_new) failed in ggml_new_tensor during llama_model_deepseek4::graph
construction, when combining --ctx-size 393216 with --ubatch-size 1024 (--batch-size
4096). Stable at --ubatch-size 512 with identical ctx-size. Full backtrace: [paste]

Question

Is the 63-tensor CPU fallback expected/known, or a bug? If known, is there a
workaround (different quant packaging, a build flag I'm missing) to get these
tensors onto CUDA0?

First Bad Commit

No response

Relevant log output

Logs

log-details.txt