CUDA: 4-bit KV cache (q4_1/q4_0) collapses prefill to ~34 t/s on qwen35 hybrid (RTX 3090, MMQ guard passes)

Name and Version

$ ./llama-server --version
version: 0.1.0-dev (build 10178, commit e9e3fdd15)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux (kernel 7.0.0-29-generic)

Which llama.cpp modules do you know to be affected?

libllama (core library)
llama-server

Command line

llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -a qwen3.8-27b \
  --host 127.0.0.1 --port 8080 -ngl 99 -c 130000 \
  --cache-type-k q4_1 --cache-type-v q8_0 \
  --flash-attn on -b 2048 -ub 512 -np 1 --threads-batch 12 \
  --temp 0.6 --min-p 0.05 --repeat-penalty 1.0 --top-p 0.95 --top-k 20 \
  --spec-type draft-mtp --spec-draft-n-max 4 --mlock --metrics --cache-ram 2048

Problem description & steps to reproduce

Switching the K cache from q8_0 to q4_1 collapses prompt processing (prefill) speed by ~20x on a qwen35 hybrid architecture model (attention + recurrent memory layers, 65 blocks, 17 attention layers, GQA 4x256). Generation speed is unaffected.

Measured on an RTX 3090 (compute capability 8.6), driver 580.173.02, CUDA 13.0, full GPU offload, --flash-attn on, cache_prompt=false:

Config	Prefill (4-10K prompt)	Generation
q8_0 / q8_0	991-1276 t/s	~60 t/s
K=q4_1, V=q8_0	34-106 t/s	~60 t/s

At 65K+ prompt lengths prefill is slow enough that requests hit the 600s timeout (a 90K prompt takes ~44 min at ~34 t/s).

Important: the MMQ shared-memory guard from #26141 (caa596ab3) is NOT the trigger here — it passes on this system. Measured: cudaDevAttrMaxSharedMemoryPerBlockOptin returns 101376 B (99 KiB) on the RTX 3090, so smpbo < 48 * 1024 is false and MMQ stays enabled. Issue #26285 reports a similar symptom from an older build; on the current build the guard does not cause the fallback.

A 4-bit V cache showed the same collapse earlier (31 Jul, older build): PP ~195 t/s with V=q4_0 vs ~1174 t/s after returning V to q8_0. So the slow path appears to be the CUDA flash-attention / quantized-KV dequant kernel selection for 4-bit KV on this hybrid architecture, independent of the shared-memory guard.

Steps to reproduce:

Start llama-server with --cache-type-k q4_1 --cache-type-v q8_0 (+ --flash-attn on, full offload) on a qwen35 hybrid model.
Send a ~8K token prompt with cache_prompt=false.
Observe prefill at ~40-100 t/s instead of ~1000+ t/s.
Switch to q8_0/q8_0, restart, same prompt -> ~1000+ t/s.

First Bad Commit

Not bisected. Reproduces on e9e3fdd15 (14 Aug 2026 upstream + cherry-picked PR #25592). The same 4-bit KV prefill collapse was observed on 31 Jul with 4-bit V on an older build.

Relevant log output

Logs

I slot print_timing: id  0 | task 1152 | prompt processing, n_tokens =   6144, progress = 0.07, t =  57.95 s / 106.03 tokens per second
I slot print_timing: id  0 | task 1152 | prompt processing, n_tokens =   8192, progress = 0.09, t = 103.49 s /  79.16 tokens per second
I slot print_timing: id  0 | task 1152 | prompt processing, n_tokens =  10240, progress = 0.12, t = 165.77 s /  61.77 tokens per second
I slot print_timing: id  0 | task 1152 | prompt processing, n_tokens =  16384, progress = 0.19, t = 481.66 s /  34.02 tokens per second

With q8_0/q8_0 the same server logs ~1174-1276 t/s at these context sizes.

Root cause found: CUDA FA kernel does not support q4_1 KV (and K != V) in default builds

The prefill collapse is not caused by the model, the MMQ guard, or a slow dequant path. It is a silent backend fallback: when the CUDA flash-attention kernel cannot handle the requested KV types, the scheduler places the whole FLASH_ATTN_EXT op on the CPU backend. Prefill then runs on CPU at ~34 t/s.

The kernel selection logic in ggml/src/ggml-cuda/fattn.cu (ggml_cuda_get_best_fattn_kernel) is gated by the GGML_CUDA_FA_ALL_QUANTS compile flag, which is OFF by default:

// fattn.cu:338-356
static bool ggml_cuda_fattn_kv_type_supported(ggml_type type) {
    switch (type) {
        case GGML_TYPE_F32:
        case GGML_TYPE_F16:
            return true;
        case GGML_TYPE_Q4_1:
        case GGML_TYPE_Q5_0:
        case GGML_TYPE_Q5_1:
#ifndef GGML_CUDA_FA_ALL_QUANTS
            return false;              // <- q4_1 unsupported in default build
#endif
        case GGML_TYPE_Q4_0:
        case GGML_TYPE_Q8_0:
        case GGML_TYPE_BF16:
            return true;
        ...
    }
}

// fattn.cu:442-446 — also requires K == V in default builds
#ifndef GGML_CUDA_FA_ALL_QUANTS
    if (K->type != V->type) {
        return BEST_FATTN_KERNEL_NONE; // <- K=q4_1, V=q8_0 fails here
    }
#endif

So on a default build:

K=q4_1, V=q8_0 -> unsupported (q4_1) and K != V -> BEST_FATTN_KERNEL_NONE -> op runs on CPU
K=q4_0, V=q8_0 -> K != V -> same fallback
K=q8_0, V=q8_0 and K=q4_0, V=q4_0 -> supported, stays on GPU

ggml_cuda_flash_attn_ext_supported() returns false, so the graph scheduler moves the attention op (and the KV data it needs) to the CPU backend. No error, no offload log message — just a massive prefill slowdown.

Fix: rebuild with GGML_CUDA_FA_ALL_QUANTS=ON

cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j

This compiles the additional FA quant kernels (q4_1/q5_0/q5_1) and allows K != V type mixes.

Verification (RTX 3090, same qwen35 hybrid, build 10178 + this flag)

Context	Prefill before (q4_1, CPU fallback)	Prefill after (q4_1/q8_0, GPU)
~20K	~34 t/s	1094 t/s
~55K	timeout territory	896 t/s
~90K	600s timeout	752 t/s
~120K	>44 min (timeout)	659 t/s

Generation speed unchanged (~24-44 t/s with MTP draft, draft acceptance 41-62%). No offload messages in the log; /health and timings confirm GPU execution.

Suggestion: consider making GGML_CUDA_FA_ALL_QUANTS default ON, or at least printing a clear warning (and ideally checking ggml_cuda_flash_attn_ext_supported against the requested cache types at startup) when a quantized KV type cannot be served by the CUDA FA kernel — a silent 30x prefill regression is very hard to diagnose from the outside.

I have fixed this issue, but not fully optimized. You can try to check either PR #27140 or prebuild without compiling llama.cpp: https://github.com/Cybertiron/llama.cpp/releases/tag/b27109-quant-kv-fix — I invite you to test and confirm if it fixed your issue. Or wait for maintainers to fix it.

@Cybertiron

@Cybertiron thanks for the fix and the prebuild!

For anyone hitting this: we solved it on our side without the PR by building with -DGGML_CUDA_FA_ALL_QUANTS=ON — that removes the kernel-selection fallback that drops FLASH_ATTN_EXT to CPU (the root cause documented in my earlier comment).

Verified on 1× RTX 3090, Qwen3.8-27B IQ4_NL, K=q4_1 / V=q8_0, 130K ctx, MTP draft (n-max=4), llama_bench with shuffled real-corpus prompts (seed=42, 2-run avg):

Context	PP t/s	GEN t/s
~20K	1094	44.4
~65K	852	39.7
~120K	659	23.6

So the 34 t/s collapse is gone and we're running this in production. There's still a residual gap vs q8_0/q8_0 (~1159 t/s PP at short context) that grows with context — if that's the slow dequant path your PR targets, a merge would close it. We're not in a rush to test the cherry-pick, but happy to run it against our setup if the maintainers want more data points before merging.

Release builds without GGML_CUDA_FA_ALL_QUANTS=ON
is so frustrating.

I need q5_1 / q4_1 to get useful KV cache size in my limited VRAM . I always forget of the limitations when downloading the latest bug fix build, and get stuck with q8 (too big) or q4 (terribile for long context).

So I have to update the git repo and compile , every time. Multiply that by every user of alternative KV quants... it's such a waste!

@aand18

@aand18 same pain on our side — it's the third report of this silent fallback in this thread now.

For the record, the root cause is documented in my first comment here: without the flag, the CUDA FA kernel rejects q4_1/q5_0/q5_1 (and K != V mixes), and the scheduler silently moves the whole FLASH_ATTN_EXT op to CPU — no error, no offload message, just a ~30x prefill collapse.

Workaround until this lands: rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON (or use @Cybertiron's prebuild from his comment).

The ask to maintainers stands: either make GGML_CUDA_FA_ALL_QUANTS default ON, or at minimum print a clear warning at startup when the requested KV cache types can't be served by the CUDA FA kernel — a silent 30x regression is very hard to diagnose from the outside.

Great news I have lowered KV cache to q3_K size. Uses less vram than q4_0 about ~24%. Arguably it doesnt lose quality. I am releasing it, but it needs testing

github.com/Cybertiron/llama.cpp-rtx3090-prefill-fix/releases/tag/ampere-kv-v0.1

Add q3_K KV cache type + fork README

docs: make release notes q3_K-focused, ggml-org#27109 fix only as lin…

@Cybertiron

@Cybertiron thanks for the q3_K build — ~24% less VRAM than q4_0.

For anyone hitting this: on our side (RTX 3090, Qwen3.8-27B) we work around it without any PR by building with -DGGML_CUDA_FA_ALL_QUANTS=ON — that makes the CUDA Flash-Attention kernel support q4_1/q4_0 KV, so the silent fallback to the slow prefill path (the root cause in my first comment) never triggers.
I hope this fix will be added to the upstream...

cuda: enable GGML_CUDA_FA_ALL_QUANTS by default

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

CUDA: 4-bit KV cache (q4_1/q4_0) collapses prefill to ~34 t/s on qwen35 hybrid (RTX 3090, MMQ guard passes) #27109

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Root cause found: CUDA FA kernel does not support q4_1 KV (and K != V) in default builds

Fix: rebuild with GGML_CUDA_FA_ALL_QUANTS=ON

Verification (RTX 3090, same qwen35 hybrid, build 10178 + this flag)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CUDA: 4-bit KV cache (q4_1/q4_0) collapses prefill to ~34 t/s on qwen35 hybrid (RTX 3090, MMQ guard passes) #27109

Description

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

47Hunter47 commented on Aug 15, 2026

Root cause found: CUDA FA kernel does not support q4_1 KV (and K != V) in default builds

Fix: rebuild with GGML_CUDA_FA_ALL_QUANTS=ON

Verification (RTX 3090, same qwen35 hybrid, build 10178 + this flag)

Cybertiron commented on Aug 15, 2026

47Hunter47 commented on Aug 15, 2026

aand18 commented on Aug 17, 2026

47Hunter47 commented on Aug 17, 2026

Cybertiron commented on Aug 17, 2026

47Hunter47 commented on Aug 28, 2026

github-actions commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions