Repository navigation
Eval bug: same K and V cache type enforced for models with no V cache #26382
Description
Name and Version
version: 10216 (876a432)
built with Clang 22.1.8 for Linux x86_64
Operating systems
Linux
GGML backends
CUDA
Hardware
Models
GLM-5.2
Problem description & steps to reproduce
When loading GLM-5.2 with -ctk q5_1 and no -ctv:
llama_init_from_model: model does not support different K (q5_1) and V (f16) cache types
First Bad Commit
No response
Activity
fairydreaming commentedon Aug 1, 2026
This is intentional - it's to make sure that any existing logic in the code that depends on V cache type still works correctly (so we don't have to introduce ...and if it's a model without separate V cache then check K cache type instead... conditions). Simply set V cache type to the same value as K cache type and you are good to go.
JomboKabary commentedon Aug 2, 2026
@fairydreaming I understand the desire to simplify the logic for models without a separate V-cache (like MLA), but this enforcement is a significant regression for users targeting high context windows on limited VRAM.
Previously I could run GLM-5.2 with -ctk q8_0 -ctv q5_1 fine. Keeping K at q8_0 preserved the model's attention accuracy while reducing q5_1 for V allowed me to fit more context without any significant degradation unlike enfocring both to q5_1 the degradation is severe.
By enforcing params.type_k == params.type_v in llama-context.cpp, I am now forced into two sub-optimal choices, either I set both to q8_0 I lose around 32K of context to stay within VRAM limits. Or I set both to q5_1, which as I said, the model's output quality degrades noticeably compared to the hybrid setup.
The specific check causing this is at src/llama-context.cpp:3561:
if ((model->hparams.is_mla() || model->arch == LLM_ARCH_DEEPSEEK4) && params.type_k != params.type_v) {
LLAMA_LOG_ERROR("%s: model does not support different K (%s) and V (%s) cache types\n", func, ggml_type_name(params.type_k), ggml_type_name(params.type_v));
return nullptr;
}
Even if the architecture uses a unified latent for KV, the implementation previously allowed these flags to independently affect the quantization of the cache segments, which provided a very balanced middle ground for performance.
Could we consider removing this strict enforcement, or at least allowing it as an "at your own risk" override with a special command line flag? It would restore the ability to run long-context models that were working fine just a few versions ago.
Thank you.
fairydreaming commentedon Aug 2, 2026
Well, you only had an illusion of control there as only
-ctkvalue was taken into account. If you run the server with--verboseyou will see for example:V cache buffer was zero-length for GLM-5.2 no matter what you set in
-ctv. The reason is that there's a single common latent representation used as both K and V. It's kept in llama.cpp K buffer, but internally it's used as both K and V. That's why it's not possible to set different K and V quantizations as in DSA they are the same latent vectors kept in a single buffer. Even for MLA models since @ggerganov implemented #19067 there was no separate V cache.Yes, implementation allowed to set different quantization values, but used only K cache type and ignored the second value. Well, not really ignored as it's used for some other checks, but no cache buffer was created based on this type.
Maybe the changes that you see in memory usage are the result of #25407 that finally implemented full GLM-5.2 DSA architecture? It added lightning indexer KV cache which increased overall memory usage.
wolfpld commentedon Aug 2, 2026
There is also the MTP layer, which for some time was loaded unconditionally. All in all, I have also observed increased memory pressure with GLM 5.2 after these changes.
Seems easy to fix? Just set the V cache type to K, if model only has K, and V was not set, at ~parameter parsing stage. This gives the single code path you describe.
fairydreaming commentedon Aug 2, 2026
@wolfpld Considered this too, but the "V was not set at parameter parsing stage" doesn't seem simple to check, as V cache type is set to
GGML_TYPE_F16by default, so it's not known inside the code if someone explicitly set it to f16 or if it was simply not set by the user at all. Therefore probably another internal boolean flag likev_type_setwould be needed (likely in several places) and passed around, which sounded like PITA.fairydreaming commentedon Aug 2, 2026
@JomboKabary If you are low on VRAM try this: #26474
It shall considerably reduce indexer cache size. In my test for q8_0 K type and 65536 cells indexer cache went from 663.00 MiB to 178.50 MiB, so ~0.5GB saved.
JomboKabary commentedon Aug 2, 2026
I think that explains it then. I wasn't aware that -ctv was a placebo for MLA models in previous versions. Seeing the VRAM usage spike on my RTX 3090 (24GB) made me think the enforcement was the culprit, but it makes total sense that it was actually the DSA indexer cache from #25407 pushing the memory footprint over the edge and causing the spill into System RAM and drop my t/s generation speed.
@fairydreaming Thanks for the technical explanation and for opening #26474 to optimize the indexer cache. I guess that ~0.5GB saving is probably what I need to keep the 200k context fully on the GPU. I'll try to test the PR branch. Thank you.
fairydreaming commentedon Aug 4, 2026
@JomboKabary b10243 and later already contain this fix, let me know how it works for you.
JomboKabary commentedon Aug 7, 2026
Tested three different versions today to be absolutely sure. Turns out you were right about
-ctvbeing a placebo for this model in previous builds.Setup for reference:
Dual Xeon E5-2699 v3 | 256GB DDR4 ECC (2133MHz, Quad Channel) | RTX 3090 | Bare TTY (no X server running)
Test runs:
The memory fix in #26474 definitely worked, the latest versions actually fit slightly more tokens than the old build. The 206k context I saw before was probably due to my Kernel update (7.1.5-gentoo) eating up more baseline VRAM. I would need to re-run the tests with my old kernel to verify if that was the case then.
While I'm here, I wonder if would it be possible to add q6_0 KV cache support in the future? I think it would be a nice middle ground for 24GB cards where q8_0 is just a bit too heavy for 200k+ context, but q5_1/q4_1 starts to show quality loss.
Thanks for the technical explanation and the quick optimization on the indexer.
fairydreaming commentedon Aug 7, 2026
@JomboKabary Currently there's no Q6_0 type in llama.cpp but I see there is #10848 so perhaps make some noise there.
github-actions commentedon Sep 21, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.