Repository navigation
Add the support for deepseek architecture .gguf #36144
Description
Feature request
The current version does not support gguf under the deepseek architecture. It is hoped that the deepseek architecture will be added. [supported-model-architectures]
Motivation
In some framework based transformers (e.g. vllm) will raise error when load .gguf file of deepseek model or quantized deepseek model.
- [Usage]: Does DeepSeek-R1 1.58-bit Dynamic Quant work on VLLM? · Issue #12573 · vllm-project/vllm
- unsloth/DeepSeek-R1-GGUF · Running the model with vLLM does not actually work
Your contribution
Is there any guidance to help users add relevant support?
Activity
Rocketknight1 commentedon Feb 12, 2025
cc @SunMarc @muellerzr @MekkCyber - who's the right person to ping for GGUF loading?
MekkCyber commentedon Feb 12, 2025
I think me and @SunMarc and @Isotr0py
Isotr0py commentedon Feb 12, 2025
Seems that all deepseek-r1 gguf checkpoints are sharded, I think we should add sharded gguf support firstly.
Isotr0py commentedon Feb 12, 2025
Anyway, regardless the sharded gguf weights (we can merge weights with tool from llama.cpp), will #35926 block us currently? Because the deepseek-v3 support hasn't landed yet.
MekkCyber commentedon Feb 12, 2025
yes deepseek v3 is still not supported for now, the pr is functional but some small adjustments are needed
z7d1 commentedon Feb 13, 2025
Hello @MekkCyber ! Isn't the reason that the transformers don't support it because the deepseek .gguf file can't be merged?
MekkCyber commentedon Feb 13, 2025
Hello @zh-jp! I think it can be merged using
llama-gguf-split --mergebut i'm not sure if it's a good ideammdbhs commentedon Feb 14, 2025
we need GGUF supported! please!
Jackarry188 commentedon Feb 14, 2025
we need GGUF supported! please!
floyddcn commentedon Feb 17, 2025
we need GGUF supported! please!
htkim27 commentedon Feb 18, 2025
we need GGUF supported! please!
fonhorst commentedon Feb 18, 2025
we need GGUF supported! please!
shangdibufashi commentedon Feb 19, 2025
we need GGUF supported! please!
maguih commentedon Feb 27, 2025
we need GGUF supported! please!
tywuAMD commentedon Mar 4, 2025
We need GGUF supported! Please!
MiyaaL commentedon Mar 13, 2025
We need GGUF supported! Please!
binghanc commentedon Sep 22, 2025
We need GGUF supported! Please!
ArthurZucker commentedon Oct 20, 2026
DeepSeek GGUF is still not supported, but the path is now well trodden: add a
deepseek2entry (llama.cpp's arch name for V2/V3/R1) toGGUF_CONFIG_MAPPINGinsrc/transformers/integrations/ggml.py, map it todeepseek_v3insrc/transformers/modeling_gguf_pytorch_utils.py, and handle stacked expert tensors like the existing MoE support (see_split_moe_expert_tensorand MiniMax-M2 support in #44526, orqwen3_moe). The MLAkv_btensors will likely need a customTensorProcessor. Docs: https://huggingface.co/docs/transformers/main/en/quantization/gguf; contributions welcome.