Repository navigation
Qwen3.5: DeepSpeed ZeRO-3 fails to load weights for language_model #45313
Description
System Info
transformersversion: 5.4.0- Platform: Linux (H200 x4)
- Python version: 3.12.0
- DeepSpeed version: 0.18.5
- PyTorch version: 2.8.0+cu128 (CUDA)
Problem
When loading Qwen/Qwen3.5-27B (also tested with 9B) with DeepSpeed ZeRO-3, language_model parameters are reported as MISSING in the load report.
Key | Status | Details
---------------------------------------------------------------------+---------+--------
model.language_model.layers.{0...63}.post_attention_layernorm.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.out_proj.weight | MISSING |
model.language_model.norm.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.in_proj_qkv.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.conv1d.weight | MISSING |
model.language_model.layers.{3...63}.self_attn.q_norm.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.dt_bias | MISSING |
model.language_model.layers.{0...62}.linear_attn.A_log | MISSING |
model.language_model.layers.{0...62}.linear_attn.norm.weight | MISSING |
model.language_model.layers.{0...63}.mlp.gate_proj.weight | MISSING |
model.language_model.layers.{0...63}.mlp.down_proj.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.in_proj_a.weight | MISSING |
model.language_model.layers.{3...63}.self_attn.q_proj.weight | MISSING |
model.language_model.layers.{3...63}.self_attn.k_proj.weight | MISSING |
model.language_model.layers.{0...63}.input_layernorm.weight | MISSING |
model.language_model.layers.{0...63}.mlp.up_proj.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.in_proj_b.weight | MISSING |
model.language_model.layers.{0...62}.linear_attn.in_proj_z.weight | MISSING |
model.language_model.layers.{3...63}.self_attn.k_norm.weight | MISSING |
model.language_model.layers.{3...63}.self_attn.o_proj.weight | MISSING |
model.language_model.layers.{3...63}.self_attn.v_proj.weight | MISSING |
model.language_model.embed_tokens.weight | MISSING |
Cause hypothesis
In conversion_mapping.py, the language_model weight keys are remapped to model only. This conversion is called in _load_pretrained_model when DeepSpeed ZeRO-3 is turned on.
transformers/src/transformers/conversion_mapping.py
Lines 155 to 157 in d081c71
The problem disappears when setting target_patterns="model.language_model" or using ZeRO-2.
Reproduction
from transformers import Qwen3_5ForConditionalGeneration
from transformers.integrations import HfDeepSpeedConfig
ds_cfg = {
"bf16": {"enabled": True},
"train_micro_batch_size_per_gpu": 1,
"train_batch_size": 1,
"zero_optimization": {
"stage": 3,
}
}
dschf = HfDeepSpeedConfig(ds_cfg)
model = Qwen3_5ForConditionalGeneration.from_pretrained("Qwen/Qwen3.5-27B", torch_dtype="auto")Expected Behavior
Model weights should load correctly with DeepSpeed ZeRO-3.
Activity
zucchini-nlp commentedon Apr 8, 2026
Duplicate of #45216 and will be fixed soon in #45314
debOliveira commentedon Apr 9, 2026
I tried the recent patch up to commit 9d840ea, but the issue still remains. Note that it works fine on stage 2 of deepspeed, it only raises on stage 3.
Using the same reproduction section above, I get:
zucchini-nlp commentedon Apr 10, 2026
@Cyrilvallez , then there is definitely smth going on internally in deepspeed
debOliveira commentedon Apr 10, 2026
I tested the new patch at #45358, but still no success. The weights under
model.language_model.layersare being remapped to onlymodel.layers. From what I understand the target beingr"^model.(?!(?:language_model.|visual.))"excludeslanguage_modelstring.A bad fix that I tested (and works) is
BartekKruczek commentedon Apr 14, 2026
Hello 👋
The same problem (PEFT + LoRA + Accelerate + Deepspeed3) on 4 x H200 with Qwen3.5 model.
Edit: transformers 5.5.3, peft 0.18.1, deepspeed 0.18.9, accelerate 1.13.0
debOliveira commentedon Apr 14, 2026
Also tried the patch at #45402, which seemed like a similar issue, but no success. I'm using my bad fix at the moment.
zucchini-nlp commentedon Apr 14, 2026
Sorry for long delay, since Cyril is off until tomorrow I tried to reproduce the code and found the root cause. The prev fix is also not completely correct and works because we accidentally end up exploiting another code path
@Cyrilvallez , the regex in
qwen3_5_textstill matches and replaces sd keys in VLMs. Your PR fixed saving it back, but the loading is broken in DS. This line in loading fn reverts back any conversion, if the converted key is not found in sd (which i believe it not intended). DS load fn, in contrary, doesn't revert any conversion, so it fails to match keys 🙃transformers/src/transformers/core_model_loading.py
Lines 1219 to 1221 in 4396b1b
I will let you decide how to fix it when you're back. Personally I believe we shouldn't revert a conversion if not in state dict (I also found that this loophole affected prev CLIP issues, but it's another story)
@debOliveira we'll fix and add it in the next patch, thanks for pinging again :)
github-actions commentedon May 9, 2026
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Cyrilvallez commentedon May 13, 2026
@zucchini-nlp The fix to revert conversion if not in state_dict was originally added to fix some remote code models, but I don't really like it either. It's kind of weird, and dangerous. Do you know if based on latest changes to all that machinery we still face the issue?
zucchini-nlp commentedon May 18, 2026
Lemme check, tbh there were so many changes in conversion lately so msth might have been accidentally fixed
Agreed that it is dangerous. Do you remember which remote models those were? If it is a model with small num of donwloads, we could deliberately break it and ask owner to pin
transformersversion in model docakshansh47 commentedon May 19, 2026
We hit this exact load report on Qwen3.5-27B + DS-Z3 across 6 training runs and
shipped two complementary workarounds while waiting for the upstream fix.
LoRA serving / adapter side
PEFT writes
base_model.model.model.language_model.layers.N.lora_*keys; vLLM< 0.19 looks up
base_model.model.model.layers.N.*and silently falls back to1-D placeholders, surfacing as
IndexError: too many indices for tensor of dimension 1atvllm/lora/layers/column_parallel_linear.py:slice_lora_b.Strip the prefix in-place before upload:
vLLM 0.19+ accepts both naming conventions (vllm#37375).
Training side under ZeRO-3
Direct attribute access to
lm_head.weight/linear_attn.conv1d.weightreturnsthe rank-local 1-D shard. Wrap in
deepspeed.zero.GatheredParametersfor anyread that happens outside
model.forward():Required for chunked LM heads, hybrid Mamba conv1d kernels, and any custom loss
that reads weights directly. Same architectural mismatch as the Qwen3-Next fix
in #43926; the 2-D
language_model.*weights hit a different branch but the samezero.init()context-manager gap.
zucchini-nlp commentedon May 20, 2026
The script provided above works now for me on
mainbranch so I assume one of the many recent changes on conversion fixed it. Anyone who encounters the issue, please install from source and try againI checked with
zucchini-nlp commentedon May 20, 2026
Also closing as resolved, but if you still see issues feel free to ping me and Cyril
Fail loud on save_pretrained() for unsharded LoRA tensors
Fail loud on save_pretrained() for unsharded LoRA tensors
Fail loud on save_pretrained() for unsharded LoRA tensors
Fail loud on save_pretrained() for unsharded LoRA tensors (#3251)