Repository navigation
[QUESTION] When will model have _extra_state? #900
Description
After updating to the main branch of Megatron-LM recently, I met this error when loading model:
Unexpected key(s) in state_dict: "decoder.layers.0.self_attention.core_attention._extra_state"
The checkpoint is transformed by the tools/checkpoint/convert.py, and loaded by pretrain_gpt.py.
Reacted by Benet, lanlin, Yifei Zuo and sambar1729
Activity
sambar1729 commentedon Aug 9, 2024
I had faced this error too, recently, and had to write a downstream script to remove this from every layer -- looks like the Mcore format is out of sync with what MegatronLM consumes?