Repository navigation
[LUKE] Tie the masked LM decoder to the word embeddings - #50043
tarekziade wants to merge 2 commits into
Conversation
LukeForMaskedLM._tied_weights_keys lost the lm_head.decoder.weight entry in the v5 weight-loading refactor, so checkpoints that do not store it load a randomly initialized LM head. Add the entry and a regression test.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: luke |
CI (Nvidia + AMD)Nvidia · ✅ No new failures 🎉👏 · ⚙️ Workflow · 📊 Dashboard Failures (2 pre-existing)Pre-existing failures (still failing) — 2 total: Same error (2): - luke: AMD · ✅ No new failures 🎉👏 · ⚙️ Workflow · 📊 Dashboard Failures (2 pre-existing)Pre-existing failures (still failing) — 2 total: Same error (2): - luke: |
LukeForMaskedLM._tied_weights_keyslost thelm_head.decoder.weight -> luke.embeddings.word_embeddings.weightentry when it was converted to a dict in the v5 weight-loading refactor (#41580). LUKE/mLUKE checkpoints don't storelm_head.decoder.weight, so loadingstudio-ousia/luke-basereports it as missing and the LM head is randomly initialized. "The capital of France is ." predicts['airline', 'arbitrary', 'Tom', ...]. With the entry restored it predicts['Paris', 'Lyon', 'Nancy', 'Nice', 'Brussels'].Fix: restore the tied-weights entry, with the same mapping as RoBERTa. The bias and the entity prediction head were already tied.
Found by a model integration contract (transformers-mic) run on
studio-ousia/luke-base.🤖 Generated with Claude Code