🚨 [ALM] Add base model without head by eustlb · Pull Request #45534 · huggingface/transformers

eustlb · 2026-04-20T15:09:17Z

What does this PR do?

This is motivated to simplify changes required for vLLM compatibility as seen in vLLM PR #39330

Fix a discrepancy in ALMs design compared to VLMs that for most of them don't have a base model class because the use the causal model for the text model directly, while Llava style (which is indeed more aligned with the lib philosophy) is rather:
XxModel: encoder + projector + language base model backbone
XxForConditionalGeneration: XxModel + lm_head

why do we actually have this discrepancy?

qwen2_audio was the first ALM in transformers, modelling reflects the status of llava when it was added
all the ALMs merged since then replicated this.

llava and VLMs evolved thanks to

🚨 Breaking changes

We don't fully ensure BC, this is breaking:

AutoModel.from_pretrained("Qwen/Qwen2-Audio-7B") now returns Qwen2AudioModel, not Qwen2AudioForConditionalGeneration, some for other ALMs 🚨
Auto mappings: AutoModel now returns base model (Qwen2AudioModel)
likewise model.language_model is still returned but it now returns base model and not the causal one as before 🚨
model.language_model on ForContionalGeneation class is not accessible anymore. Use model.get_decoder()

how do we ensure BC (the more possible)

weight loading: thanks to the dynamic weight loader and conversion mapping
ensure models cards for the references hub checkpoints in doc and test don't use AutoModel (since this will be broken)

models to which this applies

TODO

make sure to remove the get/set_input_embeddings when [MultimodalLM] add language_model to the get/set_input_embeddings logic #46029 is merged

# Conflicts: # src/transformers/models/audioflamingo3/modeling_audioflamingo3.py # src/transformers/models/glmasr/modeling_glmasr.py # src/transformers/models/granite_speech/modeling_granite_speech.py # src/transformers/models/granite_speech_plus/modeling_granite_speech_plus.py # src/transformers/models/musicflamingo/modeling_musicflamingo.py # src/transformers/models/musicflamingo/modular_musicflamingo.py # src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py # src/transformers/models/voxtral/modeling_voxtral.py # src/transformers/models/voxtral/modular_voxtral.py # src/transformers/models/voxtral_realtime/modeling_voxtral_realtime.py # tests/models/audioflamingo3/test_modeling_audioflamingo3.py # tests/models/glmasr/test_modeling_glmasr.py # tests/models/granite_speech/test_modeling_granite_speech.py # tests/models/musicflamingo/test_modeling_musicflamingo.py # tests/models/qwen2_audio/test_modeling_qwen2_audio.py # tests/models/vibevoice_asr/test_modeling_vibevoice_asr.py # tests/models/voxtral/test_modeling_voxtral.py

…rmers into alm-base-model-class

github-actions · 2026-05-26T09:18:39Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, glmasr, granite_speech, granite_speech_plus, musicflamingo, parakeet, qwen2_audio, vibevoice_asr, voxtral, voxtral_realtime

* add a base model class * ensure BC via conversion mapping * auto classes * test updates * ensure BC for accessing attributes * simplify conversion mapping * convert modular * convert modular * apply to voxtral * convert modular * remove test_model_base_model_prefix overwrite * make * make * XXXModel class in doc * add GraniteSpeechPlusModel * tests now have a base_model_class * fix qwen2audio * class level base mappings * fix test_reverse_loading_mapping * fix * fix * nit * fix * deprec in 5.7 -> 5.15 * fix flaky test * fix flaky test * fix flaky test * remove forward_base_model_attrs * _supports_attention_backend in PretrainedModel * removed redundant get/set_input_embeddings * remove requires_grad_ handling * make fix-repo * fix voxtral realtime * update vibevoice_asr test * update test * Update src/transformers/models/audioflamingo3/modular_audioflamingo3.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/audioflamingo3/modular_audioflamingo3.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/glmasr/modeling_glmasr.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * make fix-repo --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fix * fix * unnecessary and misleading

* add a base model class * ensure BC via conversion mapping * auto classes * test updates * ensure BC for accessing attributes * simplify conversion mapping * convert modular * convert modular * apply to voxtral * convert modular * remove test_model_base_model_prefix overwrite * make * make * XXXModel class in doc * add GraniteSpeechPlusModel * tests now have a base_model_class * fix qwen2audio * class level base mappings * fix test_reverse_loading_mapping * fix * fix * nit * fix * deprec in 5.7 -> 5.15 * fix flaky test * fix flaky test * fix flaky test * remove forward_base_model_attrs * _supports_attention_backend in PretrainedModel * removed redundant get/set_input_embeddings * remove requires_grad_ handling * make fix-repo * fix voxtral realtime * update vibevoice_asr test * update test * Update src/transformers/models/audioflamingo3/modular_audioflamingo3.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/audioflamingo3/modular_audioflamingo3.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/glmasr/modeling_glmasr.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * make fix-repo --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fix * fix * unnecessary and misleading

* add a base model class * ensure BC via conversion mapping * auto classes * test updates * ensure BC for accessing attributes * simplify conversion mapping * convert modular * convert modular * apply to voxtral * convert modular * remove test_model_base_model_prefix overwrite * make * make * XXXModel class in doc * add GraniteSpeechPlusModel * tests now have a base_model_class * fix qwen2audio * class level base mappings * fix test_reverse_loading_mapping * fix * fix * nit * fix * deprec in 5.7 -> 5.15 * fix flaky test * fix flaky test * fix flaky test * remove forward_base_model_attrs * _supports_attention_backend in PretrainedModel * removed redundant get/set_input_embeddings * remove requires_grad_ handling * make fix-repo * fix voxtral realtime * update vibevoice_asr test * update test * Update src/transformers/models/audioflamingo3/modular_audioflamingo3.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/audioflamingo3/modular_audioflamingo3.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/models/glmasr/modeling_glmasr.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * make fix-repo --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fix * fix * unnecessary and misleading

Dependabot's transformers 5.5.4 -> 5.10.1 bump (#6602) fails every unit-test job on one test, test_espnet_model_ps2st.py::test_espnet_model_inference. Two separate consequences of the same upstream refactor -- "[ALM] Add base model without head" (huggingface/transformers#45534), which splits an audio LM the way VLMs are already split. 1. `Qwen2AudioForConditionalGeneration.language_model` is gone. The text model now lives at `.model.language_model`, so reading bos/eos off it raised AttributeError. Rather than reach through the new path, take the ids from `config.get_text_config()`: that is the config `language_model` was built from in both versions, and `get_text_config()` is byte-identical in 5.5.4 and 5.10.1, so this does not need a version branch. 2. The dummy pytest model came out with mixed dtypes. The checkpoint config asks for bfloat16 and `AutoModel.from_config` honours it, but `lm_head` and `audio_tower` are now constructed on the outer model at the default dtype. In 5.5.4 the lm_head was inside `AutoModelForCausalLM` and came out bfloat16 with everything else; in 5.10.1 it is float32 while the hidden states feeding it are bfloat16, and the matmul refuses. Pin the shrunken test config to float32 so the whole dummy model is one dtype. Only the second fix is new behaviour, and only in `pytest_mode`; a real `from_pretrained` load gets its dtypes from the checkpoint. Verified against both versions, since the pin only moves once #6602 merges: transformers 5.5.4 before: pass after: pass transformers 5.10.1 before: fail after: pass Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

eustlb added 4 commits April 20, 2026 16:59

add a base model class

4cd44e4

ensure BC via conversion mapping

79b833b

auto classes

57436db

test updates

1bd269c

This was referenced Apr 20, 2026

feat: Add audio support for the Transformers backend vllm-project/vllm#39330

Merged

audio tester class #45391

Merged

ensure BC for accessing attributes

9010366

eustlb changed the title ~~ALM base model class~~ 🚨 [ALM] Add base model without head Apr 22, 2026

simplify conversion mapping

af429ef

eustlb added 18 commits May 11, 2026 18:48

Merge branch 'main' into alm-base-model-class

e5eb9a6

convert modular

1da57e8

convert modular

7481402

apply to voxtral

cf7c5f1

convert modular

83799ce

remove test_model_base_model_prefix overwrite

464fae5

make

744567c

make

687b693

XXXModel class in doc

ff8b9f5

add GraniteSpeechPlusModel

753b755

tests now have a base_model_class

ecce440

fix qwen2audio

1f5e199

class level base mappings

6a1166b

Merge branch 'main' into alm-base-model-class

674e64c

fix test_reverse_loading_mapping

933f8c6

Merge branch 'alm-base-model-class' of github.com:huggingface/transfo…

642bdff

…rmers into alm-base-model-class

fix

5383313

eustlb added this pull request to the merge queue May 26, 2026

Merged via the queue into main with commit 027d1a9 May 26, 2026
31 checks passed

eustlb deleted the alm-base-model-class branch May 26, 2026 09:47

eustlb mentioned this pull request May 26, 2026

Support Granite Speech NAR (NLE) #46031

Open

3 of 6 tasks

ebezzam mentioned this pull request Jun 4, 2026

fix[vLLM x v5]: Default untied embeddings in AudioFlamingo3 and VibeVoice #46400

Merged

3 of 6 tasks

hmellor mentioned this pull request Jun 5, 2026

Bump Transformers version to 5.10.4 vllm-project/vllm#41359

Merged

6 tasks done

pull Bot pushed a commit to j3din00b/transformers that referenced this pull request Jun 8, 2026

[fix] regression introduced by huggingface#45534 (huggingface#46456)

b6bad75

* fix * fix * unnecessary and misleading

khushali9 pushed a commit to khushali9/transformers that referenced this pull request Jun 8, 2026

[fix] regression introduced by huggingface#45534 (huggingface#46456)

a3cabf0

* fix * fix * unnecessary and misleading

louzongzhi pushed a commit to louzongzhi/transformers that referenced this pull request Jun 10, 2026

[fix] regression introduced by huggingface#45534 (huggingface#46456)

995d794

* fix * fix * unnecessary and misleading

louzongzhi pushed a commit to louzongzhi/transformers that referenced this pull request Jun 10, 2026

[fix] regression introduced by huggingface#45534 (huggingface#46456)

94c2ebe

* fix * fix * unnecessary and misleading

ArthurZucker pushed a commit that referenced this pull request Jun 11, 2026

[fix] regression introduced by #45534 (#46456)

9caa4e4

* fix * fix * unnecessary and misleading

wentian-byte mentioned this pull request Jun 20, 2026

[CI][test] beam_search MM test: also filter audio_bos/eos, not just audio_token vllm-project/vllm#46242

Closed

ebezzam mentioned this pull request Jun 22, 2026

Add Fun-ASR-Nano model #46180

Merged

This was referenced Jun 23, 2026

[Bugfix] Qwen2-Audio: override top-level tie_word_embeddings=False vllm-project/vllm#46479

Closed

[Bug]: Qwen2-Audio inference broken on transformers >= 5.x — generates only <|audio_bos|> / <|audio_eos|> instead of text vllm-project/vllm#46250

Closed

harshaljanjani mentioned this pull request Jul 13, 2026

fix(musicflamingo): Fix dtype mismatch with _keep_in_fp32_modules_strict override #47300

Closed

4 tasks done

ebezzam mentioned this pull request Jul 14, 2026

[AudioFlamingo3] Fix bfloat16 dtype mismatch in audio encoder positional embedding #47258

Merged

stevhliu pushed a commit to stevhliu/transformers that referenced this pull request Jul 30, 2026

[fix] regression introduced by huggingface#45534 (huggingface#46456)

e402c75

* fix * fix * unnecessary and misleading

This was referenced Sep 3, 2026

Make the Qwen2-Audio ps2st model work on transformers 5.10 espnet/espnet#6622

Merged

Bump transformers from 5.5.4 to 5.10.1 espnet/espnet#6602

Merged

harshaljanjani mentioned this pull request Sep 4, 2026

fix(audio): Route the ALM conversion scripts through from_pretrained #48525

Merged

1 task done

ArthurZucker mentioned this pull request Oct 20, 2026

[nit] glmasr should be in AutoModelForMultimodalLM #45670

Closed

pszemraj mentioned this pull request Oct 23, 2026

GraniteSpeech: logits_scaling not applied since v5.10 (logits are 8x too large) #49937

Open

Navigation Menu

Repository navigation

🚨 [ALM] Add base model without head - #45534

🚨 [ALM] Add base model without head#45534
eustlb merged 52 commits into
mainfrom
alm-base-model-class

eustlb commented Apr 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

eustlb commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

why do we actually have this discrepancy?

🚨 Breaking changes

how do we ensure BC (the more possible)

models to which this applies

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eustlb commented Apr 20, 2026 •

edited

Loading