Repository navigation
fix(audio): Route the ALM conversion scripts through from_pretrained - #48525
Conversation
|
@zucchini-nlp In relation to the #37033 update, the VLM conversion scripts might need similar updates as well. I checked a few, and LLaVA and a couple of models that inherit from it don't load their weights. Happy to raise the VLM equivalent after a thorough investigation, please do let me know if you have a refactor planned :) |
|
Oke, thanks for checking that. I don't think it is a prio atm since conversion scripts aren't really maintained/used, and for new releases we try to put all the renames in |
|
Makes sense, thanks, will leave the VLM ones then! |
ebezzam
left a comment
ebezzam
left a comment
There was a problem hiding this comment.
thanks @harshaljanjani! as @zucchini-nlp said, we don't actively maintain the conversion scripts, so i don't have a strong opinion on whether to merge this. But every now and then, I point existing conversion scripts to a contributor so it could be nice that the script doesn't fail if they are trying it?
eustlb
left a comment
eustlb
left a comment
There was a problem hiding this comment.
LGTM thanks indeed this will give better refs for contributors
|
Good day; just checking in to see if there've been any updates on this :) |
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Nice optim, are there more new recent models with the same issue?
|
Thanks @vasqu; none of the other audio ones. Outside ALMs though the DETR family has the same issue. Happy to include those in this PR as well and re-title it; if not, here are the models for future reference. LW-DETR, D-FINE, RT-DETR, RT-DETRv2 and Deformable DETR, and the fix is pretty similar I think to this PR's. |
|
Let's do it in another PR then, we should keep ALMs separate to the vision models imo |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, glmasr, musicflamingo, vibevoice_asr, voxtral, voxtral_realtime |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…uggingface#48525) fix: Route the ALM conversion scripts through from_pretrained Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Add UltraVBert: UltraBERT text encoder with a SigLIP vision encoder * Cap UltraVBert page tiles and add its model doc * Trim UltraVBert comments * Document UltraVBert's image_hidden_states and image-processor kwargs, date its model doc * Add UltraVBert integration tests on the released checkpoint * Skip SDPA flash dispatch for UltraVBert's sliding-window text tower * Describe UltraVBert as extending ModernVBert in its model doc * Update docs/source/en/model_doc/ultravbert.md Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Load the receipt fixture with load_image in the UltraVBert usage example * Split the UltraVBert usage example into AutoModel and Pipeline tabs * Always RMS-normalize UltraVBert's image tokens and drop connector_norm * Batch UltraVBert's capped resize, write its image processing out in full and default the tile cap to 27 * Reuse ModernVBert's MLM head layout and forwards in UltraVBert and test it with the VLM tester * Address review: drop the PIL image processor, adopt ModernVBert task tests * Drop SigLIP's unused pooling head from UltraVBert * Default UltraVBert's vision config to no pooling head through init_kwargs * last stuff * fix model parallel bug for molmo2 model (huggingface#49816) Signed-off-by: kaixuanliu <kaixuan.liu@intel.com> * fix(audio): Route the ALM conversion scripts through from_pretrained (huggingface#48525) fix: Route the ALM conversion scripts through from_pretrained Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> --------- Signed-off-by: kaixuanliu <kaixuan.liu@intel.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> Co-authored-by: vasqu <antonprogamer@gmail.com> Co-authored-by: kaixuanliu <kaixuan.liu@intel.com> Co-authored-by: Harshal Janjani <harshaljanjani@gmail.com>
What does this PR do?
→ #45534 nested the ALM models under
model.and added the renames to conversion_mapping.py, but those are only applied byfrom_pretrained, so every one of these scripts has been loading 0% of its weights since then→ One nit with GLM-ASR, #41250 dropped
GlmAsrEncoderConfig.head_dim, so I've changed that to the way the model does itTested the following, which broke before and now produce valid outputs (transcriptions/captions).
→ zai-org/GLM-ASR-Nano-2512
→ mistralai/Voxtral-Mini-3B-2507
→ mistralai/Voxtral-Mini-4B-Realtime-2602
→ nvidia/music-flamingo-2601-hf
→ nvidia/audio-flamingo-3
→ nvidia/audio-flamingo-next-hf
→ microsoft/VibeVoice-ASR
cc: @eustlb @ebezzam
Before submitting