Repository navigation
The get_(text|image|audio|video)_features methods have inconsistent output formats, needs aligning for Sentence Transformers #42401
Description
Hello!
Bug Report/Feature Request overview
- The
get_text_features,get_image_features, etc. methods of many base models have inconsistent output formats.
Details
The multimodal architectures in transformers can process all supported modalities simultaneously through the forward method of the base (i.e. without the head) model (e.g. CLIPModel), and many of these architectures also support getting embeddings of a single modality with methods in the form of get_(text|image|audio|video)_features, e.g. get_text_features and get_image_features for CLIP.
Without these methods, it's not possible to get embeddings/hidden states of a single modality without requiring the other modalities as well, e.g. getting text embeddings for CLIP without passing an image as well.
For context, we have:
- 20 architectures with
get_text_features - 64 architectures with
get_image_features - 7 architectures with
get_audio_features - 15 architectures with
get_video_features
These methods generally output pooled embeddings, i.e. the modality-specific dimension (seq_length, width, height) are pooled away, leaving only a shape of (batch_size, num_dimensions), but this is not consistently applied in transformers.
For example, in the 21 architectures that implement get_text_features, blip_2 its get_text_features outputs the logits of the language model, e.g. with a shape (batch_size, seq_length, vocab_size).
See here a script to quickly reproduce the discrepancy/misalignment:
import torch
from transformers import AutoTokenizer, AutoModel
# model_name = "Salesforce/blip2-opt-2.7b"
model_name = "openai/clip-vit-base-patch32"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
print(type(model))
inputs = tokenizer(["a photo of a cat"], padding=True, return_tensors="pt")
with torch.inference_mode():
text_features = model.get_text_features(**inputs)
print(text_features.shape)
# torch.Size([1, 6, 50304])
# torch.Size([1, 512])I see the same with audio, where get_image_features from deepseek_vl returns (batch_size, seq_length, num_dimensions) with torch.Size([1, 576, 2048]), while the exact same script with clip its get_image_features returns (batch_size, num_dimensions) with torch.Size([1, 512])
See here a script to quickly reproduce the discrepancy/misalignment, but now for images of multi-modal models:
import requests
from PIL import Image
from transformers import AutoImageProcessor, AutoModel
import torch
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
# model_name = "deepseek-community/deepseek-vl-1.3b-base"
model_name = "openai/clip-vit-base-patch32"
processor = AutoImageProcessor.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
print(type(model))
inputs = processor(images=image, return_tensors="pt")
with torch.inference_mode():
image_features = model.get_image_features(**inputs)
print(image_features.shape)
# torch.Size([1, 576, 2048])
# torch.Size([1, 512])I didn't look into the audio and video ones, but even short glances show that there's large discrepancies there too. The get_video_features from qwel3_vl even returns a tuple of image_embeds and deepstack_image_embeds, and for audio I see pooled embeddings (clap, audioflamingo3) and unpooled embeddings (qwen2_5_omni), and tuples (gemma3n).
I think it's crucial that we align the output structures of these methods. Beyond that, I think it's very valuable if we expose the non-pooled embeddings (e.g. last_hidden_states) via these methods as well. This would allow for much stronger multi-modality support in Sentence Transformers, for example.
I propose that we introduce a new parameter to each of the get_(text|image|audio|video)_features methods, something along the lines of pooling: bool = True. This keeps the behaviour identical for most models, but aligns the expected output for all models (of a modality), while also allowing Sentence Transformers and related projects to access the modality-specific embeddings without an arbitrary pooling procedure being applied on top. This accessibility is very important for third party libraries to properly use multi-modal models for new/novel use cases.
cc @ArthurZucker @Cyrilvallez @zucchini-nlp
cc @LysandreJik
- Tom Aarsen
Activity
zucchini-nlp commentedon Nov 26, 2025
I think we have two large groups of models with different behavior and expectation from
get_xxx_featuresmethod. CLIP-like models would return a pooled embedding (if there is a pooling layer, otherwise last hidden state) so that it can be used by users for image-text matching or simply to encode an imageVision LLMs however return an image encoding ready to be concatenated with text embeddings and each model has their own way of post-processing these encodings. Some models do pooling, some do pixel shuffle and some even return several outputs (qwen3-vl). We usually use this utility 1) in the model's forward so the code is more modularized and can be easily overriden new models are dded and 2) we use it in vLLM integration which expects to get/cache image embeddings right before the
concatenationoperation.I agree that we could standardize it between the two groups of models, especially if we want to make it easier for Sentence Transformers to use VLMs as encoding models. Personally, I prefer to keep the VLM-style and we could return more options when requested in a NamedTuple format. WDYT?
tomaarsen commentedon Dec 2, 2025
With the VLM-style, do you refer to "pooled embeddings" as the default option? I.e. always a
(batch_size, hidden_size)format?And the NamedTuple, should that only be returned if some parameter is specified? I was thinking that perhaps we could create more subclasses of the
ModelOutputfor this instead (i.e. OrderedDict-style instead of NamedTuple-style).zucchini-nlp commentedon Dec 2, 2025
Yeah, I mean "pooled" outputs as default single tensor instead of returning a Tuple/Dict. Otherwise it will be quite breaking and afaik the helper is used quite commonly. So, I prefer to leave as is by default and
return_dictwhen asked similar to how model's forward worksNo big issue for me, whether it's a ModelOutput or NamedTuple. We just need to get a set of common output keys that will work for all models :)
tomaarsen commentedon Dec 2, 2025
I think returning a
(batch_size, hidden_size)format by default, with areturn_dictthat returns aModelOutputsubclass would be perfect, withpooler_outputandlast_hidden_stateas the common output keys, and more options for specific architectures that require it.github-actions commentedon Dec 27, 2025
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
tomaarsen commentedon Dec 28, 2025
Still being worked on.