AttributeError: 'Gemma3Config' object has no attribute 'vocab_size'

@ArthurZucker

System Info

v4.50.0.dev0

Who can help?

@ArthurZucker
@LysandreJik
@xenova

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
My own task or dataset (give details below)

Reproduction

I am trying to run the new Gemma3 model, using version '4.50.0.dev0'.

When loading the model I get the error: 'Gemma3Config' object has no attribute 'vocab_size'.

Looking into this it seems Gemma3Config has vocab_size nested in a "text_config" attribute.

I try to load the model as AutoModelForCausalLM, running it with Gemma3ForConditionalGeneration does not raise this issue. Am I wrong in assuming I can run Gemma 3 as AutoModelForCausalLM?

Expected behavior

Loading the model as AutoModelForCausalLM.from_pretrained without issue.

I experienced the same issue.

I encountered the same issue as well. One might assume that adding vocab_size to the model's config.json would resolve the problem, especially since there is a vocab_size present in this config.json.

However, even after making that adjustment, I faced a multitude of other errors related to missing configuration keys in the transformers loader.

The Persistent Challenge: Beyond Vocabulary Size

The initial error I encountered was a KeyError indicating a missing vocab_size in the configuration. This issue is merely the surface of a more profound systemic problem when loading large transformer models like Gemma in environments such as Text-Generation-WebUI. While adding the missing vocab_size resolves the immediate KeyError, it does not provide a comprehensive solution; instead, it triggers a cascade of subsequent errors related to other missing configuration options. This is not simply a matter of a few isolated omissions; it reflects a fundamental mismatch between the model’s expectations and the configuration capabilities of the environment.

The Cascade of Missing Configuration Options

After addressing the vocab_size issue, I began encountering a series of AttributeError and KeyError exceptions, each necessitating the addition of yet another configuration option. Some of the critical parameters include:

hidden_size: The model requires a specific hidden size, which must be explicitly included in the configuration.
num_hidden_layers: The expected number of hidden layers needs to be defined.
rope_theta & rope_base: These parameters are essential for the RoPE implementation and must be present for accurate position embedding calculations.
intermediate_size: This defines the size of the intermediate feed-forward layer within the transformer.
num_key_value_heads: This is crucial for grouped-query attention mechanisms.
hidden_activation: This specifies the activation function used in the hidden layers.
num_attention_heads: This indicates the number of attention heads.

The process quickly becomes a tedious cycle: add one option, trigger another error, add that option, and so forth. Even after meticulously including all seemingly necessary configuration parameters, the Gemma3 model may still fail to load correctly. This is often due to an internal process known as "dynamic model initialization," compounded by the use of "meta tensors."

[edit I see you mentioned this already in your OP. Leaving the workaround here for visibility]

I resolved this error by using Gemma3ForConditionalGeneration per the model card, rather than AutoModelForCausalLM:

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
)

I'm getting a warning about "The model 'Gemma3ForConditionalGeneration' is not supported for text-generation", but inference works.

@ZipingL

@ZipingL , I think the issue is mainly that by using the AutoModel initialization it somehow passes an Gemma3Config to the Gemma3TextModel, instead of a Gemma3TextConfig, here at L619:

https://github.com/huggingface/transformers/blob/v4.49.0-Gemma-3/src/transformers/models/gemma3/modeling_gemma3.py#L619

All the config args you mention would be in Gemma3Config["text_config"]. Maybe it happens because AutoModel uses AutoConfig to initialize the config, which probably defaults to Gemma3Config, instead of Gemma3TextConfig?

Edit: Looks like my hunch was right, the Gemma3ForConditionalGeneration even uses AutoModelForCausalLM to initialize the language_model but passes config.text_config to it:
https://github.com/huggingface/transformers/blob/v4.49.0-Gemma-3/src/transformers/models/gemma3/modeling_gemma3.py#L1120

In models/auto/modeling_auto.py, line 523 I've tried replacing Gemma3ForCausalLM with Gemma3ForConditionalGeneration and it works for me – Gemma 3 4B successfully loads with AutoModelForCausalLM.from_pretrained. I'm only interested in text generation though, so not sure if image processing would be affected by this quick fix.

Hugging Face maintainer here: I can reproduce the 'Gemma3Config' object has no attribute 'vocab_size' error. Raising the issue internally and we'll try to get a fix ASAP!

Is there a way to use the Gemma3ForConditionalGeneration workaround with vLLM?

Hey all! Yes, Gemma3ForCausalLM cannot be loaded with auto mapping unfortunately, and it is a known limitation. We have it noted, and the reason is that the same hub model-id is loaded with two different tasks in transformers. That in turn causes issues when we try to infer the config type for CausalLM

There could be a dirty workaround, but the idea was to not support AutoModel until a solid solution is available. Loading with Gemma3ForCausalLM should work though

I have the same error when I use vllm images :v0.7.3 with transformers v4.50.0.dev0

I have the same error when I use vllm images :v0.7.3 with transformers v4.50.0.dev0

You need to build vLLM from source. Solution is mentioned in this thread: vllm-project/vllm#14734

When you get it right your version will look like this:

$ pip list | grep vllm
vllm                              0.7.4.dev418+gd47807ba.precompiled /home/xxx/vllm/vllm

@Rocketknight1

hey @Rocketknight1 !!

any idea on when it is going to be fixed and transformers's new version is going to be released with stable gemma3 support?

running gemma3 with transformers(the official tutorial on Hugging Face), but when running it with vllm, I encounter the error: AttributeError: 'Gemma3Config' object has no attribute 'vocab_size'.

Hi folks 👋

With #36741, which will be included in the next patch release, you can load the text-only part of gemma3 with AutoModelForCausalLM:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-4b-it", device_map="auto")

Until then, you can do pip install --upgrade git+https://github.com/huggingface/transformers.git to install from main and unlock this fix.

We apologize for the slower reply: we were discussing a systemic solution for this problem, as we expect it to happen with more modalities in the near future :) #36741 is a band-aid, a more scalable solution is in our plans.

As per comment, seems like the order is intended, so that we first check this specific subclass's attributes and if not found move to global vars. I also reproduced the issue and I see that in 3.14 the class dict has __annotations_cache__ instead of __annotations__ which is the reason i believe

Can you open a PR and add a test case?

@zucchini-nlp

@zucchini-nlp, couple of remarks.

        print(f"Subclass: {cls.__name__}")
        # defined in this particular subclass
        print(f"  start             : {list(cls.__dict__.keys())}")
        child_annotation = cls.__dict__.get('__annotations_cache__', {}).get('config', None)
        print(f"  after child_annotation             : {list(cls.__dict__.keys())}")
        child_attribute = cls.__dict__.get("config_class", None)
        print(f"  after child_attribute             : {list(cls.__dict__.keys())}")

        # defined in the class (this subclass or any parent class)
        full_annotation = get_type_hints(cls).get("config", None)
        print(f"  after full_annotation             : {list(cls.__dict__.keys())}")
        full_attribute = cls.config_class
        print(f"  after full_attribute            : {list(cls.__dict__.keys())}")

prints

  start             : ['__module__', '__firstlineno__', '_tied_weights_keys', '_tp_plan', '_pp_plan', '_keys_to_ignore_on_load_unexpected', '__init__', 'forward', '__annotate_func__', '__static_attributes__', '__doc__']
  after child_annotation             : ['__module__', '__firstlineno__', '_tied_weights_keys', '_tp_plan', '_pp_plan', '_keys_to_ignore_on_load_unexpected', '__init__', 'forward', '__annotate_func__', '__static_attributes__', '__doc__']
  after child_attribute             : ['__module__', '__firstlineno__', '_tied_weights_keys', '_tp_plan', '_pp_plan', '_keys_to_ignore_on_load_unexpected', '__init__', 'forward', '__annotate_func__', '__static_attributes__', '__doc__']
  after full_annotation             : ['__module__', '__firstlineno__', '_tied_weights_keys', '_tp_plan', '_pp_plan', '_keys_to_ignore_on_load_unexpected', '__init__', 'forward', '__annotate_func__', '__static_attributes__', '__doc__', '__annotations_cache__']
  after full_attribute            : ['__module__', '__firstlineno__', '_tied_weights_keys', '_tp_plan', '_pp_plan', '_keys_to_ignore_on_load_unexpected', '__init__', 'forward', '__annotate_func__', '__static_attributes__', '__doc__', '__annotations_cache__']

__annotations_cache__ is created lazily by get_type_hints() when it evaluates and caches the annotations. That seems to be the new behaviour of Python 3.14, where the cache doesn't exist until something triggers annotation evaluation.

The following works

        import annotationlib
        print(f"Subclass: {cls.__name__}")
        # defined in this particular subclass
        # child_annotation = cls.__dict__.get('__annotations_cache__', {}).get('config', None)
        child_annotation = annotationlib.get_annotations(cls, eval_str=True, format=annotationlib.Format.VALUE).get("config", None)
        child_attribute = cls.__dict__.get("config_class", None)

        # defined in the class (this subclass or any parent class)
        # full_annotation = get_type_hints(cls).get("config", None)
        full_annotation = annotationlib.get_annotations(cls, eval_str=True, format=annotationlib.Format.VALUE).get("config", None)
        full_attribute = cls.config_class

        print(f"  child_annotation : {child_annotation}")
        print(f"  child_attribute  : {child_attribute}")
        print(f"  full_annotation  : {full_annotation}")
        print(f"  full_attribute   : {full_attribute}")

        # priority (child class_config -> child annotation -> global class_config -> global annotation)
        if child_attribute is not None:
            print("child_attribute is fired")
            cls.config_class = child_attribute
        elif child_annotation is not None:
            print("child_annotation is fired")
            cls.config_class = child_annotation
        elif full_attribute is not None:
            print("full_attribute is fired")
            cls.config_class = full_attribute
        elif full_annotation is not None:
            print("full_annotation is fired")
            cls.config_class = full_annotation
        print(cls.config_class)

which prints

Subclass: Qwen3_5ForCausalLM
  child_annotation : <class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig'>
  child_attribute  : None
  full_annotation  : <class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig'>
  full_attribute   : <class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5Config'>
child_annotation is fired
<class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig'>

In this tutorial, it is explicitly mentioned

Note: Starting from Python 3.14, you need to access the module’s .annotations attribute at least once before the annotations variable becomes visible in the global namespace.

use annotationlib.get_annotations() and typing.get_type_hints() for more robust introspection.

After those fixes, the following works now:

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B",device_map = "auto",torch_dtype = "auto",
)

I believe we found the cause root. Would you like to open a new PR using annotationlib.get_annotations(), @zucchini-nlp?

@mostafatouny

@mostafatouny i went to git blame to see the history and found that config_class is here only for BC added recently. And since the doc for get_type_hints (https://docs.python.org/3.10/library/typing.html#typing.get_type_hints) states that the current class'es annotations take precedence over other classes in mro, I think your first suggestion on changing the order was correct and the order should've been different from the beginning

We just didn't see issues because __annotations__ were always there in old python, which isn't true anymore with lazy creation

This code works btw, so AutoConfig can correctly get the right class, why it's not just being used? Or what was the intended algorithm for Auto classes, how it shall fetch the correct one automatically, and if it is looking for a text config all the time, then why not just to use get_text_config() internally as well?

Relying too much on magic, annotations and checking for attributes may not be the best approach. I believe there shall be some kinda mapping internally for model types and respective classes.

config = AutoConfig.from_pretrained(model_id)

config = config.get_text_config()

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    config=config,
    device_map="cuda",
    dtype="bfloat16"
)

@Mevrael

@Mevrael Unfortunately we can't want to enforce get_text_config() for all models as it breaks models that rely on AutoModelForCausalLM and need the whole config to be init. We already tried that in the past and had to revert it back due to a number of issues. It was decided that checking with model classes' expected config is more reliable

The "magic" will all typing and stuff comes for different reasons, from #39132. The PR removed cls.config_class in favor of cls.config, so it can resolve the typing hint. The whole block that infers correct config class is thus legacy and is there only for BC. Ideally if we could just drop config_class, it would be much easier and we won't need the "magic"

@zucchini-nlp

I think your first suggestion on changing the order was correct and the order should've been different from the beginning

@zucchini-nlp, I'd always be happy to contribute.

I see that there is a huge const CONFIG_MAPPING_NAMES and it looks up the Config class first

and then a model class is determined from the config class.

Qwen3_5ForCausalLM

which sets self.model = Qwen3_5TextModel(config) in init

where the wrong config class is passed, when it expects Qwen3_5TextConfig

Shouldn't the CONFIG_MAPPING_NAMES by default point qwen3.5 to Qwen3_5TextConfig instead of Qwen3_5Config?

And wouldn't it be better to simply let the main class determine everything and just import the right folder/module dynamically? You don't plan on keeping adding new items to the mapping dictionaries across multiple files every time someone adds a new model and a folder, don't you? :-)

So a Base interface class for instance could have a static method getConfigClass() that returns the right class, and the model class implementing the Base would have to implement this method and return a class, not an object

For example

model = AutoResolver.loadModel('qwen/qwen3.5') would look for a qwen folder and qwen3_5_model file
it would then import it dynamically and hence the model would be of a Qwen3_5_Model class and model.getConfig() would return the right configuration, or the same getConfigClass() from a parent class.

auto class could also automatically fetch the config class

config_cls = AutoResolver.getConfigClass('qwen/qwen3.5')
config = config_cls(...)
or a shortcut
config = AutoConfig('qwen/qwen3.5', ...)
model = AutoModel('qwen/qwen3.5', config=config)

or throw an error if config and model name mismatch right there in the init.

Shouldn't the CONFIG_MAPPING_NAMES by default point qwen3.5 to Qwen3_5TextConfig instead of Qwen3_5Config?

Then we won't be able to load the Qwen3_5ForConditionalGeneration model, it needs Qwen3_5Config, so no. We are trying to use a model which is initially a VLM and load it as an LLM, cropping off the vision support

You don't plan on keeping adding new items to the mapping dictionaries across multiple files every time someone adds a new model and a folder, don't you? :-)

Hope so, we have recently discussed this internally. We might infer the "mapping" dynamically as we already have all necessary bits defined in each config and model class.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@zucchini-nlp

@zucchini-nlp, is it solved or moved to another issue?

Oh sorry, gh closed the issue without any contribution. I will open PR today

I missed tat it was actually fixed in #44672, but not linked to the issue. Checked with last release and works fine :)

Navigation Menu

Repository navigation

AttributeError: 'Gemma3Config' object has no attribute 'vocab_size' #36683

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

The Persistent Challenge: Beyond Vocabulary Size

The Cascade of Missing Configuration Options

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AttributeError: 'Gemma3Config' object has no attribute 'vocab_size' #36683

Description

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

Activity

oobabooga commented on Mar 12, 2025

ZipingL commented on Mar 12, 2025

The Persistent Challenge: Beyond Vocabulary Size

The Cascade of Missing Configuration Options

sam-paech commented on Mar 12, 2025

jumelet commented on Mar 12, 2025

somerandomguyontheweb commented on Mar 13, 2025

Rocketknight1 commented on Mar 13, 2025

rdodev commented on Mar 13, 2025

zucchini-nlp commented on Mar 13, 2025

yonggui521 commented on Mar 14, 2025

bjj commented on Mar 14, 2025

rohitgr7 commented on Mar 17, 2025

ropz12138 commented on Mar 18, 2025

gante commented on Mar 19, 2025

19 remaining items

zucchini-nlp commented on Mar 20, 2026

mostafatouny commented on Mar 20, 2026

zucchini-nlp commented on Mar 20, 2026

Mevrael commented on Mar 20, 2026

zucchini-nlp commented on Mar 20, 2026

mostafatouny commented on Mar 20, 2026

Mevrael commented on Mar 20, 2026

zucchini-nlp commented on Mar 23, 2026

github-actions commented on Apr 17, 2026

mostafatouny commented on Apr 26, 2026

zucchini-nlp commented on Apr 27, 2026

zucchini-nlp commented on Apr 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions