Upgrade to HF transformers >= v5

We need to upgrade docling to use the latest transformers (major) version 5

main problem seems to be that we currently still have huggingface-hub>=1.3.0,<2.0 somehow if I understand the problem correctly.

The current blocker is that some Docling models fail running with the new transformers. We are trying to address this.

would it make sense to stage this into 2 steps

remove the constraints
fix the models

I saw some code snippets which are explicitly verifying compliance with a <5 version of transformers i.e. around phi-4.

I think it would be neat if docling itself would be less coupled to a specific version and the user could more flexibly choose based on what models they use to find a compatible setup.

We might have to make some workaround, yes. But first let's assess the situation in a few days, it might just all align properly.

If someone wants to create a checklist of sorts of the various parts of docling that need to be updated and assign tasks I'd be willing to help, just FYI. I'm familiar with the API changes if an outline would help as well?

any update here?

We might have to make some workaround, yes. But first let's assess the situation in a few days, it might just all align properly.

new models like glm-ocr or lightonocr-2 are blocked by this transformers-v4 limitation

@geoHeil

@geoHeil we are actively working on it, hopefully today should be resolved (#3200). We needed to wait for a fix in transformers v5.

Here's a guide that I've found somewhat useful:

GUIDE HERE

# Transformers v5: Unified Migration Guide

Hugging Face Transformers v5.0.0, released January 26, 2026, is the library's first major version bump in five years. It touches nearly every public API surface: TensorFlow and JAX are gone, tokenization is rebuilt from the ground up, weight loading is rearchitected, Trainer sheds dozens of parameters, generation is consolidated, and pipelines are pruned. Weekly releases (v5.1–v5.3 already shipped) continue landing breaking changes, so pinning to an exact minor version is advisable during migration.

This guide catalogs every significant change, organized by subsystem.

---

## 1. TensorFlow and JAX Removal — PyTorch Is the Sole Backend

The single largest breaking change is the **complete removal of all TensorFlow and Flax/JAX model code**. Every `TF*` and `Flax*` class — `TFAutoModel`, `TFBertModel`, `FlaxAutoModel`, `FlaxBertModel`, and hundreds of others — no longer exists. Any import of these classes raises `ImportError`. If your codebase touches TensorFlow or JAX through Transformers, you must rewrite it in PyTorch.

This decision allows the team to focus fully on `torch` and greatly reduces maintenance cost. Work with Jax ecosystem tools (such as MaxText) continues to explore compatibility while keeping `torch` as the only backend.

Linked PR: https://github.com/huggingface/transformers/pull/40760

### Minimum Requirements

- **Python ≥ 3.10** is required from v5.0.0 onward (`setup.py` specifies `python_requires=">=3.10.0"`)
- **PyTorch ≥ 2.4** is the minimum required version
- **`huggingface_hub` ≥ 1.3.0, < 2.0** — see [Requirements Update](#17-requirements-update) below
- **`typer-slim`** added as a required dependency (powers the redesigned CLI)

---

## 2. Dynamic Weight Loading (WeightConverter API)

A new weight loading API significantly improves on the previous one. Instead of loading a checkpoint exactly as serialized, operations can reshape, merge, and split layers — a necessity for quantization and parallelism algorithms.

The API centers on the `WeightConverter` class:

```python
class WeightConverter(WeightTransform):
    operations: list[ConversionOps]
    source_keys: Union[str, list[str]]
    target_keys: Union[str, list[str]]

For example, fusing query/key/value attention layers:

conversion = WeightConverter(
    ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"],  # Input layers
    "self_attn.qkv_proj",  # Single output layer
    operations=[Concatenate(dim=0)],
)

Key improvements:

Much cleaner definition of transformations applied to checkpoints
Reversible transformations — loading and saving should round-trip cleanly
Faster model loading via scheduled tensor materialization
Enables complex transformation combinations (quantization + MoEs, TP + MoEs)

Expect varying levels of support across release candidates while implementation continues.

Linked PR: huggingface/transformers#41580

3. Model Loading Default Changes

The from_pretrained() method looks the same but behaves differently in three critical ways:

3a. Default dtype is now `auto` instead of `float32`

Models load in whatever dtype they were saved in. Code that assumed float32 and never specified torch_dtype may now get float16 or bfloat16 weights, silently changing numerical behavior. Specify torch_dtype=torch.float32 explicitly if you need the old behavior.

3b. Only safetensors serialization is supported

The safe_serialization parameter has been removed from all save_pretrained and push_to_hub methods. Passing safe_serialization=False raises an error. You cannot save pickle-based .bin files through the API — use torch.save(model.state_dict(), ...) manually if needed.

Linked PR: huggingface/transformers#42556

3c. Default shard size increased to 50 GB

The default shard size went from 5 GB to 50 GB, avoiding tens or hundreds of weight files for large models. This was made possible by the Xet backend allowing efficient serving of very large files. The decision was taken after careful benchmarking of load speed and optimization.

Linked PR: huggingface/transformers#42556

3d. `use_auth_token` removed everywhere

The use_auth_token parameter is replaced by token in all methods (from_pretrained(), push_to_hub(), etc.). A global search-and-replace handles it.

Linked PR: huggingface/transformers#41666

4. Tokenization Overhaul

Design Philosophy

Just as models moved to a single backend, tokenizers are being made much more intuitive. With v5, tokenizer definition is simpler: you can initialize an empty tokenizer and train it directly on your corpus, making tokenizers behave like models — trained or empty, defined entirely by their class.

Backend Architecture: Ending the Slow/Fast Split

Up to now, transformers maintained two parallel implementations:

"Slow" tokenizers (tokenization_<model>.py) — Python-based, often using SentencePiece
"Fast" tokenizers (tokenization_<model>_fast.py) — Rust-based, using 🤗 tokenizers

In v5, these consolidate to a single file per model: tokenization_<model>.py, using the most appropriate backend:

TokenizersBackend (preferred): Rust-based from 🤗 tokenizers. Provides optimal performance plus features like added token handling, full Python API, automatic parallelization, offsets, customization, and training.
SentencePieceBackend: For tokenizers requiring sentencepiece. Inherits from PythonBackend.
PythonBackend (formerly PreTrainedTokenizer): Python implementation of tokenizer features. Allows adding tokens and custom implementations without relying on external backends.
MistralCommonBackend (formerly MistralCommonTokenizer): Relies on MistralCommon's tokenization library.

AutoTokenizer automatically selects the appropriate backend based on available files and dependencies — transparent to the user.

Defining Custom Tokenizers

For the highest-level custom implementation, PreTrainedTokenizerBase (what PythonBackend inherits from) provides the core API: encode, decode, vocab_size, get_vocab, convert_tokens_to_ids, convert_ids_to_tokens, from_pretrained, save_pretrained, and others.

Example of defining a new tokenizer:

from transformers import TokenizersBackend, generate_merges
from tokenizers import pre_tokenizers, Tokenizer
from tokenizers.model import BPE

class Llama5Tokenizer(TokenizersBackend):
    def __init__(self, unk_token="<unk>", bos_token="<s>", eos_token="</s>",
                 vocab=None, merges=None):
        if vocab is None:
            self._vocab = {
                str(unk_token): 0, str(bos_token): 1, str(eos_token): 2,
            }
        else:
            self._vocab = vocab
        self._merges = merges or []
        self._tokenizer = Tokenizer(
            BPE(vocab=self._vocab, merges=self._merges, fuse_unk=True)
        )
        self._tokenizer.pre_tokenizer = pre_tokenizers.Metaspace(
            replacement="▁",
            prepend_scheme=_get_prepend_scheme(self.add_prefix_space, self),
            split=False,
        )
        super().__init__(
            tokenizer_object=self._tokenizer,
            unk_token=unk_token, bos_token=bos_token, eos_token=eos_token,
        )

Note for SentencePiece-based tokenizers: Override the convert_from_spm class method in your converter to customize vocabulary structure, normalizers, regexes, and anything passed to the tokenizers you are converting.

Tokenizer API Changes

Direct initialization with vocab and merges

You can now initialize blank, untrained tokenizers:

from transformers import LlamaTokenizer

tokenizer = LlamaTokenizer()  # Empty, trainable

Or initialize from vocab and merges:

vocab = {"<unk>": 0, "<s>": 1, "</s>": 2, "hello": 3, "world": 4}
merges = [("h", "e"), ("l", "l"), ("o", " ")]
tokenizer = LlamaTokenizer(vocab=vocab, merges=merges)

Simplified file loading: vocab and merges accept file paths directly. The tokenizer auto-detects format (SentencePiece .model, Tekken tekken.json, or plain files). For BPE tokenizers, if vocab is provided without merges, merges are auto-generated (excluding special tokens). Note: File-path loading is primarily for quick testing.

Unified `decode` method

batch_decode and decode are unified. Both single and batch decoding now use decode:

tokenizer = AutoTokenizer.from_pretrained("t5-small")
inputs = ["hey how are you?", "fine"]
tokenizer.decode(tokenizer.encode(inputs))
# v4: 'hey how are you?</s> fine</s>'
# v5: ['hey how are you?</s>', 'fine</s>']

`encode_plus` → `call`

The encode_plus method is deprecated in favor of the single __call__ method.

`apply_chat_template` returns `BatchEncoding`

Previously returned raw input_ids; now consistently returns a BatchEncoding dict:

messages = [
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there!"},
]
outputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
print(outputs.keys())  # dict_keys(['input_ids', 'attention_mask'])

Legacy file saving removed

special_tokens_map.json — special tokens now stored in tokenizer_config.json
added_tokens.json — added tokens now stored in tokenizer.json
added_tokens_decoder — only stored when there is no tokenizer.json
add_bos_token / add_eos_token — no longer saved in tokenizer_config.json when tokenizer.json exists

Backend synchronization removed: Automatic post-initialization synchronization of backend settings (add_prefix_space, do_lower_case, strip_accents, tokenize_chinese_chars) is gone. Behavior is now fully determined by tokenizer.json or the class definition at initialization time.

Model-specific consolidations

Models with identical tokenizers now import from their base:

LayoutLM → BertTokenizer
LED → BartTokenizer
Longformer → RobertaTokenizer
LXMert → BertTokenizer
MT5 → T5Tokenizer
MVP → BartTokenizer

These wrapper modules will eventually be removed.

Removed: The internal _eventually_correct_t5_max_length method. T5 tokenizers now handle max length consistently with other models.

Special tokens restructured

SpecialTokensMixin merged into PreTrainedTokenizerBase
special_tokens_map now only stores named special tokens (bos_token, eos_token, etc.)
additional_special_tokens → extra_special_tokens (auto-converted during init)
additional_special_tokens_ids → extra_special_tokens_ids
special_tokens_map_extended and all_special_tokens_extended → removed
extra_special_tokens only accepts list/tuple and is for initialization; for model-specific named tokens (e.g., image_token), pass directly as keyword arguments

# v4
tokenizer.special_tokens_map  # Included 'additional_special_tokens'

# v5
tokenizer.special_tokens_map  # Only named tokens
tokenizer.extra_special_tokens  # Additional tokens

Deprecated and removed methods

Deprecated:

sanitize_special_tokens() — removed in v5 (was already deprecated in v4)
prepare_seq2seq_batch() — use __call__() with text_target:

# v4
model_inputs = tokenizer.prepare_seq2seq_batch(src_texts, tgt_texts, max_length=128)

# v5
model_inputs = tokenizer(src_texts, text_target=tgt_texts, max_length=128, return_tensors="pt")
model_inputs["labels"] = model_inputs.pop("input_ids_target")

BatchEncoding.words() → use word_ids() instead

Removed from base class:

create_token_type_ids_from_sequences() — subclasses needing it must implement directly
prepare_for_model(), build_inputs_with_special_tokens(), truncate_sequences() — moved to tokenization_python.py for PythonBackend only
_switch_to_input_mode(), _switch_to_target_mode(), as_target_tokenizer() — use __call__() with text_target:

# v4
with tokenizer.as_target_tokenizer():
    labels = tokenizer(tgt_texts, ...)

# v5
labels = tokenizer(text_target=tgt_texts, ...)

parse_response() — removed from base class

Testing changes

Model-specific test files now focus on integration tests
Common API tests (add_tokens, encode, decode) are centralized and auto-applied across all tokenizers
Legacy BERT Python tokenizer code (WhitespaceTokenizer, BasicTokenizer, etc.) is preserved in bert_legacy.py for reference

Linked PRs

5. Quantization Changes

The load_in_4bit and load_in_8bit shortcut arguments to from_pretrained() are gone. You must construct a BitsAndBytesConfig explicitly:

# v4 (removed):
model = AutoModelForCausalLM.from_pretrained("model", load_in_4bit=True)

# v5 (required):
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    device_map="auto",
    quantization_config=quantization_config,
)

Quantization is now integrated at the core weight-loading level through the WeightConverter system, enabling combinations like quantization + MoE or tensor parallelism + quantization that weren't previously possible.

New quantization backends added in v5.3: MLX for Apple MPS, NVFP4 (Four Over Six), and CPU MXFP4. Minimum bitsandbytes version is 0.46.1.

6. Configuration Changes

from_xxx_config() factory methods deleted. Use the __init__ method directly. See #41314.
Config loading from URLs removed. Configs must be loaded from a local path or a Hub repo. See #42383.
RoPE parameters unified. All rotary embedding parameters are now under config.rope_parameters (a dict), including rope_theta and rope_type. Accessing config.rope_theta directly throws AttributeError. For models like Gemma3 and ModernBert, rope_parameters is a nested dict with per-layer-type parameterizations. See #39847 and #42255.
Qwen-VL family uses nested configuration. config.vocab_size throws an error — use config.text_config.vocab_size.
Non-generative models no longer carry a generation_config. Accessing model.config.generation_config on encoder-only models throws AttributeError.

7. Attention and Model Architecture Changes

Removed attention features

Several features are removed because they were only supported in a few old models and never integrated into current model additions:

Head masking — allowed turning off certain heads during attention (eager only). #41076
Relative positional biases in BERT-like models — introduced for relative position scores within attention (similar to T5); barely used in official models, eager-only. #41170
Head pruning — allowed pruning heads within attention layers. #41417

If you need output_attentions=True, you must now explicitly set attn_implementation="eager".

Removed torch APIs

TorchScript — #41688
torch.fx — #41683
BetterTransformer — use torch.compile or native SDPA instead

These were deprecated by the PyTorch team; the focus is now on torch.dynamo and torch.export.

New abstractions

A new AttentionInterface provides centralized abstraction across eager, Flash Attention 1/2/3, FlexAttention, and SDPA.
All model forward() methods now accept **kwargs for vLLM integration compatibility (PR #42603, introduced in RC1). Code that introspects forward() signatures or subclasses with strict forward() overrides may break.

RotaryEmbeddings changes

Some RotaryEmbeddings layers now return a dict of tuples when the model uses several RoPE configurations (Gemma2, ModernBert). Each value is a (cos, sin) tuple per RoPE type. Backward compatibility will be maintained as much as possible during the transition.

Vision-Language model access

The shortcut model.language_model no longer works for VLMs. Use model.model.language_model or model.get_decoder(). See #42156.

Feature extraction helpers: `get_*_features`

Multi-modal model methods (get_text_features, get_image_features, get_audio_features, get_video_features) now return a BaseModelOutputWithPooling (or subclass) instead of only a pooled embedding tensor. (This change is tracked in issue #42401 and PR #42564; verify merge status for your version.)

last_hidden_state: unpooled embeddings
pooler_output: pooled representation (what was previously returned)
hidden_states: all layer hidden states (when output_hidden_states=True)
attentions: attention maps (when output_attentions=True)

Note: There is no single universal shape for these fields. Inspect a small forward pass before making assumptions.

Migration:

# v4
text_embeddings = model.get_text_features(**inputs)

# v5
outputs = model.get_text_features(**inputs, return_dict=True)
text_embeddings = outputs.pooler_output

Linked PR: huggingface/transformers#42564

8. Generation API Changes

Output types consolidated

Old deprecated aliases like GreedySearchEncoderDecoderOutput and BeamSearchDecoderOnlyOutput are removed. Only four output classes remain, defined by two axes: decoder-only vs. encoder-decoder, and beam vs. no-beam. See #40998.

Beam scoring and constraints moved to the Hub

Deprecated beam scoring and constraint classes were moved to transformers-community/group-beam-search, requiring trust_remote_code=True for features like Diverse Beam Search with num_beam_groups. See #41223.

Default KV cache is model-defined

If generate receives no KV Cache argument, the default cache class is now defined by the model rather than always being DynamicCache. The legacy cache format is removed entirely. See #41505.

Generation parameters moved off model config

Generation parameters are no longer accessible via model.config. If serialized in config.json for an old model, they are loaded into model.generation_config. Use model.generation_config exclusively (e.g., model.generation_config.do_sample = True).

v5.1 generation changes

Generation input preparation was refactored to pass pre-sliced input_ids/inputs_embeds directly to prepare_inputs_for_generation instead of relying on cache_position.
Models with sliding window attention now enforce window size limits during generation (PR #43679), potentially changing output behavior.

9. Processing Classes

Processor attributes are now serialized as nested dicts in processor_config.json instead of separate config files. Loading supports all old formats. See #41474.
All vision XXXFeatureExtractor classes are completely removed in favor of XXXImageProcessor. For vision models, use AutoImageProcessor instead of AutoFeatureExtractor. Audio feature extractors (e.g., Wav2Vec2FeatureExtractor) are retained. See #41174.
Fast image processors now require torchvision as a dependency.
XXXFastImageProcessorKwargs → unified XXXImageProcessorKwargs (shared between fast and slow). See #40931.

10. Auto-Classes

AutoModelWithLMHead → removed. Use AutoModelForCausalLM, AutoModelForMaskedLM, or AutoModelForSeq2SeqLM.
AutoModelForVision2Seq → removed. Use AutoModelForImageTextToText.

11. Trainer Changes

Constructor and method renames

tokenizer parameter → processing_class
model_path in train() → resume_from_checkpoint

`report_to` default changed

The default changed from auto-detection to "none". Logging integrations (WandB, TensorBoard, etc.) must now be explicitly specified.

`use_cache` default

use_cache in the model config is now set to False by default during training. You can override it via TrainingArguments's use_cache argument.

Parameters removed without deprecation cycle (low usage)

mp_parameters — legacy SageMaker param
_n_gpu — not intended for users; auto-initialized
overwrite_output_dir — replaced by resume_from_checkpoint; was only used in example scripts
logging_dir — set TENSORBOARD_LOGGING_DIR env var instead
jit_mode_eval — use use_torch_compile instead
tpu_num_cores — set TPU_NUM_CORES env var instead
past_index — only used for a handful of models with special architectures
ray_scope — set RAY_SCOPE env var instead
warmup_ratio — merged into warmup_step, which now accepts float values

Previously deprecated parameters removed

Removed	Replacement
`fsdp_min_num_params`, `fsdp_transformer_layer_cls_to_wrap`	`fsdp_config`
`tpu_metrics_debug`	`debug`
`push_to_hub_token`	`hub_token`
`push_to_hub_model_id`, `push_to_hub_organization`	`hub_model_id`
`include_inputs_for_metrics`	`include_for_metrics`
`per_gpu_train_batch_size`	`per_device_train_batch_size`
`per_gpu_eval_batch_size`	`per_device_eval_batch_size`
`use_mps_device`	Auto-detected
`no_cuda`	`use_cpu`
`fp16_backend`, `half_precision_backend`	Removed (only `torch.amp`)
`include_tokens_per_second`	`include_num_input_tokens_seen`
`use_legacy_prediction_loop`	Removed (only `evaluation_loop`)

Removed integrations

SigOpt hyperparameter search — library archived, API stopped working
SageMaker support requires API ≥ 1.10
Minimum Accelerate version bumped to 1.1.0

12. Pipelines

Text pipelines removed

question-answering, Text2TextGenerationPipeline, SummarizationPipeline, and TranslationPipeline are removed. For nearly all text-to-text or Q&A tasks, a modern chat model via TextGenerationPipeline provides much higher quality output:

import torch
from transformers import pipeline

summarizer = pipeline("text-generation", model="Qwen/Qwen3-4B-Instruct-2507")
message_history = [
    {"role": "user", "content": "Summarize the following text:\n\n[TEXT_TO_SUMMARIZE]"}
]
print(summarizer(message_history)[0]["generated_text"][-1]["content"])

Vision pipelines removed

image-to-text and visual-question-answering pipelines are removed. Use a VLM via the image-text-to-text pipeline:

import torch
from transformers import pipeline

captioner = pipeline("image-text-to-text", model="Qwen/Qwen3-VL-4B-Instruct")
message_history = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "[IMAGE_URL_HERE]"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
print(captioner(message_history)[0]["generated_text"][-1]["content"])

Other pipeline changes

image-to-image pipeline removed — use 🤗 Diffusers instead.
Image-text-to-text pipelines no longer accept images as a separate argument — image data must be embedded in the chat message's "content" field. See #42359.
The grouped_entities parameter was removed.
The transformers.onnx module was removed entirely.

13. PushToHubMixin

Removed parameters: organization, repo_url, ignore_metadata_errors, use_temp_dir, safe_serialization
push_to_hub no longer accepts **kwargs — all accepted parameters are explicitly documented
All arguments except repo_id are now keyword-only
You must pass a repo_id instead of organization + repo_url

Linked PR: huggingface/transformers#42391

14. CLI Changes

The deprecated transformers-cli command is gone. transformers is the only CLI entry point.

The CLI was rebuilt on Typer with improved help and autocompletion.

`transformers chat` redesigned

No longer starts a server (that moved to transformers serve). New signature:

Usage: transformers chat [OPTIONS] BASE_URL MODEL_ID [GENERATE_FLAGS]...

Works hand in hand with transformers serve:

# Default endpoint
transformers chat HuggingFaceTB/SmolLM3-3B

# Any OpenAI API compatible endpoint
transformers chat HuggingFaceTB/SmolLM3-3B https://router.huggingface.co/v1

`transformers run` removed

Was undocumented, untested, and not part of public documentation. See #42447.

Linked PRs:

15. Environment Variables

Legacy cache variables removed:

Removed	Replacement
`TRANSFORMERS_CACHE`	`HF_HOME`
`PYTORCH_TRANSFORMERS_CACHE`	`HF_HOME`
`PYTORCH_PRETRAINED_BERT_CACHE`	`HF_HOME`

Removed constants:

HUGGINGFACE_CO_EXAMPLES_TELEMETRY
HUGGINGFACE_CO_PREFIX
HUGGINGFACE_CO_RESOLVE_ENDPOINT

Use huggingface_hub.constants.ENDPOINT instead.

Linked PR: huggingface/transformers#42391

16. Hidden Breakage: `trust_remote_code` Models and Internal API Removal

A particularly insidious class of breaks affects models loaded with trust_remote_code=True:

is_torch_fx_available was removed from transformers.utils.import_utils, causing ImportError on Hub models like deepseek-ai/deepseek-moe-16b-base that copied this utility in their v4-era code (GitHub issue #44561, filed March 10, 2026).
Removed tokenization_*_fast.py file paths break projects (e.g., Microsoft's VibeVoice) that hardcoded fast tokenizer imports.
The HybridCache import path in cache_utils changed.
When both embed_tokens.weight and lm_head.weight exist in a checkpoint, they are no longer automatically tied — potentially changing model behavior silently.

Import path changes

Paths like transformers.tokenization_utils and transformers.tokenization_utils_fast no longer exist. They redirect to transformers.tokenization_utils_sentencepiece and transformers.tokenization_utils_tokenizers respectively.

17. Requirements Update

transformers v5 pins huggingface_hub to ≥ 1.3.0, < 2.0. Key changes from that major release:

HTTP backend switched from requests to httpx. Catch httpx.HTTPError instead of requests.HTTPError.
Proxy configuration moved exclusively to environment variables (HTTP_PROXY / HTTPS_PROXY). Setting proxies from scripts is no longer possible.
hf_transfer replaced by hf_xet — HF_HUB_ENABLE_HF_TRANSFER is dropped. Should be transparent for most users.
typer-slim added as a required dependency for the hf and transformers CLIs.

See the huggingface_hub migration guide for full details.

18. RC0 Disclaimers and Known Issues

PEFT + MoE

The switch from naive MoE (nn.ModuleList for experts) creates issues with MoEs that have adapters. See huggingface/transformers#42491 (comment).

Tensor Parallel and Expert Parallel + MoE

Tensor parallelism and expert parallelism aren't working as expected during the vLLM MoE support streamlining. Known and actively worked on.

Custom pretrained models

For anyone inheriting from a PreTrainedModel, weights are automatically initialized with a general scheme (normal distribution for Linear, Conv, Embedding; ones/zeros for norms). To avoid this, override _init_weights:

class CustomModel(Qwen3VLForConditionalGeneration):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.action_head = nn.Linear(1024, 7)
        self.positional_embedding = nn.Parameter(torch.randn(16, 1152))
        self.post_init()

    def _init_weights(self, module):
        pass

Tracker: huggingface/transformers#42418

QLoRA + MoE regression

The MoE refactoring in v5 — fusing experts into single modules with stacked nn.Parameter tensors — can cause quantization to not apply correctly in some configurations, leading to OOM errors on setups that previously worked under v4.x. See axolotl #3374 and trl #5222 for community reports.

19. Additional v5.1–v5.3 Breaking Changes

Several additional breaking changes landed in post-v5.0 weekly releases:

is_torch_sdpa_available removed/relocated from transformers.utils.import_utils, causing breakage in downstream projects like LLaMA-Factory that imported it directly.
Unified 3D position IDs for VLMs (#43972, v5.3) — changes how position information is passed to vision-language models.
New model architectures continue to ship exclusively targeting v5 (EuroBERT, VibeVoice ASR, TimesFM2.5, OlmoHybrid, ModernVBert, and others in v5.3).

Quick Reference: Most Likely Breakage Points

The three changes most likely to break existing code immediately:

TF/Flax import removal — instant ImportError
Default dtype change to auto — silent numerical behavior shift
Removal of load_in_4bit/load_in_8bit shortcuts — runtime error on quantized model loading

The next wave will catch: Trainer parameter purge, generation output type consolidation, feature extractor removal, and use_auth_token removal.

Many downstream libraries have responded by pinning transformers<5.0 — pragmatic short-term, but with a ticking clock as all new model architectures target v5 exclusively.

Upgrade to HF transformers >= v5 #3090

Description

Activity

geoHeil commented on Mar 9, 2026

dolfim-ibm commented on Mar 10, 2026

geoHeil commented on Mar 10, 2026