[albertvillanova]

What does this PR do?

align_special_tokens copies the tokenizer's BOS/EOS/PAD ids onto model.config and model.generation_config. When the tokenizer does not define one of them, the None is copied over the value the config declares, and save_pretrained persists the removal.

This PR propagates only the values the tokenizer actually defines, as suggested in #48592. Removing a token stays an explicit action on the config.

Fixes #48592

Why

_special_tokens_map is initialized with None for every special token, so a tokenizer that never declared one is indistinguishable from a tokenizer whose token was deliberately cleared:

smol = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
smol.bos_token = None                                                # deliberate removal
qwen = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")   # never declared one

smol._special_tokens_map["bos_token"] == qwen._special_tokens_map["bos_token"]
# True, both None

Resolving that ambiguity in favour of removal means any model family whose tokenizer simply does not manage a token loses it from the config on the first train() call. Qwen is the widest case: config.json declares bos_token_id=151643 while tokenizer_config.json declares no bos_token, so every Qwen fine-tune was saved with bos_token_id: null.

Before this PR, on Qwen/Qwen2.5-0.5B-Instruct:

before, config.bos_token_id = 151643
  generate() with no inputs OK, shape (1, 4)
after,  config.bos_token_id = None
  ValueError: `bos_token_id` has to be defined when no `input_ids` are provided.

and the nulled value was written to config.json. After this PR the declared id survives, generate() keeps working, and the alignment still reports the one genuine change:

after,  config.bos_token_id = 151643
  generate() with no inputs OK, shape (1, 4)
Updated tokens: {'pad_token_id': 151643}

The same failure mode was reported for Whisper in #45584, closed as stale, and worked around in the still-open #45570 and #45917.

What this does not change

Values the tokenizer does define are still propagated, including onto a config that lacks them. A tokenizer that adds an end-of-turn token at fine-tuning time, the case #38441 was written for, behaves exactly as before.

The diff is 82 lines but only three are substantive: one guard per token, plus a comment. git diff -w shows the rest is indentation.

Tests

test_special_token_alignment sets all three tokens to real values, so it is unaffected and still passes. Added test_special_token_alignment_keeps_tokens_the_tokenizer_does_not_define next to it, asserting that a config-declared BOS survives a tokenizer that does not define one.

Who can review?

@SunMarc

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

Merged main: #48847 landed in the meantime and moves the comparison to model.config.get_text_config(). The two changes are complementary, that one fixes where the ids are read from for composite models, this one stops a token the tokenizer does not define from removing the id the config declares. The net diff here is still three guards and a comment.

Measured downstream: on TRL's suite the realignment message goes from 457 lines to 145, with {'bos_token_id': None} going 311 to 0 and the mixed variants losing only that key. Cases where the tokenizer does define a value the config lacks are unchanged, as intended.

@SunMarc you endorsed this direction on #48592, would you have a moment? @lewtun tagging you for visibility, since the original alignment came out of your report.

[SunMarc]

Thanks, just a nit that will improve the readability of the code

[SunMarc]

we should warn/inform about this no somehow ? maybe we already do but just checking

[albertvillanova]

I don't think we can, and I think it would be counterproductive.

The two cases are indistinguishable at this point: _special_tokens_map is initialized with None for every special token, so a tokenizer that never declared one looks exactly like a tokenizer whose token was deliberately cleared. The case worth informing about is the second, and it is the one we cannot detect.

What we can detect is the first, and it is the common case: every Qwen model defines no BOS, so informing there would fire on every train() for a very large share of users, which is the noise this PR removes. If someone does want the removal, they can set it on the config directly, which is the explicit opt-in you suggested on #48592.

The previous behaviour was that warning: when the tokenizer did not define a token, the config value was overwritten with None and reported as an update. On a TRL CI run that produced 311 Updated tokens: {'bos_token_id': None} lines out of 457 warning lines, one per model whose tokenizer defines no BOS, which is most of them.

Now nothing is changed in that case, so there is nothing to report. The warning still covers every token that is actually modified.

[SunMarc]

small idea, let's create a helper function so that we clean this redundant code that is copied pasted three times with minimal changes. in the helper function, it should also deal with the case if the generation config may hold more than one token.

[albertvillanova]

Done. The helper takes the token name and looks it up on the tokenizer, the config and the generation config, so the three calls are identical apart from several=True for EOS, which is the case where the generation config may hold more than one token: there the tokenizer's id is prepended to the ones already defined instead of replacing them.

One behaviour change worth flagging: the old code turned generation_config.eos_token_id from an int into a one-element list as a side effect of the comparison, even when the tokenizer's EOS was already there and nothing was aligned. The helper only writes when a token actually changes, so that no-op normalization is gone. I compared the old and the new implementation over every combination of tokenizer, config and generation config values, and this is the only difference between them.

It should be safe: eos_token_id is documented as Union[int, list[int]], generate normalizes it once in _prepare_special_tokens and reads _eos_token_tensor everywhere afterwards, and the other direct readers branch on the type. The gain is that a checkpoint shipping "eos_token_id": 2 keeps that value after training instead of being rewritten to [2].

[SunMarc]

thanks

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 35693923315:1
Result: success | Jobs: 16 | Tests: 191,007 | Failures: 2 | Duration: 16h 41m