chore(deps): bump tokenizers from 0.22.1 to 0.23.1 in /worker_plan

[dependabot]

Bumps tokenizers from 0.22.1 to 0.23.1.

Release notes

Sourced from tokenizers's releases.

Release v0.23.1

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

---

🚨 Breaking changes

• Drop Python 3.9 (#1952) — requires-python = ">=3.10"; 3.9 users stay on 0.22.x.
• add_tokens normalizes content at insertion (#1995) — re-saved tokenizer.json may differ in the added_tokens block. Existing files load unchanged.
• Type stubs are precise (#1928, #1997) — methods that returned Any now return real types; mypy --strict may surface previously-hidden errors. Stub layout also moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi. This breaks the surface of some of the processors like RobertaProcessign's __init__ .
• 3.14t-only: setters/getters return PyResult<T> because of Arc<RwLock<Tokenizer>>; a poisoned lock surfaces as PyException instead of a panic.

---

⚡ Performance — measured locally on this Mac, not lifted from PRs

Run with cargo bench --bench <name> -- --save-baseline v0_22_2 on v0.22.2, then --baseline v0_22_2 on v0.23.1. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.

Added-vocabulary deserialize — the headline win (#1995, #1999)

bench: improve added_vocab_deserialize to reflect real-world workloads (#2000) is now representative of how transformers actually loads tokenizer.json files. The combined effect of daachorse for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:

benchmark	v0.22.2	v0.23.1	change
100k tokens, special, no norm	~410 ms	248 ms	−40%
100k tokens, non-special, no norm	~7.1 s	273 ms	−96%
100k tokens, special, NFKC	~395 ms	235 ms	−40%
100k tokens, non-special, NFKC	~7.4 s	290 ms	−96%
400k tokens, special, no norm	~15 s	980 ms	−94%

Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".

BPE encode

benchmark	v0.22.2	v0.23.1	change
BPE GPT2 encode batch, no cache	530 ms	446 ms	−16%
BPE GPT2 encode batch (cached)	690 ms	685 ms	noise
BPE GPT2 encode (single)	1.95 s	1.94 s	noise
BPE Train (small)	32.6 ms	31.5 ms	−3%
BPE Train (big)	1.01 s	988 ms	−2%

The BPE per-thread cache PR (#2028) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.

Llama-3 encode

... (truncated)

Commits

• 7f1623b Bump version to 0.23.1
• bbe43ad ci: release workflow fixes (node + python) (#2043)
• ab0c5d8 Fix node release (#2034)
• decd8e0 bindings/python: free-threaded Python (3.14t) support (#2041)
• 3992692 update for release (#2033)
• bcdd25b BPE cache: per-thread read-through cache to avoid RwLock atomics on hits (#2028)
• 618eb38 Bump follow-redirects in /tokenizers/examples/unstable_wasm/www (#2024)
• b6b1688 chore: bump doc-builder SHA for PR upload workflow (#2025)
• 19015d6 fix: use uvx --with cairosvg instead of uv pip install --system (#2021)
• efbcc68 Ci benchmarks (#2019)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[neoneye]

CI fails here due to a real dependency conflict, not a transient/rebase issue:

planexe 2025.12.31 depends on tokenizers==0.23.1   (this PR)
transformers 4.57.3 depends on tokenizers<=0.23.0 and >=0.22.0

The pinned transformers 4.57.3 caps tokenizers at <=0.23.0, so bumping to 0.23.1 makes the environment unresolvable (ResolutionImpossible). Merging would break installs, so holding this PR.

Path forward: either wait for a transformers release that permits tokenizers 0.23.1, or bump transformers to a compatible version in the same change before merging.

[neoneye]

Why CI is failing

The tests and typecheck jobs fail during pip install with ResolutionImpossible:

planexe 2025.12.31 depends on tokenizers==0.23.1
transformers 4.57.3 depends on tokenizers<=0.23.0 and >=0.22.0

worker_plan/pyproject.toml pins both tokenizers==0.22.1 and transformers==4.57.3. This PR bumps tokenizers to 0.23.1, but transformers 4.57.3 only accepts tokenizers<=0.23.0 — the resolver cannot satisfy both pins.

Note: upgrading transformers does not help — even the latest release (5.10.2, checked 2026-06-07) still declares tokenizers<=0.23.0,>=0.22.0. So tokenizers 0.23.1 is currently incompatible with every transformers release.

Ways to resolve

1. Bump to 0.23.0 instead (recommended): tokenizers==0.23.0 is the newest version transformers accepts. Dependabot won't propose it (it only proposes the latest), so this needs a small manual PR changing the pin in worker_plan/pyproject.toml.
2. Tell dependabot to skip this version: comment dependabot ignore this version (with the @-mention) on this PR to close it; dependabot will re-open when 0.23.2+ lands — though that will hit the same wall until transformers widens its constraint.
3. Wait for transformers to release a version allowing tokenizers>0.23.0, then merge a rebased version of this PR.
4. Loosen the pin to a range (e.g. tokenizers>=0.22.1,<0.24) and let the resolver pick the compatible version — at the cost of less reproducible installs.

Options 1 and 2 combine well: bump to 0.23.0 manually, then ignore this version here.

🤖 Generated with Claude Code