[inureyes]

Summary

The SentencePiece fallback (tokenizer.model, no tokenizer.json) registered only "special": true added tokens for encoding, so "special": false ones decoded fine but split into 3 to 5 ordinary pieces on encode. It now matches every added_tokens_decoder entry on encode, mirroring the transformers slow tokenizer: leftmost-longest matching, then each token's rstrip / lstrip / single_word applied to its neighbours (new src/tokenizer/sp_added_vocab.rs). special keeps governing only skip_special_tokens on decode and what parse_special: false stops matching (non-special tokens are still matched there, as llama.cpp and the tokenizers backend do). With no decoder table, added_tokens.json supplies plain non-special tokens, as read_added_tokens_sorted does on the tokenizers path.

normalized is ignored (the slow tokenizer reads it only under do_lower_case). IQuest declares </s> single_word: true, so d</s>e is now text, as in transformers.

Survey

Local ~/models/mlx (184 checkpoints): only the two IQuest checkpoints take the SentencePiece path; Baichuan-M1, ERNIE-4.5-0.3B and InternLM3 ship a local tokenizer.json. Remote survey of every docs/model-catalog.tsv repo (153 with a repo id) found seven SentencePiece-path snapshots:

checkpoint	non-special entries
IQuest-Coder-V1-7B-Instruct-8bit	8 (<think>, </think>, six tool tags)
IQuest-Coder-V1-40B-Loop-Instruct-4bit (not catalogued)	8, same layout
Baichuan-M1-14B-Instruct-4bit (mlx-community snapshot)	20 (<tool_call>, FIM/repo markers, <C_A>, <B_FUNC>, <function_calling>, ...)
ERNIE-4.5-VL-28B-A3B-Thinking-4bit	1015 (`<
ERNIE-4.5-0.3B-PT-4bit, internlm3-8b-instruct-4bit, TeleChat3-36B-Thinking-4bit	0

No surveyed checkpoint has an added_tokens.json entry missing from its decoder.

Verification

• Tokenization diff, old vs new binary, on all seven tokenizers (base prompts, each non-special token embedded mid-word and space-delimited, plain and tool-calling prompts rendered through each chat template), checked against a transformers 4.57.1 slow-tokenizer reference: new matches it on every prompt; old mismatched on every prompt containing a non-special token. Prompts without one are unchanged.
• Greedy ids (mlxcel-server /completion, temp 0, MLXCEL_SDPA_DETERMINISTIC=1, old binary self-parity checked):
  • ~/models/mlx/iquest-coder-v1-7b-instruct-8bit: old self-parity holds. Prompts without non-special tokens give identical prompt and generated ids. The tool-calling chat prompt drops from 193 to 181 tokens and still produces the get_weather call.
  • ~/models/mlx/iquest-coder-v1-40b-loop-instruct-4bit: same pattern; the tool-calling chat prompt goes from 205 to 185 tokens, same tool call.
  • Baichuan-M1 (local weights plus the snapshot's tokenizer files): old self-parity fails on CUDA even for unchanged prompt ids, so its greedy diff is not usable. Its chat template's <C_A> is non-special: the old path echoed the question and a literal <C_A>, while the new one answers.
  • ERNIE-4.5-VL: tokenizer-only (weights not on this box). The image placeholder and its framing markers are now single ids, so the image expansion finds the template's placeholder instead of splicing a second block. Not run end to end with weights.
• New tests: sp_added_vocab_tests (splitter flags, plus an in-memory SentencePiece model covering token_to_id, encode, skip_special_tokens and parse_special: false) and iquest_tool_prompts_encode_each_tool_tag_as_one_id; the_iquest_loop_coder_tokenizer_round_trips_its_added_tokens now asserts the round trip. With the fix reverted (non-special entries dropped from the encode table), 5 of the new or inverted tests fail. Review follow-ups added splitter edge cases (single_word folds, multibyte contents, a duplicated content) and IQuest's single_word </s>. tokenizer:: (137 pass), fmt, cargo clippy --profile test-fast --features cuda --lib --tests -D warnings and the contract checks pass.

Closes #1758