Repository navigation
unicode: include '~' in collapsed symbol class - #26972
Merged
Merged
Conversation
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
tarruda
commented
Aug 12, 2026
| set_property(TEST ${TEST_TARGET} PROPERTY LABELS ${LLAMA_TEST_LABEL}) | ||
| endfunction() | ||
|
|
||
| llama_build_and_test(test-unicode.cpp) |
Contributor
Author
There was a problem hiding this comment.
Adding a new test module for this as the existing test-unicode.cpp is focused on peg parser unicode handling.
CISC
approved these changes
Aug 12, 2026
| { unicode_cpt_flags::PUNCTUATION, "\x21-\x23\x25-\x2A\x2C-\x2F\x3A-\x3B\x3F-\x40\\\x5B-\\\x5D\x5F\\\x7B\\\x7D" }, // !-#%-*,-/:-;?-@\[-\]_\{\} | ||
| { unicode_cpt_flags::ACCENT_MARK, "" }, // no sub-128 codepoints | ||
| { unicode_cpt_flags::SYMBOL, "\\\x24\\\x2B\x3C-\x3E\x5E\x60\\\x7C" }, // $+<=>^`| | ||
| { unicode_cpt_flags::SYMBOL, "\\\x24\\\x2B\x3C-\x3E\x5E\x60\\\x7C\\\x7E" }, // $+<=>^`|~ |
Member
There was a problem hiding this comment.
Should this also include at least £ and €?
Contributor
Author
There was a problem hiding this comment.
Honestly I have no idea. Should I include it?
Member
There was a problem hiding this comment.
I'm not sure what the intention is here with the limited set, best leave it, just wondered. :)
Contributor
|
Another individual added a dedicated pretokenizer, might be worth adopting? #26966 |
aldehir
approved these changes
Aug 13, 2026
thecodacus
pushed a commit
to thecodacus/llama.cpp
that referenced
this pull request
Sep 7, 2026
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
zbrad
pushed a commit
to zbrad/llama.cpp
that referenced
this pull request
Sep 10, 2026
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
pl752
pushed a commit
to pl752/llama.cpp
that referenced
this pull request
Sep 15, 2026
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
2 of 4 tasks
midagedev
added a commit
to midagedev/bloomery
that referenced
this pull request
Sep 24, 2026
…ed-category split, as mainline llama.cpp's k_ucat_map (ggml-org/llama.cpp#26972) and HF tokenizers have it — `~/` is one id (71520) instead of two; the oracle's V4.1 reference moves to the ik build that carries the same list and refuses a reference whose `~/` is not [71520]; five `~` cases; the gate's failure line prints both id windows and their decoded text at the first difference Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: midagedev <midagedev@gmail.com>
ikawrakow
pushed a commit
to ikawrakow/ik_llama.cpp
that referenced
this pull request
Sep 24, 2026
Port of ggml-org/llama.cpp#26972. Signed-off-by: midagedev <midagedev@gmail.com>
frostyautumnleaf
pushed a commit
to frostyautumnleaf/llama.cpp
that referenced
this pull request
Oct 5, 2026
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The collapsed
\p{S}class was missing~, which split~into separate pre-tokens and prevented theĠ~BPE merge used by DeepSeek V4. This caused re-tokenized prompts to diverge from sampled tokens and broke KV cache reuse.Additional information
While using DSv4 Flash locally, I noticed that long thinking blocks followed by a tool call was being reprocessed when a client returned the tool response. More information here
I've used Deepseek V4 Flash to debug and find the issue, and later verified that upstream tokenizer behavior matches this fix (
~being a separate token).Requirements