Skip to content

convert : note MiniCPM5-2B shares the MiniCPM5-1B tokenizer - #28594

Closed
cyxu0401 wants to merge 1 commit into
ggml-org:masterfrom
cyxu0401:minicpm5-2b-tokenizer-note
Closed

cyxu0401 wants to merge 1 commit into
ggml-org:masterfrom
cyxu0401:minicpm5-2b-tokenizer-note

Conversation

@cyxu0401

@cyxu0401 cyxu0401 commented Sep 8, 2026

Copy link
Copy Markdown

Follow-up to #23384 (MiniCPM5-1B), now that
openbmb/MiniCPM5-2B and
openbmb/MiniCPM5-2B-GGUF
are public.

Comment-only change — no behaviour change.

MiniCPM5-2B ships a byte-identical tokenizer.json to 1B
(md5 ee55db96827d21929c7c5db2092596a8), so running the
convert_hf_to_gguf_update.py recipe against it reproduces the hash that is
already in the table:

36f3066e97b7f3994b379aaacde306c1444c6ae84e81a5ae3cd2b7ed3b8c42d4   (187 tokens on CHK_TXT)

Verified against four copies of the file — HF 1B, HF 2B, and two local
checkpoints — all identical.

So 2B already converts correctly today and needs no code change. This commit
only records why, because the obvious move for the next person adding 2B is
to append a second entry, and that would be a silent regression:

  • entries in get_vocab_base_pre() are sequential if statements, not
    elif, so a second entry carrying the same hash placed later wins for
    both models — see the existing mpt/olmo and bert-bge/jina-v2-en
    pairs, where the later entry already shadows the earlier one;
  • renaming the pre type away from minicpm5 would break every GGUF already
    published with tokenizer.ggml.pre = "minicpm5", including openbmb's own
    1B and 2B releases.

The note is deliberately kept on one line: get_existing_models() in
convert_hf_to_gguf_update.py matches the if chkhsh == ... / res = ...
pair with a regex that tolerates only a single line between them, so splitting
the comment across two lines breaks regeneration. Confirmed by trying it.

The chat template for 2B is a separate matter and is in its own PR.

openbmb/MiniCPM5-2B ships a byte-identical tokenizer.json to
openbmb/MiniCPM5-1B (md5 ee55db96827d21929c7c5db2092596a8), so it hashes
to the chkhsh already registered for "minicpm5" and needs no new entry.

Document this on both the generated entry in conversion/base.py and the
model list in convert_hf_to_gguf_update.py, which is the source the block
is generated from. Kept to a single comment line so the get_existing_models
regex still matches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cyxu0401
cyxu0401 requested a review from CISC as a code owner September 8, 2026 06:52
@ggml-gh-bot

ggml-gh-bot Bot commented Sep 8, 2026

Copy link
Copy Markdown

Hi @cyxu0401, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

  • AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@ggml-gh-bot ggml-gh-bot Bot added the draft PR will be changed to draft by github-actions bot label Sep 8, 2026
@github-actions
github-actions Bot marked this pull request as draft September 8, 2026 07:00
@github-actions github-actions Bot removed the draft PR will be changed to draft by github-actions bot label Sep 8, 2026
@CISC

CISC commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

It doesn't work like that, the comment is autogenerated and we will notice if anyone tries to add a duplicate.

@CISC CISC closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants