Add new model: Kimi2-6 by zucchini-nlp · Pull Request #45630 · huggingface/transformers

zucchini-nlp · 2026-04-24T12:26:43Z

What does this PR do?

The moonshot team is not very active and haven't yet replied, so I kept all the model_type naming to match the remote code. That allows us to load a model with local code. There are still a few problems such as tokenizer needing conversion and jinja chat template being non-consistent though

They have saved a tiktoken tokeninzer and tbh I am not sure if we can convert it, and nudge them to update. So as a workaround we convert from tiktoken at load-time

Tested with below script but uses a smaller/dummy model. Logits and eager generation are equivalent with official implementation.

import torch
from transformers import AutoProcessor, AutoTokenizer, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained('moonshotai/Kimi-K2.6', trust_remote_code=False)
model = AutoModelForImageTextToText.from_pretrained(
    'moonshotai/Kimi-K2.6',
    dtype=torch.bfloat16,
    device_map="cuda:0",
    trust_remote_code=False,
    attn_implementation="eager",
)


messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "image.png"},
            # {"type": "video", "video_url":{"url"  "video.mp4"}}, # note that videos need different layout due to jinja template
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(device=model.device, dtype=torch.bfloat16)

generated_ids = model.generate(**inputs, max_new_tokens=64)
generated_text = processor.batch_decode(generated_ids[:, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)[0]
print(generated_text)

zucchini-nlp · 2026-04-24T16:51:28Z

Oke, I think the model part looks same as the custom code, though I didn't test it. Also we need to check what is the exact chat template format and how to add placeholder tokens per modality, again with the real checkpoint

Kimi team might reply on Monday, coming back then

vasqu

Some initial comments from my side, I wanted to focus on things that we could simplify - mainly the RoPE I think.

I know it's WIP in a way so don't mind comments you already want to address yourself

zucchini-nlp · 2026-05-11T17:01:29Z

+        if image_processor_type is None:
+            try:
+                if not isinstance(config, PreTrainedConfig):
+                    config = AutoConfig.from_pretrained(
+                        pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
+                    )
+
+                image_processor_type = getattr(config, "image_processor_type", None)
+                if hasattr(config, "auto_map") and "AutoImageProcessor" in config.auto_map:
+                    image_processor_auto_map = config.auto_map["AutoImageProcessor"]
+            except ValueError:
+                # Config loading failed (unrecognized model_type, invalid config, etc.)
+                # Continue to fallback logic below (AutoTokenizer, AutoImageProcessor, etc.)
+                pass

        # Derive base_class_name from image_processor_type
        is_legacy_fast = False
        base_class_name = None
        if image_processor_type is not None:
            is_legacy_fast = image_processor_type.endswith("Fast")
            base_class_name = image_processor_type[:-4] if is_legacy_fast else image_processor_type

        backend = _resolve_backend(backend_kwarg, use_fast, base_class_name)

        image_processor_class = None
        if base_class_name is not None:
            image_processor_class = _load_backend_class(base_class_name, backend, is_legacy_fast)

        # Handle remote code
        has_remote_code = image_processor_auto_map is not None
        has_local_code = image_processor_class is not None or type(config) in IMAGE_PROCESSOR_MAPPING
        explicit_local_code = has_local_code and not (
            image_processor_class or _load_class_with_fallback(IMAGE_PROCESSOR_MAPPING[type(config)], backend)
        ).__module__.startswith("transformers.")
        if has_remote_code:
            class_ref = _resolve_auto_map_class_ref(image_processor_auto_map, backend)
            upstream_repo = class_ref.split("--")[0] if "--" in class_ref else None
            trust_remote_code = resolve_trust_remote_code(


copied from processing_auto.pt btw. The idea is to infer local class name if possible from config, even if we found a remote code

The remote code will still be loaded with higher prio if users pass trust_remote_code, so nothing should break

vasqu · 2026-07-03T11:04:00Z

run-slow: kimi_k25

github-actions · 2026-07-03T11:05:14Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/kimi_k25"]
quantizations: []

IlyasMoutawwakil · 2026-07-03T11:35:18Z

+        outputs = []
+        running_length = 0
+        for t, h, w in grid_thw.tolist():
+            # Get the current sequence
+            seq = hidden_states[running_length : running_length + t * h * w]
+            # Reshape along self.merge_kernel_size and concat to the last dimension
+            new_height, new_width = h // kernel_height, w // kernel_width
+            reshaped_seq = seq.view(t, new_height, kernel_height, new_width, kernel_width, hidden_dim)
+            reshaped_seq = reshaped_seq.transpose(2, 3).mean(dim=0)  # temporal pooling
+            padded_seq = reshaped_seq.reshape(new_height * new_width, kernel_height * kernel_width, -1)
+            outputs.append(padded_seq)
+            running_length += t * h * w
+
+        return torch.cat(outputs, dim=0)


can you please use the compile/export friendly vision_utils here 🙏

there is no utility to permute and pooling each image, no?

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

vasqu · 2026-07-03T13:46:12Z

run-slow: kimi_k25

zucchini-nlp · 2026-07-03T13:54:40Z

run-slow: kimi_k25

zucchini-nlp · 2026-07-03T13:59:38Z

run-slow: kimi_k25

github-actions · 2026-07-03T14:00:58Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/kimi_k25"]
quantizations: []

…nto kimi

github-actions · 2026-07-03T14:22:37Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	487da00f	workflow commit (merge commit)
PR	0a1fcd4d	branch commit (from PR)
main	5d063058	base commit (on `main`)

Model CI Report

❌ 4 new failed tests from this PR 😭

kimi_k25:
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_compilation_all_outputs (✅ ⟹ ❌)
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_compile_model_forward_fullgraph (✅ ⟹ ❌)
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_from_inputs_embeds_with_static_cache (✅ ⟹ ❌)
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_with_static_cache (✅ ⟹ ❌)

This reverts commit 5952534.

vasqu · 2026-07-03T14:39:06Z

Looks like it was not reverted?

yeah, weird, it didn't get reverted

github-actions · 2026-07-03T15:20:41Z

CI recap

Dashboard: View test results in Grafana
Latest run: 28667688910:1
Result: failure | Jobs: 15 | Tests: 171,114 | Failures: 1 | Duration: 25h 54m

github-actions · 2026-07-03T15:28:38Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, kimi_k25

* add kimi * modular run conversion * add processing for image for now * video processor, maybe like this * i think this is defi gemma4 rope * add model tester * convert weights * fix processing * looks fine with dummy weights * chunk/permute rope dynamically * fix repo * fix some tests * . * rename to match remote code config * rename tests as well * oops * update * some comments * more comments about model code * fast tokenizer loaded, thasnk to moonshot * . * move my ugly code from cluster to GH, needs mocing to proper conversion * easier to read, yet not in conversion mapping * add as conversion * small fix * have to go, finish next week! * resolve todos * push the ep plan and copy from mixtral for now * Fix compressed-tensors MoE decompression under tensor/expert parallelism `DecompressExperts` unpacked each expert with the checkpoint's per-expert `weight_shape`. Under TP/EP the loader shards every source sibling mapped to the experts target (`weight_packed`/`weight_scale`/`weight_shape`) with the same op, leaving the 2-element `weight_shape` empty on most ranks (stale on rank 0), so `unpack_from_int32` raises `IndexError`. Rebuild each shard's `[out, in]` from the packed tensor (`cols * pack_factor`): correct for every shard, and identical to the stored shape in the non-distributed path (which is therefore unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix repo * memory fragmentation * tokenizer * nit * . * even the official repo has this bug lol * deepseek sybc with main, it was refactored * docs and the last bits * and this * tester uses MoE layers * fix reverse rename * some comments * video tests, flaky for now * tests! * fix tests * fix repo * no integration tests for now, too huge ckpts * maybe? looks like same * ! * more comments * run check repo * unused attr, delete * dont import these unless torch is installed * skip export tests * fix modular nit * aya * erm, oke, mvoe it to another folder following fpquant * use the correct ckpt for tests * Update docs/source/en/model_doc/kimi_k25.md Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * fix kimi k26 export * Revert "fix kimi k26 export" This reverts commit 5952534. * copy from main * skip --------- Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: vasqu <antonprogamer@gmail.com> Co-authored-by: IlyasMoutawwakil <moutawwakil.ilyas.tsi@gmail.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

zucchini-nlp added 5 commits April 24, 2026 14:25

add kimi

7eea624

modular run conversion

89d373b

add processing for image for now

44c2063

video processor, maybe like this

1f940fd

i think this is defi gemma4 rope

f19aa2d

zucchini-nlp force-pushed the kimi branch from 2184255 to f19aa2d Compare April 24, 2026 15:34

add model tester

aacd808

zucchini-nlp force-pushed the kimi branch from 36c642d to aacd808 Compare April 24, 2026 16:49

zucchini-nlp added 10 commits April 30, 2026 12:52

convert weights

eb3d46b

fix processing

4a81176

looks fine with dummy weights

0249fa1

chunk/permute rope dynamically

2bc9732

fix repo

fe72152

fix some tests

1e2d306

.

86029a5

rename to match remote code config

a4bc583

rename tests as well

0c3267d

oops

8694a52

vasqu reviewed May 7, 2026

View reviewed changes

zucchini-nlp added 4 commits May 11, 2026 14:31

update

ef766b6

some comments

c17f932

more comments about model code

7a248c2

fast tokenizer loaded, thasnk to moonshot

533a7aa

zucchini-nlp commented May 11, 2026

View reviewed changes

zucchini-nlp added 2 commits May 12, 2026 12:26

.

43d9b76

move my ugly code from cluster to GH, needs mocing to proper conversion

1267030

zucchini-nlp commented Jul 3, 2026

View reviewed changes

Comment thread src/transformers/models/kimi_k25/modular_kimi_k25.py Resolved

erm, oke, mvoe it to another folder following fpquant

7959646

IlyasMoutawwakil reviewed Jul 3, 2026

View reviewed changes

IlyasMoutawwakil self-requested a review July 3, 2026 11:59

use the correct ckpt for tests

8a5c933

huggingface deleted a comment from github-actions Bot Jul 3, 2026

Merge remote-tracking branch 'origin/main' into kimi

29d43b0

vasqu reviewed Jul 3, 2026

View reviewed changes

Comment thread docs/source/en/model_doc/kimi_k25.md Resolved

Update docs/source/en/model_doc/kimi_k25.md

f9f0d12

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

Merge branch 'main' into kimi

0a1fcd4

IlyasMoutawwakil added 2 commits July 3, 2026 16:09

fix kimi k26 export

5952534

Merge branch 'kimi' of https://github.com/zucchini-nlp/transformers i…

cb30f09

…nto kimi

Revert "fix kimi k26 export"

762aacf

This reverts commit 5952534.

vasqu reviewed Jul 3, 2026

View reviewed changes

copy from main

ba7f97c

skip

448c296

vasqu merged commit 0983570 into huggingface:main Jul 3, 2026
99 of 100 checks passed

This was referenced Aug 27, 2026

NVFP4 MoE experts are dequantized without weight_global_scale, silently #48371

Open

Carry weight_global_scale when decompressing NVFP4 MoE experts #48372

Closed

tarekziade mentioned this pull request Oct 27, 2026

Deprecate BridgeTower #50045

Open

Conversation

zucchini-nlp commented Apr 24, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

zucchini-nlp commented Apr 24, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zucchini-nlp May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasqu commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

IlyasMoutawwakil Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasqu commented Jul 3, 2026

Uh oh!

zucchini-nlp commented Jul 3, 2026

Uh oh!

zucchini-nlp commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

CI Results

Commit Info

Model CI Report

Uh oh!

vasqu Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 3, 2026

CI recap

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zucchini-nlp commented Apr 24, 2026 •

edited by github-actions Bot

Loading