Skip to content

fix(models): lift transformers 5 rope_parameters for every family #2448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 3 commits into from
Oct 19, 2026
Merged

fix(models): lift transformers 5 rope_parameters for every family #2448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0681d42
fix(models): lift transformers 5 rope_parameters for every family
inureyes Oct 18, 2026
1684f15
fix(models): lift DeepSeek-V4 main/compress rope_parameters
inureyes Oct 18, 2026
51e261b
docs: add technical report for PR #2448
inureyes Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Fixed

- Configs saved by transformers 5, which keep `rope_theta`, `rope_scaling` and `partial_rotary_factor` only under a nested `rope_parameters` block, now load with their real RoPE base and scaling in every family that reads the flat keys, not only Qwen3 and GraniteMoeHybrid (#2443). Before, about 80 config structs (Llama and everything sharing its args, Qwen2/Qwen3-MoE, Gemma, Phi, OLMo, DeepSeek, GLM, the VLM text configs, the Qwen3-Omni speech stack and the DFlash drafter) silently ran at the serde default base, decoding fluently on short prompts and drifting on long ones, and a Llama 3 config lost its `llama3` scaling. The per-layer-type forms (Gemma 3, Gemma 3n and OLMo 3 `full_attention` / `sliding_attention`, DeepSeek-V4 `main` / `compress`) and the MRoPE `mrope_section` are lifted too. A flat key still wins when both are present; EXAONE-MoE and Mistral 4 used to let the nested base win and now follow the same rule.
- Checkpoints that load through the SentencePiece fallback (a `tokenizer.model` with no `tokenizer.json`) now encode their `"special": false` added tokens as the one id each was declared with, as `transformers` does, instead of splitting them into ordinary pieces (#1758). IQuest-Coder and IQuest-Coder Loop tool-calling prompts now carry `<tools>`, `<tool_call>`, `<tool_response>` and their closers as single ids, Baichuan-M1's `<C_A>` turn marker and ERNIE-4.5-VL's `<|IMAGE_PLACEHOLDER|>` and `<|LOC_n|>` tokens do the same, and each added token's `lstrip`, `rstrip` and `single_word` flags are honored. `skip_special_tokens` still drops only the special ones. On this path `parse_special: false` (`/tokenize`, logit-bias text keys) now also recognizes non-special added tokens, as llama.cpp does, `added_tokens.json` is read when `added_tokens_decoder` is absent, and a `single_word` special token glued to a word (IQuest's `</s>` in `d</s>e`) encodes as text, as in `transformers`.
- On CUDA, a bf16 model on an fp16 TurboQuant KV cache (`--kv-cache-mode fp16+turbo4` and the other turbo modes) no longer aborts with `Mask type must promote to output type float32` when attention carries an array mask, which for Gemma 4 meant any prompt past the 1024-token sliding window (#2336). Additive masks now reach SDPA in the attention dtype, `result_type(q, k, v)`, instead of the query's dtype.
- Mixed bf16/fp16 attention follow-ups to #2336 (#2422). On CUDA, the query-chunking gate for materializing SDPA now judges cuDNN eligibility and score size from the attention dtype, `result_type(q, k, v)`, instead of q's dtype, so a bf16 model on an fp16 TurboQuant KV cache, which attends in float32, chunks long prefills at 4 bytes per score instead of skipping the chunking at head_dim 128 or budgeting 2 bytes. The softcap attention composite (Gemma 2) casts its mask to and returns the attention dtype, as fast SDPA does, instead of q's and v's. Model-state snapshot donation checks the KV cache mode before the keys/values layout, so `--kv-cache-mode fp16+turbo4` logs `cache mode Turbo4Asym is not supported` instead of `has only one of keys/values`, and a populated symmetric `turbo4` layer is refused instead of being left out of the snapshot. Same-dtype attention is unchanged.
Expand Down
72 changes: 72 additions & 0 deletions TECHNICAL_REPORTS/pr-2448-rope-parameters-lift.en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# PR #2448: Lift transformers 5 rope_parameters for every family

**Date**: 2026-10-19
**Status**: Implemented and verified on CUDA (GB10) with real checkpoints; PR open. The items under "Not addressed" remain open

## Summary

Closes #2443. transformers 5 saves `rope_theta`, `rope_scaling` and `partial_rotary_factor` only under a nested `rope_parameters` block. Only Qwen3 and GraniteMoeHybrid lifted it (#2439). Every other struct that reads the flat keys fell back to its serde default base without an error. This PR applies the lift at the config type, through one macro line per struct, for 86 structs plus the call sites that parse outside serde.

## Problem

- Config structs that read a flat `rope_theta` with a serde default (10000 for llama3, qwen3_moe, gemma2, phi3, olmo2) load a transformers 5 checkpoint without error, decode fluently on short prompts, and drift on long ones. A Llama 3 config also loses its `rope_type: "llama3"` scaling.
- On `main`, a transformers-5 copy of Llama 3.2 1B or Qwen3-30B-A3B produces garbage on a 20 KB prompt. The Llama copy degenerates into `and, and, ...` and the Qwen3 copy into `. The. The. ...`.
- mlx-lm 0.32.2 lifts only exaone_moe, gemma4, glm_moe_dsa, laguna, lfm2, mellum, ministral3, mistral4, muse_glimmer and qwen3_5, which mlxcel already handles, so no other family gets a fix from parity.
- EXAONE-MoE and Mistral 4 let the nested base win, which was inconsistent with the families that let a flat key win.
- `src/loading/vlm_qwen.rs` carried two ad hoc call-site lifts (GLM-OCR and GLM-4V MoE).

## Technical decisions

- **The type is the choke point.** The PR author looked for a shared parse helper that every site already calls and found none: about 125 family `load` functions read `config.json` directly, and VLMs nest the block under `text_config`. The issue rejects a top-level lift in `sanitize_config_json` for both reasons (it is a string replace that most family `load` functions skip, and it would miss `text_config`).
- **Shared lifter in `mlxcel-core`.** `lift_rope_parameters` / `deserialize_lifting_rope_parameters` move to `mlxcel_core::rope_parameters` (re-exported from `models::rope_utils`), so the DFlash drafter config can use them. An `impl_rope_lifting_deserialize!(Type)` macro gives each uncovered struct `#[serde(remote = "Self")]` plus one line. This covers 86 structs, including the VLM text configs and the three Qwen3-Omni speech configs. Pipeline and tensor-parallel sites parse these same types, so they are covered. No struct gains a field.
- **Lifter rules.**
- A flat non-null key wins.
- It lifts `partial_rotary_factor`.
- It reads the per-layer-type forms, meaning a block whose values are all objects. For Gemma 3 / Gemma 3n / OLMo 3, `full_attention` gives the base and scaling and `sliding_attention` gives `rope_local_base_freq`. For DeepSeek-V4, `main` gives the base and `compress` gives `compress_rope_theta` and the YaRN `rope_scaling`.
- It builds `rope_scaling` from a non-default scheme (Llama 3 `llama3` included) or from MRoPE extras like `mrope_section`.
- The scheme goes under `type` only (it reads `type` first, then `rope_type`), because the deepseek_v32 / deepseek_v4 / nemotron `alias = "type", alias = "rope_type"` fields reject a block that spells both.
- **Call sites outside serde use the shared lifter.** `DFlashConfig::from_json`, the Idefics2 Mistral defaults (which used to shadow the nested base) and both `vlm_qwen.rs` ad hoc lifts. `JinaVlmTextConfig::from_json` falls back to the nested base, and the Phi-4 text-field copy now carries `rope_parameters`.
- **Flat wins everywhere.** EXAONE-MoE and Mistral 4 now let a flat key win, as the issue specifies. transformers 5 itself gives the nested value priority, but the two only disagree when a config carries both with different values, and none of the reference checkpoints does.
- **`exaone4`.** It was the one reader that checked only `rope_type`, and it now accepts `type` too. Its `rope_scaling` is now `#[serde(default)]`, because a default-scheme transformers 5 save has none.

## Changes

- `src/lib/mlxcel-core/src/rope_parameters.rs` (new, 187 lines), `rope_parameters_tests.rs` (new, lifter unit tests), `lib.rs`: the shared lifter and macro.
- `src/models/rope_utils.rs`: lifter code removed and re-exported from `mlxcel-core`.
- About 85 config files under `src/models/` and `src/audio/qwen3_omni_moe/`: `#[serde(remote = "Self")]` and one `impl_rope_lifting_deserialize!` line each (typically 3 lines per file).
- `src/models/mistral4.rs`, `src/models/exaone_moe.rs`, `src/models/exaone4.rs`, `src/models/jina_vlm.rs`: flat-wins, `type` handling and fallback changes described above.
- `src/loading/vlm_qwen.rs`, `src/lib/mlxcel-core/src/drafter/dflash/config.rs`: call sites moved to the shared lifter.
- `src/models/rope_parameters_lift_tests.rs` (new, 417 lines): table-driven tests.

## Verification

- `rope_parameters_lift_tests::every_flat_rope_theta_config_lifts_rope_parameters_and_flat_wins` is table-driven over all 89 parse paths (the 86 lifted structs, Mistral 4, and the DFlash and Jina `from_json` paths). It checks that nested-only gives 123456 and flat 500000 plus nested gives 500000. Required non-RoPE fields are filled from serde's own errors, never `rope_theta` / `rope_scaling`.
- More tests: Llama 3 nested `llama3` resolves `RopeScalingKind::Llama3`, Gemma 3 per-layer-type, Phi `partial_rotary_factor`, and DeepSeek-V4 `compress` YaRN (added in review; not revert-run). The lifter unit tests are in `mlxcel-core`.
- With the fix stashed and only the tests kept, the first four tests all fail, and every one of the 89 rows reports a failure.
- Real checkpoints: transformers-5 copies of `~/models_archive/mlx/llama-3.2-1b-instruct-4bit` and `~/models_archive/mlx/qwen3-30b-a3b-4bit`, with only `config.json` rewritten (flat keys moved into a `rope_parameters` block identical to the one transformers 5.20.1 emits for that config) and the weights symlinked. Command: `MLXCEL_SDPA_DETERMINISTIC=1 gpu-lock run -- mlxcel generate -m <dir> -p "<first 20000 bytes of CONTRIBUTING.md + a question>" -n 64 --temp 0 --show-reasoning`. On the branch, both copies match their originals byte for byte, and the originals match `main`. On `main` (self-parity checked), the Llama copy degenerates into `and, and, ...` and the Qwen3 copy into `. The. The. ...`.
- fmt, clippy (`--lib --tests -D warnings` for mlxcel and mlxcel-core), the affected-family test filters (647 tests) and the repo contract checks pass locally.

## Changes during review

The review found no CRITICAL or HIGH issues. Two follow-ups were applied, in commit `1684f151`:

- The DeepSeek-V4 `main` / `compress` layout, which the first version turned into an all-`None` `rope_scaling`.
- A `"type": null` key that hid the `rope_type` scheme.

## Not addressed

- Vision-encoder configs (Pixtral, Llama 4 vision, etc.) and `mlxcel-xla`, which the issue does not list.
- DBRX is lifted at `attn_config`, where transformers 5 still keeps `rope_theta`, but its top-level `rope_parameters` is not copied there.

## Statistics

| Item | Value |
|------|-------|
| Files changed | 96 |
| Lines added | 1134 |
| Lines deleted | 116 |

| Commit | Message |
|--------|---------|
| `0681d42b` | fix(models): lift transformers 5 rope_parameters for every family |
| `1684f151` | fix(models): lift DeepSeek-V4 main/compress rope_parameters |
72 changes: 72 additions & 0 deletions TECHNICAL_REPORTS/pr-2448-rope-parameters-lift.ko.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# PR #2448: 모든 모델 계열에서 transformers 5 rope_parameters 끌어올리기

**작성일**: 2026-10-19
**상태**: CUDA(GB10)에서 실제 체크포인트로 구현 및 검증 완료. PR은 열려 있음. "미해결 사항"의 항목은 계속 열려 있음

## 요약

#2443을 닫는다. transformers 5는 `rope_theta`, `rope_scaling`, `partial_rotary_factor`를 중첩된 `rope_parameters` 블록 안에만 저장한다. 지금까지 Qwen3와 GraniteMoeHybrid만 이를 끌어올렸고(#2439), 평평한(flat) 키를 읽는 나머지 struct는 오류 없이 serde 기본 base로 되돌아갔다. 이 PR은 설정 타입에서 끌어올리기를 적용한다. struct 86개에 매크로 한 줄씩 추가하고, serde 밖에서 파싱하는 호출 지점도 함께 처리한다.

## 1. 문제 정의

- serde 기본값이 있는 flat `rope_theta`를 읽는 설정 struct(llama3, qwen3_moe, gemma2, phi3, olmo2는 10000)는 transformers 5 체크포인트를 오류 없이 로드하고, 짧은 프롬프트에서는 유창하게 디코딩하지만 긴 프롬프트에서는 어긋난다. Llama 3 설정은 `rope_type: "llama3"` 스케일링도 잃는다.
- `main`에서 Llama 3.2 1B 또는 Qwen3-30B-A3B의 transformers-5 사본은 20 KB 프롬프트에서 깨진 출력을 만든다. Llama 사본은 `and, and, ...`로, Qwen3 사본은 `. The. The. ...`로 퇴화한다.
- mlx-lm 0.32.2는 exaone_moe, gemma4, glm_moe_dsa, laguna, lfm2, mellum, ministral3, mistral4, muse_glimmer, qwen3_5만 끌어올리며 mlxcel은 이미 이들을 처리하므로, 다른 계열은 parity로는 수정을 얻지 못한다.
- EXAONE-MoE와 Mistral 4는 중첩된 base가 우선했는데, flat 키가 우선하는 다른 계열과 일관되지 않았다.
- `src/loading/vlm_qwen.rs`에 임시방편 호출 지점 끌어올리기가 두 곳 있었다(GLM-OCR, GLM-4V MoE).

## 2. 기술적 선택과 그 이유

- **타입이 병목 지점이다.** PR 작성자는 모든 지점이 이미 호출하는 공유 파싱 헬퍼를 찾았으나 없었다. 약 125개의 계열별 `load` 함수가 `config.json`을 직접 읽고, VLM은 블록을 `text_config` 아래에 중첩한다. 이슈는 `sanitize_config_json`의 최상위 끌어올리기를 두 이유로 배제한다(대부분의 계열 `load` 함수가 건너뛰는 문자열 치환이며, `text_config`를 놓친다).
- **`mlxcel-core`의 공유 lifter.** `lift_rope_parameters` / `deserialize_lifting_rope_parameters`를 `mlxcel_core::rope_parameters`로 옮기고(`models::rope_utils`에서 re-export), DFlash drafter 설정도 사용할 수 있게 한다. `impl_rope_lifting_deserialize!(Type)` 매크로가 미처리 struct마다 `#[serde(remote = "Self")]`와 한 줄을 제공한다. VLM 텍스트 설정과 Qwen3-Omni speech 설정 3개를 포함해 86개 struct를 다룬다. 파이프라인과 tensor-parallel 지점도 같은 타입을 파싱하므로 포함된다. 어떤 struct에도 필드가 추가되지 않는다.
- **Lifter 규칙.**
- null이 아닌 flat 키가 우선한다.
- `partial_rotary_factor`를 끌어올린다.
- 레이어 종류별 형식, 즉 값이 모두 객체인 블록을 읽는다. Gemma 3 / Gemma 3n / OLMo 3는 `full_attention`이 base와 스케일링을, `sliding_attention`이 `rope_local_base_freq`를 준다. DeepSeek-V4는 `main`이 base를, `compress`가 `compress_rope_theta`와 YaRN `rope_scaling`을 준다.
- 기본이 아닌 스킴(Llama 3 `llama3` 포함) 또는 `mrope_section` 같은 MRoPE 추가 값으로 `rope_scaling`을 만든다.
- 스킴은 `type` 아래에만 넣는다(`type`을 먼저 읽고 `rope_type`을 읽는다). deepseek_v32 / deepseek_v4 / nemotron의 `alias = "type", alias = "rope_type"` 필드가 둘을 모두 적은 블록을 거부하기 때문이다.
- **serde 밖의 호출 지점도 공유 lifter를 쓴다.** `DFlashConfig::from_json`, Idefics2 Mistral 기본값(기존에는 중첩 base를 가렸다), 그리고 `vlm_qwen.rs`의 임시 끌어올리기 두 곳이다. `JinaVlmTextConfig::from_json`은 중첩 base로 폴백하고, Phi-4 텍스트 필드 복사는 이제 `rope_parameters`를 함께 옮긴다.
- **모든 곳에서 flat 우선.** 이슈의 지시대로 EXAONE-MoE와 Mistral 4도 flat 키가 우선한다. transformers 5 자체는 중첩 값에 우선순위를 주지만, 둘이 다른 값을 가진 설정에서만 차이가 나며 참조 체크포인트 중에는 그런 것이 없다.
- **`exaone4`.** `rope_type`만 확인하던 유일한 reader였으며 이제 `type`도 받아들인다. 기본 스킴으로 저장한 transformers 5 설정에는 `rope_scaling`이 없으므로 `#[serde(default)]`로 바꿨다.

## 3. 변경 내용

- `src/lib/mlxcel-core/src/rope_parameters.rs`(신규, 187줄), `rope_parameters_tests.rs`(신규, lifter 단위 테스트), `lib.rs`: 공유 lifter와 매크로.
- `src/models/rope_utils.rs`: lifter 코드를 제거하고 `mlxcel-core`에서 re-export.
- `src/models/`와 `src/audio/qwen3_omni_moe/` 아래 약 85개 설정 파일: `#[serde(remote = "Self")]`와 `impl_rope_lifting_deserialize!` 한 줄(보통 파일당 3줄).
- `src/models/mistral4.rs`, `src/models/exaone_moe.rs`, `src/models/exaone4.rs`, `src/models/jina_vlm.rs`: 위에서 설명한 flat 우선, `type` 처리, 폴백 변경.
- `src/loading/vlm_qwen.rs`, `src/lib/mlxcel-core/src/drafter/dflash/config.rs`: 호출 지점을 공유 lifter로 이동.
- `src/models/rope_parameters_lift_tests.rs`(신규, 417줄): 테이블 기반 테스트.

## 4. 검증

- `rope_parameters_lift_tests::every_flat_rope_theta_config_lifts_rope_parameters_and_flat_wins`는 89개 파싱 경로(끌어올린 struct 86개, Mistral 4, DFlash와 Jina의 `from_json` 경로)를 테이블로 순회한다. 중첩만 있으면 123456, flat 500000에 중첩이 있으면 500000인지 확인한다. RoPE가 아닌 필수 필드는 serde 자체 오류로 채우며 `rope_theta` / `rope_scaling`은 채우지 않는다.
- 추가 테스트: Llama 3 중첩 `llama3`가 `RopeScalingKind::Llama3`로 해석됨, Gemma 3 레이어 종류별, Phi `partial_rotary_factor`, DeepSeek-V4 `compress` YaRN(리뷰에서 추가, revert 실행은 하지 않음). lifter 단위 테스트는 `mlxcel-core`에 있다.
- 수정을 stash하고 테스트만 남기면 앞의 테스트 4개가 모두 실패하고, 89개 행 모두 실패를 보고한다.
- 실제 체크포인트: `~/models_archive/mlx/llama-3.2-1b-instruct-4bit`와 `~/models_archive/mlx/qwen3-30b-a3b-4bit`의 transformers-5 사본. `config.json`만 다시 쓰고(flat 키를 transformers 5.20.1이 해당 설정에 내보내는 것과 동일한 `rope_parameters` 블록으로 이동) 가중치는 심볼릭 링크로 연결했다. 명령: `MLXCEL_SDPA_DETERMINISTIC=1 gpu-lock run -- mlxcel generate -m <dir> -p "<CONTRIBUTING.md 앞 20000 바이트 + 질문>" -n 64 --temp 0 --show-reasoning`. 이 브랜치에서는 두 사본 모두 원본과 바이트 단위로 일치하고, 원본은 `main`과 일치한다. `main`(자기 일치 확인함)에서는 Llama 사본이 `and, and, ...`로, Qwen3 사본이 `. The. The. ...`로 퇴화한다.
- fmt, clippy(mlxcel과 mlxcel-core에 `--lib --tests -D warnings`), 영향받는 계열 테스트 필터(647개 테스트), 저장소 contract 검사가 로컬에서 통과한다.

## 5. 리뷰 중 변경

리뷰에서 CRITICAL 또는 HIGH 이슈는 발견되지 않았다. 커밋 `1684f151`에서 후속 수정 두 건을 적용했다.

- DeepSeek-V4 `main` / `compress` 레이아웃. 첫 버전은 이를 전부 `None`인 `rope_scaling`으로 만들었다.
- `rope_type` 스킴을 가리던 `"type": null` 키.

## 6. 미해결 사항

- 비전 인코더 설정(Pixtral, Llama 4 vision 등)과 `mlxcel-xla`. 이슈에 나열되지 않았다.
- DBRX는 transformers 5가 여전히 `rope_theta`를 두는 `attn_config`에서 끌어올리지만, 최상위 `rope_parameters`는 그곳으로 복사되지 않는다.

## 7. 변경 요약

| 항목 | 값 |
|------|----|
| 변경된 파일 | 96 |
| 추가된 줄 | 1134 |
| 삭제된 줄 | 116 |

| 커밋 | 메시지 |
|------|--------|
| `0681d42b` | fix(models): lift transformers 5 rope_parameters for every family |
| `1684f151` | fix(models): lift DeepSeek-V4 main/compress rope_parameters |
Loading
Loading