feat(vlm): add MiniCPM-V 4.7 (minicpmv4_7) support #2398

Summary

Add MiniCPM-V 4.7 (model_type: minicpmv4_7, MiniCPMV4_7ForConditionalGeneration) in both released sizes: 1B (Qwen3.5 dense hybrid text, text_config.model_type: qwen3_5_text) and 35B-A3B (Qwen3.5 MoE hybrid text, qwen3_5_moe_text, 256 experts, top-8). The vision tower and mergers are MiniCPM-V 4.6's, which mlxcel already runs. What is new: 3-D "canvas" M-RoPE positions for thumbnails, slices and frames (4.6 runs the text model on 1-D RoPE), new marker token ids, the 4x downsample mode, and the MoE text variant.

Upstream reference

• Port: Add MiniCPM-V 4.7 1B and 35B-A3B with a custom processor Neywa-Labs/mlx-vlm#2499 (commit e928101a, merged 2026-10-13). Follow-ups: Fix MiniCPM vision dtype with quantized embeddings Neywa-Labs/mlx-vlm#2511 (keep pixel and merged-feature dtype float when token embeddings are quantized) and Simplify MiniCPM-V 4.7 vision cache handling Neywa-Labs/mlx-vlm#2514 (commit 5229dbdd, Python-side cache-handling refactor with no numeric change).
• Code: https://github.com/Blaizzy/mlx-vlm/tree/main/mlx_vlm/models/minicpmv4_7 (minicpmv4_7.py, rope.py with CanvasRoPE, config.py, language.py, processing_minicpmv4_7.py), built on https://github.com/Blaizzy/mlx-vlm/tree/main/mlx_vlm/models/minicpmv4_6.
• Test checkpoints: mlx-community/MiniCPM-V-4.7-1B-4bit (2.7 GB, affine) and mlx-community/MiniCPM-V-4.7-35B-A3B-mxfp4 (22.5 GB, quantization.mode: mxfp4, experts already in switch_mlp layout). Originals: openbmb/MiniCPM-V-4.7-1B, openbmb/MiniCPM-V-4.7-35B-A3B.

Current mlxcel state

• src/models/detection.rs:1100 maps "minicpmv4_6" to ModelType::MiniCPMV46VLM; minicpmv4_7 is rejected.
• src/loading/vlm_special.rs:495 (load_minicpmv4_6_vlm) parses text_config as qwen3_5::Qwen35Config, calls validate_supported(), remaps weights (remap_minicpmv4_6_weights, line 442) and builds models::Qwen35Model.
• src/vision/minicpmv4_6_vl.rs:30 (MiniCPMV46VLModel) documents that 4.6 uses standard 1-D RoPE; images reuse the MiniCPM-o processor and prompt (src/vision/processors/minicpmo.rs, src/multimodal/minicpmo_prompt.rs).
• src/vision/encoders/minicpmv4_6.rs always applies the VitMerger after insert_layer_id (no 4x mode). Both the VitMerger (line 343) and the final Merger (line 393) use mlxcel_core::gelu_approx, which is erf-based despite its name (src/lib/mlxcel-core/cpp/mlx_cxx_bridge.cpp:1891-1902).
• src/models/qwen3_5.rs already carries the MRoPE path the Qwen3.5 VLM uses: set_mrope (line 2222), set_mrope_state / set_mrope_state_for_sequence (lines 2121-2148), src/models/qwen_mrope_state.rs. MoE is selected per layer by Qwen35Config::is_moe_layer (line 489).
• The pixel cast at src/vision/minicpmv4_6_vl.rs:84 uses the dtype of get_embed_tokens output (src/models/qwen3_5.rs:2196, which dequantizes), so the defect fixed by Fix MiniCPM vision dtype with quantized embeddings Neywa-Labs/mlx-vlm#2511 (casting to the packed uint32 weight dtype) does not occur in mlxcel; keep it that way and cover it with the test below.

Proposed solution

1. Detection and registry: "minicpmv4_7" => Ok(ModelType::MiniCPMV47VLM) beside line 1100, src/model_metadata.rs entry, src/loading/mod.rs dispatch. One variant covers dense and MoE; the text kind follows text_config.model_type (qwen3_5_text dense, qwen3_5_moe_text MoE, anything else rejected with the value named).
2. Loader load_minicpmv4_7_vlm in src/loading/vlm_special.rs, reusing remap_minicpmv4_6_weights and minicpmv4_6_text_weights. Configure MRoPE on the Qwen35Model from text_config.rope_parameters (mrope_section [11,11,10], mrope_interleaved: true, partial_rotary_factor 0.25, rope_theta 1e7). Validate mrope_mode == "canvas" and downsample_mode in {"4x","16x"} at load, as upstream config.py does. For a native (unconverted) MoE checkpoint, split fused experts with the existing Qwen3.5 MoE sanitize path; converted checkpoints already ship switch_mlp.*.
3. Canvas M-RoPE: new src/vision/minicpmv4_7_rope.rs, a direct port of upstream CanvasRoPE (_group_visual_frames, get_vision_position_ids, get_rope_index). Inputs: token ids, a per-token modality map (image token 1, video token 2), and the per-span crop grids tgt_sizes; output: [3, B, L] position ids and rope_deltas = max(pos) + 1 - L. Thumbnail and slices of one image share a canvas; grid coordinates are rint(linspace(0, canvas-1, n)). Feed them through set_mrope_state_for_sequence. Text-only requests keep the 1-D path.
4. Span validation (upstream get_input_embeddings): merge factor is 2 for 4x and 4 for 16x; every placeholder span must have one crop grid, each grid dimension must be positive and divisible by the factor, and prod(grid // factor) must equal the span length. Each violation is an error naming the span index and both numbers.
5. Downsample mode: honor downsample_mode from config (16x on all four checkpoints above). 4x skips the VitMerger (upstream use_vit_merger = mode != "4x") and uses factor 2 in the processor and the span check. A per-request override is out of scope.
6. GELU: upstream 4.6 builds both mergers with nn.GELU(approx="precise") and 4.7 sets vit_merger.act = nn.GELU(approx="tanh"); in MLX both strings select the tanh approximation, so 4.7 is not a change from 4.6 upstream. mlxcel's shared encoder computes erf at lines 343 and 393. For 4.7, add an activation selector (pattern: MoonViTMlpActivation, src/vision/encoders/kimi_vl.rs:131) and select tanh, using the existing tanh form gemma3_backbone::gelu_approx (src/models/gemma3_backbone.rs:59). Leave 4.6's default unchanged in this PR; the 4.6 discrepancy has been reported separately and is not part of this issue.
7. Prompt and processor: reuse the MiniCPM-o slicing processor and prompt builder with the 4.7 ids from config (image_token_id 248056, video_token_id 248057, image_start_id 248078, image_end_id 248079, slice_start_id 248088, slice_end_id 248089, newline_id 198), and emit the tgt_sizes the canvas needs alongside the placeholders. EOS from config ([248044, 248046]).

Scope

In scope: 1B and 35B-A3B, text, single and multiple images, CLI and server, docs/supported-models.md row. Out of scope: native video (it keeps the ordered-frames fallback; do not add the type to model_type_has_native_video), per-request 4x/16x override, DSpark/MTP drafters, the 4.6 GELU fix.

Acceptance criteria

• mlxcel arch lists minicpmv4_7; detection test for both configs; a qwen3_5_text/qwen3_5_moe_text mismatch is rejected with the value named.
• Canvas RoPE parity test in src/vision/minicpmv4_7_rope_tests.rs: position tensors for (a) one image with thumbnail plus a 2x2 slice grid and (b) two images separated by text equal fixtures dumped from upstream CanvasRoPE.get_rope_index (commit the dump script's output as test data, with the upstream commit noted). Fails if positions fall back to 1-D.
• Span validation tests: wrong grid count, grid not divisible by the factor, and product mismatch each return their specific error; 4x and 16x use factors 2 and 4.
• Unit test: with a quantized (4-bit) token embedding table, the pixel input and merged features stay floating point and keep their fractional values (regression guard for Fix MiniCPM vision dtype with quantized embeddings Neywa-Labs/mlx-vlm#2511).
• mlxcel generate -m models/MiniCPM-V-4.7-1B-4bit --image <photo> -p "Describe this image." -n 64 and the same for models/MiniCPM-V-4.7-35B-A3B-mxfp4 are coherent, an OCR-style image returns its text, and a two-image prompt references both; the same through mlxcel-server. Record outputs in the PR.
• Teacher-forced logit trace (examples/logit_trace.rs + scripts/compare_logit_traces.py, per docs/benchmarks.md) against mlx-vlm for the 1B on an image prompt, gating on disagreement at decided positions; the arm with canvas positions replaced by 1-D positions must fail, and the arm with erf merger GELU is run and reported.
• Existing MiniCPM-V 4.6 and MiniCPM-o tests pass unchanged.

Verification

cargo build --release --features metal,accelerate
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --check
./target/release/mlxcel download mlx-community/MiniCPM-V-4.7-1B-4bit
./target/release/mlxcel download mlx-community/MiniCPM-V-4.7-35B-A3B-mxfp4
./target/release/mlxcel generate -m models/MiniCPM-V-4.7-1B-4bit --image <photo> -p "Describe this image." -n 64