Repository navigation
feat(models): add MiniCPM-V 4.6 VLM support - #82
Merged
Merged
Conversation
Add MiniCPM-V 4.6, a vision-language model that pairs a Qwen 3.5 text backbone (standard 1D RoPE) with a two-stage visual compression pipeline. Ported from the upstream mlx-vlm MiniCPM-V 4.6 model (PR #1212). The vision pipeline compresses tokens in two stages: a VitMerger doing cross-attention spatial 2x2 pooling inserted after ViT layer 6, followed by a pixel-shuffle Merger MLP. For a 448x448 image this produces 64 visual tokens, matching `query_num`. New files: - `src/vision/encoders/minicpmv4_6.rs`: VitMerger, MergerBlock, Merger, and MiniCPMV46VisionModel with per-layer ViT access for mid-tower injection. - `src/vision/minicpmv4_6_vl.rs`: MiniCPMV46VLModel wrapping Qwen35Model. Wiring: - `src/models/mod.rs`: register the MiniCPMV46VLM variant, its display name, and the `metadata()` arm, and add it to `ALL_MODEL_TYPES` so it appears in `mlxcel list`. - `src/models/detection.rs`: map the `minicpmv4_6` model-type string to `ModelType::MiniCPMV46VLM`. - `src/loading/vlm_special.rs`: add the MiniCPM-V 4.6 weight remap, wired in through `src/loading/vlm.rs` and `src/loading/mod.rs`. - `src/vision/encoders/minicpmo.rs`: expose forward_embeddings / layer / post_layernorm / num_layers for per-layer ViT access by the 4.6 path. - `src/loaded_model.rs`, `src/loaded_model_capabilities.rs`, `src/model_metadata.rs`, `src/multimodal/vlm_runtime.rs`, `src/distributed/tensor_parallel/inference.rs`: route the new model through capabilities, metadata, the multimodal runtime, and tensor-parallel inference. Verified with cargo fmt, cargo clippy (metal,accelerate, all-targets, -D warnings), 17 unit tests (vision encoder config and merger token-reduction, the weight remap, and the model-type registry integrity tests), and `mlxcel list` showing the new model.
inureyes
added
type:enhancement
inureyes
added a commit
that referenced
this pull request
May 26, 2026
Bump version to 0.0.31. Highlights: MiniCPM-V 4.6 and RT-DETRv2 model support plus an Anthropic-style /v1/messages API endpoint (#82, #80, #74); fixes for missing or null chat-message content, Gemma 3n VLM per_layer_inputs, Qwen3.5 MTP verify attention, and batched quantized KV cache mask offsets (#91, #86, #78, #76); and a pinned 1.93.1 Rust toolchain (#87, #90).
sjswerdloff
added a commit
to sjswerdloff/mlxcel
that referenced
this pull request
Jun 25, 2026
) Add per-layer m3_idx_k cache to KVCache for MiniMax-M3's MSA indexer projection (index_k_proj). The indexer scores key BLOCKS for top-k selection; in chunked-prefill / multi-turn it must see the FULL cached prefix idx_k, not just the current chunk's projection. Current code (pre-lablup#80) projects from x only, so top-k selection is blind to cached positions — the architectural gap behind the cycle-65 latent comment. API additions on KVCache: - m3_idx_k_update_and_fetch(new_idx_k) -> full cached idx_k Concatenates the new chunk to prior cached state, advances m3_idx_offset, returns an independently-owned slice of the full cache. - m3_idx_offset() — accessor for the indexer's accumulated length. - has_m3_idx_k_state() — defensive check for whether the indexer cache has been written. Fields added to KVCache struct (initialized to None / 0 in both new() constructors at lines 514 and 562; default state for non-M3 models): - m3_idx_k: Option<UniquePtr<MlxArray>> - m3_idx_offset: i32 5 unit tests in cache.rs covering: initial-empty state, single-chunk write, two-chunk concat along seq axis, three-chunk accumulation (precursor to task lablup#82's four-chunk gate), independence from main K/V offset. All pass. Wired into M3's forward() by task lablup#81; until then the methods exist but are unused by mlxcel-M3. KVCache rationale (cf. struct doc): the model-specific fields are documented as opt-in state with Option<> types, consistent with the Turbo/INT8 sidecar pattern. Avoids per-model branching in the trait at the cost of a small per-cache memory footprint when unused. Per-request scoping (KVCache is request-scoped) inherits naturally.
sjswerdloff
added a commit
to sjswerdloff/mlxcel
that referenced
this pull request
Jun 25, 2026
The architectural completion: SparseAttention::forward() now correctly handles chunked-prefill / cached cases. Key changes: - Compute kv_len = cache_offset + l BEFORE update_and_fetch, so the asymmetric block counts are derived from the FULL cached length, not just the current chunk's length. - num_query_blocks vs num_key_blocks throughout: Q padded to num_query_blocks*block_size, K/V padded to num_key_blocks*block_size, reshape sites updated accordingly. - cache.m3_idx_k_update_and_fetch(&idx_k) (from lablup#80) swaps the current-chunk-only idx_k for the FULL cached idx_k. This closes the architectural gap behind the cycle-65 latent comment: top-k block selection now scores ALL num_key_blocks blocks, including the cached prefix that pre-lablup#81 was invisible to the indexer. - New apply_causal_block_mask_asymmetric: per-query-block max attendable absolute key block = floor((cache_offset + (i+1)*block_size - 1) / block_size). Over-permissive at block level; sparse_sdpa's unified mask trims within-block future positions. - New ensure_local_block_score_asymmetric: forces top-k to include the cached-prefix's last few absolute blocks (the query's local neighbourhood), which sit BEFORE the current chunk in absolute coords. - Dispatch dense when num_key_blocks <= top_k (top-k saturates → no sparsity benefit, dense is at least as good). - selected has shape [b, num_kv_heads, num_query_blocks, top_k] with ABSOLUTE key block indices in [0, num_key_blocks). sparse_sdpa's asymmetric path consumes them correctly. Symmetric apply_causal_block_mask + ensure_local_block_score marked #[allow(dead_code)] for regression value; the asymmetric forms subsume them at cache_offset=0. Deletion planned after lablup#78-lablup#83 lands and the asymmetric forms have lived without regressions. Cycle 65's "Unreachable under default prefill_chunk_size=512, hence latent until now" comment in cycle 65's code is now obsolete — the cached-prefill path is exercised on every forward with non-empty cache. 14 mask tests pass (7 symmetric regression + 7 asymmetric). Integration test for the actual four-chunk forward-attention path is task lablup#82; this commit wires the path, the test in lablup#82 verifies correctness against dense single-shot.
sjswerdloff
added a commit
to sjswerdloff/mlxcel
that referenced
this pull request
Jun 25, 2026
Proves the asymmetric MSA path produces correct output across cached prefill — the cycle-65 latent comment turned into a live gate. Three test variants on a synthetic SparseAttention layer (small dims: num_heads=4, num_kv_heads=2, head_dim=4, block_size=2, top_k=2, sparse_local_block=1, index_dim=4) with deterministic weights and input: 1. test_msa_four_chunk_chunked_vs_single_shot_no_crash The minimum gate: four forwards with growing KVCache complete cleanly, per-chunk output shapes match expected, final cache.offset and m3_idx_offset both equal total tokens (lockstep main + indexer cache). 2. test_msa_four_chunk_matches_single_shot_within_tolerance The correctness gate (Stuart's "no-crash + dense-differential within fp tol" bar). Four-chunk path (with growing cache) vs single-shot path (no cache) on the same 24-token input. Compare via relative L2: chunked_concat against single_out. Tolerance 5% — fp accumulation order differs across paths; selection determinism keeps the residual numerical-only, not structural. 3. test_msa_four_chunk_determinism_bit_exact Cycle 64 lesson: rule out MLX nondeterminism before declaring fp issues real. Run the chunked path twice, assert L2 diff < 1e-6. l_chunk=6 (3 blocks of 2 per chunk) chosen so num_key_blocks > top_k at chunk 1, making MSA fire on every chunk in both paths — neither path falls into the dense dispatch. 3 passing tests; cycle 65's existing 14 MSA mask tests still pass; total 17 MSA tests in minimax_m3.rs. The cycle-65 latent comment is no longer latent — task lablup#82 is the regression gate for any future MSA path modification.
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
Add MiniCPM-V 4.6, a vision-language model that pairs a Qwen 3.5 text backbone (standard 1D RoPE) with a two-stage visual compression pipeline. Ported from the upstream mlx-vlm MiniCPM-V 4.6 model (PR lablup#1212). The vision pipeline compresses tokens in two stages: a VitMerger doing cross-attention spatial 2x2 pooling inserted after ViT layer 6, followed by a pixel-shuffle Merger MLP. For a 448x448 image this produces 64 visual tokens, matching `query_num`. New files: - `src/vision/encoders/minicpmv4_6.rs`: VitMerger, MergerBlock, Merger, and MiniCPMV46VisionModel with per-layer ViT access for mid-tower injection. - `src/vision/minicpmv4_6_vl.rs`: MiniCPMV46VLModel wrapping Qwen35Model. Wiring: - `src/models/mod.rs`: register the MiniCPMV46VLM variant, its display name, and the `metadata()` arm, and add it to `ALL_MODEL_TYPES` so it appears in `mlxcel list`. - `src/models/detection.rs`: map the `minicpmv4_6` model-type string to `ModelType::MiniCPMV46VLM`. - `src/loading/vlm_special.rs`: add the MiniCPM-V 4.6 weight remap, wired in through `src/loading/vlm.rs` and `src/loading/mod.rs`. - `src/vision/encoders/minicpmo.rs`: expose forward_embeddings / layer / post_layernorm / num_layers for per-layer ViT access by the 4.6 path. - `src/loaded_model.rs`, `src/loaded_model_capabilities.rs`, `src/model_metadata.rs`, `src/multimodal/vlm_runtime.rs`, `src/distributed/tensor_parallel/inference.rs`: route the new model through capabilities, metadata, the multimodal runtime, and tensor-parallel inference. Verified with cargo fmt, cargo clippy (metal,accelerate, all-targets, -D warnings), 17 unit tests (vision encoder config and merger token-reduction, the weight remap, and the model-type registry integrity tests), and `mlxcel list` showing the new model.
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
Bump version to 0.0.31. Highlights: MiniCPM-V 4.6 and RT-DETRv2 model support plus an Anthropic-style /v1/messages API endpoint (lablup#82, lablup#80, lablup#74); fixes for missing or null chat-message content, Gemma 3n VLM per_layer_inputs, Qwen3.5 MTP verify attention, and batched quantized KV cache mask offsets (lablup#91, lablup#86, lablup#78, lablup#76); and a pinned 1.93.1 Rust toolchain (lablup#87, lablup#90).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #81
Adds MiniCPM-V 4.6, a vision-language model that pairs a Qwen 3.5 text backbone (standard 1D RoPE) with a two-stage visual compression pipeline. Ported from the upstream mlx-vlm MiniCPM-V 4.6 model (PR #1212).
Architecture
The vision pipeline compresses tokens in two stages: a VitMerger doing cross-attention spatial 2x2 pooling inserted after ViT layer 6, followed by a pixel-shuffle Merger MLP. For a 448x448 image this produces 64 visual tokens, matching
query_num.New files
src/vision/encoders/minicpmv4_6.rs: VitMerger, MergerBlock, Merger, and MiniCPMV46VisionModel with per-layer ViT access for mid-tower injection.src/vision/minicpmv4_6_vl.rs: MiniCPMV46VLModel wrapping Qwen35Model.Wiring
src/models/mod.rs: register theMiniCPMV46VLMvariant, display name,metadata()arm, andALL_MODEL_TYPESentry (so it appears inmlxcel list).src/models/detection.rs: map theminicpmv4_6model-type string to the variant.src/loading/vlm_special.rs(+vlm.rs,loading/mod.rs): MiniCPM-V 4.6 weight remap.src/vision/encoders/minicpmo.rs: expose per-layer ViT access for reuse.loaded_model.rs,loaded_model_capabilities.rs,model_metadata.rs,multimodal/vlm_runtime.rs,distributed/tensor_parallel/inference.rs: capabilities, metadata, runtime, and tensor-parallel routing.Verification
cargo fmt --checkclean.cargo clippy --features metal,accelerate --all-targets -- -D warningsclean.mlxcel listshowsMiniCPM-V 4.6 (SigLIP + VitMerger + Merger + Qwen3.5 text).