You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
mlxcel has no MiniCPM-V 4.6 support. MiniCPM-V 4.6 is a vision-language model that uses a Qwen 3.5 text backbone with a two-stage visual compression pipeline, so it needs both a new vision encoder and VLM wiring.
Architecture
MiniCPM-V 4.6 uses Qwen 3.5 (standard 1D RoPE) as the text backbone and compresses vision tokens in two stages:
VitMerger: cross-attention spatial 2x2 pooling inserted after ViT layer 6.
Merger: a pixel-shuffle MLP after the ViT.
For a 448x448 image this yields 64 visual tokens, matching query_num.
Scope
New vision encoder module implementing VitMerger, the merger block, the merger, and the MiniCPM-V 4.6 vision model with per-layer ViT access for mid-tower injection.
New VLM wrapper binding the vision pipeline to the Qwen 3.5 text model.
Expose per-layer ViT access (embeddings, single layer, post-layernorm, layer count) on the existing MiniCPM-O encoder for reuse by the 4.6 path.
A weight-remap step for the MiniCPM-V 4.6 checkpoint layout.
Register the model type and display name, and wire it into VLM loading, capabilities, metadata, the multimodal runtime, and the tensor-parallel inference path.
Acceptance criteria
The model type is registered, appears in mlxcel list, and resolves from the minicpmv4_6 model-type string.
A MiniCPM-V 4.6 checkpoint loads through the VLM loader, with the two-stage visual compression producing the expected visual-token count.
Unit tests cover the vision encoder blocks and the weight remap.
Activity
feat(M3): IndexerCache storage on KVCache for chunked-prefill (lablup#80
feat(M3): wire forward() end-to-end for cached MSA (lablup#81)