Skip to content

feat(models): add MiniCPM-V 4.6 VLM support #81

Closed

Description

@inureyes

mlxcel has no MiniCPM-V 4.6 support. MiniCPM-V 4.6 is a vision-language model that uses a Qwen 3.5 text backbone with a two-stage visual compression pipeline, so it needs both a new vision encoder and VLM wiring.

Architecture

MiniCPM-V 4.6 uses Qwen 3.5 (standard 1D RoPE) as the text backbone and compresses vision tokens in two stages:

  • VitMerger: cross-attention spatial 2x2 pooling inserted after ViT layer 6.
  • Merger: a pixel-shuffle MLP after the ViT.

For a 448x448 image this yields 64 visual tokens, matching query_num.

Scope

  • New vision encoder module implementing VitMerger, the merger block, the merger, and the MiniCPM-V 4.6 vision model with per-layer ViT access for mid-tower injection.
  • New VLM wrapper binding the vision pipeline to the Qwen 3.5 text model.
  • Expose per-layer ViT access (embeddings, single layer, post-layernorm, layer count) on the existing MiniCPM-O encoder for reuse by the 4.6 path.
  • A weight-remap step for the MiniCPM-V 4.6 checkpoint layout.
  • Register the model type and display name, and wire it into VLM loading, capabilities, metadata, the multimodal runtime, and the tensor-parallel inference path.

Acceptance criteria

  • The model type is registered, appears in mlxcel list, and resolves from the minicpmv4_6 model-type string.
  • A MiniCPM-V 4.6 checkpoint loads through the VLM loader, with the two-stage visual compression producing the expected visual-token count.
  • Unit tests cover the vision encoder blocks and the weight remap.

Metadata

Metadata

Assignees

Labels

area:modelsModel architectures, weights, loading, metadatapriority:mediumMedium prioritystatus:doneCompletedtype:enhancementNew features, capabilities, or significant additions

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions