[inureyes]

Summary

Adds end-to-end support for the InternVL VLM family (internvl_chat, e.g. internvl3-1b) so it loads and runs in both text-only and image modes, wired into the real CLI and server VLM dispatch rather than left as standalone modules.

Architecture

Verified against the real config.json and safetensors tensor names, which override the "reuse the InternLM backbone" assumption:

• Language model is Qwen2, not InternLM. text_config.model_type == "qwen2" (hidden 896, 24 layers, 14 heads, 2 KV heads, QKV bias, untied embeddings, rope_theta 1e6), so the existing Qwen2 backbone (models::Qwen2Model) is reused rather than reimplementing a transformer. The bogus top-level llm_config (vocab 151936, 32B-template dims) is ignored in favor of the real text_config (vocab 151674).
• InternViT vision tower (vision_model.*): a standard SDPA encoder with fused QKV (+bias), per-layer layer-scale (ls1/ls2), LayerNorm, a Conv2d patch embed, a CLS token, and a learned position embedding. qk_normalization is false for InternVL3 so q/k norms are skipped. select_layer=-1 returns the last hidden state and the CLS token is stripped before the connector. Every tile is exactly 448x448, so the patch grid matches the trained position-embed grid and interpolation is the identity.
• Connector mlp1.*: pixel_shuffle(downsample_ratio=0.5) reshapes (B, 1024, 1024) into (B, 256, 4096), then LayerNorm -> Linear(4096->896) -> GELU -> Linear(896->896), yielding 256 image tokens per tile in the Qwen2 hidden size.
• Dynamic-tiling processor: closest-aspect-ratio tiling (min 1, max 12 tiles, plus a thumbnail when split), ImageNet normalization, channels-first output.
• Image-token merge replaces <IMG_CONTEXT> (151667, resolved from the tokenizer) positions with the connector output through the shared merge_llava path.

Mixed precision: the top-level quantization block makes the common VLM weight loader treat the checkpoint as quantized and skip its bf16->f16 conversion, keeping the 4-bit LLM and connector scales/biases as bf16 for quantized_matmul. The InternViT tower and connector LayerNorm are plain bf16 (which JIT-crash on M5), so the loader explicitly converts every remaining bf16 tensor to f16 except .scales/.biases, mirroring the standard Apple Silicon non-quantized-weight policy.

RoPE: standard 1D RoPE (not Qwen2-VL MRoPE). The config's rope_scaling dynamic-NTK factor is parsed but not applied in the rope path; for the 1B at short context this is a no-op and is noted as a known limitation.

Integration

• models/detection.rs: model_type == "internvl_chat" -> ModelType::InternVLChatVLM.
• models/mod.rs, model_metadata.rs: new InternVLChatVLM ModelType plus its Vlm metadata descriptor so the CLI and server both route it as a VLM.
• loaded_model.rs: LoadedModel::InternVLChatVLM variant with LanguageModel delegation.
• loaded_model_capabilities.rs, multimodal/vlm_runtime.rs: VlmRuntimeRef::InternVL runtime arm with tile-count-driven <img> + <IMG_CONTEXT>*N + </img> token expansion.
• loading/vlm_internvl.rs: load_internvl_vlm() (config parse, mixed-precision weight handling, Qwen2 + InternViT + connector + processor construction, token-id/EOS resolution).
• New modules: vision/encoders/internvl.rs, vision/internvl.rs, vision/processors/internvl.rs, multimodal/internvl_prompt.rs.
• distributed/tensor_parallel/inference.rs: maps InternVL onto the llama-family planner placeholder (TP is already refused for VLM-kind models upstream).

M5 embedding-prefill fixes

InternVL is the first VLM short enough to exercise the single-pass prefill path on the M5 Neural Accelerator, which surfaced two embedding-prefill bugs fixed here:

• Skip NA-tile prefill padding for embedding requests. NA prefill alignment padded the token-id vector and built a matching padded causal mask, but VLM image requests run through forward_with_embeddings with pre-merged embeddings that are not padded, so a 293-token image prompt met a 320-wide mask and aborted with a broadcast-shape error. Alignment is now skipped whenever a sequence carries pre-merged embeddings; text-only prefill is unchanged.
• Route VLM embedding prefill to full prefill, not chunked. Prompts longer than prefill_chunk_size (default 512) were sent to the chunked path, which re-triggered the same broadcast abort and is incoherent for embedding-based prefill (the whole embedding sequence is consumed while the prefill offset advances one chunk, corrupting cache bookkeeping). A multi-tile image (e.g. a 2:1 photo at 768+ image tokens) trivially exceeds 512 and hit this path; the chunked route is now gated on the absence of pre-merged embeddings, mirroring the batched-prefill dispatch that already forces VLMs onto full prefill.

Testing

• cargo fmt --all -- --check — clean.
• cargo clippy --features metal,accelerate --all-targets -- -D warnings — clean.
• 13 new/related unit tests pass: pixel_shuffle shape collapse (1024x1024 -> 256x4096), dynamic-tiling processor (single-tile square, multi-tile + thumbnail, ImageNet normalization, per-image tile counts), prompt-token insertion/expansion, and the two VLM prefill routing guards (exercised without M5 hardware).
• Model-gated tests/internvl_parity.rs covers detection, processor shapes, and finite text-only and image-conditioned logits of the correct vocab dimension (runs when the model is present).