feat(models): add LocateAnything (locateanything) VLM support #847

Summary

LocateAnything is a ~3B generative grounding VLM: a MoonViT vision tower feeds an MLP connector into a Qwen2-style decoder-only text backbone. Standard decode is ordinary autoregressive text generation whose output interleaves coordinate control tokens (box / coord / ref markers) encoding located regions; the checkpoint also ships a specialized parallel-box-decoding path (pbd) with multi-token-prediction heads (n_future_tokens: 6) for fast box emission. The text side (model_type: qwen2) is already fully supported in mlxcel, so the new work is the MoonViT tower, the connector, image-token splicing, and the grounding output handling. This is the strongest first VLM pick of the batch because the language model is done.

Concretizes the LocateAnything entry recorded as tracking-only in #524, filing it as a port work item.

Upstream reference

• Directory: https://github.com/Blaizzy/mlx-vlm/tree/main/mlx_vlm/models/locateanything
• Vision (MoonViT tower): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/vision.py
• Language (Qwen2 decoder): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/language.py
• Top-level model + connector: https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/locateanything.py
• Parallel box decoding (grounding heads): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/pbd.py
• Config: https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/config.py
• Processor + image processor: https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/processing_locateanything.py , https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/locateanything/image_processing_locateanything.py

Public checkpoint

Confirmed available on HuggingFace:

• nvidia/LocateAnything-3B (original)
• mlx-community/LocateAnything-3B-4bit (MLX, primary validation target)
• andai-labs/LocateAnything-3B-MLX (alternate MLX conversion)

Architecture notes

• model_type: locateanything with a qwen2 text sub-config and a moonvit vision sub-config; image_token_index: 151665.
• Vision: MoonViT tower with a SwiGLU MLP; a connector projects pooled ViT features into the text embedding space.
• Text: Qwen2 decoder-only stack, already supported in mlxcel.
• Grounding contract: output text carries coordinate / box control tokens. The value of the model is the grounding output, so coordinate-token handling matters. The pbd path adds multi-token-prediction heads (n_future_tokens: 6) for parallel box emission; a minimal port can start with plain autoregressive decode (raw coordinate tokens in the text) and add the parallel path later.

Implementation plan

• Reuse the existing Qwen2 decoder in src/models/qwen2.rs for the language model verbatim; the text arm already exists in src/models/detection.rs ("qwen2" => Ok(ModelType::Qwen2)).
• MoonViT already exists in-tree for Kimi-VL: reuse src/vision/encoders/kimi_vl.rs (encoder) and the VLM wiring in src/vision/kimi_vl.rs + src/loading/vlm_kimi_vl.rs as the closest template. Confirm the LocateAnything MoonViT config keys against Kimi-VL and diverge only where they differ.
• Reuse qwen2_vl-style patch / RoPE ViT helpers (src/vision/encoders/qwen2_5_vl.rs, src/vision/qwen2_vl.rs) where the MoonViT patch / RoPE logic overlaps.
• Add a new src/vision/locateanything.rs VLM module + src/loading/vlm_locateanything.rs loader following the shape of the Kimi-VL pair.
• Image-token splice: expand image_token_index: 151665 placeholders and scatter projected features via the generic merge path in src/vision/merge.rs / src/models/multimodal_placeholders.rs.
• Grounding: for the first landing, emit coordinate tokens as plain text (no post-processing) and validate against a grounded prompt; scope the pbd parallel-box path and coordinate-to-box post-processing as a follow-up.

Touchpoints & acceptance criteria

• Text config (qwen2) + vision config (moonvit) parse from config.json.
• MoonViT vision encoder + MLP connector implemented (reusing the Kimi-VL MoonViT).
• Image-token splice / merge: image_token_index: 151665 placeholder expansion + feature scatter.
• from_weights + sanitize handle mixed precision (vision often plain bf16 while text is quantized) and any conv / patch weight channels-last remap.
• Arch-string arm "locateanything" added in src/models/detection.rs.
• Registration in src/model_metadata.rs via for_each_model_registration!.
• TP / distributed arch-string + generate_vlm summary updated if applicable.
• _tests.rs unit tests (config parse, connector shapes, placeholder expansion).
• docs/supported-models.md updated.
• Real-checkpoint validation: ./target/release/mlxcel generate -m models/LocateAnything-3B-4bit --image <img> -p "<grounding prompt>" produces correct grounded output (coordinate tokens referencing the right region). Follow docs/adding-models.md.
• Follow-up scoped (not blocking this issue): pbd parallel-box-decoding path + coordinate-token-to-box post-processing.

Effort

MEDIUM-HIGH. The language side is already done; the delta is the MoonViT tower (largely present via Kimi-VL), the connector, and the grounding output contract. Best first VLM pick of this batch.