[inureyes]

Summary

Adds native support for Jina AI's small multilingual VLM (model_type: jvlm, jinaai/jina-vlm / jinaai/jina-vlm-mlx). A 27-layer SigLIP-so400m-class ViT feeds a Molmo-style connector into a Qwen2-class decoder with an OLMo tensor layout. Validated end to end on the real 4-bit MLX checkpoint through both mlxcel generate and mlxcel-server.

model_type is jvlm, not jina_vlm

The issue body says jina_vlm. No checkpoint declares that: jinaai/jina-vlm-mlx says jvlm at the top level and in text_config.model_type and vision_config.model_type, with architectures: ["JinaVLMForConditionalGeneration"] and auto_map pointing at modeling_jvlm.JinaVLM. jina_vlm is only the upstream mlx-vlm module name. Detection routes on jvlm, and accepts jina_vlm as an alias so a hand-edited config still lands in the right loader; a unit test pins both.

Three places where the checkpoint's own code and the mlx-vlm port disagree

mlx-vlm's models/jina_vlm/ is the obvious reference, but it is a reimplementation, and on three points it does not match the vendor code shipped with the checkpoint (modeling_jvlm.py, blocks_jvlm.py, image_processing_jvlm.py, processing_jvlm.py). The vendor code wins here, because that is what the model was trained with. All three are silent when wrong: they degrade the description rather than raise.

vit_layers resolves against a list that includes the post-norm output. JinaVLMVisionModel.get_visual_features appends the post_lnorm output to hidden_states, so the list is 28 long for a 27-layer tower and [-4, -10] selects the outputs of layers 24 and 18, not 23 and 17. Layers 25/26 and post_norm are then never consumed, so they are not loaded at all. The concatenation order is vit_layers order, not sorted, because it fixes which half of vl_connector.pooling.q's 2304-wide input each layer feeds.

The token grid rounds each crop window up separately. _molmo_get_patches_from_tiling rounds crop_window + margin up to a multiple of the pooling size per crop, then sums. The tiles * window + margins-then-round shortcut agrees only when the crop window is already even, which it is for Molmo (16 patches) and is not here (19): a 2x2 tiling comes out 46 where the reference says 48, one pooled column short per axis.

The coverage mask gets a trailing -1 row, not a leading 1.0 row. Upstream appends the sentinel after the crop masks even though the thumbnail was prepended to the pixel values, so mask row k belongs to crop k+1 and the last crop reads as partially padded (pad_embed[1] is added to it). That is an upstream off-by-one, reproduced deliberately and pinned by a test, for the same reason mlxcel's Molmo v1 processor reproduces its own version of it.

The scatter is additive, and that is exactly the reference's assignment

JinaVLMModel._encode_images adds the <im_patch> (151938) token embedding to every connector feature and then assigns the sum over the target rows. Every target row is an <im_patch> token by construction, because image_input_idx is built from nonzero(tokens == <im_patch>), so its embedding is already exactly the term being added and embed + feature equals feature + embed. Adding avoids a second embedding lookup and a masked write. A test pins the arithmetic so a future switch to an overwrite has to be deliberate.

The Falcon-OCR trap

sequence_state_layout() is delegated to the text model explicitly rather than left to the trait default. The default infers a model-owned placeholder from supports_batching(), and a wrapper that ever returns false there makes the server allocate an empty external cache; forward's layers.iter().zip(caches.iter_mut()) then runs zero decoder layers with no panic and no log line. That is what shipped in #848. There is a regression test asserting DenseKvCache and a layer count matching num_layers(), and the server path below was exercised for real rather than assumed.

The chat template the MLX conversion dropped

jinaai/jina-vlm ships a chat_template.jinja; the MLX conversion does not copy it, and tokenizer_config.json has no chat_template either. The server then falls back to the generic User:\n\nAssistant: template, which renders a prompt shape the model never saw. The difference is not subtle: asked for the capital of France it answers 17 under the generic template and Paris under its own.

So that template is now built in and consulted only after every declared source has come up empty (tokenizer_config.json, chat_template.jinja, chat_template.json), keyed on model_type, so it cannot shadow what a checkpoint actually ships. Three tests cover the fallback, the no-shadowing rule, and that an unknown template-less model still reports no template. With it, the server's image request renders the same 824-token prompt as the CLI.

What changed

• src/models/jina_vlm.rs: nested text_config normalization plus the decoder (fused attn.qkv, per-head Q/K RMSNorm, fused ffn.gate_up ordered [up, gate], rotate-half RoPE at theta 1e6, untied 4-bit lm_head). The split base-plus-extension embedding is Molmo's Molmo2Embedding, reused rather than re-declared.
• src/vision/encoders/jina_vlm.rs: the ViT tower (linear patch embed over patchified pixels, learned absolute positions, no class token, fused QKV, tanh-GELU MLP) and the connector (multi-layer concat, pad_and_partial_pad, 2x2 mean-query cross-attention pooling, SwiGLU projector). Not reusing encoders::siglip because that module expects q_proj/k_proj/v_proj and layer_norm1/2.
• src/vision/processors/jina_vlm.rs: smart_resize with Python's banker's rounding, select_tiling, the overlap-and-resize cropper, the <im_start> [<im_patch> * w <im_col>] * h <im_end> block, and build_image_input_idx. Resampling is bilinear with align_corners=false and no antialiasing, matching the checkpoint's interpolation: bilinear / antialias: false rather than the PIL bicubic the mlx-vlm port hardcodes.
• src/vision/jina_vlm.rs: the runtime (feature scatter, cache-layout declaration, structural-token suppression).
• src/loading/vlm_jina_vlm.rs: config normalization for the nested schema, processor construction from preprocessor_config.json, EOS resolution, and the <|image|> id.
• src/multimodal/vlm_runtime.rs: prompt assembly (User: + image block + text + Assistant:), the BOS the reference processor prepends, and the image_input_idx offset. Multi-image prompts concatenate their crops and blocks.
• Control-plane wiring: ModelType::JinaVLM, the "jvlm" / "jina_vlm" detection arm, for_each_model_registration!, LoadedModel, VlmRuntimeRef::JinaVlm, try_load_vlm_model_from_dir, the generate_vlm summary line, and the TP arch-string table.
• src/server/chat_template.rs: the built-in fallback described above.
• docs/supported-models.md.

No weight-key remapping step: the checkpoint already spells keys as language_model.* / vision_model.* / vl_connector.*. Mixed precision (language layer 0, every norm, both embedding tables, the ViT patch embedding and positional table, and vision_model.layers.N.ffn.down in bf16; the rest 4-bit affine) needs no special handling because UnifiedLinear::from_weights decides per prefix by probing for .scales, but the top-level quantization block does have to be threaded into the vision side too, not just the text stack.

Real-checkpoint validation

GB10, CUDA (sm_121), release build, models/mlx/jina-vlm-mlx (jinaai/jina-vlm-mlx, 4-bit / group 64). Image: a 640x480 photo of two cats lying on a pink couch with two TV remotes between them.

$ ./target/release/mlxcel generate -m models/mlx/jina-vlm-mlx --image coco_cats.jpg \
    -p "Describe the image." -n 128 --temp 0.0
Jina VLM: inserted 1 image block(s) (814 image tokens, 824 total tokens)
Generating...
Describe the image. In the image, two cats are comfortably resting on a pink couch. One cat is lying on its side, while the other cat is lying on its back. Both cats are surrounded by two remote controls, one positioned closer to the left cat and the other closer to the right cat. The scene appears to be a cozy and peaceful moment between the cats and their human companions.

[Generated 75 tokens in 2.44s = 30.75 tok/s]

Two cats, pink couch, two remotes: every grounded detail is right. A second, more pointed question rules out a generic caption:

$ ./target/release/mlxcel generate -m models/mlx/jina-vlm-mlx --image coco_cats.jpg \
    -p "How many cats are in the picture, what color is the couch, and what objects are lying next to them?" -n 96 --temp 0.0
There are two cats in the picture, and they are lying on a pink couch. Next to the cats, there are two remote controls, one on each side of the cats.

The 814 image tokens are the arithmetic working out end to end: smart_resize takes 640x480 to 644x476, select_tiling picks 2x2 against the 266-pixel crop window, the crop block is 24 patch + 1 col per row over 24 rows plus framing (602) and the thumbnail block is 14 + 1 over 14 rows plus framing (212). 5 crops of 729 patches, 980 pooled features, 772 of them valid, which is exactly the <im_patch> count.

./target/release/mlxcel arch lists Jina VLM (SigLIP-so400m + 2x2 attention pooling + Qwen2 text).

Server

$ ./target/release/mlxcel-server --model models/mlx/jina-vlm-mlx --port 18653
$ curl -X POST localhost:18653/v1/chat/completions -d @image_request.json
prompt_tokens 824
'In the image, two cats are comfortably resting on a pink couch. One cat is lying down on its side, while the other cat is lying down on its back. Both cats are surrounded by two remote controls, one positioned closer to the left cat and the other closer to the right cat. The scene appears to be a cozy and peaceful moment between the two cats.'

824 prompt tokens on the server, the same count the CLI reports, which is the check that the two front ends render the same prompt. Text-only on the same server:

$ curl -X POST localhost:18653/v1/chat/completions \
    -d '{"messages":[{"role":"user","content":"What is the capital of France? Answer in one word."}],"max_tokens":24,"temperature":0}'
prompt_tokens 16
'Paris.'

Two concurrent image requests were also run through the batch scheduler and both answered correctly (There are two animals in the image, which are cats. / The couch is pink.).

Test plan

• MLX_ENABLE_TF32=0 cargo test --profile test-fast --features cuda -p mlxcel --lib jina_vlm -- --test-threads=1 -> 39 passed
• MLX_ENABLE_TF32=0 cargo test --profile test-fast --features cuda -p mlxcel --lib server::chat_template -- --test-threads=1 -> 95 passed, 1 ignored
• MLX_ENABLE_TF32=0 cargo test --profile test-fast --features cuda -p mlxcel --lib multimodal::vlm_runtime -- --test-threads=1 -> 37 passed
• MLX_ENABLE_TF32=0 cargo test --profile test-fast --features cuda -p mlxcel --lib models::detection -- --test-threads=1 -> 26 passed
• MLX_ENABLE_TF32=0 cargo test --profile test-fast --features cuda -p mlxcel --lib model_metadata -- --test-threads=1 -> 8 passed
• MLX_ENABLE_TF32=0 cargo test --profile test-fast --features cuda -p mlxcel --lib models::metadata_tests -- --test-threads=1 -> 4 passed
• cargo clippy --features cuda --lib --bins --tests -- -D warnings clean
• cargo fmt --all clean
• cargo build --release --features cuda --bin mlxcel --bin mlxcel-server plus the CLI and server runs above

The synthetic tests cover the pieces that fail quietly: that negative vit_layers resolve against the post-norm slot and keep their order, that a fully padded crop is zeroed before the connector (with a live control crop so the check cannot pass by producing zeros everywhere), that the pad embedding fires only where the mask says so, that the pooled-patch to <im_patch> mapping is a bijection, that incremental decode lands on the same logits as a full-sequence pass (which is what catches a wrong RoPE offset or a QK norm on the wrong axis), and that the runtime declares the dense KV layout.

Not verified

No numeric parity against a PyTorch or mlx-vlm oracle: neither runs on this box, so the evidence is grounded generation on a real photo plus the token-count arithmetic above, not a logit diff. Multi-image prompts are implemented (crops and blocks concatenate, image_input_idx offsets accumulate) but were only exercised with one image. Tensor parallelism is refused for VLM-kind models before the arch-string table is read, so the qwen2 entry there is only kept for totality.

Closes #849

[inureyes]

Implementation Review Summary

Intent

Add native Jina VLM (model_type: jvlm) support: SigLIP-class ViT, Molmo-style 2x2 attention-pooling connector, Qwen2-class OLMo-layout decoder, overlap-and-resize processor, and full control-plane wiring, validated on jinaai/jina-vlm-mlx.

Findings Addressed

• The built-in chat template could not render typed content, so every server image request failed to render and silently fell back to the generic prompt shape (HIGH). Confirmed by reproduction: Chat template render (raw) failed ... tried to use + operator on unsupported types string and sequence on each image request, and the CLI and server generated different text ("lying on its side" vs "lying down on its side") despite both reporting 824 prompt tokens. The template now mirrors upstream's content['type'] loop and emits <|image|>.
• format_jina_vlm_prompt stripped every marker and re-templated, recognizing an already-rendered prompt only when it started with User: (HIGH). A system turn was wrapped a second time into a doubled trailing Assistant: Assistant:, and image blocks were forced to the head of the first user turn regardless of marker position. Replaced by split_jina_vlm_prompt, which splits at each marker the way the reference _interleave_text_and_image_tokens replaces each marker token in place.
• build_processor divided by patch_size / pooling_h / pooling_w read straight from preprocessor_config.json, so a zero panicked the process during load (LOW). Now clamped to at least 1.
• The output_suppressed_token_ids doc comment listed <im_slice>, which the returned vec never contained (LOW).

Remaining Items

• pool_and_project computes the pooling pad as grid % pooling rather than (pooling - grid % pooling) % pooling (LOW). This reproduces the reference VisionLanguageConnector.forward exactly, and both agree for the only released pooling size (2). A non-2 pooling size would fail loudly on the reshape in both implementations, so this is left matching the reference rather than silently diverging from it.
• hidden_state_slots() counts the post_lnorm slot only when post_lnorm is true, whereas the reference appends that slot unconditionally (LOW). No released config sets post_lnorm: false, so the resolved [-4, -10] to [24, 18] mapping is unaffected.
• JinaVlmTextConfig::vocab_size and additional_vocab_size are parsed but never read; the split embedding takes its shapes from the tensors (LOW, cosmetic).
• No numeric parity oracle. Evidence remains grounded generation plus token-count arithmetic, as the PR body states.

Verification of the three highest-risk areas

1. Chat-template safety. The built-in is reached only after tokenizer_config.json, the typed standalone, chat_template.jinja and chat_template.json have all come up empty, and builtin_chat_template matches only jvlm / jina_vlm with _ => None. No other family can be shadowed, and no pre-existing test was modified or weakened (the only deletions in the file were the two lines of the old let Some(template) = template else binding). The defect was in the template body, not the resolution order.

2. The three deliberate divergences from the mlx-vlm port. All three were checked against modeling_jvlm.py / blocks_jvlm.py / image_processing_jvlm.py, re-downloaded from jinaai/jina-vlm and byte-identical to the copies used during implementation. All three are correct: get_visual_features appends the post_lnorm output so hidden_states is 28 long and [-4, -10] selects layers 24 and 18; _molmo_get_patches_from_tiling rounds crop_window + margin up per crop, giving 48 for a 2x2 tiling of a 19-patch window where the flat shortcut gives 46; and img_mask = np.pad(img_mask, [[0, 1], [0, 0]], constant_values=-1) appends the sentinel after the crop masks even though the thumbnail was prepended to the pixel values, so the last crop reads as partially padded. The arithmetic reproduces end to end: 640x480 to 644x476, tiling 2x2, 602 + 212 = 814 image tokens, 980 pooled features of which 772 are valid, matching the <im_patch> count exactly.

3. sequence_state_layout() and server-path correctness. The override is real and delegates to the text model, so DenseKvCache with 28 layers is declared regardless of what the wrapper reports for supports_batching(), which is what the Falcon-OCR failure mode needed. Verified live rather than by inspection: the server prefills 824 prompt tokens and produces grounded output. After the fix, the CLI and server continuations are byte-identical.

Post-fix validation (GB10, CUDA sm_121, release)

• CLI: Jina VLM: inserted 1 image block(s) (814 image tokens, 824 total tokens), grounded caption (two cats, pink couch, two remotes). Counts unchanged from before the fix.
• Server: image request 824 prompt tokens with output byte-identical to the CLI; system message plus image 830 prompt tokens, grounded, no doubled generation marker; text-only 16 prompt tokens answering Paris..
• grep -c 'Chat template render' on the server log: 1 before the fix, 0 after.
• Tests: jina_vlm 40, server::chat_template 97, multimodal::vlm_runtime 42, loading::vlm 158, server::chat_request 75, models::detection 26, model_metadata 8, tokenizer:: 62, --bin mlxcel 178. 0 failed.
• cargo clippy --features cuda --lib --bins --tests -- -D warnings clean, cargo fmt --all clean.

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow (detection arm, for_each_model_registration!, LoadedModel, VlmRuntimeRef, try_load_vlm_model_from_dir, generate_vlm summary, TP arch string, docs)
• Project conventions followed
• Existing modules reused where applicable (Molmo2Embedding, UnifiedLinear mixed-precision probing, LayerNorm / RMSNorm, compiled_swiglu_activation, InputEmbeddings, the shared loader helpers)
• No unintended structural changes
• Tests pass

[inureyes]

Security and performance review

Focus: the server-reachable surface (chat template rendering, image preprocessing on untrusted uploads), the split embedding bounds, quantized checkpoint loading, and the decode hot path. Three commits pushed: 18e1a32e, 58efdef4, 6b8849c5.

Verdict on attacker-controlled chat content

Attacker-controlled chat content cannot destabilize the server. The built-in JINA_VLM_CHAT_TEMPLATE is a const &'static str; user content reaches minijinja only as data through Value::from_serialize, never as template source, so there is no server-side template injection primitive. template_content (src/server/chat_request.rs:630-653) normalizes every request into a closed set of four server-built shapes ({"type":"text","text":<String>}, {"type":"image"}, {"type":"video"}, {"type":"audio"}), and ContentPart is an internally-tagged serde enum with a required text: String, so an unknown type, a missing key, a wrong type, or nesting is a 400 at the HTTP boundary and never reaches the template. Runaway renders are bounded by the pre-existing env.set_fuel(Some(50_000_000)), and a render failure degrades to render_simple_fallback with a warn rather than propagating (chat_request.rs:222-231). The fallback path drops the <|image|> marker, and split_jina_vlm_prompt self-heals that by re-inserting markers, so the degraded path is safe too.

The change to the shared resolution path is behaviour-preserving for every other model: builtin_chat_template is keyed on model_type, so a non-jvlm checkpoint with no template still takes the original return Ok(None).

Fixed

HIGH (perf, 18e1a32e): 622 MB embedding table re-materialized on every decode step. Molmo2Embedding::forward concatenated the base and extension tables inside forward, which runs once per generated token. On this checkpoint the base table is BF16 [151936, 2048] = 622 MB, so every token allocated and copied a fresh ~623 MB array to gather one row. Hoisted to from_weights with a load-time eval. Measured 83.1/83.1/83.9 -> 148.2/148.9/149.4 tok/s (1.79x), greedy output byte-identical (same md5). Resident memory unchanged. molmo, molmo2 and molmo_point share the fix; molmo2-4b and molmo-7b were re-verified.

HIGH (58efdef4): unbudgeted full-resolution f32 image copy. preprocess_image built a Vec<f32> sized by the source image (12 bytes per source pixel) before resizing. The decoder's 512 MB image::Limits budget is charged in the decoder's native colour type, so a 16384x16384 8-bit grayscale PNG of a constant value compresses to a few hundred KB, is charged 268 MB, and then becomes 3.22 GB at that line, with the decoded image and the RGB8 copy still live: ~4.3 GB peak from a sub-1 MB request part, up to 16 images per request. smart_resize already bounds the target to ~1.0 Mpx, so the full-resolution buffer never needed to exist. The first resize now samples the u8 buffer and converts at read time. Measured peak RSS at 12000x12000: 3.11 GB -> 1.69 GB, and the curve is now flat in source resolution. Every sibling processor already resized before widening.

HIGH (58efdef4): one-hot matmul scatter, quadratic in image count. The vision feature merge built a dense [seq_len, n_targets] matrix and ran a full GEMM to move n_targets rows. Both dimensions scale with image count, so 16 images of ~3304x252 (a request body well under 1 MB) give seq_len ~33k and n_targets ~31k: about 2 GB of scratch and ~4.2 TFLOP on the serialized model thread, head-of-line blocking every other client. Replaced with mlxcel_core::scatter_add. The one-hot pattern is pre-existing in the Molmo family, but those runtimes only process images[0]; Jina VLM is the first to route all 16 through it, which is where the amplification comes from.

MEDIUM (58efdef4): unchecked feature-row gather. position was bounds-checked against seq_len but row was never checked against the feature row count before mlxcel_core::take. MLX's CUDA gather does not bounds-check: absolute_index only wraps negatives, then out[out_idx] = src[src_loc] is an unchecked device read, so an out-of-range row is silent garbage or a CUDA_ERROR_ILLEGAL_ADDRESS that poisons the context and kills the process. Not reachable today (processor and tower agree), but the agreement was implicit across two separately-parsed config sources.

CRITICAL-class robustness (6b8849c5): geometry config silently defaulted, producing fluent garbage. num_attention_heads, num_key_value_heads, head_dim and intermediate_size were read only from the nested block_config path, with no HF alias, despite the doc comment claiming both spellings are accepted. A flat-HF text_config therefore silently fell back to the released 3B defaults; the resulting slice_last_dim offsets stay in range, so attention runs on the wrong halves of Q/K/V with no error, no panic and no warning. Added the missing aliases, plus a load-time cross-check of (num_attention_heads + 2 * num_key_value_heads) * head_dim against the fused attn.qkv row count. The row axis is unpacked in both the dense ([4096, 2048] BF16) and 4-bit affine ([4096, 256] U32) cases on this checkpoint, so one check covers both; input width goes through the existing validate_quantized_packing.

HIGH robustness (6b8849c5): vision_config divisors unvalidated. patch_size, pooling_h and pooling_w reached four division and modulo sites unbounded. patch_size: 0 loaded fine and then panicked on the first image request; a negative pooling_h produced a negative pad width, which throws inside MLX and is an uncatchable abort. This was an internal inconsistency: the same loader already clamps the processor's copies of these three keys 75 lines below, with a comment saying exactly why. Now rejected at load, following vlm_locateanything's documented reject-rather-than-clamp reasoning.

HIGH robustness (6b8849c5): trusted tensor ranks. A 1-D pos_embed or pad_embed panicked on pe_shape[1]; a 1-row pad_embed was silently clamped by MLX slice to zero rows and then aborted in the following reshape. Both now rank-checked in from_weights. pad_embed stays optional, now with a doc comment stating what is lost when it is absent.

MEDIUM (6b8849c5): usize underflow hanging the worker. crop_h/crop_w used plain subtraction where the same function used saturating_sub 33 lines earlier. With oversized overlap_margins this wrapped to ~1.8e19 in release, making for py in 0..pooled_h an effectively infinite loop with a bounds-guarded body: the worker thread spins forever and the server stops serving entirely. molmo.rs does this arithmetic in i64 and cannot wrap. Fixed at both sites and rejected at load.

Verified clean

• Split base-plus-extension embedding bounds. MLX's CUDA gather does not clamp, so an out-of-range id would be an OOB device read. Not reachable: there is no raw-token-id endpoint, so ids come only from the tokenizer, whose maximum id on this checkpoint is 151941 against a 152064-row table.
• Vision path is prefill-only. JinaVlmModel::forward touches only self.text_model; neither self.vision_model nor self.processor is reachable from any decode entry point.
• No prompt-length scaling in decode. RoPE takes a scalar cache.offset, not a positions vector, so the known prior-port bug of that shape is not present. No eval, item_*, or slice readback anywhere in the three hot-path files. Per-step work is O(layers).
• Crop count is bounded. select_tiling only emits pairs with i * j <= max_crops, so at most 13 crops and ~22.3 MB of pixel_values per image, and the bound is enforced before any allocation proportional to it. smart_resize outputs are capped at exactly max_pixels. Float-to-int casts saturate rather than wrap.
• Malformed images are rejected cleanly; the zero-images case cannot reach the Jina arm.

Not fixed, worth a follow-up

1. Greedy image-path generation on this checkpoint is already nondeterministic, before any of these changes. Twenty runs of the unmodified binary produced two distinct completions (17/3, then 10/10 after). Text-only is stable, and scatter_add and the one-hot matmul were each isolated at the real problem size (seq_len 434, 424 targets, bf16) and produced exactly one result over 64 repeats, so neither is the source. It is somewhere in the tower and is worth its own issue. Practical consequence for reviewers: a single-run byte-identity check on the image path is a coin flip, so the bitwise unit tests added in 58efdef4 are the reliable signal.
2. forward_batched falls back to the serial trait default while supports_batching() returns true, so batched decode is B separate forward passes with no weight-bandwidth amortization. Molmo does the same, but the Qwen2-class sibling this port is modelled on overrides it. Either implement it or document the gap.
3. No prompt-length check against the context window before the KV cache is sized. A 70-byte 1x1 PNG buys 424 tokens, and 16 images can reach ~33k tokens and ~3.8 GB of KV. Pre-existing repo-wide, but Jina VLM has the highest per-image token count of the crop-tiling processors and is the first to accept multiple images.
4. Literal <|image|> in user text steals an image block's splice position. No crash: the reconciliation loops in split_jina_vlm_prompt guarantee exactly n_images + 1 segments, so no feature-less block is ever emitted. Consequence is prompt-structure confusion within the attacker's own request. The repo already has validate_no_reserved_media_sentinels for this class on the audio path; extending it is the cleaner home than escaping in the template.
5. Decode misses the fused-RoPE and Turbo-KV dispatch the Qwen2-class sibling has, partly justified by the per-head Q/K norms sitting between projection and RoPE. Worth a note in the module header so it does not read as an oversight.
6. max_crops, min_pixels, and a non-square base_input_size are still unclamped in the loader and can produce an index panic or a hang on the first image request. Config-tier and below the fix threshold here.
7. Pre-existing shared-code gap: UnifiedLinear::from_weights never checks that a 2-D quantized weight's scales leading dimension matches the weight's, and MLX's own batch-shape check compares zero elements for a 2-D weight, so a wrong leading dim is a silent out-of-bounds read. validate_quantized_packing has this check and is not called by any VLM loader. This PR widens exposure by adding call sites rather than introducing it.

Verification

Release build with --features cuda, cargo clippy --features cuda --lib -- -D warnings clean, cargo fmt --all clean. --lib jina_vlm 52 passed, --lib vision::processors 171 passed, server::chat_template + multimodal::vlm_runtime + models::detection 165 passed, all with --test-threads=1 and confirmed non-zero counts. On the real 4-bit checkpoint: image path still reports 1 image block(s) (424 image tokens, 434 total tokens) and produces a coherent description; text-only is unchanged at 144 tok/s; molmo2-4b and molmo-7b re-verified for the shared embedding change.