feat(models): video input for the Qwen VL families (Qwen3.5-VL / Qwen3.8, then the rest) #1166

Summary

--video is rejected for every Qwen VL family with "--video input is currently only supported by Gemma 4 and Kimi-VL VLMs" (src/commands/generate_vlm.rs:447), even though the checkpoints ship a full video contract and upstream mlx-vlm supports video on this path end to end. Add video input for the Qwen VL families, starting with qwen3_5 (Qwen3.5-VL / Qwen3.8-27B).

The rejection is honest and fails closed, so this is a missing capability, not a correctness bug.

Evidence that the checkpoints expect video

mlx-community/Qwen3.8-27B-4bit (revision 3e6447f0) ships all of:

• config.json: video_token_id: 248057, vision_start_token_id: 248053, vision_end_token_id: 248054
• video_preprocessor_config.json: video_processor_type: "Qwen3VLVideoProcessor", patch_size 16, temporal_patch_size 2, merge_size 2, size.longest_edge 25165824
• processor_config.json: fps: 2.0, min_frames: 4, max_frames: 768
• chat_template.jinja: emits <|vision_start|><|video_pad|><|vision_end|> for item.type == 'video'

The Qwen3.8-27B model card advertises "Native support for image and video understanding, from STEM diagrams and documents to hour-scale videos."

mlxcel already reads video_token_id for these families (src/loading/vlm.rs:611-626, src/loading/vlm_qwen.rs), so the id plumbing is half there; nothing consumes it.

What upstream implements

mlx_vlm/models/qwen3_5/vision.py is five lines subclassing qwen3_vl's vision model, so all video behavior comes from the shared Qwen3-VL stack:

• Token merge treats image and video alike: special_image_mask = (input_ids == image_token_index) | (input_ids == video_token_index), then masked_scatter. get_input_embeddings falls back to pixel_values_videos when pixel_values is None and picks grid_thw = image_grid_thw if image_grid_thw is not None else video_grid_thw.
• 3-D MRoPE with a real temporal axis: get_rope_index counts image vs video spans by inspecting the token following each vision_start_token_id, walks them in positional order, and for a video expands t_index = mx.arange(llm_grid_t) broadcast over llm_grid_h * llm_grid_w. h/w are divided by spatial_merge_size; t is not.
• Temporal patching in the tower: a 3-D conv with kernel_size = [temporal_patch_size, patch_size, patch_size]; cu_seqlens repeats the per-frame h*w length grid_thw[i,0] times. Still images are handled by duplicating along T up to temporal_patch_size, so grid_t = 1 flows through the same conv.
• A real video preprocessor: processing_qwen3_vl.py defines a BaseVideoProcessor subclass with model_input_names = ["pixel_values_videos", "video_grid_thw"], _smart_resize_video (rounds the frame count up to a multiple of temporal_factor), bicubic per-frame resize, defaults fps 2.0, min_frames 4, max_frames 768.

Current mlxcel gap

• VlmPreparationSummary::QwenVlm (src/multimodal/vlm_runtime.rs:67-70) carries only image_blocks and total_image_tokens. There is no video branch anywhere in the Qwen VL preparation path; grid_thw is derived from images only.
• compute_vlm_embeddings (src/commands/generate_vlm.rs:403-447) dispatches video to Gemma 4, Gemma 4 Unified, and Kimi-VL, then errors for everything else.
• cli_video_content_part_count (src/commands/generate.rs:793-801) returns num_videos only for Gemma4Unified and 0 otherwise, so the CLI never renders a {"type":"video"} content part for a Qwen model. That is what keeps an unexpanded <|video_pad|> out of the prompt today, and it must stay in lockstep with whatever this issue adds.
• The server has no video request path for these families either.

Implementation plan

1. Add a Qwen video processor: aspect-preserving _smart_resize_video on the 16-pixel grid, frame count rounded up to a multiple of temporal_patch_size, bicubic per-frame resize, (pixel/255 - mean)/std with the checkpoint's image_mean/image_std (both [0.5, 0.5, 0.5]), producing pixel_values_videos and video_grid_thw.
2. Honor fps, min_frames, and max_frames from processor_config.json, with --fps overriding. Reject a clip that would exceed max_frames with a named error rather than silently truncating.
3. Extend the Qwen VL placeholder expansion to video_token_id, asserting strict lockstep between declared clips, decoded clips, <|video_pad|> placeholders, projected feature rows, and final embedding positions. A cardinality mismatch must be an invalid-request error before generation; media must never be silently dropped.
4. Extend the MRoPE position builder to the video case: temporal index from arange(llm_grid_t), spatial indices divided by spatial_merge_size, spans ordered by position and classified by the token after vision_start_token_id.
5. Add a VlmPreparationSummary::QwenVlm video count (or a new variant) so the CLI reports clips and frame slots the way the Gemma 4 and Kimi-VL paths already do.
6. Update cli_video_content_part_count and the chat-template content-part builder together, so the rendered prompt and the expansion agree.
7. Extend the same path to the server so CLI and server produce byte-identical prompts, and ensure usage.prompt_tokens counts the expanded visual prefix.
8. Update docs/supported-models.md.

Validation constraints

Video decoding shells out to ffmpeg (mlxcel::video::load_videos). ffmpeg is not installed on the box this investigation ran on, so no video behavior of any kind was measured here — the rejection message above is from reading the dispatch, and the whole video path is unmeasured. Whoever picks this up needs ffmpeg on PATH and should say so in the closing comment.

Suggested matrix: text-only control, single clip, clip + image in one prompt, multiple clips, a clip that trips max_frames, and B>1 concurrent requests with different expanded lengths.

Scope

• In scope: qwen3_5 (Qwen3.5-VL, Qwen3.8-27B) first, since that is the checkpoint driving this.
• Adjacent: qwen3_vl, qwen3_vl_moe, qwen2_vl, qwen2_5_vl share the same upstream vision stack and processor, so the work should generalize. Land one family with tests, then extend.
• Out of scope: audio. Qwen3.8-27B has no audio tower (confirmed against the upstream config's top-level keys and weight map); it is a vision-only VLM.

Acceptance criteria

• mlxcel generate --video clip.mp4 produces a correct description on a Qwen3.5-VL or Qwen3.8-27B checkpoint
• Server video requests produce a byte-identical prompt to the CLI for the same input
• Placeholder count, projected feature rows, and usage.prompt_tokens agree exactly; a mismatch is a pre-generation error
• MRoPE temporal positions verified against the upstream reference on a fixed clip, including the spatial_merge_size division applying to h/w and not t
• Mixed image + video prompt ordered correctly
• max_frames overflow rejected with a named error, not truncated
• cli_video_content_part_count and the expansion path covered by a test that fails if they drift apart
• docs/supported-models.md updated; the closing comment names what was and was not measured