fix(youtu_vl): multi-window images still described wrong after the patch-order fix #1618

Problem / Background

#1610 fixed the Youtu-VL image processor's patch emission order (raster with (c, dy, dx) features, now merge-block-major with (dy, dx, c)). That fix is confirmed correct by the single-window control fixture, which went from being described as black to being described as orange. It is not sufficient: both multi-object fixtures still get a description that has nothing to do with their contents, so at least one further defect remains in this family's vision path.

This issue records what was observed so the next person does not have to re-derive it. It does not claim a diagnosis.

Evidence

Measured on models/mlx/youtu-vl-4b-instruct (tencent/Youtu-VL-4B-Instruct), M1 Ultra, release build with metal,accelerate, greedy (-t 0 -n 48), at fix/issue-1610-youtu-vl-patch-order.

models/mlx/youtu-vl-4b-instruct has patch_size=16, spatial_merge_size=2 and window_size=256, so vit_merger_window_size is 8 merged units and smart_resize snaps every edge to a multiple of 32. The merged grid is therefore one eighth of the resized pixel edge.

For comparison, the same three runs before #1610 gave "completely black ... a solid black square or rectangle" for the orange fixture and "a single, solid black circle on a white background" for both shape fixtures.

Observations worth keeping

The one fixture that is correct is the only one whose merged grid fits inside a single attention window. Both fixtures that need more than one window are wrong, and they are wrong in different ways from each other (the 336 rendering inverts the polarity relative to the 448 rendering), which is not what a size-independent defect in the embedding or the merger would produce.

Both wrong answers collapse three objects in three colors into one achromatic object. That is the signature of tokens carrying content from the wrong spatial location rather than of content being erased: the model is confidently describing something, it is just not this image.

Candidate areas, in the order the evidence points at them, none of them verified:

• window::get_window_index and the cu_window_seqlens it returns in src/vision/encoders/youtu_vl.rs, including how partial windows at the right and bottom edges are padded and how the segment boundaries are deduplicated. 11x11 and 14x14 both leave a partial window; 7x7 does not.
• The interaction between the window reordering and fullatt_block_indexes in YoutuVLVisionEncoder::forward_with_spatial: window-ordered hidden states are fed to the full-attention layers with full_cu boundaries, so the two orderings have to agree about what a segment is.
• The window inverse itself is out of scope here. fix(youtu_vl): reverse_window_indices returns the permutation, not its inverse #1600 / fix(youtu_vl): build the window inverse as argsort #1603 fixed it to argsort(window_index) and that fix is at HEAD; it cannot explain the 224 result either way, since the permutation is the identity at that size.

Reproduce

cargo build --release --features metal,accelerate
./target/release/mlxcel generate -m models/mlx/youtu-vl-4b-instruct --image tests/fixtures/test_image_shapes.png -p "What shapes and colors are in this image? Answer briefly." -n 48 -t 0
./target/release/mlxcel generate -m models/mlx/youtu-vl-4b-instruct --image tests/fixtures/test_image_shapes_336.png -p "What shapes and colors are in this image? Answer briefly." -n 48 -t 0

tests/youtu_vl_parity.rs, added in #1610, already carries both of these as #[ignore]d content assertions written at the full correct answer and labelled as known failing against this issue. Fixing this issue is what turns them green; do not weaken them.

Acceptance Criteria

• Root cause identified and named, with the reasoning that rules out the alternatives above.
• test_image_shapes.png and test_image_shapes_336.png are both described with all three shapes and all three colors by the real checkpoint, with before and after text recorded in the PR.
• The 224x224 control still reads as orange, so the fix does not trade one size for another.
• The two known-failing tests in tests/youtu_vl_parity.rs are un-gated: their #[ignore] reason drops the "known failing" clause and keeps only the checkpoint requirement.
• A checkpoint-free unit test covers whatever the defect turns out to be, on a grid that spans more than one window.

Technical Considerations

Related: #1610 (the patch emission order, which this issue is the residual of), #1600 / #1603 (the window inverse in the same encoder), #1611 (the processor's patch-count cap), #1596 / #1601 (the Qwen2.5-VL pair of defects this family mirrors).