Skip to content

Keep at least one pixel per axis when resizing an image - #289

Merged
juliendenize merged 2 commits into
mistralai:mainfrom
arthi-arumugam-git:fix/extreme-aspect-ratio-image-crash
Aug 25, 2026
Merged

juliendenize merged 2 commits into
mistralai:mainfrom
arthi-arumugam-git:fix/extreme-aspect-ratio-image-crash

Conversation

@arthi-arumugam-git

Copy link
Copy Markdown
Contributor

The bug

An image with a very extreme aspect ratio crashes the encoder on valid input. At the production config (image_patch_size=16, max_image_size=1024), a 4 by 10000 pixel image raises a bare AssertionError:

cfg = ImageConfig(image_patch_size=16, max_image_size=1024, spatial_merge_size=1)
enc = ImageEncoder(cfg, SpecialImageIDs(img=10, img_break=11, img_end=12))
enc(ImageChunk(image=Image.new("RGB", (4, 10000))))   # AssertionError

Same for 1 by 4096 and 3 by 8000. Nothing about those images is invalid: they are just tall.

Why

In _image_to_num_tokens, when the image is larger than max_image_size the two sides are divided by the ratio and rounded. For a ratio above twice the shorter side, round() takes that side to 0, so the token count for that axis is (0 - 1) // patch + 1, which is 0, and __call__ then trips its own assert w > 0.

The change

Both axes clamp to a minimum of one pixel after the resize, so an extremely thin image encodes as a single row or column of tokens instead of crashing. Nothing else changes: any image whose shorter side survives the resize is unaffected, and the existing expectations in test_image_to_num_tokens still hold.

Tests

test_image_to_num_tokens_extreme_aspect_ratio covers 1x512, 4x10000, 10000x4 and 2x4096 at both spatial_merge_size values, asserting both that the axis counts are at least 1 and that the encoder produces the expected token count. It fails on main at both parameter values and passes here. The rest of tests/test_image.py is green (22 passed).

An image whose aspect ratio is more extreme than max_image_size to 1
crashed the encoder on valid input: the resize rounded the shorter side
to 0 pixels, the token count for that axis came out 0, and the encoder
hit its own assert. A 4 by 10000 image raises AssertionError at the
production config today.

Both axes now clamp to one pixel, so such an image encodes as a single
row or column of tokens.

@juliendenize juliendenize left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution !

Looks good but could you make parametrized tests and clean up a bit the comments before i approve ?

Comment thread tests/test_image.py Outdated
Comment on lines +46 to +48
# For a very wide or very tall image the resize used to round the shorter side
# down to 0 pixels, which made the token count for that axis 0 and crashed the
# encoder on valid input. The shorter side must clamp to at least one token.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be removed imo :)

Suggested change
# For a very wide or very tall image the resize used to round the shorter side
# down to 0 pixels, which made the token count for that axis 0 and crashed the
# encoder on valid input. The shorter side must clamp to at least one token.

Comment thread tests/test_image.py Outdated
)
image_encoder = ImageEncoder(image_config, special_token_ids)

for size in [(1, 512), (4, 10000), (10000, 4), (2, 4096)]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you use parametrized instead ?

@arthi-arumugam-git

Copy link
Copy Markdown
Contributor Author

Both done, thanks.

The four sizes are parametrized now, stacked with the existing spatial_merge_size one, so it runs as 8 cases rather than one test with a loop inside it. That also means a failing size shows up in the test name instead of only in an assert message, so I dropped that message too.

Comment block in the test is gone, and I cut the one in image.py down to a single line.

All 8 cases still fail on main and pass here, and tests/test_image.py is 28 passed.

@juliendenize juliendenize left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks for iterating :)

@juliendenize
juliendenize merged commit 3430cbf into mistralai:main Aug 25, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants