Skip to content

mtmd: use pillow-accurate algo, correct resize_algo for all models - #27594

Merged
ngxson merged 2 commits into
masterfrom
xsn/mtmd_pillow
Aug 23, 2026
Merged

ngxson merged 2 commits into
masterfrom
xsn/mtmd_pillow

Conversation

@ngxson

@ngxson ngxson commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Overview

  • remove all non-pillow resize variants because HF impl always uses pillow
  • track down all models's config.json and correct them one by one, see a complete table in comment

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: most code written + verified by AI. this is a mechanical task

@ngxson
ngxson requested a review from a team as a code owner August 23, 2026 10:01
@github-actions github-actions Bot added the mtmd Related to multimodal functionality (video/image/audio) label Aug 23, 2026
@ngxson

ngxson commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author
model (projector) need correct? correct algo link to config
LLaVA 1.5 (MLP) yes (was BILINEAR) BICUBIC preprocessor_config.json
LLaVA 1.6 (MLP, uhd) yes (ov was BILINEAR) BICUBIC (main/rf/ov) preprocessor_config.json
MobileVLM (LDP/LDPV2) yes (was BILINEAR) BICUBIC image_processing_clip.py (CLIP default)
CogVLM2 (COGVLM) yes (was BILINEAR) BICUBIC modeling_cogvlm.py
GLM-Edge (GLM_EDGE) yes (was BILINEAR) BICUBIC preprocessor_config.json (resample=3)
MiniCPM-V 2.x/4.x (MINICPMV) yes (ov was BILINEAR) BICUBIC (all resizes) image_processing_minicpmv.py
MiniCPM-V 4.6 (MINICPMV4_6) yes (ov was BILINEAR) BICUBIC (all resizes) same processor family as MiniCPM-V 2.6 above
InternVL (INTERNVL) yes (ov was BILINEAR) BICUBIC preprocessor_config.json (resample=3)
Nemotron Nano V2 VL no BICUBIC image_processing.py (torchvision BICUBIC)
Idefics3 / SmolVLM (IDEFICS3) yes (was default BICUBIC) LANCZOS preprocessor_config.json (resample=1)
LFM2-VL (LFM2) yes (was BILINEAR x3) BICUBIC (main/rf/ov) preprocessor_config.json (resample=3)
Phi-4-multimodal (PHI4) no BILINEAR processing_phi4mm.py (torchvision resize, default interpolation)
Pixtral (PIXTRAL) yes (was BILINEAR) BICUBIC preprocessor_config.json (resample=3)
LightOnOCR (LIGHTONOCR) no BICUBIC preprocessor_config.json (resample=3)
dots.ocr (DOTS_OCR) no BICUBIC preprocessor_config.json (Qwen2VL processor, default BICUBIC)
dots3-note (DOTS3NOTE_V) no BICUBIC preprocessor_config.json (Qwen2VL processor, default BICUBIC)
Kimi-VL (KIMIVL) yes (was BILINEAR) BICUBIC image_processing_kimi_vl.py
Kimi-K2.5 (KIMIK25) no BICUBIC media_utils.py
Gemma 3 (GEMMA3) no BILINEAR preprocessor_config.json (resample=2)
Gemma 4 (GEMMA4V/GEMMA4UV) yes (was BILINEAR) BICUBIC processor_config.json (resample=3)
Gemma 3n (GEMMA3NV) yes (was default BICUBIC) BILINEAR preprocessor_config.json (resample=2)
Qwen2-VL / Qwen2.5-VL / Qwen3-VL yes (was BILINEAR) BICUBIC image_processing_qwen2_vl.py (default BICUBIC)
MiniMax-M3 (MINIMAX_M3) no BICUBIC image_processor.py
Muse Glimmer (MUSE_GLIMMER) no LANCZOS processor_config.json (resample=1)
MiMo-VL (MIMOVL) no BICUBIC preprocessor_config.json (Qwen2VL processor, default BICUBIC)
Step3-VL (STEP3VL) no (algo hardcoded in preprocessor, hparams unused) BILINEAR (PIL stage) processing_step3.py
Youtu-VL (YOUTUVL) no BILINEAR preprocessor_config.json (resample=2)
Reka Edge (YASA2) no BICUBIC preprocessor_config.json (resample=3)
GLM-4.xV (GLM4V) yes (was BILINEAR) BICUBIC image_processing_glm4v.py (default BICUBIC)
Llama 4 (LLAMA4) yes (was default BICUBIC) BILINEAR preprocessor_config.json (resample=2)
PaddleOCR-VL (PADDLEOCR) yes (was BILINEAR) BICUBIC preprocessor_config.json (resample=3)
DeepSeek-OCR / OCR-2 no BICUBIC (PIL resize default) modeling_deepseekocr.py
HunyuanOCR (HUNYUANVL) yes (was BICUBIC) LANCZOS preprocessor_config.json (resample=1)
EXAONE 4.5 (EXAONE4_5) no BICUBIC preprocessor_config.json (resample=3)
Janus Pro (JANUS_PRO) yes (was BILINEAR) BICUBIC preprocessor_config.json (resample=3)
Granite 4 vision (GRANITE4_VISION) yes (ov was BILINEAR) BICUBIC (main/rf/ov) preprocessor_config.json (resample=3)

@ngxson

ngxson commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

Speed optimization (last commit)

Bicubic

case master legacy bicubic master pillow bicubic this PR vs legacy vs master pillow
4032x3024 -> 1024x1024 60.6 18.4 12.6 4.8x 1.5x
1920x1080 -> 784x784 33.9 5.4 3.2 10x 1.7x
800x600 -> 336x336 6.6 1.16 0.75 8.8x 1.5x
512x512 -> 1024x1024 (up) 62.0 5.0 2.2 28x 2.3x
336x336 -> 3072x3072 (up) 536 29.4 10.0 53x 2.9x

Lanczos

case master this PR speedup
4032x3024 -> 1024x1024 26.3 19.1 1.4x
1920x1080 -> 784x784 7.6 5.3 1.4x
800x600 -> 336x336 1.81 1.26 1.4x
512x512 -> 1024x1024 (up) 6.0 3.0 2.0x
336x336 -> 3072x3072 (up) 38.6 13.0 3.0x

Bilinear

Master's legacy bilinear is not a fair baseline: it had no antialiasing and used align-corners mapping,
so it did far less work and did not match Pillow. Numbers for reference:

case master legacy bilinear (not PIL-correct) this PR (PIL-exact)
4032x3024 -> 1024x1024 2.6 8.7
1920x1080 -> 784x784 1.5 3.0
800x600 -> 336x336 0.28 0.69
512x512 -> 1024x1024 (up) 2.6 1.6
336x336 -> 3072x3072 (up) 22.8 7.5

@ngxson
ngxson merged commit 56db501 into master Aug 23, 2026
24 of 26 checks passed
therealkenc pushed a commit to therealkenc/llama.cpp that referenced this pull request Aug 24, 2026
…gml-org#27594)

* mtmd: use pillow-accurate resize algo, correct resize_algo for all models

* speed optimization
thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026
…gml-org#27594)

* mtmd: use pillow-accurate resize algo, correct resize_algo for all models

* speed optimization
zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026
…gml-org#27594)

* mtmd: use pillow-accurate resize algo, correct resize_algo for all models

* speed optimization
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mtmd Related to multimodal functionality (video/image/audio)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant