Repository navigation
fix(models): correct Molmo v1 image decoding seams - #1099
Merged
Merged
Conversation
Default omitted Molmo v1 rope_impl to the checkpoint effective LLaMA rotate-half layout instead of the stale interleave dataclass default, so flat Molmo-7B-D-0924 configs no longer select the wrong MLX RoPE layout. Match Molmo v1 preprocessing to the reference by padding raw black before normalization and preserving fractional patch coverage for pad_and_partial_pad instead of thresholding masks to booleans. Validation: cargo fmt --all -- --check; cargo test -p mlxcel models::molmo::tests; cargo test -p mlxcel vision::processors::molmo::tests; cargo test -p mlxcel --test molmo_parity; cargo clippy -p mlxcel --lib --tests -- -D warnings. Closes #1087
inureyes
added
status:review
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
## Summary Fixes two Molmo v1 seams that corrupt image-conditioned generation: flat Molmo-7B-D-0924 configs now default omitted `rope_impl` to the checkpoint's effective LLaMA rotate-half layout, and image preprocessing now preserves reference padding and fractional mask values for `pad_and_partial_pad`. ## What changed - Default missing Molmo v1 `rope_impl` to `llama` while preserving explicit `interleave` as MLX traditional RoPE. - Pad raw black before normalization so padded pixels become normalized black values instead of zero-valued normalized-space pixels. - Preserve per-patch float coverage in `image_masks`, including partial border patches, instead of thresholding masks to booleans. - Add deterministic regression tests for the affected flat config omission, explicit RoPE controls, normalized black padding, and the known 640x480 `9/14` partial border patch. ## Validation - [x] `cargo fmt --all -- --check` - [x] `cargo test -p mlxcel models::molmo::tests` - [x] `cargo test -p mlxcel vision::processors::molmo::tests` - [x] `cargo test -p mlxcel --test molmo_parity` - [x] `cargo clippy -p mlxcel --lib --tests -- -D warnings` - [x] `cargo build --release --features cuda --bin mlxcel --bin mlxcel-server --locked` on NVIDIA GB10 (sm_121) - [x] Real-checkpoint CLI A/B with `/home/inureyes/models/mlx/molmo-7b` and the 640x480 COCO cats image: the pre-fix 749-token prompt reproduced the issue output byte-for-byte; the fixed build generated a coherent description of two cats sleeping on a red couch under a pink blanket. - [x] Real-checkpoint `mlxcel-server` validation through `POST /v1/chat/completions` with the same image and prompt on the normal BatchScheduler path (`--parallel 1`): coherent output matched the CLI, with `prompt_tokens=749`, `completion_tokens=40`, and `total_tokens=789`. The legacy `--no-batch` server mode was also attempted and failed before generation with its existing zero-capacity `CachePool` admission error. That path is independent of the Molmo decoder/preprocessor changes; the normal production scheduler path completed the required server generation successfully. Closes lablup#1087
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two Molmo v1 seams that corrupt image-conditioned generation: flat Molmo-7B-D-0924 configs now default omitted
rope_implto the checkpoint's effective LLaMA rotate-half layout, and image preprocessing now preserves reference padding and fractional mask values forpad_and_partial_pad.What changed
rope_impltollamawhile preserving explicitinterleaveas MLX traditional RoPE.image_masks, including partial border patches, instead of thresholding masks to booleans.9/14partial border patch.Validation
cargo fmt --all -- --checkcargo test -p mlxcel models::molmo::testscargo test -p mlxcel vision::processors::molmo::testscargo test -p mlxcel --test molmo_paritycargo clippy -p mlxcel --lib --tests -- -D warningscargo build --release --features cuda --bin mlxcel --bin mlxcel-server --lockedon NVIDIA GB10 (sm_121)/home/inureyes/models/mlx/molmo-7band the 640x480 COCO cats image: the pre-fix 749-token prompt reproduced the issue output byte-for-byte; the fixed build generated a coherent description of two cats sleeping on a red couch under a pink blanket.mlxcel-servervalidation throughPOST /v1/chat/completionswith the same image and prompt on the normal BatchScheduler path (--parallel 1): coherent output matched the CLI, withprompt_tokens=749,completion_tokens=40, andtotal_tokens=789.The legacy
--no-batchserver mode was also attempted and failed before generation with its existing zero-capacityCachePooladmission error. That path is independent of the Molmo decoder/preprocessor changes; the normal production scheduler path completed the required server generation successfully.Closes #1087