Repository navigation
feat(vision): add Inkling HMLP image support - #1535
Merged
Merged
Conversation
inureyes
added
status:review
inureyes
added a commit
that referenced
this pull request
Aug 31, 2026
## Summary - add a native B=1 Inkling chained-MTP drafter that reuses the target decoder-layer implementation, target embedding/final norm/LM head, and exact KV plus four-convolution cache snapshots - capture pre-norm target hidden states, restore the pre-verify target snapshot, and replay the accepted prefix so recurrent convolution and attention state match non-speculative decoding exactly - detect actual `model.mtp.layers.*` tensors, load only indexed matching or unindexed auxiliary safetensors, register both standalone Inkling and public-checkpoint `InklingVLM` targets across offline, legacy burst, and tick-cooperative server lifecycles, and derive the default verify width as `num_nextn_predict_layers + 2` - route text-only speculative requests on `InklingVLM` through `vlm.text` while retaining the classic HMLP prepared-embedding prefill path for image requests - add safe repeatable downloader `--include` globs, clear standalone-head diagnostics, user documentation, deterministic tiny-model parity/rollback and prepared-image-prefill coverage, and source-scanned dispatch coverage - track English and Korean technical reports under `TECHNICAL_REPORTS/` ## Validation - `cargo check --lib` - `cargo test -p mlxcel-core inkling_mtp --lib` (7 passed) - `cargo test --lib inkling -- --test-threads=1` after reconciling PR #1535 and the InklingVLM hardening (48 passed) - `cargo test --lib every_mtp_dispatch_site_covers_every_capable_variant` - `cargo test --lib burst_declined_for_vlm_embeddings` - `cargo test --lib resolve_draft_block_size_derives_inkling_default_from_the_mtp_layer_count` - `cargo test --lib isolated_inkling_mtp_download_is_not_a_standalone_model` - `cargo test --lib include_globs_select_only_safe_allow_list_files` - `cargo clippy -p mlxcel-core --lib -- -D warnings` - `cargo clippy --lib -- -D warnings` - `cargo clippy --lib --tests -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` The orchestrated focused-validation lane did not run the broad workspace Metal/Accelerate test or all-target clippy commands. The public 153.5 GB Inkling-Small target, 4.46 GB MTP shard, and the 0.6B checkpoint were not present, so real-checkpoint greedy parity, acceptance length, and throughput remain unmeasured; deterministic tiny target and drafter tests cover the same snapshot/replay and chained-forward contracts without making performance claims. ## Review findings Final correctness and security review fixed three silent integration gaps before submission: the public repository keeps `mtp.safetensors` outside its target-only index; the tick-cooperative scheduler repeats the adapter dispatch separately from the legacy burst; and public checkpoints with vision weights load as `LoadedModel::InklingVLM`, which the initial text-only variant gates omitted. The loader now inspects unindexed auxiliary safetensors without opening indexed target shards, every B=1 lifecycle covers both Inkling variants, the wrapper adapter delegates speculative text decode to `vlm.text`, and the multimodal gate preserves classic HMLP prepared-embedding prefill. User glob filters remain subordinate to the existing safe path/type allow-list, index-selected shard names retain path-traversal validation, and the avoidable raw-pointer concatenate was removed. Closes #1315
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Review findings
Validation
cargo test --lib inkling --profile test-fast --features metal,accelerate -- --test-threads=1(40 passed)cargo check --lib --features metal,acceleratecargo clippy -p mlxcel --lib --tests -- -D warningscargo fmt --all -- --checkgit diff --checkThe public 153.5 GB Inkling-Small affine MLX checkpoint and the roughly 170.7 GB native NVFP4 checkpoint were unavailable on the validation host, so real image-answer quality and checkpoint parity were not claimed. The implementation instead uses deterministic reference tests for HMLP planning, fold order, preprocessing, feature shape, prompt expansion, loading, and detection.
Closes #1327