Repository navigation
feat: add token-exact Phi4MM audio reference path (#874) - #887
Merged
Merged
Conversation
Port the pinned Phi-4 Multimodal audio stack so the MLX runtime provides a trustworthy reference for the Cascades Conformer, audio projection, placeholder replacement, and modality adapters. Implement SpeechLib-compatible feature extraction, strict audio architecture and weight validation, named speech and vision LoRA selection, request-scoped modality state, and exact image/audio embedding replacement. Wire the path through both CLI generation and OpenAI-compatible streaming serving, including multiple server audio clips and mixed media. Handle streaming WAV length sentinels, keep Phi4MM on dense external KV state while preserving single-slot decode, and add pinned real-checkpoint parity fixtures covering features, encoder/projection values, KV state, and greedy tokens. Validation includes formatting, all-target compilation, warnings-as-errors Clippy for core and mlxcel library/test targets, focused Phi3/Phi4MM regressions, CUDA CLI and server audio/mixed smoke tests, and release CUDA parity against revision 93f923e1a7727d1c4f446756212d9d3e8fcc5d81. Refs #874
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
cargo fmt --all -- --checkcargo check --all-targetscargo clippy -p mlxcel-core --lib --tests -- -D warningscargo clippy -p mlxcel --lib --tests -- -D warningsmicrosoft/Phi-4-multimodal-instructrevision93f923e1a7727d1c4f446756212d9d3e8fcc5d81: feature/intermediate/KV checks passed and greedy tokens matched[13347, 382](what is)what is, image-onlyA solid, mixed image/audioA solid; prompt lengths matched 53/553/599what is, and mixed media streamedA solidCloses #874