Skip to content

feat: add token-exact Phi4MM audio reference path (#874) - #887

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-874-phi4mm-audio
Jul 23, 2026
Merged

inureyes merged 1 commit into
mainfrom
feature/issue-874-phi4mm-audio

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

  • Port the pinned Phi-4 Multimodal SpeechLib feature extractor, Cascades Conformer encoder, speech/vision audio projections, and complete modality LoRA weights with fail-closed shape and completeness checks.
  • Normalize and expand numbered audio/image placeholders, replace the exact media embedding rows, and keep language, speech, vision, and mixed adapter modes request-scoped through decode.
  • Wire audio-only and mixed media through the CLI and OpenAI-compatible streaming server, including multiple server audio clips, bounded streaming-WAV sentinel handling, and the correct dense external KV layout for single-slot Phi4MM serving.
  • Add pinned revision documentation, deterministic unit fixtures, and a gated real-checkpoint CUDA parity test for features, encoder/projection values, merged prefill, KV state, and greedy tokens.

Validation

  • cargo fmt --all -- --check
  • cargo check --all-targets
  • cargo clippy -p mlxcel-core --lib --tests -- -D warnings
  • cargo clippy -p mlxcel --lib --tests -- -D warnings
  • Focused LoRA, streaming WAV, scheduler gate, Phi3, and Phi4MM tests: 48 passed
  • Release CUDA parity against microsoft/Phi-4-multimodal-instruct revision 93f923e1a7727d1c4f446756212d9d3e8fcc5d81: feature/intermediate/KV checks passed and greedy tokens matched [13347, 382] (what is)
  • CUDA CLI smoke: audio-only what is, image-only A solid, mixed image/audio A solid; prompt lengths matched 53/553/599
  • OpenAI-compatible streaming server smoke: default warmup passed, audio streamed what is, and mixed media streamed A solid

Closes #874

Port the pinned Phi-4 Multimodal audio stack so the MLX runtime provides a trustworthy reference for the Cascades Conformer, audio projection, placeholder replacement, and modality adapters.

Implement SpeechLib-compatible feature extraction, strict audio architecture and weight validation, named speech and vision LoRA selection, request-scoped modality state, and exact image/audio embedding replacement. Wire the path through both CLI generation and OpenAI-compatible streaming serving, including multiple server audio clips and mixed media.

Handle streaming WAV length sentinels, keep Phi4MM on dense external KV state while preserving single-slot decode, and add pinned real-checkpoint parity fixtures covering features, encoder/projection values, KV state, and greedy tokens.

Validation includes formatting, all-target compilation, warnings-as-errors Clippy for core and mlxcel library/test targets, focused Phi3/Phi4MM regressions, CUDA CLI and server audio/mixed smoke tests, and release CUDA parity against revision 93f923e1a7727d1c4f446756212d9d3e8fcc5d81.

Refs #874
@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:medium Medium priority area:models Model architectures, weights, loading, metadata area:inference Generation, sampling, decoding (incl. speculative, DRY) labels Jul 23, 2026
@inureyes
inureyes merged commit 64c5095 into main Jul 23, 2026
5 checks passed
@inureyes inureyes added status:done Completed and removed status:review Under review labels Jul 23, 2026
@inureyes
inureyes deleted the feature/issue-874-phi4mm-audio branch July 23, 2026 03:39
@inureyes inureyes self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata priority:medium Medium priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(vlm): add a token-exact Phi4MM audio reference path

1 participant