Skip to content

feat(xla): support Phi4MM audio and per-slot adapters - #914

Merged
inureyes merged 8 commits into
mainfrom
feature/issue-877-phi4mm-xla-audio
Jul 24, 2026
Merged

inureyes merged 8 commits into
mainfrom
feature/issue-877-phi4mm-xla-audio

Conversation

@inureyes

@inureyes inureyes commented Jul 24, 2026 •

Copy link
Copy Markdown
Member

Summary

  • emit the pinned Phi4MM cascaded Conformer and speech/vision projection as a static IREE audio.main module
  • implement complete per-row language/speech/vision LoRA selection in prefill and decode without global mutable adapter state
  • validate complete LoRA A/B pairs, rank, scale, shapes, projection lengths, placeholders, and artifact identity before native execution
  • preserve adapter mode through ragged batching, cancellation, and slot reuse
  • integrate bounded audio-only, image-only, and approved mixed image/audio preparation into the shared CLI and streaming-server XLA path
  • keep only the filtered text embedding and qualified vision producer on MLX; the full decoder and audio encoder remain IREE-owned
  • fix the production audio-only CLI path so an empty image list does not enter image decoding

Validation

  • cargo fmt --all -- --check
  • git diff --check
  • cargo check --features xla-iree --lib --tests
  • cargo clippy --features xla-iree --lib --tests --no-deps
  • Phi4MM XLA focused tests: 12 passed, 5 diagnostic tests intentionally ignored
  • root Phi4MM focused tests: 32 passed
  • mixed scheduler admission covers simultaneous audio-only, image-only, and image+audio requests with Speech/Vision/Vision per-slot modes and one-token completion
  • cancellation and slot-reuse tests prove adapter state is cleared and cannot leak
  • official revision 93f923e1a7727d1c4f446756212d9d3e8fcc5d81: IREE projection produced 44 rows and passed the feat(vlm): add a token-exact Phi4MM audio reference path #874 eight-probe gate at max abs 0.01486496 < 0.025
  • actual production XLA CLI with the official WAV generated token IDs [13347, 382] (what is), exactly matching the pinned oracle
  • artifact-hardening tests: 9 passed, 4 environment-only diagnostics ignored; the cache identity includes the compiler executable's raw SHA-256 and decoded audio weights reject NaN/Inf before native upload

Notes

Closes #877

@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:low Low priority area:models Model architectures, weights, loading, metadata area:architecture Architecture and code structure changes area:inference Generation, sampling, decoding (incl. speculative, DRY) labels Jul 24, 2026
@inureyes
inureyes merged commit 775bcee into main Jul 24, 2026
5 checks passed
@inureyes inureyes added status:done Completed and removed status:review Under review labels Jul 24, 2026
@inureyes
inureyes deleted the feature/issue-877-phi4mm-xla-audio branch July 24, 2026 08:19
@inureyes inureyes self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata priority:low Low priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(xla): support Phi4MM audio and per-slot speech adapters

1 participant