design: scope the multimodal / VLM track (Qwen2-VL/2.5-VL/3-VL, Gemma3n, Phi4MM, Molmo, and more) #503

Part of #493

Context

Multimodal / VLM models add vision or audio encoders and projectors that feed embeddings into the language model. The OpenXLA path is text-only today: the serve worker rejects multimodal input. Target set: Qwen2-VL / Qwen2.5-VL / Qwen3-VL, Gemma3n, Phi4MM, Molmo / Molmo2, Youtu-VL.

Scope

• Write a design doc covering encoder execution (running the vision/audio tower on the XLA/IREE path or alongside it), projector handling, embedding injection into the LM token stream, and the serve-path changes needed to accept multimodal requests.
• Then spin out a dedicated follow-up epic for implementation.

Acceptance criteria

• Design doc merged, OR an explicit, recorded decision to defer with the follow-up epic created. Design doc added at spike/openxla/MULTIMODAL_VLM_DESIGN.md; recorded decision to DEFER; follow-up epic created: epic: OpenXLA multimodal / VLM architecture support (Qwen2-VL, Gemma3n, Phi4MM, Molmo, and more) #566.

Resolution

Design doc merged with a recorded decision to defer implementation. spike/openxla/MULTIMODAL_VLM_DESIGN.md covers all four deliverables the scope asks for, grounded in the current codebase: encoder execution (host-encode-first reusing the MLX vision stack, then an on-XLA ViT/SigLIP graph), projector handling, embedding injection into the LM token stream (the load-bearing emitter change: a prefill-from-embeddings graph entry, since the text graph only accepts token ids and gathers params['embed'] internally today), and the serve-path change to accept multimodal requests instead of rejecting them at admit. Implementation is a foundation-sized effort spanning the emitter, the IREE shim, the engine, and the serve worker, so it is scoped into the follow-up epic #566 rather than landed here.

Dependencies

None hard. Builds on the text-only LM path already in place. Spun out into follow-up epic #566.