Repository navigation
fix(server): classify projector flags and align multimodal media handling #1451
Copy link
Copy link
Closed
Labels
area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadatamodeltype:vlmVision-language modelVision-language modelpriority:highHigh priorityHigh prioritystatus:doneCompletedCompletedtype:bugBug fixes, error corrections, or issue resolutionsBug fixes, error corrections, or issue resolutions
Description
Parent
Problem
b10621 loads a separate multimodal projector and exposes projector URL/auto/offload/device controls plus --media-path for local file:// inputs. mlxcel has direct MLX VLM checkpoints and multimodal request handling, so exact feature equivalence requires translation at the model capability and request boundary rather than accepting projector flags as no-ops.
Scope
- Classify
--mmproj,--mmproj-url,--mmproj-auto,--mmproj-offload,--mmproj-device, their aliases, and environment variables. - Translate only inert or capability-equivalent forms for integrated MLX VLM checkpoints; explicitly reject requests to load a separate GGUF projector.
- Implement compatible image/audio content-part decoding, media limits, model modality checks, and error envelopes on shared chat/responses/transcription routes.
- Align
--image-min-tokens,--image-max-tokens, and--mtmd-batch-max-tokenswith integrated MLX vision encoders where exact token-budget translation is possible. - Implement
--media-pathonly with a confined root and b10621-compatible relativefile://handling, or reject it with a supported upload alternative. - Keep remote URL policy explicit and protect local files, loopback services, metadata services, redirects, and decompression limits.
Acceptance criteria
- Every projector and media option is classified with a parser or diagnostic test.
- Text-only models reject media before generation, while supported VLM/audio models accept compatible content-part shapes.
- Local media paths cannot traverse or escape the configured root through symlinks, encoding, or URL normalization.
- Remote media handling has request-size, decode-size, timeout, redirect, and network-address protections.
- Image-token minima, maxima, and batch limits either constrain the real encoder path or fail before serving with an architecture-specific diagnostic.
- Help does not imply that a GGUF
mmprojfile can be attached to an MLX checkpoint.
Validation
- Smoke-test one real VLM with an uploaded image and one audio-capable model with a short clip; include malformed and oversized media cases.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadatamodeltype:vlmVision-language modelVision-language modelpriority:highHigh priorityHigh prioritystatus:doneCompletedCompletedtype:bugBug fixes, error corrections, or issue resolutionsBug fixes, error corrections, or issue resolutions
Activity
test(server): add pinned b10621 compatibility manifest and gates
fix(server): translate model-source flags and reject GGUF semantics (#…
fix(server): classify projector flags and confine request media
fix(server): give the image cardinality route test a model that takes…
fix(server): classify projector flags and confine request media (labl…