Repository navigation
fix(vision): make Gemma 3n VLM per_layer_inputs sequence-aware - #86
Merged
Merged
Conversation
Gemma 3n VLM stashed its projected per_layer_inputs tensor in a single RefCell<Option<...>> cell that get_input_embeddings wrote and forward_with_embeddings drained via take().unwrap(). The batch scheduler runs embedding-prep at enqueue time but prefill asynchronously, so a burst of concurrent Gemma 3n VLM requests could overwrite the cell before the first prefill consumed it — the first prefill then saw the wrong row's tensor, or panicked on None when the timing flipped. Replace the cell with Gemma3nPerLayerInputsState, a per-SequenceId map plus a fallback slot, mirroring the existing Gemma4PerLayerInputsState container. Override forward_with_embeddings_and_sequence_id on Gemma3nVLModel so prefill resolves the tensor from the seq-keyed map first and only falls back to the legacy slot for CLI / bench / single-row callers; route the legacy forward_with_embeddings through it with seq_id=None. Wire release_sequence_state_by_id to drain the map alongside the scheduler's per-sequence cache cleanup, add the matching LoadedModel capability methods, and hook them into drain_incoming_requests and the preemption round trip next to the Gemma 4 calls. Unit tests cover the burst-enqueue race, the fallback round trip, release semantics, and the preemption round trip; they are marked #[ignore] for serial MLX execution like the Gemma 4 helper tests.
inureyes
added a commit
that referenced
this pull request
May 26, 2026
Bump version to 0.0.31. Highlights: MiniCPM-V 4.6 and RT-DETRv2 model support plus an Anthropic-style /v1/messages API endpoint (#82, #80, #74); fixes for missing or null chat-message content, Gemma 3n VLM per_layer_inputs, Qwen3.5 MTP verify attention, and batched quantized KV cache mask offsets (#91, #86, #78, #76); and a pinned 1.93.1 Rust toolchain (#87, #90).
inureyes
added a commit
that referenced
this pull request
May 26, 2026
The v0.0.31 changelog and debian entries described the Gemma 3n VLM per_layer_inputs fix as keeping per-layer input shapes correct, which undersells it: the real bug was a batch-scheduler concurrency race where a burst of concurrent VLM requests overwrote a single shared cell before prefill consumed it, reading the wrong sequence's tensor. Reword both entries to match the GitHub release note. The v0.0.31 tag is unchanged, so this lands on main for the next release.
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
…p#86) Gemma 3n VLM stashed its projected per_layer_inputs tensor in a single RefCell<Option<...>> cell that get_input_embeddings wrote and forward_with_embeddings drained via take().unwrap(). The batch scheduler runs embedding-prep at enqueue time but prefill asynchronously, so a burst of concurrent Gemma 3n VLM requests could overwrite the cell before the first prefill consumed it — the first prefill then saw the wrong row's tensor, or panicked on None when the timing flipped. Replace the cell with Gemma3nPerLayerInputsState, a per-SequenceId map plus a fallback slot, mirroring the existing Gemma4PerLayerInputsState container. Override forward_with_embeddings_and_sequence_id on Gemma3nVLModel so prefill resolves the tensor from the seq-keyed map first and only falls back to the legacy slot for CLI / bench / single-row callers; route the legacy forward_with_embeddings through it with seq_id=None. Wire release_sequence_state_by_id to drain the map alongside the scheduler's per-sequence cache cleanup, add the matching LoadedModel capability methods, and hook them into drain_incoming_requests and the preemption round trip next to the Gemma 4 calls. Unit tests cover the burst-enqueue race, the fallback round trip, release semantics, and the preemption round trip; they are marked #[ignore] for serial MLX execution like the Gemma 4 helper tests.
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
Bump version to 0.0.31. Highlights: MiniCPM-V 4.6 and RT-DETRv2 model support plus an Anthropic-style /v1/messages API endpoint (lablup#82, lablup#80, lablup#74); fixes for missing or null chat-message content, Gemma 3n VLM per_layer_inputs, Qwen3.5 MTP verify attention, and batched quantized KV cache mask offsets (lablup#91, lablup#86, lablup#78, lablup#76); and a pinned 1.93.1 Rust toolchain (lablup#87, lablup#90).
be-free-snowboy
pushed a commit
to freestylejs/snowbrain-mlxcel
that referenced
this pull request
Oct 27, 2026
…y fix The v0.0.31 changelog and debian entries described the Gemma 3n VLM per_layer_inputs fix as keeping per-layer input shapes correct, which undersells it: the real bug was a batch-scheduler concurrency race where a burst of concurrent VLM requests overwrote a single shared cell before prefill consumed it, reading the wrong sequence's tensor. Reword both entries to match the GitHub release note. The v0.0.31 tag is unchanged, so this lands on main for the next release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gemma 3n VLM stashed its projected
per_layer_inputstensor in a singleRefCell<Option<…>>cell thatget_input_embeddingswrote andforward_with_embeddingsdrained viatake().unwrap(). The batch scheduler runs embedding-prep at enqueue time but prefill asynchronously, so a burst of concurrent Gemma 3n VLM requests could overwrite the cell before the first prefill consumed it — the first prefill then saw the wrong row's tensor, or panicked onNonewhen the timing flipped. This affects only the batchedmlxcel-serverpath; single-row CLI / bench paths are unaffected.Fix
Replace the cell with
Gemma3nPerLayerInputsState(a per-SequenceIdmap plus a fallback slot), mirroring the existingGemma4PerLayerInputsState:forward_with_embeddings_and_sequence_idonGemma3nVLModelso prefill resolves from the seq-keyed map first and falls back to the legacy slot for CLI / bench / single-row callers; the legacyforward_with_embeddingsroutes through it withseq_id = None.release_sequence_state_by_idto drain the map alongside the scheduler's per-sequence cache cleanup.LoadedModelcapability methods (bind/take/install_gemma3n_per_layer_inputs_*) and hook them intodrain_incoming_requestsand the preemption round trip, next to the Gemma 4 calls.Testing
make verify-clippy(--all-targets --features metal,accelerate -- -D warnings) — cleanmake verify-fmt— cleanGemma3nPerLayerInputsStateunit tests (burst-enqueue race, fallback round trip, release semantics, preemption round trip) — 8/8 pass (--ignored, serial MLX)Closes #85