fix(perf): fuse QKV and add text-only fast path for Qwen3-VL decode by inureyes · Pull Request #29 · lablup/mlxcel

inureyes · 2026-05-18T17:45:56Z

Summary

Two related optimizations to qwen3_vl and qwen3_vl_moe:

QKV fusion. Replace the three separate q_proj / k_proj / v_proj matmuls per layer with a single FusedQKVLinear (qkv_proj). The shared layer is already used by Llama3 / Qwen2/3 / Gemma / Mistral / Cohere2 / StarCoder2 / InternLM3; docstring updated to list Qwen3-VL and Qwen3-VL-MoE.
Text-only fast path. New forward_text_only route on Attention, DecoderLayer, and the top-level model. Taken when input_embeddings is None and the sequence has no live MRoPE state, visual position mask, or DeepStack embeddings — i.e. a normal text prompt with no image. Uses plain fast_rope keyed on the cache offset instead of computing 3D MRoPE position ids, and skips visual-state propagation entirely. MRoPE / DeepStack state is explicitly cleared at the start of any fresh sequence (cache_offset == 0, no input embeddings) so the gate is sound across speculative-decoding turns.

Performance (M5 Max, 4-bit, decode tok/s)

Model	Before	After	Speedup	mlx-lm parity
Qwen3-VL-30B-A3B-Instruct-4bit	56.00	146.29	2.61x	38% → 99%
Qwen3-VL-32B-Instruct-4bit	18.79	27.33	1.45x	66% → 94%
Qwen3-VL-2B-Instruct-4bit	—	unchanged	—	—

Changes

src/models/qwen3_vl.rs, src/models/qwen3_vl_moe.rs — switch Attention to FusedQKVLinear; add forward_text_only on Attention/DecoderLayer/Model; auto-clear MRoPE + DeepStack state at sequence start.
src/lib/mlxcel-core/src/layers.rs — extend FusedQKVLinear "Used by" docstring; minor use-ordering fix.

Test plan

cargo check --features metal,accelerate
Manual (M5 Max): Qwen3-VL-30B-A3B-Instruct-4bit text-only decode reproduces ~146 tok/s.
Manual (M5 Max): Qwen3-VL-32B-Instruct-4bit text-only decode reproduces ~27 tok/s.
Manual: image-in-prompt path on Qwen3-VL-2B-Instruct-4bit (or any Qwen3-VL with an actual image) still produces correct output — fast path must NOT engage when visual state is present.

Qwen3-VL and Qwen3-VL-MoE text-only decode went through the full VLM forward path: three separate `q_proj`/`k_proj`/`v_proj` matmuls per layer, MRoPE delta computation against per-sequence position state, visual position masks, and DeepStack visual-embedding fan-in — all of it dead weight when there is no image in the prompt. This change introduces two optimizations for both `qwen3_vl` and `qwen3_vl_moe`: 1. Fuse Q/K/V into a single `FusedQKVLinear` (`qkv_proj`). One concatenated matmul per layer replaces three, which is the dominant decode-time saving on Apple Silicon where the per-call MLX overhead matters more than raw FLOPs. The shared `FusedQKVLinear` already serves Llama3/Qwen2/Qwen3/Gemma/Mistral/Cohere2/StarCoder2/InternLM3; the docstring is updated to include the Qwen3-VL variants. 2. Add a `forward_text_only` fast path on `Attention`, `DecoderLayer`, and the top-level model. It is taken when `input_embeddings` is `None` and the model has no live MRoPE state, visual position mask, or DeepStack embeddings for the sequence. The fast path uses plain `fast_rope` keyed on the cache offset instead of computing 3D MRoPE position ids, and skips the visual-state propagation entirely. To keep the gate sound across speculative-decoding sequences, MRoPE and DeepStack state are explicitly cleared at the start of any fresh (`cache_offset == 0`, no input embeddings) sequence. Measured on M5 Max (decode tok/s, 4-bit): - Qwen3-VL-30B-A3B-Instruct-4bit: 56.00 -> 146.29 (2.61x; 38% -> 99% of mlx-lm parity) - Qwen3-VL-32B-Instruct-4bit: 18.79 -> 27.33 (1.45x; 66% -> 94% of mlx-lm parity) - Qwen3-VL-2B-Instruct-4bit: unchanged (already on the fast path the small model never left) Image-in-prompt and DeepStack flows are unchanged: the original `forward` path is still taken whenever any visual state is present, so VLM correctness on multimodal inputs is preserved.

This release focuses on Apple Silicon decode performance for VLM and linear-attention model families, plus a CLI fix for `mlxcel list`. Performance: - Gemma3n bf16 prefill path: gemma3n-e4b-bf16 decode 11.16 to 38.81 tok/s on M5 Max (#28). - Qwen3-VL text-only decode: QKV fusion + `forward_text_only` fast path. Qwen3-VL-30B-A3B-Instruct-4bit decode 56.00 to 146.29 tok/s (2.61x, 99% of mlx-lm parity); Qwen3-VL-32B-Instruct-4bit decode 18.79 to 27.33 tok/s (1.45x, 94% parity); image-in-prompt and DeepStack paths unchanged (#29). - GatedDeltaNet fast RMSNorm in Qwen3.5 / Qwen3-Next / KimiLinear: `mlx::fast::rms_norm` replaces the expanded square / mean / sqrt / divide graph for Q/K and gated output. qwen3.5-0.8b-4bit decode 425.16 to 535.43 tok/s on M5 Max (+25.9%, 96% parity); Qwen3.5 speculative-decoding verify pass updated alongside prefill / decode to preserve draft / verify dtype agreement (#30). Fixed: - Rewrite `mlxcel list` output to drive from the `ModelType` registry: fixes stale count, missing VLM family, missing ~30 model types, and removes a broken docs link (#27). Docs: - Refresh README performance snapshot and `docs/benchmarks.md` to match the latest M5 Max sweep.

…xisting on base, stride-gate + supervisor approved

…truction] msa_fetch/fp16_gathered (Clement seat closed: approve, no must-fixes; Xander's seat open at his wake) Env instruments demote to default seeds; store is single source of truth at every dispatch site. Runtime msa_core makes the G core axis boot-free for the morning A/B leg (task lablup#28); construction keys are boot-frozen with warn-on-divergence. Header echo: path=..; core=..; v=N. Both gathered cores carry first-dispatch witnesses. Latch-order property (review note a): init_and_watch's boot-time file load precedes first dispatch on the server path; TOML-less processes latch env-seeded defaults at first touch by design. Verification (sharded per lablup#29 convention, isolated target): decode_config 26, server::routes 77, m3 50, core cache 480, bins build. Full-suite SIGTRAP reproduced on BASE without this change — pre-existing, lablup#29 updated with the datum.

…ge (draft for Xander's seat) Drives the REAL engine write path (update_kvarn8, fresh v_bits=4 cache per harvest event) on the SAME real-tile population the boot-night screen licensed (RESULTS_kvarn4_realtile_2026-07-11), and pins every stored field — K side (5) and V side (4) plus 4 structure pins — BITWISE against the dual-approved math layer (kvarn_quantize@8 / kvarn_quantize_v4@gs32) on the identical event batch. Structural first act: assert_round_half_even_parity() is the gating test's first statement (Violet PM pin 1, Xander confirmation). Scope (coverage mapping, so the green is never over-read): - gates the STORAGE roundtrip ONLY; gather_qmm fused consumption stays §4.2 tolerance-gated - live-session bit-equality out of scope BY CONSTRUCTION: dumps sampled ≤8 spread tiles/event AND sinkhorn best-so-far is batch-global — the contract is engine-vs-reference on identical per-event batches - ALL comparisons bitwise; no dequant-level tolerance door opened (cancellation-pricing birth constraint honored by construction) Population pinned, not assumed: replicates the screen's load_pairs() exactly (sorted K sidecars, np.linspace truncation semantics — pinned by unit test — seq+1 V pairing on matching cache+offset); asserts exactly 512 events / 4096 tiles; loader fail-louds on role/shape/size/non-finite. Self-pins (in-suite, no bank needed) prove the harness can go red: - comparator catches a single flipped byte (and stays quiet on identical) - comparator refuses length mismatch (no zip-shortest false green) - linspace selection matches exact integer floor for the bank geometry - synthetic-event roundtrip: harness helpers reproduce the engine bitwise end-to-end (10222 wiring idiom re-proven THROUGH the helpers) Evidence: cache:: scope 483/483 green (479 prior + 4 self-pins), gating test ignored pending review. Full-suite SIGTRAP is the known upstream MLX clear_streams teardown double-free (lablup#29, stash-proven not ours). SEQUENCING: the ignored gating run does NOT execute until Xander's review clears it (standing commitment: his seat before it gates anything). Invocation, post-review: MLXCEL_KVARN_GOLDEN_DIR=$HOME/kvarn_harvest_20260710 \ cargo test -p mlxcel-core --release k8v4_golden_harness_real_tiles \ -- --ignored --nocapture Co-Authored-By: clement-7074f29f <clement-7074f29f@sjstargetedsolutions.co.nz>

The gate's record (2026-07-11, Violet's verdict 20:06): completions BIT-IDENTICAL to blocked across kernel schedules at 295K (1,500 greedy tokens, diff empty), sdpa +3.4–8.1% tok/s. Safe everywhere by construction — the masked SDPA core consumes fp16 compact windows regardless of source (fp16-gathered or kvarn-dequanted). Env instrument keeps switch semantics against the NEW default: MLXCEL_MSA_CORE=blocked selects the non-default; anything else (including parse garbage) means sdpa. The two default-pinning tests flipped WITH the default, deliberately, citing the gate — exactly the red a default flip owes. Evidence: decode_config 26/26, msa 29/29, C contracts 2/2 (their gathered reference now runs the sdpa core live — still 1e-3-matched, consistent with the gate's bit-identical finding), root lib 1,876 passed / 0 failed before the known lablup#29 teardown SIGTRAP. Co-Authored-By: clement-7074f29f <clement-7074f29f@sjstargetedsolutions.co.nz>

… wider / store-admission narrower), Violet server-premise refuted, lablup#29 mechanism found Co-Authored-By: violet-14057653 <violet@sjstargetedsolutions.co.nz> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ablup#29) * fix(perf): fuse QKV and add text-only fast path for Qwen3-VL decode Qwen3-VL and Qwen3-VL-MoE text-only decode went through the full VLM forward path: three separate `q_proj`/`k_proj`/`v_proj` matmuls per layer, MRoPE delta computation against per-sequence position state, visual position masks, and DeepStack visual-embedding fan-in — all of it dead weight when there is no image in the prompt. This change introduces two optimizations for both `qwen3_vl` and `qwen3_vl_moe`: 1. Fuse Q/K/V into a single `FusedQKVLinear` (`qkv_proj`). One concatenated matmul per layer replaces three, which is the dominant decode-time saving on Apple Silicon where the per-call MLX overhead matters more than raw FLOPs. The shared `FusedQKVLinear` already serves Llama3/Qwen2/Qwen3/Gemma/Mistral/Cohere2/StarCoder2/InternLM3; the docstring is updated to include the Qwen3-VL variants. 2. Add a `forward_text_only` fast path on `Attention`, `DecoderLayer`, and the top-level model. It is taken when `input_embeddings` is `None` and the model has no live MRoPE state, visual position mask, or DeepStack embeddings for the sequence. The fast path uses plain `fast_rope` keyed on the cache offset instead of computing 3D MRoPE position ids, and skips the visual-state propagation entirely. To keep the gate sound across speculative-decoding sequences, MRoPE and DeepStack state are explicitly cleared at the start of any fresh (`cache_offset == 0`, no input embeddings) sequence. Measured on M5 Max (decode tok/s, 4-bit): - Qwen3-VL-30B-A3B-Instruct-4bit: 56.00 -> 146.29 (2.61x; 38% -> 99% of mlx-lm parity) - Qwen3-VL-32B-Instruct-4bit: 18.79 -> 27.33 (1.45x; 66% -> 94% of mlx-lm parity) - Qwen3-VL-2B-Instruct-4bit: unchanged (already on the fast path the small model never left) Image-in-prompt and DeepStack flows are unchanged: the original `forward` path is still taken whenever any visual state is present, so VLM correctness on multimodal inputs is preserved. * style: apply cargo fmt to layers.rs use ordering

This release focuses on Apple Silicon decode performance for VLM and linear-attention model families, plus a CLI fix for `mlxcel list`. Performance: - Gemma3n bf16 prefill path: gemma3n-e4b-bf16 decode 11.16 to 38.81 tok/s on M5 Max (lablup#28). - Qwen3-VL text-only decode: QKV fusion + `forward_text_only` fast path. Qwen3-VL-30B-A3B-Instruct-4bit decode 56.00 to 146.29 tok/s (2.61x, 99% of mlx-lm parity); Qwen3-VL-32B-Instruct-4bit decode 18.79 to 27.33 tok/s (1.45x, 94% parity); image-in-prompt and DeepStack paths unchanged (lablup#29). - GatedDeltaNet fast RMSNorm in Qwen3.5 / Qwen3-Next / KimiLinear: `mlx::fast::rms_norm` replaces the expanded square / mean / sqrt / divide graph for Q/K and gated output. qwen3.5-0.8b-4bit decode 425.16 to 535.43 tok/s on M5 Max (+25.9%, 96% parity); Qwen3.5 speculative-decoding verify pass updated alongside prefill / decode to preserve draft / verify dtype agreement (lablup#30). Fixed: - Rewrite `mlxcel list` output to drive from the `ModelType` registry: fixes stale count, missing VLM family, missing ~30 model types, and removes a broken docs link (lablup#27). Docs: - Refresh README performance snapshot and `docs/benchmarks.md` to match the latest M5 Max sweep.

inureyes added status:review Under review type:bug Bug fixes, error corrections, or issue resolutions type:performance Performance improvements priority:high High priority area:models Model architectures, weights, loading, metadata labels May 18, 2026

inureyes self-assigned this May 18, 2026

style: apply cargo fmt to layers.rs use ordering

b00e4c6

inureyes merged commit c7b28e4 into main May 18, 2026
4 checks passed

inureyes deleted the fix/qwen3-vl-text-only-decode branch May 18, 2026 17:56

inureyes added this to the 0.1 milestone Jun 21, 2026

sjswerdloff added a commit to sjswerdloff/mlxcel that referenced this pull request Jul 10, 2026

docs: PM board — migration pushed (in review), lablup#29 pinned pre-e…

a389d0e

…xisting on base, stride-gate + supervisor approved

inureyes added status:done Completed and removed status:review Under review type:performance Performance improvements labels Sep 6, 2026

Navigation Menu

Repository navigation

fix(perf): fuse QKV and add text-only fast path for Qwen3-VL decode - #29

fix(perf): fuse QKV and add text-only fast path for Qwen3-VL decode#29
inureyes merged 2 commits into
mainfrom
fix/qwen3-vl-text-only-decode

inureyes commented May 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

inureyes commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Performance (M5 Max, 4-bit, decode tok/s)

Changes

Test plan

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

inureyes commented May 18, 2026 •

edited

Loading