perf(cuda): audit L=1 depthwise conv1d dispatch across SSM/hybrid decode paths and apply the #751 elementwise step where confirmed #752

Summary

Issue #748 / PR #751 found that MLX 0.32.1's CUDA backend dispatches a single-output-position (L=1) bf16 depthwise conv1d to cuDNN's generic per-channel engine (convolve_common_engine_float_NHWC, one kernel launch per channel). On lfm2-350m-8bit that was ~1024 launches per conv op, 88.6% of decode GPU time, and a 10x decode regression; the fix replaced the L=1 step with a broadcast multiply plus axis sum against a decode weight precomputed at load (39.84 -> 393.84 tok/s, greedy byte-identical). The same call pattern (rolling-window pad + depthwise conv1d executed every decode step) exists in 12 other model files. This issue audits each family at L=1 on CUDA and applies the #751 elementwise decode step where measurement confirms the slow engine.

Background (from #748 / PR #751)

• Mechanism: at L=1 with bf16 activations the CUDA conv dispatch falls off the fast conv1d_c1_k1_nhwc kernel into the per-channel grouped-conv engine; f32 activations keep the fast kernel (why the f32 lfm2-8b sibling never regressed); prefill (L > 1) keeps the fast kernel too.
• Detection: nsys kernel histogram on a warm decode (look for convolve_common_engine_float_NHWC instances), or a per-token latency A/B against an elementwise prototype.
• Fix pattern: precompute a time-major [1, L_cache, hidden] decode weight at load (skip on Metal), compute the L=1 step as broadcast multiply + axis sum, keep conv1d for multi-position outputs (prefill, speculative verify) and Metal. Equivalence pinned by channel- and tap-asymmetric kernel unit tests (f32 + bf16), see src/models/lfm2_tests.rs.
• The fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 helpers build_conv_decode_weight / short_conv_decode_step live in src/models/lfm2.rs as pub(crate) and are the natural extraction seed for a shared utility.

Candidate call sites (decode-path depthwise conv1d)

What the 2026-07-12 sweep did and did not show

• lfm2-350m-8bit was the only 10x case (fixed by fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751).
• mamba2-130m (bf16) decode moved -10.4% (181.05 -> 162.23) and is a live candidate; its 4-bit 1.3b sibling improved (+4.9%), so magnitude clearly depends on dtype/size mix.
• PR fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751's body hypothesized gemma-4-26b-a4b-it(-qat) (-10 to -14%) and glm4-flash-4bit (-14%) as the same pattern, but code inspection refutes that: neither is a conv architecture (gemma4 MoE and glm4_moe_lite respectively; neither file calls conv1d). Their drops need separate attribution and are OUT OF SCOPE here; this issue corrects the PR fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 hypothesis on the record.
• The SSM/hybrid cluster also carried an unattributed environmental 2-3x sweep-over-sweep uplift on this host, so sweep deltas alone cannot confirm or refute the dispatch cost; only per-family kernel-level measurement can.

Plan

1. Extract the fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 decode-step helpers from src/models/lfm2.rs into a shared utility; switch lfm2 to it with behavior unchanged (existing lfm2 tests must pass untouched).
2. For each candidate family with a local checkpoint: measure a warm L=1 decode on GB10 (nsys kernel histogram or per-token A/B) and record confirmed/refuted in the PR.
3. Apply the elementwise decode step to confirmed families with the same gating as fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 (single-position output only, off Metal).
4. Add channel- and tap-asymmetric parity unit tests (f32 + bf16, checkpoint-free) per adapted family.
5. Re-bench affected models with ./scripts/bench_decode.sh models/<name> --cooldown 0 singles and refresh the corresponding rows/notes in docs/benchmark_results/model_tests_gb10.md.

Acceptance Criteria

• Shared decode-step helper extracted (src/models/conv_decode.rs); lfm2 switched to it with unchanged behavior (existing 11 lfm2 tests pass unmodified; greedy output byte-identical).
• Every candidate family with a local checkpoint measured on GB10 under nsys; all REFUTED (fast conv1d_c1_k1_nhwc engine). Verdict table + kernel counts in the PR.
• N/A: no family was CONFIRMED, so none is adapted (adapting a refuted family would add the elementwise path for no benefit). Generalized f32+bf16 asymmetric parity tests still added in conv_decode_tests.rs so the shared helper is guarded for future adopters.
• Refuted families documented with per-family kernel evidence (no code change).
• docs/benchmark_results/model_tests_gb10.md refreshed with the audit outcome and the mamba2-130m annotation. No singles CSVs committed because no model's behavior changed (nothing to re-bench).
• gemma-4-26b-a4b / glm4-flash sweep drops recorded as unrelated to conv dispatch (PR fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 hypothesis corrected).

References

• Issue perf(cuda): lfm2-350m-8bit decode regressed ~10x on GB10 (409 -> ~40 tok/s) between 0.3.1 and 0.4.0-rc.1 #748, PR fix(cuda): restore lfm2-350m-8bit decode via elementwise short conv #751 (3fa2789), PR fix: bind gemma-4 NVFP4 vision tower so image input works #750 (3fa7cc1)
• TECHNICAL_REPORTS/751-lfm2-elementwise-short-conv-decode-20260712.en.md (local)
• benchmarks/cuda_gb10_2026-07-12.csv (sweep evidence), benchmarks/cuda_gb10_2026-07-12_single_lfm2-350m-8bit_fix748.csv