[inureyes]

Summary

qmv, the CUDA quantized GEMV behind every single-stream decode step, chose its accumulator from the weight bit width alone: float at bits >= 8, the element type below that. For a bf16 checkpoint at 4 bits that is a bf16 accumulator, and no pre-Ampere part has a bf16 ALU to run it on, so every FMA in the k-loop ran as convert-to-float, fma, convert-back. Below Ampere the kernel now accumulates bf16 in float at every bit width. On a Tesla V100 that makes qmv 2.01x faster and qwen3.8-27B-4bit decode 1.87x faster end to end.

Route taken, and why

The __CUDA_ARCH__ guard the issue preferred, not a host-side dispatch. qmv is AOT-compiled: the host takes &qmv_kernel<...> as a plain function pointer (qmv.cu:617, :662) and launches it through add_kernel_node_raw (:790, :845), and the accumulator appears in neither the kernel's template signature nor its mangled name. nvcc emits one device pass per entry in MLX_CUDA_ARCHITECTURES, so a guard inside device code gives exactly per-architecture behavior inside a fat binary with nothing for the single host pass to disagree with. The fallback bool FloatAccum template parameter from #1537's capability probe was not needed.

One deliberate narrowing against the issue text, which says "regardless of bit width". The promotion is conditioned on the element type being bfloat16_t, not applied to every type below Ampere. sm_70 and sm_75 do have a native fp16 FMA at twice the fp32 rate, so promoting f16 accumulators there would spend throughput to buy precision, the opposite of the trade this makes for bf16, and f16 policy below Ampere is #1542's subject. Every checkpoint measured here is bf16, so this costs nothing measured and avoids pre-empting #1542. The narrowing is on element type; the "regardless of bit width" part is implemented exactly as written.

What changed

• src/lib/mlx-cpp/patches/mlx/backend/cuda/quantized/qmm/qmv.cu: new qmv_accumulator<bits, T> trait selecting float below sm_80 for bfloat16_t and keeping upstream's bits >= 8 rule everywhere else, used by both qmv_kernel_impl and qmv_multirow_kernel_impl. gather_qmv_kernel inherits it through qmv_kernel_impl. The multirow kernel had to move with the single-row one: perf(cuda/quant): amortizing small-M quantized GEMM for batched decode on Blackwell #725 guarantees a multirow launch is bit-identical to the per-row launches it replaces.
• src/lib/mlxcel-core/src/ffi_tests.rs: qmv_matches_qmm_across_bits_and_group_sizes, comparing qmv at M = 1 against qmm at M = 8 on identical operands across bits 4 and 8 and group sizes 32, 64 and 128.
• docs/benchmark_results/qmv-float-accum-v100-2026-08-31.md: the full before and after record.
• docs/benchmark_results/volta-sm70-baseline-2026-08-31.md: the #1539 row of the post-program comparison table is filled in, and the "prefer 8-bit on Volta" guidance is re-measured, which that section had asked for.
• CHANGELOG.md: entry recording the pre-Ampere output change explicitly.

Measured

Every number follows the six methodology rules of the baseline record: decode rate is a slope over -n 40 to -n 120 at a fixed prompt with generated_tokens == n asserted, warm PTX cache, --cuda-graph-trace=node, profiles reconciled against unprofiled wall clock, and absolute kernel times compared only at matching instance counts. The before arm reproduces the committed baseline to within 1.2% on every checkpoint and to four significant figures on qmv kernel time.

Checkpoint	Before	After	Speedup
qwen3.8-27B-4bit	219.96 ms/tok (4.55 tok/s)	117.83 ms/tok (8.49 tok/s)	1.87x
gemma-4-12B-it-4bit	124.26 ms/tok (8.05 tok/s)	71.66 ms/tok (13.95 tok/s)	1.73x
gemma-4-12b-it-8bit (control)	66.75 ms/tok	66.73 ms/tok	1.00x
gemma-4-26b-a4b-it-4bit (MoE)	35.45 ms/tok	29.56 ms/tok	1.20x, ordinal only

qmv in nsys at identical launch counts: 5.2165 to 2.5984 s over 11,431 instances on the qwen run (2.01x) and 12.8369 to 6.5654 s over 39,151 on the dense run (1.96x). Reconciliation 101.3% to 101.9% and 100.7% to 103.3%. Attribution closes: the qmv delta is 52.26 ms/token against a measured slope delta of 52.60 on the dense pair, that is 99.4%. qmm_naive, which already accumulated in float and cannot be reached by the guard, moves by -0.12% and +0.55%, and the 8-bit checkpoint is unmoved inside its own repeat spread. Those two controls are what confine the result to the accumulator.

The issue's 25%-of-roofline acceptance criterion is not met, and that is reported rather than worked around. qmv on qwen3.8-27B-4bit goes from 7.37% to 14.80% of the 900 GB/s roofline. Reaching 25% would need another 1.69x on top. The criterion was written before the controlled measurement existed; that measurement put the ceiling at 2.14x and said in terms it was an upper bound, and 2.01x is 94% of it. The two figures are not consistent with each other: the 8-bit arm itself only reaches 26.7% of roofline while reading 1.9x the bytes, so no accumulator change alone could put a 4-bit arm at the same fraction. The remaining headroom belongs to the later items in #1536.

The 4-bit against 8-bit inversion the baseline record found is now nearly closed: the 8-bit arm's advantage falls from 1.886x to 1.074x while still reading 1.9x the bytes.

Occupancy

cuobjdump -res-usage on the sm_70 object, before and after, for the <8, 16, 64> bf16 / uint4b instantiation the profile runs:

Kernel	REG	Blocks/SM	Warps/SM
qmv_kernel, has_residue_k = false	55 -> 61	4 -> 4	32 -> 32
qmv_kernel, has_residue_k = true	59 -> 66	4 -> 3	32 -> 24
qmv_multirow_kernel, max_x_rows = 8	110 -> 168	2 -> 1	16 -> 8

Nothing spills: STACK and LOCAL are zero across all 378 qmv-family instantiations in both arms. The residue-k variant is the one this workload runs (gemma-4-12B-it-4bit has k = 3840 and 3840 % 512 = 256), so the measured 1.96x already includes its lost block per SM. elems_per_thread was not re-tuned, because the issue conditions that on occupancy dropping enough to eat the win and 24 warps per SM did not come close. The multirow kernel halves its occupancy and is the one unquantified risk here: it serves batched decode with 2 to 8 rows, which no measurement in this PR exercises, it is the most weight-load-amortized member of the family, and its row window is already autotuner-tunable (#906).

Numerics

Float accumulation is strictly more accurate than the emulated bf16 accumulation it replaces, so quality improves on pre-Ampere parts. It also changes them: greedy decode on a Volta host is no longer token-identical to a build from before this change. No published Volta baseline exists for it to regress against, and the changelog entry states it explicitly.

Determinism run to run is unaffected: three repetitions of a greedy 32-token generation (--temp 0 --seed 0) give one distinct output on the before build and one on the after build, and the two builds produce the same tokens as each other on that prompt.

tests/cuda_qmm_determinism.rs does not pass on this host, and it did not pass before this change either. Pointed at gemma-4-12B-it-4bit (its default models/llama-3.2-1b-4bit is not present here, or in CI, and the test skips rather than fails when the directory is missing, so every CI run of it so far has been a skip) it diverges at step 0, the prefill. That was verified against an origin/main build with qmv.cu reverted: same test, same checkpoint, same divergence at the same step. Prefill is M = 64, so it dispatches to qmm_naive and never reaches qmv. The divergence is between iterations inside one process rather than between the two builds, and iteration 0's prefill logits hash to the same value (5078372879123904733) on both builds, so prefill output is bit-identical across this change. Quantized prefill on sm_70 not being bitwise reproducible is the same class of defect #910 fixed for qmm_sm80 on sm_121 and deserves its own issue; it is out of scope here and is written up in the benchmark record so it is not lost.

Test plan

• Repeated greedy decode byte-identical run to run on the V100, three repetitions on the before build and three on the after build
• MLXCEL_TEST_QMM_MODEL_DIR=models/mlx-community/gemma-4-12B-it-4bit cargo test --release --features cuda --test cuda_qmm_determinism on the V100: fails, and fails identically on origin/main with qmv.cu reverted, at the prefill step this change cannot reach. See the numerics section
• cargo test --release --features cuda -p mlxcel-core --lib qmv_ on the V100, 3 passed, including qmv_multirow_matches_per_row_qmv_bitwise (perf(cuda/quant): amortizing small-M quantized GEMM for batched decode on Blackwell #725 bit-identity) and the new qmv_matches_qmm_across_bits_and_group_sizes
• MLX_CUDA_ARCHITECTURES=70 make release-cuda, and cuobjdump --list-elf reports 96 cubins, every one sm_70
• Decode slopes on four checkpoints, three repetitions each (five for MoE), before and after
• nsys kernel attribution at matching instance counts, both runs reconciled against unprofiled wall clock
• cuobjdump -res-usage occupancy comparison at sm_70
• cuobjdump --dump-sass diff at sm_80 and sm_121, byte-identical before and after
• cargo fmt -p mlxcel-core -- --check

Deferred to GB10

No sm_80-or-later device exists on this host, so three of the issue's acceptance criteria cannot be closed here. They belong to the ## GB10 (sm_121) continuation section of epic #1536.

• Byte-identical greedy output on GB10 for a fixed prompt and seed. Not run: there is no sm_80+ part here.
• GB10 throughput unmoved. Not run, same reason.
• cargo test --features cuda green on an sm_121 host. Only the sm_70 half was run here.

The third criterion, that the emitted SASS for the sm_121 pass is unchanged, was verified and is not deferred. nvcc does not need the target part to emit code for it, so qmv.cu was compiled from both the unpatched and the patched source at one architecture with the production flag set and the disassembly diffed: cuobjdump --dump-sass is byte-identical at sm_80 (389,998 lines) and at sm_121 (437,519 lines), and cuobjdump -res-usage is identical at both. Identical SASS is strong evidence for the two runtime criteria above as well, since a device executing the same instructions on the same inputs produces the same outputs, but it is evidence and not the measurement, so they stay open.

Closes #1539