ggml-cuda: fuse GLU into the MMQ Q8_1 activation quantization

[SongXiaoXi]

Overview

CUDA already fuses the GLU on the decode (MMVQ/MMVQ-MMID) path. The batched (MMQ) path still runs the GLU as a standalone kernel whose F32 output is written to global memory and read back by the Q8_1 activation quantizer of the down projection: one extra kernel launch plus a full write+read of [n_ff, n_expert_used, n_tokens] of F32 per FFN per ubatch.

This PR computes the gated activation (SWIGLU / GEGLU / SWIGLU_OAI / SWIGLU_CLAMP) in registers of the Q8_1 quantize kernel: the standalone GLU kernel is no longer launched and the F32 intermediate is never written or re-read. The activation is quantized exactly once, as before. Covers both shapes: {GLU, MUL_MAT} (dense/shared FFN) and {GLU, MUL_MAT_ID} (routed MoE down, including the MoE-cache slot map).

The consumer is found in a small forward window: the concurrent-event stream reordering can place independent branches (e.g. a shared expert) between the GLU and its down projection. The fused matmul is computed at the GLU's position, so fusion is skipped unless every intervening node and its sources are disjoint from the output's allocated memory, and the matmul's other inputs (including view chains) are already computed. MUL_MAT_ID is additionally restricted to non-broadcast activations (glu->ne[1] == ids->ne[0]); the scatter path is left unfused. All three conditions have targeted repros in the new backend-op cases.

Additional information

Performance (RTX 4090, SM clock locked to 2235 MHz, llama-bench -ngl 99 -r 5):

model	master t/s	PR t/s	change
GLM-4.7-Flash Q4_K, pp2048	8437.7	8587.5	+1.8%
Qwen3.5-35B-A3B Q4_K_S, pp2048	11554.2	11665.8	+1.0%
Qwen3.8-27B UD-Q4_K_XL, pp4096	2950.6	3000.7	+1.7%

Numerics: bit-identical down-projection outputs with and without the fusion - 52,428,800 floats on the routed path (gate/up IQ2_S + down IQ4_NL, 512 experts x 10 used, 2048 tokens) and 5,242,880 on the dense path, 0 differences. New GLU_MUL_MAT / SWIGLU_MUL_MAT_ID backend-op cases pass; GLM-4.7-Flash wikitext-2 PPL identical to master.

Decode is unaffected (already fused on the MMVQ path).

Related work: #28702 and #29948 fuse the FFN gate/up matmuls (and the GLU) into one MMQ kernel for dense FFN; this PR covers the down-projection side, the routed MoE path (mul_mat_id), and all Q8_1-MMQ quant types.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - running benchmark

[JohannesGaessler]

Sorry, but I think that #29948 is the better approach for GLU fusion. So I would go with that PR instead and close this one.

[am17an]

That PR is different from this. From what I understand this is fusing the GLU + quantization for the down projection. So both are valid fusions.

[JohannesGaessler]

The approach in #29948 is to do the up and gate matrix multiplications as one fused operation and to do the GLU before writing back the data to VRAM. That data can then be quantized and directly used for the down matrix multiplication.

To my understanding this PR would take the individual results from the up and gate matrix multiplications, do the GLU during the quantization step, and then use that as input for the down matrix multiplication.

I don't see how you could possibly do both at the same time.

[am17an]

Hmm, perhaps some modification is required but both these PRs work together #29948 (comment)

[JohannesGaessler]

Context: Aman and me had an off-site discussion regarding this PR. There is some merit if the up and gate tensors have different quantization types.