Feature Request: generalize MMQ CUDA kernel for floating-point data

[JohannesGaessler]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Generalize the kernel in mmq.cuh:

• Optionally use FP16 activations instead of q8_1.
• Implement function for loading FP16 data from src0.
• Implement function for FP16, FP16 -> FP32 matrix multiplication.
• Move and rename some parts of the code to different headers to organize it better, this should be done in a follow-up PR.

Same implementation can be done for BF16 and FP32, for FP8 and NVFP4 we should first standardize how to treat them in ggml and add CPU support.

This feature should be tackled by experienced CUDA programmers only.

Motivation

I wrote a matrix multiplication CUDA kernel for quantized data that currently resides mmq.cuh. I wrote it because it allows for a fused matrix multiplication using custom data types since a library like cuBLAS only supports standard data types like FP16. I don't think it would be worthwhile to try and compete with cuBLAS for such data types when it comes to conventional GEMM. However, I at some point added optimizations for MoE models which I think will result in better performance than cuBLAS.

Possible Implementation

No response

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[bssrdf]

There could be potential overflow risk (actually happening for some Qwen Image gguf models when quantizing activations into q8_1_mmq due to using half2 for ds4. Should it be addressed?

[JohannesGaessler]

I don't remember whether or not those values are actually used as FP16 or if they are always upcast to FP32 anyways. If it's the latter I would say to just use BF16.

[bssrdf]

I don't remember whether or not those values are actually used as FP16 or if they are always upcast to FP32 anyways. If it's the latter I would say to just use BF16.

Thanks, @JohannesGaessler.
The values could have already been overflowed here

llama.cpp/ggml/src/ggml-cuda/quantize.cu

Line 267 in 451ef08

y[ib].ds4[iqs/32] = make_half2(d, sum);

before they were upcast to f32 for use.