[KakaruHayate]

Overview

ggml_cuda_should_use_mmq() currently selects MMQ purely from the quantization type. The current MMQ configurations are designed and maintained against a minimum of 48 KiB per-block shared memory, the limit provided by NVIDIA Pascal GPUs and later. On devices that report less, no supported MMQ tile fits and mul_mat_q_switch_J() aborts when every tile size exceeds the device's per-block shared memory budget.

This PR disables MMQ when smpbo < 48 KiB so the caller falls back to the BLAS path instead of hitting GGML_ABORT. Some current MUSA QY1 devices report only 28 KiB and are covered by this guard.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, This is an issue that Claude found and fixed when I was adapting LLAMA.CPP for MUSA RWKV7 downstream

[ggml-gh-bot]

Hi @KakaruHayate, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[am17an]

What device do you face this problem in?

[KakaruHayate]

What device do you face this problem in?

MooreThreads MTT S70. The sharedMemPerBlock of S70 is only 28 KiB.

[am17an]

Would it be better if you use cublas for all MUSA GPUs? None of the maintainers have access to these GPUs and cannot guarantee MMQ will keep working with these GPUs in the future.

[KakaruHayate]

Would it be better if you use cublas for all MUSA GPUs? None of the maintainers have access to these GPUs and cannot guarantee MMQ will keep working with these GPUs in the future.

I understand the concern — maintainers don't have access to Moore Threads hardware, so long-term guarantees for the MMQ path on these devices are difficult.

That said, completely disabling MMQ for all MUSA GPUs may be overly broad. On the current S70/S80/S3000 (28 KiB sharedMemPerBlock), some quantizations (Q4_K_M, FP16, etc.) still fit and run with MMQ without issues. Only certain types like Q8_0 exhaust the available shared memory and need the fallback. Newer cards (S4000/S5000) may also have larger shared memory budgets.

Note that this particular abort path essentially never triggers on NVIDIA GPUs. All modern NVIDIA architectures that llama.cpp targets for MMQ (Pascal and later) report at least 48 KiB sharedMemPerBlock by default, which is the baseline the current tile selection was designed around. The 28 KiB case is unique to the Moore Threads hardware.

The current PR only adds a safety check inside ggml_cuda_should_use_mmq(): if no tile size fits into the device's smpbo, it returns false and falls back to cuBLAS instead of aborting. This is a general defensive measure that would also protect any future low-smem devices, not just MUSA.

If the maintainers prefer a more conservative stance, I'm happy to:

• Add an explicit #ifdef GGML_USE_MUSA path that forces cuBLAS, or
• Document that MMQ support on Moore Threads is best-effort / untested by the core team.

Just let me know which direction you prefer and I'll adjust the PR accordingly.

[am17an]

Note that this particular abort path essentially never triggers on NVIDIA GPUs. All modern NVIDIA architectures that llama.cpp targets for MMQ (Pascal and later) report at least 48 KiB sharedMemPerBlock by default, which is the baseline the current tile selection was designed around. The 28 KiB case is unique to the Moore Threads hardware

Probably just add this check then? If smpbo is report less than 48kb then MMQ cannot be used at the moment. Leave a comment about MUSA devices having 28kib

[KakaruHayate]

Note that this particular abort path essentially never triggers on NVIDIA GPUs. All modern NVIDIA architectures that llama.cpp targets for MMQ (Pascal and later) report at least 48 KiB sharedMemPerBlock by default, which is the baseline the current tile selection was designed around. The 28 KiB case is unique to the Moore Threads hardware

Probably just add this check then? If smpbo is report less than 48kb then MMQ cannot be used at the moment. Leave a comment about MUSA devices having 28kib

This is also feasible. It's very pragmatic and reasonable. I will make changes based on this suggestion.