metal : gate mul_mm_id src1 rescale behind ggml_prec by mdegans · Pull Request #29029 · ggml-org/llama.cpp

mdegans · 2026-09-17T13:25:13Z

Overview

This is a follow-up to #26223 per request from @ggerganov to use ggml_prec....

Set f32 src precision on the Mistral 4 down projection and rescale on Metal only when it is set.
With Mistral Small 4, the down projection input peaks around 2.9e5, gate/up input near 59.
On metal, MUL_MAT_ID 849 of 849, MUL_MAT 1265 of 1265, and the batched path is ~1% faster than master for unflagged models. See details:

Details

Hardware: Apple M2 Max
Perf Command: test-backend-ops perf -b MTL0 -o MUL_MAT_ID
master 4ff829e against this branch, eight passes in ABBA BAAB order, median of per-case medians, negative means this branch is faster.

N	cases	median	best	worst
1	15	-0.41%	-1.49%	+2.07%
4	15	+0.04%	-1.74%	+3.07%
8	15	+0.11%	-1.67%	+1.36%
32	14	-1.45%	-4.28%	+0.39%
64	14	-1.30%	-4.99%	+0.71%
128	14	-1.22%	-3.88%	+1.50%
256	14	-1.04%	-3.26%	+0.85%
512	15	-1.74%	-3.22%	-0.49%
1024	7	-1.01%	-2.06%	-0.34%
2048	7	-1.50%	-2.53%	-0.76%
4096	7	-0.91%	-1.97%	-0.63%
8192	7	-0.50%	-1.86%	-0.28%
all	144	-0.87%
n>=32 (mul_mm_id)	99	-1.22%

Please note that N >= 1024 are local cases only and not committed.

Additional information

Six amax=1e5 MUL_MAT_ID cases that were commented out in metal: fix NaN in mul_mm_id when activations exceed f16 range #26223 are re-enabled ~~which will likely break on CUDA, Vulkan and WebGPU~~ (now likely fixed). ~~Let me know if these should be commented out. My guess is they should be, but that putting the test change in a seperate commit might be desirable. Please advise.~~

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES Assisted-by: Claude Fable 5.1 -- Claude made the code changes as requested and helped with wording.

Assisted-by: Claude Fable 5.1

ggerganov

Thanks! We expect some backends to fail the new tests now - let's see.

ggerganov · 2026-09-19T11:04:57Z

@mdegans We should now update the support_op implementation of the failing backends to return false when the specified src1 precision is F32.

@ggml-org/ggml-cuda @ggml-org/ggml-vulkan @ggml-org/ggml-webgpu fyi

mdegans · 2026-09-19T22:39:57Z

@mdegans We should now update the support_op implementation of the failing backends to return false when the specified src1 precision is F32.

I will take care of that tomorrow, no problem.

0cc4m · 2026-09-20T08:49:28Z

I was already looking into this for Vulkan, will open a PR soon.

mdegans · 2026-09-20T12:57:37Z

I was already looking into this for Vulkan, will open a PR soon.

I was about to tackle this but will hold off in this case. @ggerganov and @0cc4m please let me know which bits you want me to handle, if any.

yomaytk · 2026-09-20T14:03:34Z

@mdegans
For the WebGPU backend, I changed supports_op to return false for this case, so could you cherry-pick the following commit?
I confirmed that the MUL_MAT_ID tests pass on both Apple and NVIDIA machines.

fcb7238

I'll open a separate PR once WebGPU can actually compute this case.

mdegans · 2026-09-20T21:37:52Z

@mdegans For the WebGPU backend, I changed supports_op to return false for this case, so could you cherry-pick the following commit? I confirmed that the MUL_MAT_ID tests pass on both Apple and NVIDIA machines.

fcb7238

I'll open a separate PR once WebGPU can actually compute this case.

Will do tomorrow along with the rest of the backends unless anybody objects by then.

0cc4m · 2026-09-21T05:52:31Z

I was already looking into this for Vulkan, will open a PR soon.

I was about to tackle this but will hold off in this case. @ggerganov and @0cc4m please let me know which bits you want me to handle, if any.

You can update supports_op here for failing cases, I'll open a follow-up PR with fixes for Vulkan.

mdegans · 2026-09-21T11:19:43Z

You can update supports_op here for failing cases, I'll open a follow-up PR with fixes for Vulkan.

Will do. Cherry picking WebGPU as asked and fixing supports_op for all failing backends. Going to try to test as much as I can but I do not have Intel Windows, Intel Linux, rocm, and WebGPU on Nvidia. Will rely on CI for that.

fix `supports_op` to return false for failing backends when the specified src1 precision is f32 Assisted-by: Claude Fable 5.1

CISC · 2026-09-21T11:37:58Z

Going to try to test as much as I can but I do not have Intel Windows, Intel Linux, rocm, and WebGPU on Nvidia. Will rely on CI for that.

FYI, Intel CIs are down ATM.

mdegans · 2026-09-21T11:42:32Z

Going to try to test as much as I can but I do not have Intel Windows, Intel Linux, rocm, and WebGPU on Nvidia. Will rely on CI for that.

FYI, Intel CIs are down ATM.

Thanks for the heads-up but I had not looked at the causes of the failures on Intel. I think they're unrelated. So it's fine even if it runs now. Vulkan passes locally on macOS and Linux. CUDA passes on Linux. WebGPU I am assuming is workign based on @yomaytk but I will run locally as well to confirm. Edit: Confirmed on macOS.

* metal : gate mul_mm_id src1 rescale behind ggml_prec Assisted-by: Claude Fable 5.1 * ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32 * cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32 fix `supports_op` to return false for failing backends when the specified src1 precision is f32 Assisted-by: Claude Fable 5.1 --------- Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>

* metal : gate mul_mm_id src1 rescale behind ggml_prec Assisted-by: Claude Fable 5.1 * ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32 * cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32 fix `supports_op` to return false for failing backends when the specified src1 precision is f32 Assisted-by: Claude Fable 5.1 --------- Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com> (cherry picked from commit 0f8a414)

Same supports_op gate CUDA, Vulkan and WebGPU got in ggml-org#29029: SYCL quantizes src1 to q8_1 or converts it to f16, so activations beyond the f16 range turn into NaN (MUL_MAT_ID q8_0 n_mats=8 n_used=2 n=64 amax=1e5). Assisted-by: Claude Opus 5.5

297 upstream commits. Conflicts and how they were resolved: - ggml ops: upstream added GGML_OP_TOP_K_MASK next to the fork's GGML_OP_TURBO_WHT; both are kept, upstream's first, GGML_OP_COUNT 103. - MoE down projection: upstream replaced the MISTRAL4 F32 src1 precision hint (its own, ggml-org#29029) with a per-row amax rescale (ggml-org#30861) for MISTRAL4 and MELLUM. It now sits in build_moe_ffn_exps, so every tiered call (hot, cold, each arena window) rescales its own rows; rows skipped by a partial mul_mat_id are zero, so their scale is 1. - MoE post_exp_norm (new upstream): applied in build_moe_ffn after the hot/cold merge and before the router weights; the two halves fill disjoint expert slots, so the norm sees whole rows. - build_lora_mm_id: upstream's Hadamard input transform (prism) after the fork's partial/slots assert. Expert tensors split by LLAMA_MOE_HOT are not in the Hadamard map, so prism and tiering do not stack on one layer. - kv-cache attention rotation: the fork's default (off unless LLAMA_ATTN_ROT_DISABLE=0) is kept; upstream's llama_model_set_attn_rot can still turn it off per model. - model loader: hugetlb per-tensor populate kept before upstream's multi-buffer lookup of the mmap buffer. - llama-model: MoE arena setup and upstream's load_tensors_hadamard both run at the end of load_tensors. - qwen4exp: turbo Q pre-rotation kept; the selection mask is now `sel`. - server: upstream added draft-token logprobs (ggml-org#27196) read back from the batch logits, without post_sampling_probs. The fork's on_sample path, which records them during verification and covers post_sampling_probs, is kept; upstream's test_draft_token_probs passes with it. Upstream's local spec_i_batch is taken. - mmq: fork's IQ1_M MMQ gains upstream's prec_src1 parameter. - fattn-tile: the fork's D=640 case and HIP guard kept, upstream's default-case style. - argsort.cu: upstream's CCCL guard; the fork's hipcub arm was dead since HIP does not define GGML_CUDA_USE_CUB. - tests/CMakeLists.txt: fork tests kept, upstream's LINK and consolidated test-llama-context taken. Tested on gfx1100: ctest 78/80 (the two failures are environmental: the ggml-vocabs clone has git-lfs pointers, and system python has no jinja2). All fork tests pass (turbo-kv x8, moe-tier, fattn-turbo4, fattn-long-kv, mmvq-src1-cache, fusions, mul-mat-f16-range, top-k-graph-capture, op-offload-devices). test-llama-archs 390/390. Server test_speculative.py 10/10, completion/probs pytests 14/14. Needle at 20.5k tokens, Qwen3.8-27B IQ4_NL, turbo4 K/V: correct, prefill 925 tok/s vs 909 before the merge (single run).

metal : gate mul_mm_id src1 rescale behind ggml_prec

abd41ad

Assisted-by: Claude Fable 5.1

mdegans requested review from a team, CISC and ggerganov as code owners September 17, 2026 13:25

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Sep 17, 2026

ggerganov approved these changes Sep 17, 2026

View reviewed changes

ggerganov self-assigned this Sep 17, 2026

ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32

c7bb099

cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32

dea99e1

fix `supports_op` to return false for failing backends when the specified src1 precision is f32 Assisted-by: Claude Fable 5.1

mdegans requested review from a team as code owners September 21, 2026 11:34

github-actions Bot added Vulkan Issues specific to the Vulkan backend CUDA Related to the CUDA backend WebGPU labels Sep 21, 2026

yomaytk approved these changes Sep 22, 2026

View reviewed changes

ggerganov merged commit 0f8a414 into ggml-org:master Sep 22, 2026
39 of 41 checks passed

ggerganov mentioned this pull request Sep 23, 2026

ggml : bump version to 0.25.0 ggml-org/ggml#1635

Merged

0cc4m mentioned this pull request Oct 4, 2026

cuda: do not reject MUL_MAT_ID with GGML_PREC_F32 for quantized weights #29911

Closed

eduralph mentioned this pull request Oct 14, 2026

vulkan : keep F32-precision MUL_MAT_ID on the GPU for mat-vec, skip MMVQ #30544

Open

This was referenced Oct 15, 2026

Eval bug: Mellum NaN on Metal with Q8_0 (ffn_moe_down input exceeds F16 range) #30579

Closed

graph : add precision handling for MELLUM architecture #30584

Merged

0cc4m mentioned this pull request Oct 16, 2026

vulkan: ggml src1 precision support #30656

Open

ruben-salas20 mentioned this pull request Oct 17, 2026

Eval bug: Mellum 8x slower on CUDA since #30584 (ffn_moe_down MUL_MAT_ID falls back to CPU) #30704

Open

leehack mentioned this pull request Oct 20, 2026

Eval bug: Laguna repeats <assistant> on Metal for ubatches >= 32 tokens (ffn_moe_down input exceeds F16 range) #30896

Open

Navigation Menu

Repository navigation

metal : gate mul_mm_id src1 rescale behind ggml_prec - #29029

metal : gate mul_mm_id src1 rescale behind ggml_prec#29029
ggerganov merged 3 commits into
ggml-org:masterfrom
mdegans:metal-mul-mm-id-prec

mdegans commented Sep 17, 2026 •

edited

Loading

Uh oh!

ggerganov left a comment

Uh oh!

ggerganov commented Sep 19, 2026

Uh oh!

mdegans commented Sep 19, 2026

Uh oh!

0cc4m commented Sep 20, 2026

Uh oh!

mdegans commented Sep 20, 2026

Uh oh!

yomaytk commented Sep 20, 2026

Uh oh!

mdegans commented Sep 20, 2026

Uh oh!

0cc4m commented Sep 21, 2026

Uh oh!

mdegans commented Sep 21, 2026 •

edited

Loading

Uh oh!

CISC commented Sep 21, 2026

Uh oh!

mdegans commented Sep 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mdegans commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Sep 19, 2026

Uh oh!

mdegans commented Sep 19, 2026

Uh oh!

0cc4m commented Sep 20, 2026

Uh oh!

mdegans commented Sep 20, 2026

Uh oh!

yomaytk commented Sep 20, 2026

Uh oh!

mdegans commented Sep 20, 2026

Uh oh!

0cc4m commented Sep 21, 2026

Uh oh!

mdegans commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CISC commented Sep 21, 2026

Uh oh!

mdegans commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mdegans commented Sep 17, 2026 •

edited

Loading

mdegans commented Sep 21, 2026 •

edited

Loading

mdegans commented Sep 21, 2026 •

edited

Loading