Repository navigation
CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size - #29633
Merged
ynankani merged 2 commits intoOct 5, 2026
Merged
Conversation
github-actions
Bot
added
ggml
Contributor
|
can you measure on more devices? Preferably ampere and ada too |
JohannesGaessler
requested changes
Sep 29, 2026
JohannesGaessler
left a comment
JohannesGaessler
left a comment
Contributor
There was a problem hiding this comment.
Modify the logic in ggml_cuda_should_use_mmvf instead of adding a second check in ggml_cuda_mul_mat. You can dynamically adjust the threshold for ne11 based on the return value of ggml_cuda_should_use_mmf.
Contributor
Author
Performance RTX-4090
Performance RTX-3090
|
Contributor
|
it says thin F16 and BF16, have we measured the F16 path? because that would I think have a different performance profile. |
Contributor
Author
yup need to collect that, let me change the outtype to fp16 and collect perf sweep for that. |
Contributor
Author
Performance RTX-PRO-6000-BLACKWELLSeeing similar gains with FP16
|
Signed-off-by: ynankani <ynankani@nvidia.com>
ynankani
force-pushed
the
ynankani/cuda_bf16_mmvf_small_batch
branch
from
September 30, 2026 06:15
e56cd33 to
bbbb1d0
Compare
Comment on lines
+830
to
+833
| // MMF needs full row tiles, for other row counts MMVF still beats cuBLAS at small batch size | ||
| if (src0_small && !ggml_cuda_should_use_mmf(type, cc, warp_size, src0_ne, src0_nb, ne11, /*mul_mat_id =*/ false)) { | ||
| return ne11 <= MMVF_MAX_BATCH_SIZE; | ||
| } |
Contributor
There was a problem hiding this comment.
My intuition is that the correct logic is to put this in front of the switch statement for the type. I'll check the AMD performance.
Contributor
|
Note: these numbers are with a patch on top that force-disables MMF. It seems that only for RDNA a speedup can be achieved vs. hipBLAS by using MMVF more often:
I did not touch the NVIDIA-specific logic. |
JohannesGaessler
force-pushed
the
ynankani/cuda_bf16_mmvf_small_batch
branch
from
September 30, 2026 20:28
44dab7b to
ebae262
Compare
JohannesGaessler
approved these changes
Sep 30, 2026
Contributor
Author
|
Hi @am17an, Could i please get a 2nd review and approval for this? |
am17an
approved these changes
Oct 5, 2026
1 task
benjigill
added a commit
to benjigill/llama.cpp
that referenced
this pull request
Oct 6, 2026
matcher Upstream ggml-org#29633 added a warp_size parameter; the fused Q4_K gate/up SwiGLU matcher still used the old signature and failed to compile.
Wizard815
pushed a commit
to Wizard815/mx-llama.cpp-Rocm10
that referenced
this pull request
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Use MMVF for thin f16/bf16 mul_mat at small batch size instead of slow cublass path. Credits to @am17an for this suggestion #29187 (comment)
Additional information
Performance RTX-6000 PRO-BLACKWELL
Collected with -p 256 -n 0 -ub 1,2,3,4,5,6,7,8 -r 5 <style> </style>Requirements