metal: FWHT kernels for block widths above 512 - #29095
Merged
Merged
Conversation
github-actions
Bot
added
testing
bri-prism
force-pushed
the
up/fwht-metal-wide
branch
7 times, most recently
from
September 19, 2026 18:09
13c2ed8 to
b25c008
Compare
The Metal FWHT covers widths 64 to 512, one row per simdgroup with N/32 values per lane. Wider blocks need more registers per lane than that layout allows. kernel_fwht_tg runs one row per threadgroup with 256 threads, so each thread keeps N/256 values. Butterflies below the simdgroup width still shuffle, those up to the threadgroup width go through threadgroup memory, and the rest stay in registers. Same butterfly and sign convention as the simdgroup kernel. Widths 64 to 512 keep the simdgroup kernel. 1024 through 8192 use the new one, for both F32 and F16 sources. The wide kernels allocate float[N] of threadgroup memory, 32 KB at 8192, so the size check takes the device limit and reports those widths as unsupported where they would not fit. Without that a device with less threadgroup memory would accept the op and then abort on a nil pipeline. test-backend-ops on M5 Pro: MUL_MAT_HADAMARD 26/26, MUL_MAT 1265/1265.
bri-prism
force-pushed
the
up/fwht-metal-wide
branch
from
September 21, 2026 15:12
b25c008 to
a599d6e
Compare
Contributor
Author
|
@ggerganov this one's reconciled and mergeable now, ready for another look whenever you have time. |
ggerganov
reviewed
Sep 25, 2026
feal87
added a commit
to feal87/myllama.cpp
that referenced
this pull request
Sep 25, 2026
Merge upstream commits: - llama: add llama_prec_policy + model-driven W4A4 path (ggml-org#24364) - llama: fix tensor split for fused qkv with uneven K/V head sizes (ggml-org#29294) - metal: split fa kernels into per-dtype libraries (ggml-org#29329) - metal: FWHT kernels for block widths above 512 (ggml-org#29095) - CUDA: fuse RMS_NORM + SCALE into one kernel (ggml-org#29393) - common: extract shared unicode path/string helpers (ggml-org#29415) - common,rpc: simplify fs_create_directory_with_parents() (ggml-org#29432) - rpc: include nb in the get_alloc_size cache key (ggml-org#29283) - [SYCL] support sparse FA (ggml-org#28796) - musa: fix PH1 operator failures and build issues (ggml-org#29193) - HIP: bump HIP_VERSION required for fp8 (ggml-org#29231) - opencl: add q5_k bin kernel (ggml-org#29401) - hexagon: add q5_k quant type support (ggml-org#29123) - hexagon: use DMA for contiguous dim1 CONCAT (ggml-org#29404) - mtmd: fix mel preprocessor in LFM2 audio (ggml-org#29403) - vulkan: fix legacy GLSLC without cooperativeMatrix (ggml-org#29409) - gguf-py: ByteLevel processing defaults bos/eos to False (ggml-org#29422) - gguf-py: TemplateProcessing has final word on add_special_token (ggml-org#29417) Assisted-by: Pi
sky-mighty
pushed a commit
to sky-mighty/llama.cpp
that referenced
this pull request
Sep 26, 2026
* metal: FWHT kernels for block widths above 512 The Metal FWHT covers widths 64 to 512, one row per simdgroup with N/32 values per lane. Wider blocks need more registers per lane than that layout allows. kernel_fwht_tg runs one row per threadgroup with 256 threads, so each thread keeps N/256 values. Butterflies below the simdgroup width still shuffle, those up to the threadgroup width go through threadgroup memory, and the rest stay in registers. Same butterfly and sign convention as the simdgroup kernel. Widths 64 to 512 keep the simdgroup kernel. 1024 through 8192 use the new one, for both F32 and F16 sources. The wide kernels allocate float[N] of threadgroup memory, 32 KB at 8192, so the size check takes the device limit and reports those widths as unsupported where they would not fit. Without that a device with less threadgroup memory would accept the op and then abort on a nil pipeline. test-backend-ops on M5 Pro: MUL_MAT_HADAMARD 26/26, MUL_MAT 1265/1265. * cont : add TODOs --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The Metal FWHT covers block widths 64 to 512. It runs one row per simdgroup and keeps
N/32values per lane, so wider blocks need more registers per lane than that layoutallows.
kernel_fwht_tgruns one row per threadgroup with 256 threads, so each thread keepsN/256values instead. Butterflies below the simdgroup width still shuffle, those up tothe threadgroup width go through threadgroup memory, and the rest stay in registers. The
butterfly and sign convention match the existing kernel.
Widths 64 to 512 keep the simdgroup kernel and are untouched. 1024 through 8192 use the
new one, for both F32 and F16 sources.
Additional information
Stacked on #29094, which adds the F16 source type. The last commit is the one to review here.
8192 allocates 32 KB of threadgroup memory, which is the limit on the devices I can test.
ggml_metal_fwht_supported_sizestops there for that reason, and anything wider keeps theexisting behaviour of falling back to the generic matmul path.
Tested on an M5 Pro with
test-backend-ops:MUL_MAT_HADAMARD26/26, including the new 1024/2048/4096/8192 cases in F32 and F16MUL_MAT1265/1265, no regressionsThe previous "too big" test case moved from 1024 to 16384, since 1024 is now covered.
Requirements
structure of the existing simdgroup kernel, and to format this description to the PR template.
I reviewed every line and take full responsibility for the changes.