Repository navigation
cuda : add BF16 support for XIELU - #29955
Merged
Merged
Conversation
The XIELU CUDA kernel template is already generic over the element type; only the F32/F16 type assertion and the else-if dispatch were missing. Add the nv_bfloat16 branch to the launcher, and drop the temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU. test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]). docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row flips from no to yes as well, i.e. the previous record was stale. Tested: - Mac CPU: xIELU F32/F16/BF16, 6/6 - Mac Metal: existing F32/F16, 4/4; BF16 still unsupported - RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6 - RTX 4090 CUDA BF16-only: 2/2 - git diff --check passes
github-actions
Bot
added
documentation
Qiao12-pixel
commented
Oct 5, 2026
Qiao12-pixel
left a comment
Qiao12-pixel
left a comment
Contributor
Author
There was a problem hiding this comment.
I have approved the code, it can be merged
Contributor
Author
|
@ggerganov could you help approve the CI workflows and review this BF16 XIELU CUDA support? All test-backend-ops passed on RTX4090 and Mac CPU. |
JohannesGaessler
approved these changes
Oct 6, 2026
JohannesGaessler
added
the
merge ready
CISC
approved these changes
Oct 6, 2026
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 8, 2026
The XIELU CUDA kernel template is already generic over the element type; only the F32/F16 type assertion and the else-if dispatch were missing. Add the nv_bfloat16 branch to the launcher, and drop the temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU. test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]). docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row flips from no to yes as well, i.e. the previous record was stale. Tested: - Mac CPU: xIELU F32/F16/BF16, 6/6 - Mac Metal: existing F32/F16, 4/4; BF16 still unsupported - RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6 - RTX 4090 CUDA BF16-only: 2/2 - git diff --check passes (cherry picked from commit 5ad1c5d)
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The XIELU CUDA kernel template is already generic over the element type; only the F32/F16 type assertion and the else-if dispatch were missing. Add the nv_bfloat16 branch to the launcher, and drop the temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU.
test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]). docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row flips from no to yes as well, i.e. the previous record was stale.
Tested:
Overview
Next, I will present the test results of this PR
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24091 MiB):
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24091 MiB
Testing 2 devices
Backend 1/2: CUDA0
Device description: NVIDIA GeForce RTX 4090
Device memory: 24091 MB (23699 MB free)
XIELU(type=f32,ne=[10,5,4,3]): SUPPORTED
XIELU(type=f16,ne=[10,5,4,3]): SUPPORTED
XIELU(type=bf16,ne=[10,5,4,3]): SUPPORTED
XIELU(type=f32,ne=[512,16,1,1]): SUPPORTED
XIELU(type=f16,ne=[512,16,1,1]): SUPPORTED
XIELU(type=bf16,ne=[512,16,1,1]): SUPPORTED
Backend CUDA0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24091 MiB):
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24091 MiB
Testing 2 devices
Backend 1/2: CUDA0
Device description: NVIDIA GeForce RTX 4090
Device memory: 24091 MB (23699 MB free)
XIELU(type=bf16,ne=[10,5,4,3]): OK
XIELU(type=bf16,ne=[512,16,1,1]): OK
2/2 tests passed
Backend CUDA0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
root@autodl-container-060c4ba85a-dba89d5f:~/autodl-tmp/llama.cpp# ./build-cuda/bin/test-backend-ops test -b CUDA0 -o XIELU
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24091 MiB):
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24091 MiB
Testing 2 devices
Backend 1/2: CUDA0
Device description: NVIDIA GeForce RTX 4090
Device memory: 24091 MB (23699 MB free)
XIELU(type=f32,ne=[10,5,4,3]): OK
XIELU(type=f16,ne=[10,5,4,3]): OK
XIELU(type=bf16,ne=[10,5,4,3]): OK
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
XIELU(type=f32,ne=[512,16,1,1]): OK
XIELU(type=f16,ne=[512,16,1,1]): OK
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
XIELU(type=bf16,ne=[512,16,1,1]): OK
6/6 tests passed
Backend CUDA0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
root@autodl-container-060c4ba85a-dba89d5f:~/autodl-tmp/llama.cpp# ./build-cuda/bin/test-backend-ops support -b CUDA0 -o XIELU --output csv
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24091 MiB):
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24091 MiB
"backend_name","op_name","op_params","test_mode","supported","error_message","backend_reg_name"
"CUDA0","XIELU","type=f32,ne=[10,5,4,3]","support","1","yes","CUDA"
"CUDA0","XIELU","type=f16,ne=[10,5,4,3]","support","1","yes","CUDA"
"CUDA0","XIELU","type=bf16,ne=[10,5,4,3]","support","1","yes","CUDA"
"CUDA0","XIELU","type=f32,ne=[512,16,1,1]","support","1","yes","CUDA"
"CUDA0","XIELU","type=f16,ne=[512,16,1,1]","support","1","yes","CUDA"
"CUDA0","XIELU","type=bf16,ne=[512,16,1,1]","support","1","yes","CUDA"
Additional information
Requirements