Repository navigation
cuda : enable i16 and i32 for DUP - #28897
Merged
Merged
Conversation
ggerganov
approved these changes
Sep 14, 2026
am17an
approved these changes
Sep 15, 2026
dzannotti
added a commit
to halo-box/llama.cpp
that referenced
this pull request
Sep 15, 2026
* upstream/master: (72 commits) HIP: Enable AllReduce for ROCm (ggml-org#27825) opencl: choose the MoE expert matmul by batch size for speculative decoding/MTP (ggml-org#27637) ci: build MUSA for only 1 arch (ggml-org#28944) docs: Rule of thumb for AI review time [no ci] (ggml-org#28945) rpc : hash-cache only weights (ggml-org#28789) cuda: support row-contiguous SUM_ROWS (ggml-org#26308) models : move build_arch_graph() after graph() template specialization (ggml-org#28934) vulkan: support sparse Flash Attention (ggml-org#28105) OpenVINO: optimize stateful decode and GPU MoE inference (ggml-org#28638) opencl: add generic ssm_scan (ggml-org#28881) ci: bump kleidiai runners from 22.04 to 24.04 (ggml-org#28885) metal : add FA kernels for HSK=96, HSV=64 (MiniCPM3) (ggml-org#28599) ci: Bump CUDA Windows x64 builds to 13.4.1 (ggml-org#28930) ci : fix android release (ggml-org#28936) cuda : enable i16 and i32 for DUP (ggml-org#28897) cmake : use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR (ggml-org#28771) webui: stop re-probing disabled /tools endpoint on every message (ggml-org#28646) ci : reuse build tag name when used instead of safe one (ggml-org#28911) CI: hip-quality-check: ignore spill added in bfdc321 (ggml-org#28909) HIP: fattn-mma: use fp32 accumulation on MFMA devices (ggml-org#28576) ...
quimmedes
pushed a commit
to quimmedes/cafe-llama.cpp
that referenced
this pull request
Sep 16, 2026
* cuda : enable i16 and i32 for DUP * docs : update ops table for DUP on CUDA
adromir
pushed a commit
to adromir/llama-cpp-turboquant
that referenced
this pull request
Sep 17, 2026
* cuda : enable i16 and i32 for DUP * docs : update ops table for DUP on CUDA
1 task done
roverdrubber
added a commit
to roverdrubber/llama.cpp
that referenced
this pull request
Oct 15, 2026
…backend changes A quantized weight that takes the dequantize and hipBLAS path was dequantized to F16, so any activation past the F16 range came back as inf. On RDNA3 the 27B crosses into that path at 129 rows, which is why a long prompt returned NaN logits while a short one did not. Quantized types now dequantize to F32 on AMD; NVIDIA keeps F16 where its hardware is fast at it. The regression test added later in this series (activations in [-4000, 4000], both sides of the 128-row cutover) returned inf for Q2_K, Q6_K and Q8_0 before this change. The same commit carries the backend half of the upstream sync: the MMQ and vec-dot changes (ggml-org#28552, ggml-org#26705), fast_bf16_hardware_available and the BF16 fallback (ggml-org#28846), the GGML_CUDA_FA_QUANTS build option, and ggml-org#26308 and ggml-org#28897. The integrated-GPU host-buffer path stays disabled on HIP as well, because it corrupts output there (upstream ggml-org#26209, reverted in ggml-org#28604). MUL_MAT and MUL_MAT_ID pass on gfx1100 (1304/1304 and 892/892 at the tip of this series). Assisted-by: Claude Opus 5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
i32 and i16 DUP ops were silently falling back to the CPU. The i32 path already existed under the hood but was blocked by the support gate. This PR adds the missing i16 branch and fixes the gate to allow both.
Additional information
Testing: test-backend-ops -o DUP passes 10/10 on dual T4s. The full suite passes with zero regressions (16097/16097).
test-backend-ops -o DUP
Docs: Manually updated the 4 DUP rows in docs/ops/CUDA.csv. I skipped a full regen since running it on a T4 (compute 7.5) would incorrectly downgrade rows for ops that require newer hardware.
Ref: #14909
Requirements