Repository navigation
[NVPTX] Set default value of nvptx-allow-ftz-atomics to false - #231429
Merged
Merged
Conversation
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
@llvm/pr-subscribers-llvm-ir @llvm/pr-subscribers-backend-nvptx Author: Christian Sigg (chsigg) ChangesThis restores the strict FTZ default originally introduced in #200732 Frontends can now explicitly opt in to native FTZ-mismatched atomics on a Update test cases to verify the new default disallow behavior. Full diff: https://github.com/llvm/llvm-project/pull/231429.diff 8 Files Affected:
diff --git a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
index 30b2dddc305fe..b7fa3c44bb91d 100644
--- a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
@@ -112,15 +112,16 @@ static cl::opt<bool> UsePrecSqrtF32(
cl::init(true));
// PTX atom.add.f32 has fixed FTZ behavior that may not match the function's
-// (see shouldExpandAtomicRMWInIR), so we'd normally fall back to a CAS loop
-// when they disagree. This option (enabled by default) allows using atom.add
-// anyway, trading correct denormal handling for the speed of the native
-// instruction.
+// (see shouldExpandAtomicRMWInIR), so by default we fall back to a CAS loop
+// when they disagree (strict mode), unless declared insignificant via
+// !atomic.ignore.denormal.mode metadata or this option. This option allows
+// using atom.add anyway, trading correct denormal handling for the speed of the
+// native instruction.
static cl::opt<bool> AllowFTZAtomics(
"nvptx-allow-ftz-atomics", cl::Hidden,
cl::desc("NVPTX Specific: Lower atomicrmw fadd to atom.add even when its "
"FTZ behavior does not match the function's denormal mode."),
- cl::init(true));
+ cl::init(false));
/// Whereas CUDA's implementation (see libdevice) uses ex2.approx for exp2(), it
/// does NOT use lg2.approx for log2, so this is disabled by default.
diff --git a/llvm/test/CodeGen/NVPTX/atomicrmw-allow-ftz-atomics.ll b/llvm/test/CodeGen/NVPTX/atomicrmw-allow-ftz-atomics.ll
index 78c86bfb08e2f..e78f2428bbde9 100644
--- a/llvm/test/CodeGen/NVPTX/atomicrmw-allow-ftz-atomics.ll
+++ b/llvm/test/CodeGen/NVPTX/atomicrmw-allow-ftz-atomics.ll
@@ -2,16 +2,16 @@
; -nvptx-allow-ftz-atomics forces atomicrmw fadd to lower to atom.add even when
; the hardware's FTZ behavior doesn't match the function's denormal mode, where
; we'd otherwise emit a CAS loop. Compare the lowering with the option disabled
-; (DISALLOW) against the option enabled or default (ALLOW).
+; or default (DISALLOW) against the option enabled (ALLOW).
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=CHECK,DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=CHECK,ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | FileCheck %s --check-prefixes=CHECK,ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | FileCheck %s --check-prefixes=CHECK,DISALLOW
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=false | %ptxas-verify -arch=sm_90 %}
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=true | %ptxas-verify -arch=sm_90 %}
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | %ptxas-verify -arch=sm_90 %}
; global f32: atom.add flushes, so we'd normally use a CAS loop (function is not FTZ)
-; when the option is disabled; when enabled (default), it lowers to atom.add.
+; when the option is disabled (default); when enabled, it lowers to atom.add.
define float @fadd_f32_global(ptr addrspace(1) %addr, float %val) {
;
; DISALLOW-LABEL: fadd_f32_global(
@@ -51,7 +51,7 @@ define float @fadd_f32_global(ptr addrspace(1) %addr, float %val) {
}
; generic f32: we can't know the memory space, so we'd normally use a CAS loop
-; when the option is disabled; when enabled (default), it lowers to atom.add.
+; when the option is disabled (default); when enabled, it lowers to atom.add.
define float @fadd_f32_generic(ptr %addr, float %val) {
;
; DISALLOW-LABEL: fadd_f32_generic(
diff --git a/llvm/test/CodeGen/NVPTX/atomicrmw-ignore-denormal-mode.ll b/llvm/test/CodeGen/NVPTX/atomicrmw-ignore-denormal-mode.ll
index f3cf67199e8b9..1d40575176670 100644
--- a/llvm/test/CodeGen/NVPTX/atomicrmw-ignore-denormal-mode.ll
+++ b/llvm/test/CodeGen/NVPTX/atomicrmw-ignore-denormal-mode.ll
@@ -1,6 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=CHECK,STRICT
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | FileCheck %s --check-prefixes=CHECK,DEFAULT
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=CHECK,DEFAULT
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | FileCheck %s --check-prefixes=CHECK,STRICT
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=false | %ptxas-verify -arch=sm_90 %}
; PTX atom.add has a fixed FTZ behavior that need not match the function's
@@ -13,9 +14,7 @@
; atom.add.ftz.f16)
; When the two disagree the atomic is expanded into a CAS loop, unless the
; mismatch is explicitly declared insignificant -- either globally via
-; -nvptx-allow-ftz-atomics (on by default, hence the STRICT run lines above,
-; without which the metadata would never be the deciding factor) or per
-; instruction via !atomic.ignore.denormal.mode.
+; -nvptx-allow-ftz-atomics or per instruction via !atomic.ignore.denormal.mode.
; f32, global: the native instruction flushes, so an IEEE function needs the
; metadata to avoid the CAS loop.
diff --git a/llvm/test/CodeGen/NVPTX/atomicrmw-sm60.ll b/llvm/test/CodeGen/NVPTX/atomicrmw-sm60.ll
index fedb1ba54f8a8..85d7f8f2f5243 100644
--- a/llvm/test/CodeGen/NVPTX/atomicrmw-sm60.ll
+++ b/llvm/test/CodeGen/NVPTX/atomicrmw-sm60.ll
@@ -1,10 +1,10 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM60,SM60-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM60,SM60-NOFTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 | FileCheck %s --check-prefixes=SM60,SM60-NOFTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 | FileCheck %s --check-prefixes=SM60,SM60-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM60,SM60-FTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM60,SM60-FTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM60,SM60-FTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM60,SM60-FTZ-DISALLOW
; RUN: %if ptxas-sm_60 && ptxas-isa-5.0 %{ llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -nvptx-allow-ftz-atomics=false | %ptxas-verify -arch=sm_60 %}
; RUN: %if ptxas-sm_60 && ptxas-isa-5.0 %{ llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 -nvptx-allow-ftz-atomics=true | %ptxas-verify -arch=sm_60 %}
; RUN: %if ptxas-sm_60 && ptxas-isa-5.0 %{ llc < %s -march=nvptx64 -mcpu=sm_60 -mattr=+ptx50 | %ptxas-verify -arch=sm_60 %}
diff --git a/llvm/test/CodeGen/NVPTX/atomicrmw-sm70.ll b/llvm/test/CodeGen/NVPTX/atomicrmw-sm70.ll
index 12ba9b5fb4f8c..e40d78b4effaa 100644
--- a/llvm/test/CodeGen/NVPTX/atomicrmw-sm70.ll
+++ b/llvm/test/CodeGen/NVPTX/atomicrmw-sm70.ll
@@ -1,10 +1,10 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM70,SM70-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM70,SM70-NOFTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 | FileCheck %s --check-prefixes=SM70,SM70-NOFTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 | FileCheck %s --check-prefixes=SM70,SM70-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM70,SM70-FTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM70,SM70-FTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM70,SM70-FTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM70,SM70-FTZ-DISALLOW
; RUN: %if ptxas-sm_70 && ptxas-isa-6.3 %{ llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -nvptx-allow-ftz-atomics=false | %ptxas-verify -arch=sm_70 %}
; RUN: %if ptxas-sm_70 && ptxas-isa-6.3 %{ llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 -nvptx-allow-ftz-atomics=true | %ptxas-verify -arch=sm_70 %}
; RUN: %if ptxas-sm_70 && ptxas-isa-6.3 %{ llc < %s -march=nvptx64 -mcpu=sm_70 -mattr=+ptx63 | %ptxas-verify -arch=sm_70 %}
diff --git a/llvm/test/CodeGen/NVPTX/atomicrmw-sm90.ll b/llvm/test/CodeGen/NVPTX/atomicrmw-sm90.ll
index ab31e4a297bfa..599d775d45da7 100644
--- a/llvm/test/CodeGen/NVPTX/atomicrmw-sm90.ll
+++ b/llvm/test/CodeGen/NVPTX/atomicrmw-sm90.ll
@@ -1,10 +1,10 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM90,SM90-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM90,SM90-NOFTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | FileCheck %s --check-prefixes=SM90,SM90-NOFTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | FileCheck %s --check-prefixes=SM90,SM90-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM90,SM90-FTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM90,SM90-FTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM90,SM90-FTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM90,SM90-FTZ-DISALLOW
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=false | %ptxas-verify -arch=sm_90 %}
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 -nvptx-allow-ftz-atomics=true | %ptxas-verify -arch=sm_90 %}
; RUN: %if ptxas-sm_90 && ptxas-isa-8.7 %{ llc < %s -march=nvptx64 -mcpu=sm_90 -mattr=+ptx87 | %ptxas-verify -arch=sm_90 %}
diff --git a/llvm/test/CodeGen/NVPTX/atomicrmw.py b/llvm/test/CodeGen/NVPTX/atomicrmw.py
index cad1573550a6c..03edcb1eb8d84 100644
--- a/llvm/test/CodeGen/NVPTX/atomicrmw.py
+++ b/llvm/test/CodeGen/NVPTX/atomicrmw.py
@@ -62,10 +62,10 @@
run_statement = Template(
"""; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM${sm},SM${sm}-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM${sm},SM${sm}-NOFTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} | FileCheck %s --check-prefixes=SM${sm},SM${sm}-NOFTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} | FileCheck %s --check-prefixes=SM${sm},SM${sm}-NOFTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=SM${sm},SM${sm}-FTZ-DISALLOW
; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -denormal-fp-math-f32=preserve-sign -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=SM${sm},SM${sm}-FTZ-ALLOW
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM${sm},SM${sm}-FTZ-ALLOW
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -denormal-fp-math-f32=preserve-sign | FileCheck %s --check-prefixes=SM${sm},SM${sm}-FTZ-DISALLOW
; RUN: %if ptxas-sm_${sm} && ptxas-isa-${ptxfp} %{ llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -nvptx-allow-ftz-atomics=false | %ptxas-verify -arch=sm_${sm} %}
; RUN: %if ptxas-sm_${sm} && ptxas-isa-${ptxfp} %{ llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} -nvptx-allow-ftz-atomics=true | %ptxas-verify -arch=sm_${sm} %}
; RUN: %if ptxas-sm_${sm} && ptxas-isa-${ptxfp} %{ llc < %s -march=nvptx64 -mcpu=sm_${sm} -mattr=+ptx${ptx} | %ptxas-verify -arch=sm_${sm} %}
diff --git a/llvm/test/CodeGen/NVPTX/atomics.ll b/llvm/test/CodeGen/NVPTX/atomics.ll
index e1853f1b00cf3..8b42e10c1f65e 100644
--- a/llvm/test/CodeGen/NVPTX/atomics.ll
+++ b/llvm/test/CodeGen/NVPTX/atomics.ll
@@ -1,7 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
; RUN: llc < %s -mtriple=nvptx64 -mcpu=sm_32 -nvptx-allow-ftz-atomics=false | FileCheck %s --check-prefixes=CHECK,DISALLOW
; RUN: llc < %s -mtriple=nvptx64 -mcpu=sm_32 -nvptx-allow-ftz-atomics=true | FileCheck %s --check-prefixes=CHECK,ALLOW
-; RUN: llc < %s -mtriple=nvptx64 -mcpu=sm_32 | FileCheck %s --check-prefixes=CHECK,ALLOW
+; RUN: llc < %s -mtriple=nvptx64 -mcpu=sm_32 | FileCheck %s --check-prefixes=CHECK,DISALLOW
; RUN: %if ptxas %{ llc < %s -mtriple=nvptx64 -mcpu=sm_32 -nvptx-allow-ftz-atomics=false | %ptxas-verify %}
; RUN: %if ptxas %{ llc < %s -mtriple=nvptx64 -mcpu=sm_32 -nvptx-allow-ftz-atomics=true | %ptxas-verify %}
; RUN: %if ptxas %{ llc < %s -mtriple=nvptx64 -mcpu=sm_32 | %ptxas-verify %}
|
chsigg
force-pushed
the
users/chsigg/nvptx-default-disallow-ftz-atomics
branch
from
October 13, 2026 09:52
f381ff6 to
e911c5e
Compare
🪟 Windows x64 Test Results
✅ The build succeeded and all tests passed. |
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
chsigg
force-pushed
the
users/chsigg/nvptx-default-disallow-ftz-atomics
branch
2 times, most recently
from
October 14, 2026 04:41
7f70bc7 to
fbf3884
Compare
Contributor
There was a problem hiding this comment.
I'm not really convinced this is the best approach. If we're just going to have all frontends emit the ignore denormal mode metadata, then why bother to change the default?
I had AI do a scan of other project frontends we would probably need to audit and update:
Producer | What needs attention
-- | --
MLIR higher-level atomics | MemRef→LLVM, OpenACC→LLVM, and SCF→OpenMP reductions create atomic RMWs without setting it.
Flang CUDA Fortran | genAtomicAdd → genAtomBinOp emits unannotated llvm.atomicrmw fadd. This looks like the clearest missing counterpart to the Clang CUDA fix.
Clang OpenMP GPU reductions | The atomic reduction combiner directly calls CreateAtomicRMW, bypassing Clang’s target metadata hook.
LLVM’s legacy NVVM intrinsic upgrades | AutoUpgrade.cpp converts old nvvm.atomic.load.add.* and scoped atomic-add intrinsics into unannotated RMWs. Existing bitcode needs coverage too, independently of frontend updates.
Producer What needs attention
MLIR higher-level atomics [MemRef→LLVM](https://github.com/llvm/llvm-project/blob/975df9c2ea08/mlir/lib/Conversion/MemRefToLLVM/MemRefToLLVM.cpp#L1987), [OpenACC→LLVM](https://github.com/llvm/llvm-project/blob/975df9c2ea08/mlir/lib/Conversion/OpenACCToLLVM/ACCAtomicPatterns.cpp#L709), and [SCF→OpenMP reductions](https://github.com/llvm/llvm-project/blob/975df9c2ea08/mlir/lib/Conversion/SCFToOpenMP/SCFToOpenMP.cpp#L250) create atomic RMWs without setting it.
Flang CUDA Fortran [genAtomicAdd → genAtomBinOp](https://github.com/llvm/llvm-project/blob/975df9c2ea08/flang/lib/Optimizer/Builder/CUDAIntrinsicCall.cpp#L685) emits unannotated llvm.atomicrmw fadd. This looks like the clearest missing counterpart to the Clang CUDA fix.
Clang OpenMP GPU reductions The [atomic reduction combiner](https://github.com/llvm/llvm-project/blob/975df9c2ea08/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp#L1890) directly calls CreateAtomicRMW, bypassing Clang’s target metadata hook.
LLVM’s legacy NVVM intrinsic upgrades [AutoUpgrade.cpp](https://github.com/llvm/llvm-project/blob/975df9c2ea08/llvm/lib/IR/AutoUpgrade.cpp#L3313) converts old nvvm.atomic.load.add.* and scoped atomic-add intrinsics into unannotated RMWs. Existing bitcode needs coverage too, independently of frontend updates.
There's also some non-llvm-project open source projects.
chsigg
force-pushed
the
users/chsigg/nvptx-default-disallow-ftz-atomics
branch
from
October 16, 2026 12:48
fbf3884 to
9a618da
Compare
Contributor
Author
|
Thank you for looking into the other frontends and passes!
For the other cases from your scan, I think not setting the metadata is what we want:
Downstream compilers that emit |
llvmorg-github-actions
Bot
added
flang
This was referenced Oct 16, 2026
[NVPTX] Attach !atomic.ignore.denormal.mode when upgrading legacy NVVM atomic add intrinsics
#232725
Merged
chsigg
changed the base branch from
main
to
users/chsigg/flang-cuda-atomic-ignore-denormal-mode
October 16, 2026 14:20
chsigg
force-pushed
the
users/chsigg/nvptx-default-disallow-ftz-atomics
branch
from
October 17, 2026 15:55
9a618da to
f782788
Compare
chsigg
force-pushed
the
users/chsigg/flang-cuda-atomic-ignore-denormal-mode
branch
from
October 17, 2026 15:55
f382e48 to
69eddc6
Compare
chsigg
added a commit
that referenced
this pull request
Oct 17, 2026
…M atomic add intrinsics (#232725) Legacy NVVM floating-point atomic add intrinsics (`@llvm.nvvm.atomic.load.add.f32.p*`, `@llvm.nvvm.atomic.load.add.f64.p*`, and `@llvm.nvvm.atomic.add.gen.f.*`) specifically targeted the PTX `atom.add` instruction, which flushes f32 denormals to zero on global memory. Attach `!atomic.ignore.denormal.mode` metadata when auto-upgrading these intrinsics to `atomicrmw fadd` so that they continue to lower to native `atom.add` instructions when `-nvptx-allow-ftz-atomics` defaults to `false` (#231429), matching the Clang lowering of `__nvvm_atom_add_gen_f` (#217587).
Base automatically changed from
users/chsigg/flang-cuda-atomic-ignore-denormal-mode
to
main
October 18, 2026 06:37
chsigg
added a commit
that referenced
this pull request
Oct 18, 2026
…232726) CUDA Fortran `atomicadd` has the same device semantics as CUDA C++ `atomicAdd()`, targeting PTX `atom.add` (which flushes f32 denormals to zero on global memory). Set `ignore_denormal_mode` on `mlir::LLVM::AtomicRMWOp` when lowering CUDA Fortran `atomicadd` for floating-point types in `genAtomBinOp`, matching Clang's CUDA/HIP `atomicAdd` lowering (#217587) so it continues to emit native `atom.add.f32` when `-nvptx-allow-ftz-atomics` defaults to `false` (#231429).
This restores the strict FTZ default originally introduced in #200732 and temporarily relaxed in #206154. Frontends can now explicitly opt in to native FTZ-mismatched atomics on a per-instruction basis using `!atomic.ignore.denormal.mode` metadata (#217585, #217586, #217587, #217588, #232725, #232726), which is supported in Clang, Flang, MLIR, and XLA. Users who need the previous behavior globally can still pass `-nvptx-allow-ftz-atomics=true`. Update test cases to verify the new default disallow behavior.
chsigg
force-pushed
the
users/chsigg/nvptx-default-disallow-ftz-atomics
branch
from
October 18, 2026 06:46
f782788 to
3870e77
Compare
Member
The escape hatch option was preserving status quo because we didn't have any way to express the specific behavior customers may want. Now we have it, and it's time to switch to the correct-by-default behavior. If some front-end (or any other LLVM consumer) chooses to ignore denormals on atomics, it's up to them how they do that, but it's not LLVM's job, IMO, to codegen demonstrably incorrect code, even if it happens to be fast. |
Artem-B
approved these changes
Oct 24, 2026
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/198/builds/15825 Here is the relevant piece of the build log for the reference |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This restores the strict FTZ default originally introduced in #200732
and temporarily relaxed in #206154.
Frontends can now explicitly opt in to native FTZ-mismatched atomics on a
per-instruction basis using
!atomic.ignore.denormal.modemetadata (#217585,#217586, #217587, #217588, #232725, #232726), which is supported in Clang,
Flang, MLIR, and XLA. Users who need the previous behavior globally can still
pass
-nvptx-allow-ftz-atomics=true.Update test cases to verify the new default disallow behavior.