Repository navigation
fix: Write the layer_norm VJP bias gradient on ROCm #2632
Description
Problem
fast::LayerNormVJP::eval_gpu in src/lib/mlxcelverse/overlays/rocm/mlx/backend/rocm/layer_norm.hip:423-429 allocates gb when has_gb is true and leaves // TODO: Implement proper column reduction for gb. Nothing writes it, so the bias gradient of fast::layer_norm on ROCm is uninitialized device memory. Found by code reading; layer_norm_vjp_past_the_row_grid_cap_matches_host in tests/rocm_hip_grid_clamp/row_grid.rs:269 checks only wrt 0 and 1, which is why it passes. Reproduce it first.
Fix
Port the CUDA path (mlx/backend/cuda/layer_norm.cu, LayerNormVJP::eval_gpu): before the VJP kernel launches, col_reduce(encoder, g, gb, Reduce::ReduceType::Sum, {0}, plan) with a ContiguousStridedReduce plan over n_rows x axis_size, then encoder.set_input_array(gb) when g was donated into gx or gw_temp. The reduction must run before the kernel because a donated g is overwritten. gb keeps the dtype MLX gives it (the bias dtype); the expected value is sum(g, all axes but the last), the fallback in mlx/fast.cpp. Reuse the existing col_reduce call used for gw at layer_norm.hip:507-510.
Acceptance Criteria
-
wrt = 2matches a host reference (column sums ofg) for f32, f16 and bf16 with weight and bias; the test fails on current main. - The same check at
n_rows > 65535, extendinglayer_norm_vjp_past_the_row_grid_cap_matches_host(therow_gridsplit of fix(rocm): split per-row softmax and norm grids past the HIP limit #2627); fails on main. - Bias without weight: make the
weightargument offast_layer_norm_vjp_for_test(src/lib/mlxcel-core/cpp/mlx_cxx_bridge.cpp:4244,src/lib/mlxcel-core/src/lib.rs:1530) a nullable*const MlxArrayasfast_scaled_dot_product_attentiondoes formask, and testgbwith no weight; fails on main. - New
LOCAL_FIXES.mditem (next number after 63) naminglayer_norm.hip; noCORE_RESIDUAL.diffchange unless a core file changes.
Verification
Implement everything in one PR with one verification pass at the end and as few pushes as possible.
scripts/rocm_gpu_guard.sh --hold -- cargo test --profile test-fast --features rocm --test rocm_hip_grid_clamp -- --test-threads=1
scripts/rocm_gpu_guard.sh --hold -- make verify-rocm
Activity
fix(rocm): write the layer_norm VJP bias gradient (#2635)