You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(test): attend_batched mask test fails on ROCm on a fully masked row #2219
Since PR #2210 (KV attention dispatch moved into the cache; #2171, Phase 4a of epic #2166, implemented on a non-ROCm host), cargo test --profile test-fast --features rocm -p mlxcel-core --lib cache::attend::attend_tests on gfx1151 gives 10 passed, 1 failed: batched_masked_rows_get_their_own_mask_row (src/lib/mlxcel-core/src/cache/attend_tests.rs:213), the assert_eq!(to_vec_f32(&got), to_vec_f32(&want)) on line 252.
The batched output and the per-row reference are bit-identical where finite and both NaN at the same 256 positions, all in batch row 1. The assertion fails only because NaN != NaN, so attend_batched is correct on ROCm.
Cause
The test masks key 0 for every query of row 1 and keys s > t causally, so query t=0 of row 1 has every key masked. The f16 mask value -1e9 rounds to -inf. On ROCm an array-masked SDPA with head dim 64 has no fused kernel (supports_sdpa_vector, supports_sdpa_flash, and supports_sdpa_flash_wmma all reject array masks), so MLX's graph fallback runs add(mask) plus softmax, and softmax over an all -inf row is NaN. On Metal the same shape (qL <= 8 with an array mask) runs sdpa_vector, which skips keys whose mask is below finite_min and returns 0 for an empty row, so both sides are 0 and the test passes.
Fix
Compare the two outputs bitwise with f32::to_bits, so NaN in the same slot counts as equal.
Keep the mask shape and document why the fully masked query is there.
Separately confirm whether production batched prefill on ROCm can produce a fully masked query row.
Acceptance Criteria
cache::attend::attend_tests passes 11 of 11 with --features rocm on gfx1151
The test still fails when attend_batched uses the wrong mask row (checked with a deliberate mutation)
Production exposure to fully masked rows on ROCm is measured and reported (separate issue if real)
Activity
fix(test): compare attend_batched mask test outputs bitwise (lablup#2221