fix(cuda): add MLXCEL_SDPA_DETERMINISTIC for reproducible decode

[inureyes]

Summary

Qwen3 temp-0 output differed from run to run on GB10 once the context passed 256 tokens (#2128), and it differed within one process too. The cause is the cuDNN engine the heuristics pick for the one-row decode step: its output depends on the order of an internal reduction, which a contributor's analysis on #2128 locates in an unordered cross-warp reduction of the softmax denominator. This PR adds MLXCEL_SDPA_DETERMINISTIC=1, which makes decode bitwise reproducible. The default is unchanged, because stream-K is also the fast path at long context (decision recorded on #2128).

Root cause

• From 256 KV positions, use_cudnn_for_decoding sends a one-row SDPA to cuDNN with k and v unsliced to the cache buffer and the true lengths in a padding mask.
• On GB10 (sm_121) with cuDNN 9.27, the first plan for the head_dim 128 graph is eng8_k40=0_k41=2_k24=2_k38=1_k27=0. Knob 38 is CUDNN_KNOB_TYPE_STREAM_K.
• Isolated, with q [1,16,1,128] and k/v 300 rows of a 512-row buffer, 4000 identical calls gave 2 distinct outputs, differing by at most 7.6e-6 (ULP level). At 1500 of 2048 rows, 446 of 2000 calls differed.
• The other candidate, eng8_k40=2_k41=1_k24=1_k38=0_k27=0 (stream-K off), gave 1 distinct output. It also differs in its tile and kernel-config knobs, so knob 38 is the measured correlate, not an isolated cause.
• Jaeyeong-CHOI's analysis on fix(cuda): Qwen3 greedy decoding is not reproducible run to run on GB10 #2128, with a standalone cuDNN reproducer on 9.13 and 9.24.1, traces the order dependence to the softmax-denominator reduction and confirms it by fixing the order inside the kernel.
• cuDNN does not tag the engine NONDETERMINISTIC, so deselect_numeric_notes keeps it (tried).
• In Qwen3-1.7B, logits were bit-identical through prefill and the first 224 decode steps, then diverged between steps 225 and 247, in every comparison. The prompt was 34 tokens, so that is just past 256 cache positions.
• Ruled out: CUDA graphs (MLX_USE_CUDA_GRAPHS=0) and the KV in-place write (MLXCEL_KV_INPLACE_WRITE=0) left it unchanged. MLX_CUDA_USE_CUDNN_SDPA=0 removed it.
• Gemma-3-1B never takes this path, since cuDNN SDPA is limited to head_dim ≤ 128. Llama-3.2-1B (head_dim 64) did not show it in these runs.

Changes

• src/lib/mlx-cpp/patches/mlx/backend/cuda/scaled_dot_product_attention.cpp: with MLXCEL_SDPA_DETERMINISTIC=1:
  • use_cudnn_for_decoding returns false, so one-row decode runs sdpa_vector, whose KV split reduces in a fixed order.
  • As a precaution, build_sdpa_graph bars stream-K candidates (_k38=<non-zero> in the engine tag) on the remaining forward graphs (prefill, a bucketed verify block). Those graphs were not shown to be nondeterministic and were not separately measured.
  • If no stream-K-off plan builds, a fresh graph is built with the default selection instead of aborting. A one-time warning names the switch, and it also fires when every candidate uses stream-K or a tag cannot be read.
  • Off by default. Unset, the dispatch is identical to main.
• tests/cuda_sdpa_determinism.rs (new binary; sets the switch before main with ctor unless the environment does):
  • 2000 identical decode SDPA calls must give one output, at 300 of 512 keys and at 1500 of 2048 keys. The second case is sdpa_vector's two-pass kernel. Neither needs a checkpoint.
  • A Qwen3 decode of 64 prompt positions and 400 steps, with logits hashed per step, skips when the checkpoint is absent.
• tests/cuda_qmm_determinism.rs keeps guarding the default dispatch. Its default checkpoint path now points at models/mlx/, where checkpoints live; the old path never existed, so the bug: Gemma 4 26B A4B temp-0 decode is non-deterministic run-to-run and corrupts near-tied tokens independently of the fused MoE kernel #910 test always skipped. fix(cuda/quant): quantized prefill is not bitwise reproducible on sm_70, and its determinism test never runs #1558 still owns making a missing checkpoint fail instead of skip.
• docs/environment-variables.md: the variable, with the measured costs and what it does not cover. The MLXCEL_SDPA_FALLBACK_MAX_QUERIES row no longer says decode never enters cuDNN.
• docs/upstream/mlx-cuda-sdpa-stream-k-nondeterminism.md: upstream report draft for MLX and cuDNN.

Cost of the two deterministic options

Decode throughput on GB10, relative to the default. Median of 3 interleaved runs on a quiet host, release build, 4-bit checkpoints.

Model	Keys	cuDNN, stream-K barred	sdpa_vector (this PR's switch)
Qwen3-1.7B	~400 / 2826 / 7366 / 15653	0.99 / 0.92 / 0.90 / 0.88	1.01 / 1.01 / 0.96 / 1.00
Qwen3-4B	~800 / 2826 / 7366 / 15653	0.97 / 0.94 / 0.92 / 0.87	1.00 / 0.91 / 0.97 / 0.94
Llama-3.2-1B	~870 / 2717 / 7189 / 15122	0.99 / 0.89 / 0.77 / 0.67	1.01 / 0.95 / 0.90 / 0.87

Prefill throughput was within noise (0.95x to 1.05x) with stream-K barred. Barring stream-K alone, which was the first version of this branch, costs more than moving decode to sdpa_vector, so the switch does the latter.

Verification

• Regression tests discriminate:
  • cuda_sdpa_determinism passes 3 of 3 with the switch.
  • With MLXCEL_SDPA_DETERMINISTIC=0 it fails 3 of 3: 11 of 2000 calls differ at 300 keys, 446 of 2000 at 1500 keys, and the Qwen3 logits diverge at decode step 204.
  • cuda_qmm_determinism passes on the default dispatch.
• CLI, the issue's command (800 tokens, temp 0), 5 separate processes per model, CUDA graphs on and off:
  • With the switch, there was 1 distinct output each for Qwen3-0.6B, 1.7B and 4B, Llama-3.2-1B and Gemma-3-1B.
  • Default Qwen3-1.7B still gave 3 distinct outputs in 3 runs, as designed.
• Long context: a 7366-token prompt with the switch was identical over 3 runs, at 114.3 tok/s against 119.0 for the default. The default figure equals the pre-change number, so the default path is unchanged.
• Server: mlxcel-server (Qwen3-1.7B, pool-backed paged decode, which does not use MLX SDPA) gave the same output for one greedy request across server restarts with the switch on and off. A repeat served from the prompt-prefix cache differed from the first, uncached run, identically in both modes. That is a separate path difference, not addressed here.
• cargo clippy --profile test-fast --features cuda -D warnings on both test targets: clean. cargo fmt --check: clean. CI on the first head: all jobs green.

Changes during review

• The security review found that a graph whose stream-K-off candidates all fail support or build would abort the process under the switch. It now falls back to the default selection with a warning.
• The switch-on tests moved to their own binary so the bug: Gemma 4 26B A4B temp-0 decode is non-deterministic run-to-run and corrupts near-tied tokens independently of the fused MoE kernel #910 test keeps covering the default path.
• The documentation now credits the contributor's root-cause analysis and narrows the claims to what was measured.

Closes #2128