test: verify-test-cuda is red on clean main because the gate leaves MLX TF32 enabled

Problem / Background

make verify-test-cuda is the documented Linux/NVIDIA merge gate, and it is structurally red on a clean main. Four numeric parity tests fail only because MLX defaults TF32 on and the gate does not set MLX_ENABLE_TF32=0.

Verified first-hand on main at 0f727572 (GB10, CUDA sm_121, Linux aarch64). Full run: 7655 passed, 5 failed. Four of the five failures are TF32 artifacts, not defects. Each one fails with the gate as written and passes with MLX_ENABLE_TF32=0, verified individually with --exact:

models::bailing_moe_linear::tests::chunked_gla_matches_the_sequential_recurrence
models::deepseek_v2::tests::absorbed_mla_attention_matches_the_decompressed_block_step_for_step
models::florence2::florence2_tests::incremental_decode_matches_full_sequence
models::klear::tests::the_prefill_is_causal_without_being_handed_a_mask

The failure magnitudes are consistent with TF32's 10-bit mantissa rather than with a logic error:

klear:     logit 25 at position 0 changed when later tokens were added: -0.1357227 vs -0.13684261 (about 1.1e-3)
florence2: incremental decode diverged from full-sequence decode, max abs diff 0.00014263391

make verify-test-cuda (Makefile:572-575) resolves to cargo test --workspace --profile test-fast --features cuda --no-fail-fast -- --test-threads=1 and sets no TF32 environment variable, so it inherits MLX's default of TF32 enabled.

The fifth failure, execution::memory_estimate::tests::resolve_block_budget_explicit_bytes_floors_to_block_count, is a genuine pre-existing integer underflow and is filed separately. It is not TF32-related and fails identically with TF32 off, so it is out of scope here.

Note that src/lib/mlxcel-xla/README.md already records that MLX CUDA's default FAST_TF32 mode is an intentional, lower-precision throughput policy that is not labeled as F32 evidence. The gate as written contradicts that documented project policy.

Why this matters

The merge gate is the project's real correctness check, because CI does not run tests or clippy for this repository. A gate that is red on a clean tree trains everyone to ignore it, and it makes "did my PR break something?" unanswerable without a manual control run against main every single time. During the #847/#848/#849 VLM port chain these five failures had to be control-run on main three separate times to prove they were not introduced by the branch under review.

Proposed Solution

This is a real fork, and both options have a genuine cost. The choice belongs to the maintainer.

Option 1: set MLX_ENABLE_TF32=0 in the verify-test-cuda target. Cheapest, one line, and it matches the project's stated policy that a TF32 result is not F32 evidence. Cost: the gate then stops exercising the precision that production actually runs with by default, so a real TF32-only regression anywhere in the workspace would not be caught by the gate.

Option 2: keep TF32 on at the gate level and fix the four tests. Either widen their tolerances to something defensible for a 10-bit mantissa, or set TF32 off per-test inside just these four tests. This preserves default-precision coverage for the rest of the suite while making the four parity tests meaningful.

Recommendation: option 2, scoped per-test. These four are same-input parity comparisons where reduced mantissa is a known and irrelevant confound, so disabling TF32 inside them (or justifying a wider tolerance) targets exactly the confound. The other 7655 tests keep running at production precision, which is where a TF32-specific regression would actually show up. Whichever option is chosen, apply it deliberately rather than by fitting numbers to the current failure output.

Acceptance Criteria

make verify-test-cuda passes on a clean main on GB10, or fails only on genuine defects.
Whichever option is chosen is documented at the gate's definition in the Makefile, so the next person does not rediscover this.
If tolerances are widened, each new tolerance has a stated numeric justification (expected error given a 10-bit mantissa over the relevant reduction length) rather than being fitted to the observed failure.
The four tests still fail if the underlying math actually regresses. Verify by deliberately perturbing one of them and confirming it is caught.
The fix is verified on GB10 with a full make verify-test-cuda run, not only on the four tests in isolation.

Technical Considerations

Gate definition: Makefile:572-575 (verify-test-cuda), with the surrounding rationale comment at Makefile:528-535.
The macOS counterpart verify-test runs --features metal,accelerate and is not affected by TF32, so any change here should not silently diverge the two gates' semantics without a note.
The four affected tests live in src/lib/mlxcel-core under models::bailing_moe_linear, models::deepseek_v2, models::florence2, and models::klear.
Reduced-precision policy precedent is recorded in src/lib/mlxcel-xla/README.md, which sets MLX_ENABLE_TF32=0 for reference-parity work for exactly this reason.

test(harness): pin f32 GEMMs to full precision, and accept minijinja'…

CUDA runner validation: the gate is green, and the pin is load-bearing

This is the CUDA-runner half that #1260 explicitly did not claim. Run on the hardware this issue was filed from.

Environment. GB10, CUDA sm_121, Linux aarch64, driver 580.173.02. Commit 670512c2 (the #1260 merge), main at origin/main, working tree clean. MLX pin 9a795735. Target directory was absent, so this was a cold MLX + CUDA build: 12m51s.

1. `make verify-test-cuda` passes on a clean `main`

Run exactly as documented, unmodified, --test-threads=1 left alone.

8167 passed; 0 failed; 310 ignored   (101 test binaries/sections)

Zero FAILED, zero panicked, zero SIGABRT/SIGSEGV, zero compile errors. --no-fail-fast means nothing was hidden behind an earlier failure.

crate	passed	ignored	time
`mlxcel` (root lib)	5710	126	53.15s
`mlxcel-core`	1513	0	177.42s
`mlxcel-xla`	254	11	0.19s
`mlxcel-surgery`	129	0	0.42s
`mlxcel-mlx-pin`	26	0	0.03s

All five originally-failing tests ran (not filtered, not ignored) and passed:

models::bailing_moe_linear::tests::chunked_gla_matches_the_sequential_recurrence            ... ok
models::deepseek_v2::tests::absorbed_mla_attention_matches_the_decompressed_block_step...   ... ok
models::florence2::florence2_tests::incremental_decode_matches_full_sequence                ... ok
models::klear::tests::the_prefill_is_causal_without_being_handed_a_mask                     ... ok
execution::memory_estimate::tests::resolve_block_budget_explicit_bytes_floors_to_block...   ... ok

The fifth failure, the integer underflow this issue scoped out, was fixed separately by #1130 (8a5f061b).

2. The pin is doing real work on CUDA, verified by operator override

MLX_ENABLE_TF32=1 at 670512c2 still reproduces the red on three of the four:

models::bailing_moe_linear::tests::chunked_gla_matches_the_sequential_recurrence   ... FAILED
models::deepseek_v2::tests::absorbed_mla_attention_matches_the_decompressed_...    ... FAILED
models::florence2::florence2_tests::incremental_decode_matches_full_sequence       ... FAILED
models::klear::tests::the_prefill_is_causal_without_being_handed_a_mask            ... ok
test result: FAILED. 1 passed; 3 failed

test	failure under TF32
bailing GLA	`state[11]: chunked -0.23453808 vs recurrence -0.23468243` (about 1.4e-4)
deepseek_v2 MLA	`step 1 drifted by 0.00027007936`
florence2	`max abs diff 0.00014263391`

florence2's 0.00014263391 is identical to the value recorded in this issue at 0f727572, digit for digit. The mechanism is unchanged and deterministic, and the operator override genuinely wins over the ctor pin as designed.

So the green gate above is attributable to #1260's pin, not to something else drifting under it.

3. klear no longer reproduces under TF32, but still guards its subject

klear is the one exception, and it is worth recording why rather than leaving it as an anomaly. Its margin was always the thinnest of the four: this issue measured 1.1e-3 against a 1e-3 tolerance, roughly 10 percent over. The MLX pin has since moved from 2c46b953 to 9a795735 (2026-08-17, 168 commits), and that range touched mlx/backend/cuda/matmul.cpp. The most likely reading is that the bump nudged klear back under its bound while leaving the other three above theirs.

That makes TF32 useless as a regression probe for klear specifically, so this acceptance criterion was checked the way the issue itself proposed, by deliberately perturbing the code. src/models/klear.rs:617 selects the causal kernel with if l > 1 && mask.is_none(). Forcing that to if false makes the prefill bidirectional, which is exactly the defect the test exists to catch:

baseline    ... ok
perturbed   ... FAILED
            logit 0 at position 0 changed when later tokens were added:
            -0.34913516 vs -0.6283903. The prefill is not causal.

The delta is 0.28, about 280x the 1e-3 tolerance, so the test has enormous margin on its actual subject even though it no longer resolves a TF32-sized difference. The perturbation was reverted; the tree is back at 670512c2 clean.

Correction to this issue's body

The Technical Considerations section says the four tests live in src/lib/mlxcel-core. They do not. All four are in the root mlxcel crate, under src/models/{bailing_moe_linear,deepseek_v2,florence2/,klear}_tests.rs. This does not change the outcome, because #1260 placed the ctor in both src/lib.rs and src/lib/mlxcel-core/src/lib.rs, but a pin placed only in mlxcel-core would have left this issue open.

Acceptance criteria

make verify-test-cuda passes on a clean main on GB10. 8167 passed, 0 failed.
Documented at the gate's definition in the Makefile. Not met. The chosen approach is documented thoroughly, but only in the ctor doc comments in the two lib crates. The verify-test-cuda comment block in the Makefile does not mention TF32 or the pin at all, so somebody starting from the gate will not find it. Recommend a short paragraph in that block pointing at the ctor, which is the whole point of this criterion.
Tolerance justification. Not applicable: no tolerance was widened, the resolution was a precision pin.
The tests still fail if the math regresses. Three of four proven by the TF32 override, klear proven by deliberate perturbation.
Verified on GB10 with a full make verify-test-cuda run, not only the four tests in isolation.

Closing on the substance. The remaining Makefile documentation gap is a comment-only change and is better tracked on its own than by holding this open.

docs(make): record the TF32 pin at the verify-test-cuda gate definiti…

Acceptance criterion 2 is now met. PR #1262 (07474633) adds the missing note as a third bullet in the verify-test-cuda comment block in the Makefile, recording why neither gate sets MLX_ENABLE_TF32, that the pin lives in the two lib test binaries and therefore covers both gates without their command lines diverging, the policy it encodes, and the trap of widening a parity tolerance before checking whether the run had TF32 on. The evidence cited there is the GB10 validation above. Comment-only change; make -n verify-test-cuda emits the same recipe.

Correction: klear reproduces too, and section 3 above is wrong

The validation comment above says MLX_ENABLE_TF32=1 reds three of the four tests and treats klear as an exception, then explains that exception with the MLX pin bump. Both halves are wrong. Each test had been observed exactly once, and klear is flaky rather than fixed.

Repeating klear 10 times per mode at 07474633 on the same GB10 host, instrumented to print the max logit delta:

	`MLX_ENABLE_TF32=0` (pin active)	`MLX_ENABLE_TF32=1`
max logit delta	3.6e-7 to 7.2e-7	5.9e-4 to 2.0e-3
median	about 4.7e-7	about 1.2e-3
result	10 ok, 0 failed	2 ok, 8 failed

klear straddles its own 1e-3 tolerance under TF32 rather than sitting under it. The two green runs that produced the wrong count were the 20 percent tail. The 1.1e-3 this issue originally recorded at 0f727572 sits at the median of that distribution, so klear behaves exactly as filed and nothing about it changed.

So all four tests depend on the pin, not three. That strengthens the conclusion rather than weakening it.

The MLX-bump explanation is withdrawn. It was reasoning from a single observation, and the observation was the artifact. For the record, the elimination work behind it still holds and is worth keeping: between 0f727572 and 670512c2, src/models/klear.rs and src/models/klear_tests.rs have zero commits, the only change to causal_attention is the Metal4 predicate swap in #1196 which is Apple-only, and the layers.rs churn in that range does not touch the generic attention path. klear's CUDA numeric path is unchanged. There was simply nothing to explain.

One finding worth keeping from the repeat runs: MLX's reduced-precision kernel is nondeterministic run to run here, spanning 3.5x on this test, while the pinned full-precision path is stable to within 2e-7. The gate never sees it because the pin is active, but anyone hand-running with MLX_ENABLE_TF32=1 should expect a distribution, not a value.

PR #1263 corrects the Makefile note added by #1262 with these numbers, and adds the warning that a single green run under MLX_ENABLE_TF32=1 is not evidence that a test has stopped depending on the pin.

fix(make): correct the TF32 note, klear reproduces too and is flaky (#…

fix(make): scope the TF32 note's nondeterminism claim to klear (#1264)

fix(tests): seed synthetic model fixtures from a sorted key walk (#1266)

Navigation Menu

Repository navigation

test: verify-test-cuda is red on clean main because the gate leaves MLX TF32 enabled #1088

Problem / Background

Why this matters

Proposed Solution

Acceptance Criteria

Technical Considerations

CUDA runner validation: the gate is green, and the pin is load-bearing

1. `make verify-test-cuda` passes on a clean `main`

2. The pin is doing real work on CUDA, verified by operator override

3. klear no longer reproduces under TF32, but still guards its subject

Correction to this issue's body

Acceptance criteria

Correction: klear reproduces too, and section 3 above is wrong

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test: verify-test-cuda is red on clean main because the gate leaves MLX TF32 enabled #1088

Description

Problem / Background

Why this matters

Proposed Solution

Acceptance Criteria

Technical Considerations

Activity

inureyes commented on Aug 20, 2026

CUDA runner validation: the gate is green, and the pin is load-bearing

1. make verify-test-cuda passes on a clean main

2. The pin is doing real work on CUDA, verified by operator override

3. klear no longer reproduces under TF32, but still guards its subject

Correction to this issue's body

Acceptance criteria

inureyes commented on Aug 20, 2026

inureyes commented on Aug 20, 2026

Correction: klear reproduces too, and section 3 above is wrong

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `make verify-test-cuda` passes on a clean `main`