[inureyes]

Summary

cargo test --release --features cuda -p mlxcel-core --lib has never reached a verdict on GB10. It dies with SIGABRT partway through, at a different test and with a different CUDA error each run, so the abort reads as if whichever test happened to be running is broken. This PR makes the CUDA suite reach a verdict, makes serialization the property of the command rather than of the developer's memory, and adds the CUDA test gate that did not exist at all.

What the measurements say

Three full runs of that command on main at 6276f556, release profile, --features cuda, idle GB10 (sm_121), MLX pin 2c46b953:

Run	Threads	MLX_USE_CUDA_GRAPHS	Outcome
1	default (20)	on	SIGABRT, cudaStreamEndCapture ... failed: operation failed due to a previous error during capture
2	--test-threads=1	on	ran to a verdict: 1406 passed, 4 failed, 88.02s
3	default (20)	0	SIGABRT, cuLaunchKernelEx ... failed: invalid argument

Run 3 is the result the issue did not anticipate, and it is the one that decides the fix. Disabling graph capture does not stop the abort. It only changes which CUDA call reports it: under capture the poisoned state surfaces at cudaStreamEndCapture, without capture the same race surfaces as an invalid cuLaunchKernelEx. So the cause is the libtest harness driving MLX concurrently from many host threads, and graph capture merely selects the symptom.

The issue floated "disable capture for the test binary, for example by setting MLX_USE_CUDA_GRAPHS=0 for the suite" as a candidate remedy. It is measurably not one. This PR says so in the guard's panic message, the Makefile comment, CONTRIBUTING.md and docs/installation.md, because a future reader hitting this abort will otherwise reach for exactly that.

Two consequences worth stating plainly:

• The acceptance criterion "at least one test still exercises CUDA graph capture if capture is disabled for the rest of the suite" is moot, not skipped. Nothing gets disabled. Run 2 has capture fully on and all 1410 tests run under it, so the whole suite keeps exercising capture. There is no carve-out to arrange.
• Machine state is not the explanation, which the issue kept open as an alternative. Runs 1 and 3 abort on an idle, non-benchmarked machine and run 2 passes on the same machine minutes apart. The outcome tracks the thread count, not the machine's history.

Serialization is cheap: 88 seconds for 1410 tests. There is no throughput argument against making it the gate.

Mechanism, consistent with the note already in docs/installation.md from #642: MLX begins capture with cudaStreamCaptureModeThreadLocal (mlx/backend/cuda/device.cpp at the pinned commit), which restrains only the capturing thread and leaves the others free to issue capture-unsafe calls, while the allocator issues cudaMalloc/cudaFree on demand from whichever thread needs memory. That accounts for the moving abort site, the varying message, and scoped runs passing on the same binary.

The second hole: there was no CUDA gate to fix

make test-fast-cuda already passed --test-threads=1, but it omits --workspace. Per the workspace comment in the root Cargo.toml, the workspace root is itself the mlxcel package, so a bare cargo test resolves to -p mlxcel and never builds mlxcel-core. That target cannot run a single one of mlxcel-core's 1410 tests. make verify-test is --workspace but pins --features metal,accelerate, so it is the macOS gate.

Net: on Linux/CUDA no target ran mlxcel-core's tests, in the crate that holds the MLX cxx bridge, layers.rs, the KV cache and the quantization loaders. That is the blindness #1007 removed on macOS, still present on the other backend, and CUDA is gated nowhere in CI, not even nightly, so nothing else was covering it.

What changed

make verify-test-cuda is the Linux/NVIDIA counterpart of make verify-test, following the shape #1007 established so the two gates stay symmetrical: cargo test --workspace --profile test-fast --features cuda --no-fail-fast -- --test-threads=1. Same scope, same profile, same --no-fail-fast reporting, cuda in place of metal,accelerate, and the serialization flag embedded so the documented command cannot be run wrong.

On the --workspace question the issue raised: pulling in mlxcel-surgery and mlxcel-xla on the CUDA path is what we want, and it is close to free. Both depend on mlxcel-core, and one cargo test --workspace --features cuda unifies that into a single cuda-enabled build of it, so neither triggers a second MLX compile. mlxcel-xla's iree feature stays off, so it remains pure Rust and needs no IREE distribution. mlxcel-mlx-pin does not depend on mlxcel-core at all. Their test targets are backend-agnostic Rust, so excluding them would only mean those two crates are gated on macOS and nowhere else. This is written down in docs/installation.md next to the existing macOS explanation.

There is deliberately no verify-clippy-cuda in this PR. The CUDA lint hole is a separate hole from the CUDA test hole, #1048 is about the test gate, and cargo clippy --workspace --all-targets does not complete in a usable time on this host, so I would have been adding a target I could not run.

A harness-level guard, the_cuda_test_suite_must_run_single_threaded in the new src/lib/mlxcel-core/src/cuda_test_serialization_tests.rs, is the part that does not depend on remembering a flag. It resolves the thread count the way libtest itself does (command line --test-threads wins over RUST_TEST_THREADS, otherwise available_parallelism()) and fails by name, with the right command, whenever the CUDA suite is about to run multi-threaded. A hand-run cargo test --workspace --features cuda now gets a diagnosis instead of an anonymous abort.

Three properties worth noting:

• Being an ordinary test, it is filtered out of narrowed runs, which stay parallel. That is correct rather than a loophole: scoped subsets pass parallel, and it is whole-suite runs that abort.
• It writes the message straight to the stderr handle rather than through eprintln!. libtest captures the print macros and replays them in its end-of-run summary, and in a parallel run there may be no end of run, because the abort takes the process down first. Writing to the handle bypasses the capture. Verified: the message appears on a 20-thread run.
• MLXCEL_ALLOW_PARALLEL_CUDA_TESTS=1 downgrades it to a printed warning, mirroring MLXCEL_ALLOW_CONCURRENT_GPU_TESTS in gpu_exclusivity_tests.rs, for someone deliberately reproducing the abort.

It is gated on feature = "cuda", so macOS is untouched. The Metal hazard is the cross-process one in #1008, which gpu_exclusivity_tests already covers, and --test-threads capping is documented there as not helping it.

Documentation. CONTRIBUTING.md gains the CUDA gate command, the requirement that it be serialized, and the run-3 reason not to reach for MLX_USE_CUDA_GRAPHS=0; its Linux/CUDA sample command also gains the --test-threads=1 it was missing, so following the contributor docs literally no longer produces a SIGABRT. docs/installation.md gains the measured table, the negative result, the guard's location and the CUDA feature-resolution note. Both state that test-fast / test-fast-cuda are edit-test-loop targets and not gates. docs/environment-variables.md gains MLXCEL_ALLOW_PARALLEL_CUDA_TESTS in the Test and CI table, along with MLXCEL_ALLOW_CONCURRENT_GPU_TESTS, which the #1008 guard has always had and which was never documented there either; that file is where people grep for what an MLXCEL_* variable does, and an existing omission is a weak argument for adding a second.

Why not the other options

• .cargo/config.toml [env] RUST_TEST_THREADS = "1". Cargo's [env] cannot be conditioned on platform or feature, so this would also serialize the macOS gate, where serializing buys nothing because fix(test): mlxcel-core release test binary intermittently hangs forever, blocking later runs via the cargo lock #1008's hazard is between processes rather than between threads, and I cannot measure that cost from this host. It would also move the gate's behavior into a file the gate's own command line does not mention, which is precisely the objection the root Cargo.toml already records against using default-members for scope.
• A committed nextest profile. nextest runs each test in its own process, which recreates fix(test): mlxcel-core release test binary intermittently hangs forever, blocking later runs via the cargo lock #1008's cross-process condition, the worse of the two failure modes, and it adds a required tool to the gate.
• The Makefile target and documentation alone. That covers the documented command but leaves a hand-run cargo test --workspace --features cuda aborting anonymously, which is the case the guard exists for.

Known-failing tests: the gate is not green once it runs

Run 2 reports 4 failures. None are fixed here, and none are marked #[ignore], because hiding them would recreate exactly the blindness this issue exists to remove.

Test	Location	Message	Tracked
fused_rope_parity_tests::fused_rope_append_matches_graph_rope_across_offsets	src/lib/mlxcel-core/src/fused_rope_parity_tests.rs:87	q offset=131071: normalized rms 2.684e-3 (tol 2.0e-3), normalized max 3.663e-2 (tol 1.2e-2)	#1049
fused_rope_parity_tests::fused_rope_append_multi_token_positions_are_absolute	src/lib/mlxcel-core/src/fused_rope_parity_tests.rs:87	q seq=3 offset=131069: normalized rms 2.497e-3 (tol 2.0e-3), normalized max 3.698e-2 (tol 1.2e-2)	#1049
paged_v2::launch::launch_tests::chunk_size_does_not_change_the_answer	src/lib/mlxcel-core/src/paged_v2/launch_tests.rs:324	pages_per_chunk 1 relative error 55896.2	#1054
paged_v2::sparse::sparse_tests::an_f16_cache_matches_the_reference_within_its_own_precision	src/lib/mlxcel-core/src/paged_v2/sparse_tests.rs:424	max relative error 1.0004048 on an f16 cache	#1053

Every failure a red CUDA gate can currently report maps to a tracking issue, so the next person reading one does not have to work out which of them are accounted for. The two paged_v2 failures were unfiled when this branch was opened and are now #1053 and #1054, split rather than filed together because the triage below says they are different bugs. From a scoped serialized run on this branch (-p mlxcel-core --lib paged_v2 -- --test-threads=1, 91 passed, 1 failed):

• an_f16_cache_matches_the_reference_within_its_own_precision (fix(paged): paged_v2 sparse f16 cache decode disagrees completely with its reference #1053) reproduces in isolation. A relative error of essentially 1.0 means the output carries no relation to the reference, so this is a correctness failure rather than tolerance drift, against a tolerance already sized for f16 rounding. The assert!(outcome.is_fused(), ...) at sparse_tests.rs:421 runs before the numeric assert at 424, so the fused sparse kernel path is genuinely taken and the failure is purely numeric.
• chunk_size_does_not_change_the_answer (fix(paged): paged_v2 chunked launch fails only in the full suite, pointing at cross-test state #1054) passes both inside that 92-test scope and run entirely alone (1.73s). It fails only in the full suite. That points at leaked shared mutable state rather than a defect in the chunking path itself, which is a materially different investigation from the first one. The assert at launch_tests.rs:324 sits inside for ppc in [1, 2, 3, 4, 7, 16, 64], so the reported pages_per_chunk 1 is only the first iteration and the larger chunk sizes are never reached; the failure is not specific to chunk size 1.

Determining whether either also fails on a CPU-only build would need a separate non-CUDA MLX build, which I did not run.

Test plan

The gate itself, on GB10, run end to end.

• make verify-test-cuda: the whole workspace under [profile.test-fast] with --features cuda --no-fail-fast -- --test-threads=1, cold-building that profile first. mlxcel-core lib: 1411 passed, 4 failed, 89.35s, no SIGABRT. Every other workspace target passed, roughly 6674 further tests across the other binaries. make exited 2 on the four failures, which is what a red gate should do.

Everything below on the same host, release profile, --features cuda, narrow targets only.

• Guard passes serialized: cargo test --release --features cuda -p mlxcel-core --lib cuda_test_serialization -- --test-threads=1 gives 5 passed, 0 failed.
• Guard fails as intended at 20 threads: same command without the flag gives 4 passed, 1 failed, with the full diagnosis and both commands to run.
• The diagnosis survives libtest's capture: the direct stderr write appears in the 20-thread run's output.
• Escape hatch works: MLXCEL_ALLOW_PARALLEL_CUDA_TESTS=1 on the 20-thread run gives 5 passed with a printed warning.
• Scoped runs are unaffected: -p mlxcel-core --lib paged_v2 -- --test-threads=1 runs 92 tests with the guard filtered out.
• cargo fmt --all -- --check clean.
• cargo clippy -p mlxcel-core --release --tests --features cuda -- -D warnings clean, so the new module's test target carries no lint debt (this is the target --all-targets without --workspace would not have compiled).
• cargo tree --workspace --features cuda resolves all five members, so the gate's feature selection is valid.
• make -n verify-test-cuda emits the intended command; make help lists the new target.
• python3 scripts/ci/check_cross_repo_refs.py clean.

What the end-to-end gate run settles

Two gaps this PR opened with, both now closed by that run.

The serialization result carries across profiles. The three diagnostic runs were taken under [profile.release], while the gate builds under [profile.test-fast]. Those differ in LTO and codegen units, and this repository has a standing note that a defect can reproduce under one and not the other, so the gate could have been resting on a profile nothing had tested. It is not: 1411 passed, 4 failed, 89.35s, no abort, against 1406/4/88.02s under release.

The known-failing list is accurate for the command a contributor actually runs, not only for the release-profile command the diagnosis used. The same four failures reproduce under test-fast, at the same assertion sites: fused_rope_parity_tests.rs:87 twice (#1049), paged_v2/launch_tests.rs:324 (#1054), paged_v2/sparse_tests.rs:424 (#1053). Nothing else in the workspace is red.

One incidental result worth recording: no other test binary aborted either, so the root mlxcel suite having no guard of its own did not bite on this run. The guard covers mlxcel-core because that is where the aborts were measured; whether the root suite needs one is unresolved rather than answered.

Not verified

The fresh-boot half of the acceptance criterion is still not met. The criterion asks for a verdict "both on a fresh boot and after a benchmarking session". Every run here, the three diagnostic ones and the gate run, is on a machine that has been up for days with builds and several full suite runs the same day, which covers the "after a session" half and arguably covers it harder than the criterion intended. A fresh-boot run needs a reboot, which is the maintainer's call, and prior sessions on this host recorded driver-level instability accumulating after heavy runs.

Closes #1048

[inureyes]

Implementation Review Summary

Intent

Make the Linux/CUDA merge gate exist and reach a verdict: add make verify-test-cuda, embed the serialization the suite needs, and guard against running it parallel by hand.

Source-level review only. A make verify-test-cuda run was in flight on this host while I reviewed, so I ran no cargo command and could not re-derive anything that needs a build.

Verified against the diff

• The target is correct and complete: --workspace, --profile test-fast, --features cuda, --no-fail-fast, -- --test-threads=1. make -n verify-test-cuda emits exactly that. mlxcel and mlxcel-core both define a cuda feature, which is the same shape verify-test already relies on for metal,accelerate, so the feature selection resolves the way the docs claim.
• Discoverability is real, not just a comment: verify-test-cuda matches the help target's (test|check|lint|clippy) grep, so it lists under Test Targets as well as in the added Examples lines. make verify itself matched none of those greps before, which the new Examples line fixes.
• declared_test_threads does mirror libtest. libtest parses --test-threads with getopts::optopt and falls back to RUST_TEST_THREADS then available_parallelism(), and opt_str returns the first occurrence. The guard matches on all of that: both --test-threads=N and --test-threads N, first-occurrence wins, options accepted after positionals, CLI over env over parallelism. The divergences I could construct are all benign: a literal -- before the flag makes libtest treat it as a filter that selects zero tests, 0 and negative values make libtest exit with an error, and a malformed RUST_TEST_THREADS makes libtest panic. None of them yields a false pass on a run that would actually abort.
• No new #[ignore], no should_panic, no tolerance change anywhere in the diff. The only Rust touched is the new file plus its mod declaration.
• The MLX_USE_CUDA_GRAPHS=0 explanation reflects run 3 in every place it appears (Makefile comment, CONTRIBUTING, installation.md, module docs, panic message), not the issue's superseded capture hypothesis.
• No AI attribution and no em dash in the commit message or PR body. No new CLAUDE.md reference.

Fixed on this branch (d61f895, docs only)

• The narrow-run claim was stated as absolute (MEDIUM). Filtering the guard out is a property of libtest's substring match, so --lib paged_v2 is safe by construction, but a filter that does match the guard's name selects it and fails a scoped run that would have been fine. cuda is the realistic case: --features cuda --lib cuda is a plausible narrowing and this crate already has cuda_l2_is_an_explicitly_unimplemented_stub, cuda_graph_cache_default_matches_build_feature and selector_cuda_backend_native_on_single_slab_only, among others. CONTRIBUTING and installation.md now name the exception and point at MLXCEL_ALLOW_PARALLEL_CUDA_TESTS=1 for it.
• "CUDA is gated nowhere in CI" narrowed to CUDA tests (LOW). Step 4 of the same section says CUDA verification "stays exclusive to release.yml", which is true of the build (cargo build --release --features cuda --locked on two self-hosted NVIDIA runners) and not of tests. Without the qualifier the two sentences read as contradicting each other.

Left for you, not fixed

• The guard covers mlxcel-core only (MEDIUM). Its stated benefit is that a hand-run cargo test --workspace --features cuda "gets a diagnosis instead of an anonymous abort", but the root mlxcel test binary drives MLX too and carries no guard. If cargo runs the root binary first and that one aborts, without --no-fail-fast the run ends before mlxcel-core starts and no diagnosis is printed. The measured aborts are all from -p mlxcel-core --lib, so the claim is established for that invocation and inferred for --workspace. Worth either measuring or softening; a second copy of the guard in the root crate would settle it.
• The guard is best effort against an abort that beats it (LOW). libtest sorts tests by name before dispatching, so cuda_test_serialization_tests::... runs early, and both observed abort sites (drafter::, ffi_tests::) sort after it. But hundreds of tests sort before it, so an early abort would take the process down first. Not a false green (the process still exits non-zero), just a missing message. The module docs cover the capture problem but not this one.
• MLXCEL_ALLOW_PARALLEL_CUDA_TESTS is not in docs/environment-variables.md (LOW). It is in the module docs and, more usefully, in the failure message itself. MLXCEL_ALLOW_CONCURRENT_GPU_TESTS is not in that table either, so this is consistent rather than a regression; adding both to the "repository's own tests and automation" table would close it.
• make test, make test-verbose and make test-lib are still described as "Run all tests" (LOW). They are root package only and build no backend feature at all, so on a CUDA host they are the most misleadingly named blind targets in the file. Out of fix(test): the full mlxcel-core lib suite aborts under CUDA graph capture, so the real merge gate does not run #1048's scope, but the "not a gate" note the PR adds for test-fast* applies to them at least as strongly.
• Two #1049 rows in the PR body's known-failure table have empty Location and Message cells (LOW). The table is doing honesty work; the blanks undercut it slightly.
• installation.md now says the cuLaunchKernelEx result three times (LOW). The paragraph above the new table already carried it from chore(mlx): bump pin to e9463bb and rebase CUDA overlays (#625) #642, so the table and the paragraph after it restate the same two messages.

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow (module wired into lib.rs, target in help and in both docs)
• Project conventions followed (license header, #[path] module style, nested mod tests, inline format args, all matching gpu_exclusivity_tests.rs)
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass. Not checked and not claimable here: the gate is red by design with 4 tracked failures (fix(cuda): fused_rope_append diverges from the graph RoPE reference at offsets near 2^17 #1049 x2, fix(paged): paged_v2 sparse f16 cache decode disagrees completely with its reference #1053, fix(paged): paged_v2 chunked launch fails only in the full suite, pointing at cross-test state #1054), and I ran no cargo command.

Could not verify without a build

make verify-test-cuda end to end, whether the serialization result carries from [profile.release] (where all three measurements were taken) to [profile.test-fast] (what the gate uses), whether the four failures reproduce under test-fast, and the fresh-boot half of the acceptance criterion. The PR body already discloses the first and the last of those.