[inureyes]

Summary

Makes CUDA compute capability a first-class, testable value in mlxcel. Before this change grep -rn "compute_capability" src/ --include=*.rs returned nothing: every architecture decision was delegated to MLX's C++ gates, so mlxcel could not report which architecture it was running on, did not record which architectures it had been compiled for, and failed with an opaque CUDA load error at the first kernel launch when the two did not agree. This is plumbing and diagnostics only. No kernel or dispatch behavior changes; acting on the value is left to the consumers in epic #1536.

What changed

Runtime probe. mlxcel_core::hardware::cuda_compute_capability() -> Option<(u32, u32)> (implemented in the new src/lib/mlxcel-core/src/cuda_arch.rs, re-exported from hardware.rs) returns the running GPU's compute capability, cached in a OnceLock because it cannot change within a process. It is backed by a new gpu_compute_capability(index) bridge function that reads MLX's own device_info map, whose CUDA backend already fills compute_capability_major / compute_capability_minor from the cudaDeviceGetAttribute values it caches per device, so no second CUDA call is issued and no CUDA header is required. The same symbol links on Metal and CPU-only builds, where those keys are absent and the probe reports None. Reading device properties never loads a cubin, which is what keeps the probe callable on exactly the binaries the mismatch check has to report on.

Build-time record. build.rs now resolves the CUDA architecture list once in main via a new resolve_cuda_architectures(), passes that same string to CMake, and records it in the crate as cargo:rustc-env=MLXCEL_CUDA_ARCHITECTURES=<list>. The list CMake compiles for and the list the binary reports are therefore the same string by construction. Empty off CUDA.

Mismatch check. cuda_arch_mismatch() compares the running device against the compiled list using CUDA's own compatibility rules: a cubin carries forward across minor revisions within one major version, PTX JITs forward including across majors, a-suffixed (architecture-specific) entries cover their exact target only, and f-suffixed (family-specific) entries carry forward within their major. enforce_cuda_arch_compatibility() turns a no-coverage verdict into a named error, and the new initialize_runtime_checked() calls it at device init, so generate, chat, embed, rerank, detect and the server refuse to start with a message naming both the compiled list and the device found, instead of continuing to a CUDA load failure that names neither. Two safety valves are deliberate: an architecture list that parses to nothing (native, all-major, or a spelling this parser has not learned) is treated as unknown rather than as "covers nothing", so it can never manufacture a startup failure; and MLXCEL_DEVICE=cpu bypasses the refusal, since running on the CPU is the one workaround left to someone holding an archive built for the wrong architecture.

Diagnostics. The startup device report now carries the architecture picture next to Detected N GPU(s), in both the CLI (src/commands/generate.rs) and the server (src/server/startup.rs, which also emits it as structured fields alongside the existing Hardware capabilities detected record). MLXCEL_TRACE_ARCH (presence-enables, matching the other diagnostic switches) prints the resolved capability, the compiled list, and the coverage verdict once per process to stderr. The CUDA QuantizedMatmul overlay (src/lib/mlx-cpp/patches/mlx/backend/cuda/quantized/quantized.cpp) additionally reports which quantized-matmul path the dispatcher picked on its first call, which is the most architecture-dependent choice in decode and which MLX exposes no hook for; its environment read is cached because it sits on the decode path.

Docs. docs/environment-variables.md gains the MLXCEL_TRACE_ARCH row; docs/installation.md cross-references the new refusal from the CUDA architecture selection section, naming the two cases that produce it (a published x86_64 archive on a pre-Ampere card, and a source build made where nvidia-smi was unavailable and build.rs fell back to 90a).

Test plan

Host: Tesla V100-PCIE-32GB (compute capability 7.0, sm_70), CUDA 12.9.41, driver 575.51.03, x86_64. This is the only GPU on the machine, which bounds what could be verified; see the section below.

• cargo test -p mlxcel-core --release --features cuda --lib cuda_arch (26 passed). The parser and coverage tests cover exact match, PTX-forward-compatible match, 90a-suffixed entries, f-suffixed entries, -real / -virtual qualifiers, cubin-preferred-over-PTX, the no-match case, and the error message. None of them need a GPU.
• The same 26 tests on a build without the cuda feature (cargo test -p mlxcel-core --release --lib cuda_arch). There the recorded architecture list is empty and the capability probe reports None, which is the same device_info-without-capability-keys path a Metal build takes.
• cargo test --release --features mlxcel/cuda --lib execution::runtime (13 passed), including that the refusal tracks cuda_arch_mismatch() exactly and that it applies only when the GPU was actually requested.
• cargo clippy --release --features mlxcel/cuda -p mlxcel-core -p mlxcel --lib --tests -- -D warnings, cargo fmt --check, git diff --check.
• MLX_CUDA_ARCHITECTURES=70 cargo build --release --features cuda, then MLXCEL_TRACE_ARCH=1 mlxcel generate -m ./models/mlx-community/qwen3.8-27B-4bit -p "Hi." -n 8. The probe reports the real device: [mlxcel arch] compute capability 7.0 (sm_70); compiled for [70]; coverage: cubin, printed exactly once across 8 decode steps, followed once by [mlxcel arch] quantized matmul path on first call: qmm_naive (device sm_70). The startup device report shows Detected 1 GPU(s). then CUDA compute capability 7.0 (sm_70); compiled for [70] (cubin).
• The same command with MLXCEL_TRACE_ARCH unset prints neither trace line and generates identically.
• MLX_CUDA_ARCHITECTURES="80;86;89;90a;100;120" cargo build --release --features cuda --bin mlxcel (the x86_64 release matrix; its libmlx.a is 249 MB against 155 MB for the single-architecture sm_70 build, so the six architectures really are compiled in), then running that binary on this V100. It exits non-zero with the new named error before touching a kernel: this mlxcel binary was compiled for CUDA architectures [80;86;89;90a;100;120] and cannot run on the GPU in this host, which is compute capability 7.0 (sm_70). No compiled architecture provides a cubin for sm_70, and none is old enough for the driver to JIT its PTX forward, so every kernel launch would fail. Rebuild for this device with MLX_CUDA_ARCHITECTURES=70 ... To run this binary as it is, set MLXCEL_DEVICE=cpu.
• The same mismatched binary under MLXCEL_TRACE_ARCH=1 reports coverage: not covered before refusing, and under MLXCEL_DEVICE=cpu starts normally on the CPU with the startup line reading compiled for [80;86;89;90a;100;120] (not covered).

Not run: the full cargo test --features cuda and cargo test suites. This unit builds MLX, CUTLASS and cuDNN from source per feature set, and each cold configuration here took 30 to 47 minutes, so validation was scoped to the modules this change touches plus the end-to-end runs above. CI covers the rest.

Deferred to GB10

This machine has exactly one GPU, a Tesla V100 at compute capability 7.0, and no Ampere-or-later device of any kind. The following acceptance criteria from #1537 therefore could not be exercised here and are left unticked rather than assumed. They belong to the ## GB10 (sm_121) continuation section of epic #1536.

• cuda_compute_capability() returns Some((12, 1)) on GB10. Only the Some((7, 0)) half was verified. The unpacking of the bridge's major * 1000 + minor encoding and the 121 list entry are covered by unit tests, but no sm_121 device ran this code.
• cuda_compute_capability() returns None on a Metal build. No macOS host was available. The nearest reachable evidence is the Linux CPU-only build above, which exercises the same branch: device_info returns a map without the two capability keys and the probe reports None. That is the identical code path, but it is not a Metal run.
• No behavior change on any existing platform with the variable unset, and the GB10 baseline unmoved. Verified only on sm_70, where generation with MLXCEL_TRACE_ARCH unset is unchanged. No GB10 or Apple Silicon baseline was measured, and no throughput claim is made for either.

Closes #1537

[inureyes]

Follow-up datum on what the check replaces, since the acceptance criterion is phrased as "the new named error rather than a CUDA load failure".

mlxcel-bench-decode is the one entry point this PR deliberately leaves on the unchecked initialize_runtime(), so building it from the same x86_64 release-matrix tree gives a like-for-like before/after on one binary. Running it on this V100 with the same checkpoint and prompt, it printed [Load] wall: 0.795 s MLX peak: 0.00 GB and then produced nothing at all for 15 minutes, holding no GPU memory, until the harness timeout killed it. The checked mlxcel generate path on the same tree refuses in well under a second with the named error, and the matching sm_70 build finishes the same 2-token run in about 15 seconds.

I am not attributing the stall to a specific CUDA call, since I did not attach a debugger and MLX also JIT-compiles some kernels with NVRTC at first use, which could account for part of it. The point that stands is the one the issue makes: without the check the failure mode carries no diagnostic naming either the architectures the binary holds or the device it found.