Repository navigation
[CUDA] Deterministic illegal memory access on sm_120 (RTX 5090) with glm_moe_dsa arch (GLM-5.2) — sm_121/sm_110 unaffected #3874
Description
Summary
Running GLM-5.2 (mlx-community/GLM-5.2-mxfp4, model_type: glm_moe_dsa, DeepseekV32-style arch with DSA sparse attention + 256-expert MoE) distributed via exo's ring pipeline, the shard hosted on an RTX 5090 (sm_120, consumer Blackwell) crashes deterministically with:
RuntimeError: cudaPeekAtLastError() failed: an illegal memory access was encountered
The same code, same model, same pipeline runs fine on the other CUDA ranks in the ring: GB10/DGX Spark (sm_121) and Jetson Thor (sm_110), as well as on the Metal ranks (M4/M5 Max). Only the sm_120 device crashes.
Environment
- RTX 5090 (32 GB), CUDA 12.8, WSL2 (Ubuntu, mirrored networking)
- MLX 0.32.0 built from source with
CMAKE_ARGS="-DMLX_BUILD_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120"(the published wheels lack sm_120 cubins entirely — that part is a known separate limitation) - Model: GLM-5.2-mxfp4, 368 GB, 78 layers,
first_k_dense_replace=3,n_routed_experts=256,index_topk=2048 - 5-rank ring pipeline (2× Metal + sm_121 + sm_110 + sm_120)
Reproduction pattern (deterministic across ~6 attempts)
- sm_120 rank hosts layers [0,5) (embedding + first layers): load OK, warmup OK, first real inference request crashes with the illegal memory access.
- sm_120 rank hosts layers [44,49) (deep MoE/DSA layers): crashes already during warmup.
- After the first illegal access, the CUDA context is poisoned; subsequent kernel loads fail with:
RuntimeError: cuModuleGetFunction(&kernel, module_, mangled.c_str()) failed: an illegal memory access was encountered
- GPU itself stays healthy (standalone matmul fine after process restart; ~20 crashes total, no persistent damage).
What works standalone on the same device (ruling things out)
- Plain
matmul(bf16): OK mx.quantized_matmul(..., transpose=True, group_size=32, bits=4, mode='mxfp4')against the actual model tensors (e.g.(256, 2048, 768) uint32weights +(256, 2048, 192) uint8scales from the switch_mlp experts): compiles and runs in ~2 s for all tensors of a shard file — so the mxfp4 matmul path itself appears fine on sm_120.
This suggests the illegal access sits in another kernel family used by this arch (DSA sparse attention / MoE gather-scatter paths?) specifically on sm_120, possibly a launch-bounds/occupancy or arch-conditional issue that sm_121/sm_110 don't hit.
Happy to run instrumented builds or targeted repro snippets on the 5090 — the setup is available for testing.
Related: #3862 (ring transport fail-fast, same cluster).
Activity
Stefanfindeisen commentedon Jul 20, 2026
Correction / retraction — this is very likely VRAM exhaustion, not an sm_120 kernel bug
Follow-up after a lot more investigation. I now believe the framing of this issue is wrong and want to correct the record so nobody chases a phantom kernel bug.
Standalone, the mxfp4 MoE/DSA path runs fine on sm_120. I built an isolated repro that instantiates a real
glm_moe_dsadecoder stack with the actual 256-expert mxfp4 weights loaded from disk (not random init) and runs forward passes:index_topk) → OKNo illegal memory access in any of these. The trick to fit them was loading the already-quantized weights so the 256 unquantized experts are never materialized.
The crash correlates with VRAM, not SM version. The RTX 5090 has 32 GB; the ranks that "were unaffected" (GB10, Thor) have 128 GB. The original crash happened when the 5090 hosted a mid-pipeline shard of ~5 MoE layers (~26 GB of weights) and warmup/decode activations pushed it over 32 GB → the "illegal memory access" is what an out-of-VRAM allocation surfaces as mid-kernel. I originally attributed the difference to
sm_120; the real variable is card memory size. In a later 5-node run the 5090 loaded an 8-layer MoE-containing shard within its VRAM budget without the illegal access.Caveat (why I'm not 100% certain): I could not fully re-run inference-time decode on the 5090 in the cluster afterwards — a separate, unrelated distributed-barrier hang (#3876) blocked reaching generation. So I can't formally rule out an sm_120-specific decode-path issue, but the standalone evidence + the VRAM correlation make a kernel bug unlikely.
Closing this as not-an-mlx-bug (most likely user-side VRAM exhaustion). Happy to reopen with a clean, memory-safe decode repro if it ever resurfaces. Apologies for the noise.