CUDA error: misaligned address in ggml_cuda_op_unary<op_sigmoid> with long prompts on RTX 5090 Laptop (sm_120) — regression between d3146f2b5 and ad6c66839 #28877

Activity

1. suntao12138 commented on Sep 14, 2026

   

   suntao12138

   on Sep 14, 2026

   Author

   More actions

   Correction — please disregard the bisect conclusion above

   The "regression between d3146f2b5 and ad6c66839" claim in my original report is wrong, and I want to retract it before anyone spends time on it.

   Today build 10919 (d3146f2b5, the build I previously reported as "good") crashed with the exact same backtrace — including ggml_cuda_op_unary<&(op_sigmoid(float))> and CUDA error: misaligned address. So both builds crash; it is not a regression introduced in that commit range. It is load-dependent, which is why the shorter test on the older build appeared to pass. Apologies for the noise.

   Observed crash conditions on the failing run (build 10919, ~112 s after start):

   W srv stop: cancel task, id_task = 11004 ... 11000     (9 tasks cancelled)
   I slot release: id 0 | task 10999 | stop processing: n_tokens = 6416
   I slot release: id 1 | task 11000 | stop processing: n_tokens = 6538
   I slot launch_slot_: id 1 | task 13653 | processing task, is_child = 0
   I slot launch_slot_: id 0 | task 13652 | processing task, is_child = 0
   E CUDA error: misaligned address
     in function ggml_cuda_kernel_launch at ggml-cuda/common.cuh:1704
     cudaLaunchKernelEx(&pdl_cfg.cfg, kernel, std::forward<Args>(args)... )
   

   Under sustained use the server had processed ~13,600 tasks when it died.

   New lead: PDL kernel launch path

   The faulting launch is the PDL branch in ggml_cuda_kernel_launch (common.cuh), i.e. cudaLaunchKernelEx with cudaLaunchAttributeProgrammaticStreamSerialization. PDL is on by default and can be disabled at runtime via the GGML_CUDA_PDL env var:

   static const bool env_pdl_enabled = []() {
       const char * env = getenv("GGML_CUDA_PDL");
       return env == nullptr || std::atoi(env) != 0;
   }();
   if (env_pdl_enabled && ggml_cuda_kernel_can_use_pdl(kernel)) {
       CUDA_CHECK(cudaLaunchKernelEx(&pdl_cfg.cfg, kernel, ...));   // <-- faulting path
       return;
   }

   Current status

   I have restarted the server with GGML_CUDA_PDL=0 (falls back to plain kernel<<<...>>> launches). A first stress test of 20 rounds (4 parallel long-prompt requests per round + mid-flight request cancellation) passed with 0 CUDA errors and the server alive throughout. A longer/harder run (40 rounds, 6 parallel mixed-length requests per round, ~10 min) is in progress; I will report the result.

   Environment: RTX 5090 Laptop (sm_120, compute 12.0), driver 595.99.02, CUDA 13.0.88, GCC 13.3.0, Ubuntu 24.04.5.

   If this hypothesis is right, the programmaticStreamSerialization launch path may be unstable on sm_120 / CUDA 13.0 for some kernels (at least the unary/sigmoid one) under concurrent graph compute — happy to run any additional test or provide a fuller log if useful.
2. suntao12138 commented on Sep 14, 2026

   

   suntao12138

   on Sep 14, 2026

   Author

   More actions

   Follow-up: longer stress test passed with PDL disabled

   Result of the longer run on build 10919 with GGML_CUDA_PDL=0:

   Run	Load	Duration	CUDA errors	Server
   1 (short)	20 rounds × 4 parallel long-prompt requests, cancellation each round	~2 min	0	alive
   2 (long)	40 rounds × 6 parallel requests (mixed long + short), cancellation each round	360 s continuous	0	alive

   Combined: ~8 minutes of sustained concurrent load with PDL disabled → no faults, GPU at ~98% utilization / 60 °C throughout.

   For contrast, on the same machine with PDL enabled:

   • build 10919 → crashed after ~112 s (same backtrace as reported above)
   • build 10950 → crashed within minutes

   So GGML_CUDA_PDL=0 looks like a solid workaround here. Hypothesis to confirm: the programmaticStreamSerialization launch path (sm_120 + CUDA 13.0.88, driver 595.99.02) faults on the unary/sigmoid kernel under concurrent graph compute.

   Happy to run further targeted tests if useful, e.g.:

   • PDL enabled with --parallel 1 (to see whether concurrency is required)
   • stress with a patched ggml_cuda_kernel_can_use_pdl returning false for specific ops
   • or dump the faulting kernel/arguments with CUDA_LAUNCH_BLOCKING=1
3. github-actions

   added

   stale

   on Oct 15, 2026