[BUG] All CUDA 13 test jobs fail: RMM allocation during CUDA graph capture (cudaErrorStreamCaptureUnsupported)

[ramakrishnap-nv]

Summary

As of the 2026-08-19 nightly, every CUDA 13.x test job fails and every CUDA 12.x test job passes, across C++, Python, and server suites. All failures share one signature: an RMM device allocation is issued while a CUDA stream is capturing a graph, which is illegal for a non-stream-ordered memory resource.

19 jobs that passed in the previous nightly now fail. The split by CUDA major version is exact — there are no CUDA 13 passes and no CUDA 12 failures.

Observed in

Nightly run 32220613929 — test workflow, main @ 1dbab9cf, 2026-08-19. 31 of 51 jobs failed; 19 of those are new relative to the previous nightly 32104333527 (main @ dc7113bc, 2026-08-18).

The remaining 12 failures are pre-existing third-party ones and are unrelated — see #1746, #1747, #1485.

Job family	CUDA 13.x	CUDA 12.x
conda-cpp-tests	6 / 6 fail	7 / 7 pass
conda-python-tests	6 / 6 fail	7 / 7 pass
wheel-tests-cuopt-server	6 / 6 fail	7 / 7 pass
conda-notebook-tests / build	fails	—

The 19 newly-failing jobs

conda-cpp-tests / 13.0.3, 3.12, amd64, rockylinux8, rtxpro6000, latest-driver, latest-deps
conda-cpp-tests / 13.0.3, 3.12, arm64, rockylinux8, a100, latest-driver, latest-deps
conda-cpp-tests / 13.3.0, 3.13, amd64, ubuntu26.04, h100, latest-driver, latest-deps
conda-cpp-tests / 13.3.0, 3.13, arm64, rockylinux8, a100, latest-driver, latest-deps
conda-cpp-tests / 13.3.0, 3.14, amd64, ubuntu26.04, rtxpro6000, latest-driver, latest-deps
conda-cpp-tests / 13.3.0, 3.14, arm64, ubuntu26.04, gb300, latest-driver, latest-deps
conda-notebook-tests / build
conda-python-tests / 13.0.3, 3.12, amd64, rockylinux8, rtxpro6000, latest-driver, latest-deps
conda-python-tests / 13.0.3, 3.12, arm64, rockylinux8, a100, latest-driver, latest-deps
conda-python-tests / 13.3.0, 3.13, amd64, ubuntu26.04, h100, latest-driver, latest-deps
conda-python-tests / 13.3.0, 3.14, amd64, ubuntu26.04, rtxpro6000, latest-driver, latest-deps
conda-python-tests / 13.3.0, 3.14, arm64, rockylinux8, a100, latest-driver, latest-deps
conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, gb300, latest-driver, latest-deps
wheel-tests-cuopt-server / 13.0.3, 3.12, amd64, ubuntu24.04, h100, latest-driver, latest-deps
wheel-tests-cuopt-server / 13.0.3, 3.12, arm64, ubuntu24.04, l4, latest-driver, latest-deps
wheel-tests-cuopt-server / 13.3.0, 3.13, amd64, ubuntu26.04, h100, latest-driver, latest-deps
wheel-tests-cuopt-server / 13.3.0, 3.14, amd64, ubuntu26.04, rtxpro6000, latest-driver, latest-deps
wheel-tests-cuopt-server / 13.3.0, 3.14, arm64, ubuntu26.04, gb300, latest-driver, latest-deps
wheel-tests-cuopt-server / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps

It reproduces on both CUDA 13.0.3 and 13.3.0, on amd64 and arm64, on A100 / H100 / RTX PRO 6000 / GB300 / L4, and on Python 3.12–3.14. All affected jobs are latest-driver, but so are the passing CUDA 12 jobs, so the driver is not the discriminator.

Error signatures

C++ (conda-cpp-tests, CUDA 13.3.0):

C++ exception with description "CUDA error at: /opt/conda/envs/test/include/rmm/device_uvector.hpp:220:
cudaErrorStreamCaptureUnsupported operation not permitted when stream is capturing" thrown in the test body.
[  FAILED  ] vehicle_types_float_test_t.VEHICLE_TYPES (446 ms)

C++ exception with description "extrema failed to synchronize:
cudaErrorStreamCaptureUnsupported: operation not permitted when stream is capturing" thrown in the test body.

Then, in later tests, the capture is already poisoned:

C++ exception with description "CUDA error at:
/opt/conda/envs/test/include/rmm/mr/detail/stream_ordered_memory_resource.hpp:479:
cudaErrorStreamCaptureInvalidated operation failed due to a previous error during capture"

Python (conda-python-tests):

MemoryError: std::bad_alloc: CUDA error (failed to allocate 24 bytes) at:
.../librmm/work/cpp/src/mr/cuda_memory_resource.cpp:26:
cudaErrorStreamCaptureUnsupported operation not permitted when stream is capturing

Server (wheel-tests-cuopt-server) — the solver process is killed and the server marks itself unhealthy:

CUDA error at: .../librmm/include/rmm/device_uvector.hpp:220:
cudaErrorStreamCaptureUnsupported operation not permitted when stream is capturing
...
solver process unhealthy: CUDA error encountered at: .../cudf/cpp/src/interop/from_arrow_host.cu:161:
906 cudaErrorStreamCaptureImplicit operation would make the legacy stream depend on a capturing blocking stream
solver process exiting

Notebook (conda-notebook-tests / build):

cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED:
operation not permitted when stream is capturing
Notebook last_mile_delivery/cvrp_daily_deliveries.ipynb failed to execute. Exiting.

Scope

• C++: most routing suites in ROUTING_INTERNAL_TEST — vehicle_types_float_test_t, level0_ges/* (GES_PDP, GES_VRP), level0_base/l0_float_test_t (TSP, VRP, CVRP, VRPTW, CVRPTW), all 19 level0_retail/retail_float_test_t.CVRPTW_Retail/*, level0_vehicle_order_match/*, level0_objective_function/*, top_k/top_cand_test.test_top_k/*.
• Python: 45 failures per job — all of tests/routing/* (distance engine, host arrays, initial solutions, re-routing, gRPC serialization, solver, solver settings, vehicle properties, warnings/exceptions) plus tests/socp/test_socp.py (3 tests).
• Server: routing solves fail, then the health check trips and the solver process exits.
• Notebooks: last_mile_delivery/cvrp_daily_deliveries.ipynb.

Note on where it starts

This is not a capture leaked by an earlier test. In ROUTING_INTERNAL_TEST the very first test executed — vehicle_types_float_test_t.VEHICLE_TYPES — fails immediately with cudaErrorStreamCaptureUnsupported. The routing graph-capture path is allocating inside an active capture from the start; the later cudaErrorStreamCaptureInvalidated failures are downstream cascade.

Hypotheses — not yet confirmed

cudaMalloc (what rmm::mr::cuda_memory_resource calls) is always illegal during stream capture; a stream-ordered/async or pool resource is not. So the behaviour is consistent with the effective RMM memory resource differing between the CUDA 12 and CUDA 13 test environments, rather than with a change in cuOpt's capture logic. That would explain why the split is exactly along the CUDA major version while the cuOpt source is identical.

Two candidate directions, in rough order of likelihood:

1. A change in the CUDA 13 dependency stack. These jobs consume RAPIDS nightlies, so an RMM change to the default memory resource (or to which resource is selected on CUDA 13) between the two nightly runs would produce exactly this. Worth diffing the resolved rmm/librmm build between run 32104333527 and run 32220613929.
2. One of the seven commits merged between the two nightlies, though a cuOpt-side change breaking only CUDA 13 is harder to explain:

32bb762f  Report absent solution values instead of returning success (#1734)
092662e4  Release pslp presolve memory (#1735)
c2830ef4  wheels: avoid vendoring libnvJitLink (#1741)
5a7fcf94  Build cuopt as an abi3 wheel using the CPython Limited API (#1731)
e8c1ae0b  Routing over gRPC: VRP server + compiled C++/Cython client (#1597)
1dbab9cf  test: isolate library wheel smoke test (#1624)
2b3f4038  Update cpp/benchmarks codeowners to cuopt-solver-codeowners (#1742)

Note that #1741 and #1731 are wheel-only changes while conda-* jobs also fail, which argues against both.

Why this is P0

It is new, it is a hard break rather than a flake, and it is plausibly user-facing rather than a CI artifact — if the effective memory resource on CUDA 13 really cannot serve allocations during graph capture, then routing solves are broken for CUDA 13 users, not just for CI. That should be established or ruled out before anything else.

Follow-up

• Diff the resolved RAPIDS/RMM package versions between the 2026-08-18 and 2026-08-19 nightlies
• Confirm which RMM memory resource is active in a CUDA 13 test env vs a CUDA 12 one
• If the dependency stack is exonerated, bisect the seven commits above
• Determine whether a released CUDA 13 build of cuOpt is affected, or only the nightly test environment

Possibly related

#1729 — "cuOpt 26.8.0 CU13 build aborts on MIP solves while CU12 solves the same instance" (open, 2026-08-14). Also a CUDA 13-only failure that CUDA 12 does not reproduce. Worth checking whether the two share a root cause before treating them separately.