A comprehensive test suite for validating CUDA and OpenMPI integration, including CUDA-aware MPI capabilities. This project helps verify that your GPU computing environment is properly configured for parallel computing applications.
This project tests the integration of CUDA 12.8 and OpenMPI 5 on your system, providing automated tests for basic CUDA operations, standard MPI communication, and CUDA-aware MPI features.
- CUDA Version: 12.8
- OpenMPI Version: 5.0.6
- GPU: NVIDIA GeForce 940M (1GB VRAM, Compute Capability 5.0)
- OS: Xubuntu 22.04
-
Basic CUDA Operations
- Device detection
- Memory allocation/deallocation
- Kernel execution
- Host-Device memory transfers
- Vector addition verification
-
Standard MPI Communication
- Host-to-host data transfer
- Data integrity verification
-
CUDA-aware MPI
- GPU-to-GPU direct communication
- Tests if OpenMPI was compiled with CUDA support
cd /media/waqasm86/External1/Project-CPP/Project-Nvidia/cuda-openmpi/mkdir -p build
cd buildcmake ..Expected output should show:
- MPI Found with paths
- CUDA Toolkit version 12.8
- Building for CUDA Architecture: 50
makeIf build succeeds, you should see: Built target cuda_mpi_test
./cuda_mpi_testThis will test basic CUDA operations without MPI communication.
mpirun -np 2 ./cuda_mpi_testThis runs with 2 MPI processes and tests all features including:
- CUDA operations on each process
- Standard MPI communication
- CUDA-aware MPI
ompi_info --parsable --all | grep mpi_built_with_cuda_supportExpected output if CUDA-aware:
mca:mpi:base:param:mpi_built_with_cuda_support:value:true
If it shows false or nothing, your OpenMPI is not CUDA-aware (which is fine for basic MPI+CUDA work).
✓ Basic CUDA operations PASSED
✓ Standard MPI communication PASSED
✓ CUDA-aware MPI PASSED
Solution: Check NVIDIA driver installation
nvidia-smiCause: GPU not properly detected Solution: Verify CUDA installation
nvcc --version
nvidia-smiCause: OpenMPI not compiled with CUDA support Impact: This is NORMAL for standard OpenMPI installations Workaround: Use host memory staging:
GPU -> Host Memory -> MPI Transfer -> Host Memory -> GPU
Solution: Ensure CUDA is in PATH
export PATH=/usr/local/cuda-12.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATHSolution: Add OpenMPI to library path
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATHYour GeForce 940M has:
- 1GB VRAM: Be careful with large data allocations
- Compute Capability 5.0: Supports most CUDA features
- Maxwell Architecture: Good for learning, limited for production
Available VRAM: ~800MB (after OS overhead)
Safe allocation per process: <400MB
Test uses: ~12KB per test (very safe)
==============================================
CUDA-MPI Integration Test Suite
==============================================
MPI Processes: 2
Number of CUDA devices: 1
Rank 0 - GPU Device 0: GeForce 940M
Compute Capability: 5.0
Total Global Memory: 1024.00 MB
Max Threads per Block: 1024
Multiprocessors: 3
CUDA Capability Major/Minor version: 5.0
Rank 1 - GPU Device 0: GeForce 940M
Compute Capability: 5.0
Total Global Memory: 1024.00 MB
Max Threads per Block: 1024
Multiprocessors: 3
CUDA Capability Major/Minor version: 5.0
==============================================
Test 1: Basic CUDA Operations
==============================================
Rank 0: Testing basic CUDA operations...
Rank 0: ✓ Basic CUDA operations PASSED
Rank 1: Testing basic CUDA operations...
Rank 1: ✓ Basic CUDA operations PASSED
==============================================
Test 2: Standard MPI Communication (Host Memory)
==============================================
Rank 0: Testing standard MPI communication...
Rank 0: Sending 1000 integers to Rank 1 via host memory...
Rank 0: ✓ Send completed
Rank 1: Testing standard MPI communication...
Rank 1: ✓ Standard MPI communication PASSED
==============================================
Test 3: CUDA-aware MPI (GPU-to-GPU)
==============================================
Rank 0: Testing CUDA-aware MPI (GPU-to-GPU)...
Rank 0: Sending 1000 integers to Rank 1 directly from GPU memory...
Rank 0: ✓ CUDA-aware MPI send completed
Rank 1: Testing CUDA-aware MPI (GPU-to-GPU)...
Rank 1: ✓ CUDA-aware MPI PASSED (Your OpenMPI supports CUDA-aware MPI!)
==============================================
All tests completed!
==============================================
Once this test passes, you can:
- Scale up: Try with more data
- Multi-GPU: If you add more GPUs
- Benchmarking: Measure transfer speeds
- Real applications: Use this as a template
# Full workflow
cd /media/waqasm86/External1/Project-CPP/Project-Nvidia/cuda-openmpi/
mkdir -p build && cd build
cmake ..
make
mpirun -np 2 ./cuda_mpi_test
# Check CUDA
nvcc --version
nvidia-smi
# Check MPI
ompi_info --version
ompi_info --parsable --all | grep mpi_built_with_cuda_support
# Clean rebuild
cd /media/waqasm86/External1/Project-CPP/Project-Nvidia/cuda-openmpi/
rm -rf build
mkdir build && cd build
cmake .. && make- Always run from build directory:
./cuda_mpi_testnotbuild/cuda_mpi_test - Use absolute paths in mpirun: If relative paths fail
- Check environment variables: Especially PATH and LD_LIBRARY_PATH
- Verify GPU is not in use: Close other GPU applications
cuda-openmpi/
├── cuda_mpi_test.cu # Main test program with CUDA kernels and MPI tests
├── CMakeLists.txt # CMake build configuration
├── Makefile.direct # Alternative direct Makefile build system
├── build_and_run.sh # Interactive automated build and test script
├── quick_test.sh # Non-interactive quick test script
├── logs/ # Test execution logs
│ └── cuda-openmpi-logs.txt
└── README.md # This documentation
-
Basic CUDA Operations Testing
- GPU device detection and initialization
- Memory allocation and deallocation verification
- CUDA kernel execution (vector addition)
- Host-to-device and device-to-host memory transfers
- Result validation
-
Standard MPI Communication
- Host-to-host data transfer between MPI processes
- Data integrity verification across processes
- Support for multiple MPI ranks
-
CUDA-aware MPI Testing
- Direct GPU-to-GPU communication
- Automatic detection of CUDA-aware MPI support
- Fallback guidance for non-CUDA-aware installations
Contributions are welcome! Please feel free to submit issues or pull requests.
This project is licensed under the MIT License - see the LICENSE file for details.
- NVIDIA CUDA Toolkit
- Open MPI Project
- Community contributors
- GitHub: @waqasm86
- Repository: cuda-openmpi
If tests fail:
- Check the error messages carefully
- Verify environment variables
- Run each component test separately:
nvidia-smi(GPU driver)nvcc --version(CUDA compiler)ompi_info --version(MPI)
- Look for specific error codes in the output
- Open an issue on GitHub with your system details and error logs