Skip to content

About

CUDA-aware OpenMPI integration for GPU-accelerated distributed computing. Multi-GPU LLM inference with MPI communication, performance benchmarking, and collective operations testing.

Topics

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CUDA-MPI Integration Test Suite

CUDA OpenMPI License: MIT Platform

A comprehensive test suite for validating CUDA and OpenMPI integration, including CUDA-aware MPI capabilities. This project helps verify that your GPU computing environment is properly configured for parallel computing applications.

Overview

This project tests the integration of CUDA 12.8 and OpenMPI 5 on your system, providing automated tests for basic CUDA operations, standard MPI communication, and CUDA-aware MPI features.

System Information

  • CUDA Version: 12.8
  • OpenMPI Version: 5.0.6
  • GPU: NVIDIA GeForce 940M (1GB VRAM, Compute Capability 5.0)
  • OS: Xubuntu 22.04

What This Test Suite Checks

  1. Basic CUDA Operations

    • Device detection
    • Memory allocation/deallocation
    • Kernel execution
    • Host-Device memory transfers
    • Vector addition verification
  2. Standard MPI Communication

    • Host-to-host data transfer
    • Data integrity verification
  3. CUDA-aware MPI

    • GPU-to-GPU direct communication
    • Tests if OpenMPI was compiled with CUDA support

Build Instructions

Step 1: Navigate to project directory

cd /media/waqasm86/External1/Project-CPP/Project-Nvidia/cuda-openmpi/

Step 2: Create build directory

mkdir -p build
cd build

Step 3: Configure with CMake

cmake ..

Expected output should show:

  • MPI Found with paths
  • CUDA Toolkit version 12.8
  • Building for CUDA Architecture: 50

Step 4: Build the project

make

If build succeeds, you should see: Built target cuda_mpi_test

Run Instructions

Test 1: Single Process (Tests CUDA only)

./cuda_mpi_test

This will test basic CUDA operations without MPI communication.

Test 2: Multiple Processes (Tests CUDA + MPI)

mpirun -np 2 ./cuda_mpi_test

This runs with 2 MPI processes and tests all features including:

  • CUDA operations on each process
  • Standard MPI communication
  • CUDA-aware MPI

Test 3: Check OpenMPI CUDA Support

ompi_info --parsable --all | grep mpi_built_with_cuda_support

Expected output if CUDA-aware:

mca:mpi:base:param:mpi_built_with_cuda_support:value:true

If it shows false or nothing, your OpenMPI is not CUDA-aware (which is fine for basic MPI+CUDA work).

Interpreting Results

✓ PASSED Indicators:

✓ Basic CUDA operations PASSED
✓ Standard MPI communication PASSED
✓ CUDA-aware MPI PASSED

Common Issues and Solutions

Issue 1: "No CUDA-capable device found"

Solution: Check NVIDIA driver installation

nvidia-smi

Issue 2: "CUDA Error - invalid device"

Cause: GPU not properly detected Solution: Verify CUDA installation

nvcc --version
nvidia-smi

Issue 3: "CUDA-aware MPI FAILED"

Cause: OpenMPI not compiled with CUDA support Impact: This is NORMAL for standard OpenMPI installations Workaround: Use host memory staging:

GPU -> Host Memory -> MPI Transfer -> Host Memory -> GPU

Issue 4: Build fails with "No CUDA toolchain found"

Solution: Ensure CUDA is in PATH

export PATH=/usr/local/cuda-12.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH

Issue 5: Runtime error "libmpi.so not found"

Solution: Add OpenMPI to library path

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

Understanding Your GPU Limitations

Your GeForce 940M has:

  • 1GB VRAM: Be careful with large data allocations
  • Compute Capability 5.0: Supports most CUDA features
  • Maxwell Architecture: Good for learning, limited for production

Memory Considerations:

Available VRAM: ~800MB (after OS overhead)
Safe allocation per process: <400MB
Test uses: ~12KB per test (very safe)

Expected Test Output

==============================================
CUDA-MPI Integration Test Suite
==============================================
MPI Processes: 2

Number of CUDA devices: 1

Rank 0 - GPU Device 0: GeForce 940M
  Compute Capability: 5.0
  Total Global Memory: 1024.00 MB
  Max Threads per Block: 1024
  Multiprocessors: 3
  CUDA Capability Major/Minor version: 5.0

Rank 1 - GPU Device 0: GeForce 940M
  Compute Capability: 5.0
  Total Global Memory: 1024.00 MB
  Max Threads per Block: 1024
  Multiprocessors: 3
  CUDA Capability Major/Minor version: 5.0

==============================================
Test 1: Basic CUDA Operations
==============================================
Rank 0: Testing basic CUDA operations...
Rank 0: ✓ Basic CUDA operations PASSED
Rank 1: Testing basic CUDA operations...
Rank 1: ✓ Basic CUDA operations PASSED

==============================================
Test 2: Standard MPI Communication (Host Memory)
==============================================
Rank 0: Testing standard MPI communication...
Rank 0: Sending 1000 integers to Rank 1 via host memory...
Rank 0: ✓ Send completed
Rank 1: Testing standard MPI communication...
Rank 1: ✓ Standard MPI communication PASSED

==============================================
Test 3: CUDA-aware MPI (GPU-to-GPU)
==============================================
Rank 0: Testing CUDA-aware MPI (GPU-to-GPU)...
Rank 0: Sending 1000 integers to Rank 1 directly from GPU memory...
Rank 0: ✓ CUDA-aware MPI send completed
Rank 1: Testing CUDA-aware MPI (GPU-to-GPU)...
Rank 1: ✓ CUDA-aware MPI PASSED (Your OpenMPI supports CUDA-aware MPI!)

==============================================
All tests completed!
==============================================

Next Steps

Once this test passes, you can:

  1. Scale up: Try with more data
  2. Multi-GPU: If you add more GPUs
  3. Benchmarking: Measure transfer speeds
  4. Real applications: Use this as a template

Quick Reference Commands

# Full workflow
cd /media/waqasm86/External1/Project-CPP/Project-Nvidia/cuda-openmpi/
mkdir -p build && cd build
cmake ..
make
mpirun -np 2 ./cuda_mpi_test

# Check CUDA
nvcc --version
nvidia-smi

# Check MPI
ompi_info --version
ompi_info --parsable --all | grep mpi_built_with_cuda_support

# Clean rebuild
cd /media/waqasm86/External1/Project-CPP/Project-Nvidia/cuda-openmpi/
rm -rf build
mkdir build && cd build
cmake .. && make

Troubleshooting Tips

  1. Always run from build directory: ./cuda_mpi_test not build/cuda_mpi_test
  2. Use absolute paths in mpirun: If relative paths fail
  3. Check environment variables: Especially PATH and LD_LIBRARY_PATH
  4. Verify GPU is not in use: Close other GPU applications

Project Structure

cuda-openmpi/
├── cuda_mpi_test.cu       # Main test program with CUDA kernels and MPI tests
├── CMakeLists.txt         # CMake build configuration
├── Makefile.direct        # Alternative direct Makefile build system
├── build_and_run.sh       # Interactive automated build and test script
├── quick_test.sh          # Non-interactive quick test script
├── logs/                  # Test execution logs
│   └── cuda-openmpi-logs.txt
└── README.md              # This documentation

Features

  • Basic CUDA Operations Testing

    • GPU device detection and initialization
    • Memory allocation and deallocation verification
    • CUDA kernel execution (vector addition)
    • Host-to-device and device-to-host memory transfers
    • Result validation
  • Standard MPI Communication

    • Host-to-host data transfer between MPI processes
    • Data integrity verification across processes
    • Support for multiple MPI ranks
  • CUDA-aware MPI Testing

    • Direct GPU-to-GPU communication
    • Automatic detection of CUDA-aware MPI support
    • Fallback guidance for non-CUDA-aware installations

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

  • NVIDIA CUDA Toolkit
  • Open MPI Project
  • Community contributors

Contact

Getting Help

If tests fail:

  1. Check the error messages carefully
  2. Verify environment variables
  3. Run each component test separately:
    • nvidia-smi (GPU driver)
    • nvcc --version (CUDA compiler)
    • ompi_info --version (MPI)
  4. Look for specific error codes in the output
  5. Open an issue on GitHub with your system details and error logs

About

CUDA-aware OpenMPI integration for GPU-accelerated distributed computing. Multi-GPU LLM inference with MPI communication, performance benchmarking, and collective operations testing.

Topics

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages