RuntimeError: cuDNN version incompatibility

[cameronk]

cog version 0.4.4 (built 2022-09-23T19:52:35Z)

Problem

I'm attempting to build and run a container on one of GCP's premade machine images.

c0-deeplearning-common-cu110-v20221107-debian-10
Google, Debian 10 based Deep Learning VM with , M100, Base CUDA 11.0, Deep Learning VM Image with CUDA 11.0 preinstalled.

build:
  gpu: true
  python_version: "3.8"
  python_packages:
    - pyannote.audio==2.1.1
  system_packages:
    - "ffmpeg"
    - "libsndfile1"
predict: "predict.py:Predictor"

The container builds but fails when running:

RuntimeError: cuDNN version incompatibility: PyTorch was compiled  against (8, 5, 0) but found runtime version (8, 1, 1). 
PyTorch already comes bundled with cuDNN. 
One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.
one possibility is that there is a conflicting cuDNN in LD_LIBRARY_PATH.

Debugging

$ cog run python
>>> import torch
>>> torch.__version__
'1.13.0+cu117'

$ cog run whereis cudnn.h
cudnn: /usr/include/cudnn.h

$ cog run nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0

Verifying the installed version of cudnn (see here):

$ docker run --interactive cog-speaker-diarization sh -c "cat /usr/include/x86_64-linux-gnu/cudnn_v*.h | grep CUDNN_MAJOR -A 2   "
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 1
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#endif /* CUDNN_VERSION_H */

[cameronk]

I tried locking torch to 1.11 but encountered #814. I'll try another version like 1.12 and see if that helps.

As you noted, this error is caused by the version of cuDNN installed on your system not matching the version PyTorch was compiled against. To fix this issue, you could either update the cuDNN version installed on your system to match the version PyTorch was compiled against, or install the correct version of PyTorch.

To uninstall the installed version of cuDNN, use the command sudo apt-get remove libcudnn8. To install the correct version of cuDNN, use the command sudo apt-get install libcudnn8.

Seems like you're already trying something like this. Let me know if any of this works/helps!

[arnavmehta7]

Hey Can you please tell me how you fied this?

[arnavmehta7]

I fixed it by moving the torch cudnn version to cuda 16

[mattt]

This should be fixed by #812.