CPU environment has CUDA?!

[bhashemian]

Describe the bug
The PyTorch version that is being used for CPU only quick tests seems to be 1.9.0+cu102, which is a cuda version.
https://github.com/Project-MONAI/MONAI/runs/3667531453#step:9:435
https://github.com/Project-MONAI/MONAI/runs/3667531453#step:10:10

Also the installed torchvision: TorchVision version: 0.10.0+cu102
https://github.com/Project-MONAI/MONAI/runs/3667531453#step:10:29

This is causing issue with cucim: https://github.com/Project-MONAI/MONAI/pull/2987/checks?check_run_id=3669657801#step:10:10852
since cucim is not working in cuda<11.0 and this test environment is supposed to be CPU only where cuCIM should work but seems to be a CUDA 10.2 environment.

[Nic-Ma]

Hi @drbeh ,

I will submit a PR to skip cuCIM tests if CUDA < 11.0 today.

Thanks.

[wyli]

I think pytorch by default pip install torch is shipped with the compiled binaries, the test is simulating the commands. But it does not necessarily use the binaries during the tests -- I believe the flag torch.cuda.is_avaialble() is False in this case.

What's the issue here? please follow the bug report template to provide more information about the expected behavior.

[wyli]

Hi @drbeh ,

I will submit a PR to skip cuCIM tests if CUDA < 11.0 today.

Thanks.

Not sure how the reply is related to this ticket, I'm closing this ticket. please reopen with a detailed report following the template.

[bhashemian]

@wyli, thanks for your comments. I updated the description to include why it is causing an error.

[wyli]

the test log correctly showed that in the cpu tests there's no gpu and cuda (around line 6239 of the log https://github.com/Project-MONAI/MONAI/pull/2987/checks?check_run_id=3669657801#step:10:6239 which is generated by

MONAI/monai/config/deviceconfig.py

Lines 183 to 211 in f31b55f

	def get_gpu_info() -> OrderedDict:
	output: OrderedDict = OrderedDict()
	num_gpus = torch.cuda.device_count()
	_dict_append(output, "Num GPUs", lambda: num_gpus)
	_dict_append(output, "Has CUDA", lambda: bool(torch.cuda.is_available()))
	if output["Has CUDA"]:
	_dict_append(output, "CUDA version", lambda: torch.version.cuda)
	cudnn_ver = torch.backends.cudnn.version()
	_dict_append(output, "cuDNN enabled", lambda: bool(cudnn_ver))
	if cudnn_ver:
	_dict_append(output, "cuDNN version", lambda: cudnn_ver)
	if num_gpus > 0:
	_dict_append(output, "Current device", torch.cuda.current_device)
	if hasattr(torch.cuda, "get_arch_list"): # get_arch_list is new in torch 1.7.1
	_dict_append(output, "Library compiled for CUDA architectures", torch.cuda.get_arch_list)
	for gpu in range(num_gpus):
	gpu_info = torch.cuda.get_device_properties(gpu)
	_dict_append(output, f"GPU {gpu} Name", lambda: gpu_info.name)
	_dict_append(output, f"GPU {gpu} Is integrated", lambda: bool(gpu_info.is_integrated))
	_dict_append(output, f"GPU {gpu} Is multi GPU board", lambda: bool(gpu_info.is_multi_gpu_board))
	_dict_append(output, f"GPU {gpu} Multi processor count", lambda: gpu_info.multi_processor_count)
	_dict_append(output, f"GPU {gpu} Total memory (GB)", lambda: round(gpu_info.total_memory / 1024 ** 3, 1))

)

[bhashemian]

there's

So what's the reason that we have cuda102 version of some packages installed?

Pytorch version: 1.9.0+cu102: https://github.com/Project-MONAI/MONAI/pull/2987/checks?check_run_id=3669657801#step:10:18
TorchVision version: 0.10.0+cu102: https://github.com/Project-MONAI/MONAI/pull/2987/checks?check_run_id=3669657801#step:10:6200

[wyli]

there's

So what's the reason that we have cuda102 version of some packages installed?

Pytorch version: 1.9.0+cu102: https://github.com/Project-MONAI/MONAI/pull/2987/checks?check_run_id=3669657801#step:10:18
TorchVision version: 0.10.0+cu102: https://github.com/Project-MONAI/MONAI/pull/2987/checks?check_run_id=3669657801#step:10:6200

That's the default behaviour of pytorch on Linux for pip install torch according to the installation guide (https://pytorch.org/), it'll install 1.9.0+cu102.

I believe this is expected for our testing pipeline, so I"m closing this ticket.

(Update: it still has the issue even with a CPU version of pytorch, please see a quick test log: https://github.com/Project-MONAI/MONAI/runs/3673405225?check_suite_focus=true , so this is probably not the root cause. @gigony please advice.)

[bhashemian]

Hi @drbeh ,

I will submit a PR to skip cuCIM tests if CUDA < 11.0 today.

Thanks.

Hi @Nic-Ma , have you been able to submit a PR to skip cucim when cuda<11? It is a blocking factor in testing cucim and releasing monai. Thanks