torch 1.6 unit test test_keep_largest_connected_component

[wyli]

Describe the bug

12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_28_onehot_independent_batch_2_apply_label_1_2_connect_2 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 1 / 25 (4%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 1, 0, 0],
12:01:15          [0, 2, 1, 1, 1],
12:01:15          [2, 2, 1, 0, 0],...
12:01:15   y: array([[[0, 0, 1, 0, 0],
12:01:15          [0, 2, 1, 1, 1],
12:01:15          [2, 2, 1, 0, 0],...

Details

12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_28_onehot_independent_batch_2_apply_label_1_2_connect_2 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 1 / 25 (4%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 1, 0, 0],
12:01:15          [0, 2, 1, 1, 1],
12:01:15          [2, 2, 1, 0, 0],...
12:01:15   y: array([[[0, 0, 1, 0, 0],
12:01:15          [0, 2, 1, 1, 1],
12:01:15          [2, 2, 1, 0, 0],...
12:01:15  
12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_29_onehot_dependent_batch_2_apply_label_1_2_connect_2 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 2 / 25 (8%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [2, 0, 1, 1, 2],...
12:01:15   y: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [0, 0, 1, 1, 2],...
12:01:15  
12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_30_onehot_none_dependent_batch_2_apply_label_1_2_connect_1 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 2 / 25 (8%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [2, 0, 1, 1, 2],...
12:01:15   y: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [0, 0, 1, 1, 2],...
12:01:15  
12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_44_onehot_independent_batch_2_apply_label_1_2_connect_2 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 1 / 25 (4%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 1, 0, 0],
12:01:15          [0, 2, 1, 1, 1],
12:01:15          [2, 2, 1, 0, 0],...
12:01:15   y: array([[[0, 0, 1, 0, 0],
12:01:15          [0, 2, 1, 1, 1],
12:01:15          [2, 2, 1, 0, 0],...
12:01:15  
12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_45_onehot_dependent_batch_2_apply_label_1_2_connect_2 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 2 / 25 (8%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [2, 0, 1, 1, 2],...
12:01:15   y: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [0, 0, 1, 1, 2],...
12:01:15  
12:01:15  ======================================================================
12:01:15  FAIL: test_correct_results_46_onehot_none_dependent_batch_2_apply_label_1_2_connect_1 (tests.test_keep_largest_connected_component.TestKeepLargestConnectedComponent)
12:01:15  ----------------------------------------------------------------------
12:01:15  Traceback (most recent call last):
12:01:15    File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
12:01:15      return func(*(a + p.args), **p.kwargs)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_keep_largest_connected_component.py", line 365, in test_correct_results
12:01:15      assert_allclose(result.argmax(0)[None], result2)
12:01:15    File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/utils.py", line 107, in assert_allclose
12:01:15      np.testing.assert_allclose(actual, desired, *args, **kwargs)
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
12:01:15      assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
12:01:15    File "/usr/local/lib/python3.8/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
12:01:15      raise AssertionError(msg)
12:01:15  AssertionError: 
12:01:15  Not equal to tolerance rtol=1e-07, atol=0
12:01:15  
12:01:15  Mismatched elements: 2 / 25 (8%)
12:01:15  Max absolute difference: 2
12:01:15  Max relative difference: 0.
12:01:15   x: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [2, 0, 1, 1, 2],...
12:01:15   y: array([[[0, 0, 0, 0, 1],
12:01:15          [0, 0, 1, 1, 1],
12:01:15          [0, 0, 1, 1, 2],...
12:01:15  
12:01:15  ----------------------------------------------------------------------

Environment

Ensuring you use the relevant python executable, please paste the output of:

11:39:59  .================================
11:39:59  Printing MONAI config...
11:39:59  ================================
11:39:59  MONAI version: 0.8.1+141.g5c56b559
11:39:59  Numpy version: 1.22.2
11:39:59  Pytorch version: 1.6.0
11:39:59  MONAI flags: HAS_EXT = True, USE_COMPILED = True
11:39:59  MONAI rev id: 5c56b5599275fc9f65f5c867c5cf2c07f89bb3d8
11:39:59  
11:39:59  Optional dependencies:
11:39:59  Pytorch Ignite version: 0.4.8
11:39:59  Nibabel version: 3.2.2
11:39:59  scikit-image version: 0.19.2
11:39:59  Pillow version: 9.0.1
11:39:59  Tensorboard version: 2.8.0
11:39:59  gdown version: 4.4.0
11:39:59  TorchVision version: 0.7.0
11:39:59  tqdm version: 4.63.0
11:39:59  lmdb version: 1.3.0
11:39:59  psutil version: 5.9.0
11:39:59  pandas version: 1.4.1
11:39:59  einops version: 0.4.0
11:39:59  transformers version: 4.16.2
11:39:59  mlflow version: 1.24.0
11:39:59  
11:39:59  For details about installing the optional dependencies, please visit:
11:39:59      https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

Additional context
Add any other context about the problem here.

[wyli]

I'm able to reproduce this using nvcr.io/nvidia/pytorch:20.07-py3,
with

pip install parameterized
python -m tests.test_keep_largest_connected_component

could you help look into it? @yiheng-wang-nv @Nic-Ma

[yiheng-wang-nv]

Hi @wyli , I took some time and found the root cause. It is due to the changes of torch.argmax, where after 1.7,
if there are multiple maximal values then the indices of the first maximal value are returned. For example:

a = torch.tensor([[0], [0], [0], [0]])
print(a.argmax(0))

will get tensor([0]).
But for torch==1.6.0, the result is tensor([3]).

Therefore, the changes in this place may raise the issue.

Another issue is that the transform KeepLargestConnectedComponent will change the original input, thus we may need to copy it first in the test class.

Let me submit a PR to fix it.

[wyli]

Thanks a lot, now I remember that we went into this issue before. please help add some notes in the docstring about the argmax.

[yiheng-wang-nv]

Hi @wyli , sure. do you mean add the notes in the class TestKeepLargestConnectedComponent?

[wyli]

Yes, for the test cases..