Extensive tests fail on Summit

[yingwaili]

On Summit, all the fast tests pass but extensive tests fail. Not sure if it is the software issue or hardware at the moment. Fixes will be posted in the "summit_cmake" branch.

[ubulling]

The extensive tests that use MPI pass if --smpiargs=none is removed, which makes sense.
After the tests have passed, I still get the following error, though:
https://gist.github.com/uhaehner/35fbdeef44b2845f1f19e6dcce0dbda9

[ubulling]

Adding --smpiargs="-tcp" to the jsrun command fixed it.
Reference: horovod/horovod#308

I'll try to get some more infos on this issue at the Summit workshop tomorrow.

[ubulling]

SpAccumulatorGpuTest.SumTo still fails randomly:

64/129 Testing: sp_accumulator_gpu_test
64/129 Test: sp_accumulator_gpu_test
Command: "/opt/ibm/spectrum_mpi/jsm_pmix/bin/jsrun" "-a" "1" "-n" "1" "-g" "1" "-c" "8" "--smpiargs=none" "/gpfs/wolf/proj-shared/gen113/haehneru/build/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp/sp_accumulator_gpu_test"
Directory: /gpfs/wolf/proj-shared/gen113/haehneru/build/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp
"sp_accumulator_gpu_test" start time: Dec 05 15:28 EST
Output:

Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (156 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
/gpfs/wolf/proj-shared/gen113/haehneru/DCA/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp/sp_accumulator_gpu_test.cpp:82: Failure
Expected: (5e-7) > (diff.l_inf), actual: 5e-07 vs inf
[ FAILED ] SpAccumulatorGpuTest.SumTo (18 ms)
[----------] 2 tests from SpAccumulatorGpuTest (174 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (174 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] SpAccumulatorGpuTest.SumTo

1 FAILED TEST

Test time = 0.29 sec

Test Failed.
"sp_accumulator_gpu_test" end time: Dec 05 15:28 EST
"sp_accumulator_gpu_test" time elapsed: 00:00:00

[ubulling]

Fail, pass, fail on three consecutive executions:

bash-4.2$ jsrun -n 1 -a 1 -g 1 -c 8 --smpiargs=none ./sp_accumulator_gpu_test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (167 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
/gpfs/wolf/proj-shared/gen113/haehneru/DCA/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp/sp_accumulator_gpu_test.cpp:82: Failure
Expected: (5e-7) > (diff.l_inf), actual: 5e-07 vs inf
[ FAILED ] SpAccumulatorGpuTest.SumTo (17 ms)
[----------] 2 tests from SpAccumulatorGpuTest (184 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (184 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] SpAccumulatorGpuTest.SumTo

1 FAILED TEST
bash-4.2$ jsrun -n 1 -a 1 -g 1 -c 8 --smpiargs=none ./sp_accumulator_gpu_test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (156 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
[ OK ] SpAccumulatorGpuTest.SumTo (17 ms)
[----------] 2 tests from SpAccumulatorGpuTest (173 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (173 ms total)
[ PASSED ] 2 tests.
bash-4.2$ jsrun -n 1 -a 1 -g 1 -c 8 --smpiargs=none ./sp_accumulator_gpu_test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (204 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
/gpfs/wolf/proj-shared/gen113/haehneru/DCA/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp/sp_accumulator_gpu_test.cpp:82: Failure
Expected: (5e-7) > (diff.l_inf), actual: 5e-07 vs 5.01839
[ FAILED ] SpAccumulatorGpuTest.SumTo (18 ms)
[----------] 2 tests from SpAccumulatorGpuTest (222 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (222 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] SpAccumulatorGpuTest.SumTo

1 FAILED TEST

[yingwaili]

Extensive tests results on Summit today (Dec 6, 18):

make test
Running tests...
Test project /gpfs/alpine/stf006/scratch/ywl/ctest/DCA/build
Start 1: coarsegraining_test
1/19 Test #1: coarsegraining_test ........................ Passed 34.42 sec
Start 2: bilayer_lattice_Nc1_interband
2/19 Test #2: bilayer_lattice_Nc1_interband ..............***Failed 47.48 sec
Start 3: bilayer_lattice_Nc1_intra_plus_interband
3/19 Test #3: bilayer_lattice_Nc1_intra_plus_interband ...***Failed 166.64 sec
Start 4: bilayer_lattice_Nc1_intraband
4/19 Test #4: bilayer_lattice_Nc1_intraband ..............***Failed 87.24 sec
Start 5: square_lattice_Nc4_nn
5/19 Test #5: square_lattice_Nc4_nn ......................***Failed 69.60 sec
Start 6: square_lattice_Nc4_onSite_plus_nn
6/19 Test #6: square_lattice_Nc4_onSite_plus_nn ..........***Failed 56.51 sec
Start 7: NiO_no_change_test
7/19 Test #7: NiO_no_change_test .........................***Failed 114.02 sec
Start 8: stdthread_ctaux_tp_test
8/19 Test #8: stdthread_ctaux_tp_test ....................***Failed 29.75 sec
Start 9: stdthread_ctaux_gpu_tp_test
9/19 Test #9: stdthread_ctaux_gpu_tp_test ................ Passed 33.45 sec
Start 10: ed_cluster_solver_four_site_test
10/19 Test #10: ed_cluster_solver_four_site_test ........... Passed 1.95 sec
Start 11: lattice_mapping_sp_test
11/19 Test #11: lattice_mapping_sp_test .................... Passed 18.31 sec
Start 12: statistical_testing_integration_test
12/19 Test #12: statistical_testing_integration_test ....... Passed 1.21 sec
Start 13: analysis_dca_pp_up_down_full_test
13/19 Test #13: analysis_dca_pp_up_down_full_test .......... Passed 8.63 sec
Start 14: analysis_dca_pp_up_down_sym_test
14/19 Test #14: analysis_dca_pp_up_down_sym_test ........... Passed 8.03 sec
Start 15: analysis_dcaplus_pp_up_down_full_test
15/19 Test #15: analysis_dcaplus_pp_up_down_full_test ...... Passed 76.55 sec
Start 16: analysis_dcaplus_pp_up_down_sym_test
16/19 Test #16: analysis_dcaplus_pp_up_down_sym_test ....... Passed 68.77 sec
Start 17: dca_sp_DCA+_mpi_test
17/19 Test #17: dca_sp_DCA+_mpi_test .......................***Failed 57.99 sec
Start 18: dca_sp_DCA+_mpi_cuda_test
18/19 Test #18: dca_sp_DCA+_mpi_cuda_test ..................***Failed 57.64 sec
Start 19: dca_sp_DCA+_thread_test
19/19 Test #19: dca_sp_DCA+_thread_test ....................***Failed 261.20 sec
47% tests passed, 10 tests failed out of 19
Total Test time (real) = 1202.10 sec
The following tests FAILED:
2 - bilayer_lattice_Nc1_interband (Failed)
3 - bilayer_lattice_Nc1_intra_plus_interband (Failed)
4 - bilayer_lattice_Nc1_intraband (Failed)
5 - square_lattice_Nc4_nn (Failed)
6 - square_lattice_Nc4_onSite_plus_nn (Failed)
7 - NiO_no_change_test (Failed)
8 - stdthread_ctaux_tp_test (Failed)
17 - dca_sp_DCA+_mpi_test (Failed)
18 - dca_sp_DCA+_mpi_cuda_test (Failed)
19 - dca_sp_DCA+_thread_test (Failed)
Errors while running CTest

Modules used:

1. hsi/5.0.2.p5
2. xalt/1.1.3
3. lsf-tools/2.0
4. DefApps
5. gcc/6.4.0
6. spectrum-mpi/10.2.0.7-20180830
7. cuda/9.2.148
8. hdf5/1.10.0-patch1
9. fftw/3.3.5
10. cmake/3.11.3
11. magma/2.3.0
12. netlib-lapack/3.6.1
13. essl/6.1.0-1
14. subversion/1.9.3
15. boost/1.66.0

[yingwaili]

Updates:

1. All fast and extensive tests (except sp_accumulator_gpu_test mentioned by @uhaehner above) pass on Ascent (Note that Ascent is a small training cluster emulating Summit, but with newer libraries and modules. Hopefully the failed tests on Summit will go away after receiving new module upgrades).

2. But on Summit many extensive tests still fail (see above).

3. The --smpiargs=-mxm flag used in summit.cmake is only a temporary solution that uses Mellanox's protocol and bypasses PAMI for MPI communications. If we are sure that our tests is correct, this means a SpectrumMPI misbehavior and will need to be filed as a bug to IBM.

[yingwaili]

Updates (Jan 4, 19):

Compiled and ran tests on Summit today. The tests that failed before passed today, except sp_accumulator_gpu_test. So basically the same observation as @uhaehner on Ascent above.

@gbalduzz : Any thoughts or insights?

jsrun -n 1 -a 1 -g 1 -c 8 --smpiargs=none ./sp_accumulator_gpu_test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (4570 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
/gpfs/alpine/stf006/scratch/ywl/ctest/DCA/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp/sp_accumulator_gpu_test.cpp:82: Failure
Expected: (5e-7) > (diff.l_inf), actual: 5e-07 vs inf
[ FAILED ] SpAccumulatorGpuTest.SumTo (17 ms)
[----------] 2 tests from SpAccumulatorGpuTest (4587 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (4587 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] SpAccumulatorGpuTest.SumTo

1 FAILED TEST

[ubulling]

@yingwaili Does the test always fail the same way?
If I remember correctly, on Ascent I got inf, nan or some wrong number in a random manner.

[yingwaili]

@uhaehner @gbalduzz To me, the test alternatively passes and fails. But also, I randomly get inf and a super wrong number...

When it passes:
jsrun -n 1 -a 1 -g 1 -c 8 --smpiargs=none ./sp_accumulator_gpu_test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (296 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
[ OK ] SpAccumulatorGpuTest.SumTo (17 ms)
[----------] 2 tests from SpAccumulatorGpuTest (313 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (313 ms total)
[ PASSED ] 2 tests.

When it fails:
jsrun -n 1 -a 1 -g 1 -c 8 --smpiargs=none ./sp_accumulator_gpu_test
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from SpAccumulatorGpuTest
[ RUN ] SpAccumulatorGpuTest.Accumulate
[ OK ] SpAccumulatorGpuTest.Accumulate (263 ms)
[ RUN ] SpAccumulatorGpuTest.SumTo
/gpfs/alpine/stf006/scratch/ywl/ctest/DCA/test/unit/phys/dca_step/cluster_solver/shared_tools/accumulation/sp/sp_accumulator_gpu_test.cpp:82: Failure
Expected: (5e-7) > (diff.l_inf), actual: 5e-07 vs 5.00579
[ FAILED ] SpAccumulatorGpuTest.SumTo (17 ms)
[----------] 2 tests from SpAccumulatorGpuTest (280 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (280 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] SpAccumulatorGpuTest.SumTo

1 FAILED TEST