ERS fates hydro test failing with nag compiler due to `use_lch4`

[glemieux]

Brief summary of bug

ERS_D_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdDefHydro is failing RUN due to a ctsm5.1.dev058 update (I think 1e2729f) that sets use_lch4 = .true.. The test will pass if I over ride the setting as .false. in the testmod user_nl_clm file.

Note: This test passes B4B on cheyenne with the intel compiler.

General bug information

CTSM version you are using: ctsm5.1.dev059-4-g6837c2977

Does this bug cause significantly incorrect results in the model's science? No

Details of bug

While testing PR #1524, which updates the fates externals pointer to incorporate a separate fix for the same test (fates issue 790), I ran into this PIO error. After iterating on some combinations of the ctsm and fates branches, I determined the the issue was likely due to merging in ctsm5.1.dev058 given that the original fix for issue 790 was only tested on ctsm commit 5f3a822 which was prior to merging in the dev058 tag.

Discussing this with @ekluzek, and noting the differences in the lnd_in files between iterations of the test that passed and failed, it was determined that we should test the use_lch4 and use_nitrif_denitrif options. Of the two, setting use_lch4 = .false. resulted in a passing test.

Important output or errors that show the problem

The test aborts the run within a few timesteps with the following output in the cesm.log:

[0] Runtime Error: *** Arithmetic exception: Floating overflow - aborting
[0] /home/glemieux/ctsm/src/main/ncdio_pio.F90.in, line 2039: Error occurred in NCDIO_PIO:NCD_IO_2D_DOUBLE
[0] /home/glemieux/ctsm/src/main/histFileMod.F90, line 3547: Called by HISTFILEMOD:HFIELDS_WRITE
[0] /home/glemieux/ctsm/src/main/histFileMod.F90, line 4065: Called by HISTFILEMOD:HIST_HTAPES_WRAPUP
[0] /home/glemieux/ctsm/src/main/clm_driver.F90[0] , line 1399: Called by CLM_DRIVER:CLM_DRV
[0] /home/glemieux/ctsm/src/cpl/mct/lnd_comp_mct.F90, lin[0] e 451: Called by LND_COMP_MCT:LND_RUN_MCT
[0] /home/glemieux/ctsm/components/cpl7/driver/main/component_mod.F90, line 737: Called by COMPONENT_MOD:COMPONENT_RUN[0]
[0] /home/glemieux/ctsm/components/cpl7/driver/main/cime_comp_mod.F90, line 2855: Called by CIME_COMP_MOD:CIME_RUN
[0] /home/glemieux/ctsm/components/cpl7/driver/main/cime_driver.F90, l[0] ine 153: Called by CIME_DRIVER
[0] [i036.unified.ucar.edu:mpi_rank_0][error_sighandler] Caught error: Aborted (signal 6)

[glemieux]

This appears to be running successfully now, but is failing COMPARE_base_rest. Looking at previous fates baselines this looks like it was partially fixed prior to the dev083 update. My assumption is that it was partially fixed with #1535.

[glemieux]

Running this test on tag ctsm5.1.dev075 shows that the test passes completely so the COMPARE_base_rest must have been injected somewhere else. I'll run some more tests later to see if I can pinpoint it.
Results: /scratch/cluster/glemieux/ctsm-tests/tests_issue1525_comparefailcheck

[glemieux]

Running this test on tag ctsm5.1.dev075 shows that the test passes completely so the COMPARE_base_rest must have been injected somewhere else. I'll run some more tests later to see if I can pinpoint it. Results: /scratch/cluster/glemieux/ctsm-tests/tests_issue1525_comparefailcheck

It appears that this issue is reflective of a different problem and unrelated to this issue: NGEET/fates#854 (comment). As such I'm closing this as fixed.