Reproducibility problem with clm4_5_10_r187 -- 30% of the time randomly dies with a methane error

[ekluzek]

Erik Kluzek < erik > - 2016-08-08 13:16:05 -0600
Bugzilla Id: 2344
Bugzilla Depends: 2345,
Bugzilla CC: andre, dlawren, dll, mvertens, oleson, rfisher, sacks,

A reproducibility issue came in with clm4_5_10_r187 where about 30% of the time the following case will die with a methane error, and other times run to completion. I think simpler cases will die as well, but the simpler tests I did were successful before I realized it wasn't reproducible.

create_newcase -case clm4_5_10_r187_SP_4x5_ADspin -mach yellowstone_intel -res f45_f45 -user_compset 1850_DATM%CRU_CLM50%BGC_SICE_SOCN_MOSART_SGLC_SWAV -user_pes_setby clm

./xmlchange CLM_ACCELERATED_SPINUP="on",STOP_N=19,STOP_OPTION="nmonths",CLM_FORCE_COLDSTART="on",DATM_CLMNCEP_YR_ALIGN=1901
./xmlchange DATM_CLMNCEP_YR_END=1905,DATM_CLMNCEP_YR_START=1901,DATM_MODE=CLMGSWP3

user_nl_clm
paramfile='/glade/u/home/rfisher/Matlab/pft_files/FUNparams/TRY_default_c160708_hydr1.5.nc'
fsurdat='/glade/p/cesm/sdwg_dev/lawrence/surfdata/surfdata_4x5_16pftsmidarctic_simyr2000_c160419.nc'
use_hydrstress=.true.
use_luna=.true.

SourceMods/src.datm/namelist_defaults_datm.xml

<namelist_defaults>

CLMGSWP3.Solar,CLMGSWP3.Precip,CLMGSWP3.TPQW,CLMCRUNCEP_V5.TPQW

<strm_datvar stream="CLMCRUNCEP_V5.TPQW">
TBOT tbot
QBOT shum
</strm_datvar>

<strm_datvar stream="CLMGSWP3.TPQW">
WIND wind
PSRF pbot
FLDS lwdn
</strm_datvar>

</namelist_defaults>

An example error message in cesm.log is...

41: Negative conc. in ch4tran. c,j,deficit (mol): 3512 1
41: 8.843302222960208E-003
0: memory_write: model date = 10623 0 memory = 104.44 MB (highwater) 1946.05 MB (usage) (pe= 0 comps= ATM ICE OCN WAV ESP)
65: Methane demands exceed methane available. Error in methane competition (mol/m^3
65: /s), c,j: -2.755768946371973E-010 6818 2
65: Latdeg,Londeg= -30.0000000000000 35.0000000000000
65: ENDRUN:
65: ERROR: Methane demands exceed methane available.ERROR in ch4Mod.F90 at line 33
65: 90
65:
65:
65:
65:
65:
65:
65: ERROR: Unknown error submitted to shr_sys_abort.
65:Image PC Routine Line Source
65:cesm.exe 000000000153ABA8 Unknown Unknown Unknown
65:cesm.exe 0000000000E43893 shr_sys_mod_mp_sh 401 shr_sys_mod.F90
65:cesm.exe 0000000000509058 abortutils_mp_end 43 abortutils.F90
65:cesm.exe 0000000000A4352D ch4mod_mp_ch4_tra 3389 ch4Mod.F90
65:cesm.exe 0000000000A377CD ch4mod_mp_ch4_ 1980 ch4Mod.F90
65:cesm.exe 0000000000511D82 clm_driver_mp_clm 835 clm_driver.F90
65:cesm.exe 00000000004FDC49 lnd_comp_mct_mp_l 437 lnd_comp_mct.F90
65:cesm.exe 000000000042EDFB component_mod_mp_ 1079 component_mod.F90
65:cesm.exe 0000000000419FB2 cesm_comp_mod_mp_ 2509 cesm_comp_mod.F90
65:cesm.exe 000000000042EB92 MAIN__ 93 cesm_driver.F90
65:cesm.exe 0000000000417B7E Unknown Unknown Unknown
65:libc.so.6 00002ACAABC8FD5D Unknown Unknown Unknown
65:cesm.exe 0000000000417A89 Unknown Unknown Unknown

a precursor to the problem seems to be these ch4tran warnings....

grep ch4tran cesm.log.160806-113740
36: Negative conc. in ch4tran. c,j,deficit (mol): 2842 1
98: Negative conc. in ch4tran. c,j,deficit (mol): 11343 1
24: Negative conc. in ch4tran. c,j,deficit (mol): 1205 1
34: Negative conc. in ch4tran. c,j,deficit (mol): 2561 1
141: Negative conc. in ch4tran. c,j,deficit (mol): 17127 1
53: Negative conc. in ch4tran. c,j,deficit (mol): 5161 1
73: Negative conc. in ch4tran. c,j,deficit (mol): 7928 1
37: Negative conc. in ch4tran. c,j,deficit (mol): 2964 1
40: Negative conc. in ch4tran. c,j,deficit (mol): 3375 1
93: Negative conc. in ch4tran. c,j,deficit (mol): 10667 1

The ch4tran warnings don't occur in the simulations that were successful.

[ekluzek]

Erik Kluzek < erik > - 2016-08-08 13:17:17 -0600

Rosie tried similar submissions with r185 and r186 and didn't see runs that died with the same problem for 8 submissions. Hence we think this came in with r187.

[ekluzek]

Erik Kluzek < erik > - 2016-08-08 13:19:30 -0600

This is for a sequential layout without threading...

env_mach_pes.xml: NTASKS_ATM 15
env_mach_pes.xml: NTASKS_CPL 135
env_mach_pes.xml: NTASKS_ESP 15
env_mach_pes.xml: NTASKS_GLC 135
env_mach_pes.xml: NTASKS_ICE 15
env_mach_pes.xml: NTASKS_LND 135
env_mach_pes.xml: NTASKS_OCN 15
env_mach_pes.xml: NTASKS_ROF 135
env_mach_pes.xml: NTASKS_WAV 15
[erik@yslogin5 clm4_5_10_r187_SP_4x5_ADspin]$ ./xmlquery list | grep THRDS
env_mach_pes.xml: NTHRDS_ATM 1
env_mach_pes.xml: NTHRDS_CPL 1
env_mach_pes.xml: NTHRDS_ESP 1
env_mach_pes.xml: NTHRDS_GLC 1
env_mach_pes.xml: NTHRDS_ICE 1
env_mach_pes.xml: NTHRDS_LND 1
env_mach_pes.xml: NTHRDS_OCN 1
env_mach_pes.xml: NTHRDS_ROF 1
env_mach_pes.xml: NTHRDS_WAV 1

and with daily barriers...

env_run.xml: BARRIER_N 1
env_run.xml: BARRIER_OPTION ndays

[ekluzek]

Erik Kluzek < erik > - 2016-08-08 15:01:27 -0600

We think this problem is just because Rosie's param file doesn't have rootprof_beta as a 2D variable. So the 2nd dimension (for Carbon) isn't getting initialized to anything (and set to garbage) hence the problem.

so this file has rootprof_beta as a 1D variable rather than 2D.

/glade/u/home/rfisher/Matlab/pft_files/FUNparams/TRY_default_c160708_hydr1.5.nc

so this problem shouldn't occur for the default code where the standard params file is being used -- just for cases where a params file doesn't have rootprof_beta as a 2D variable.

As a separate issue we should fix the code so it doesn't allow such a thing to happen.

[ekluzek]

Erik Kluzek < erik > - 2016-08-22 16:33:43 -0600

This does also occur on hobart_intel and with both PIO1 and PIO2.

[ekluzek]

I think this issue should have been closed before since the comments indicate it had to do with a params file that didn't have a variable with the right dimension. We don't seem to be running into this problem of late, so I'm going to close it.