WACCMX test fails with ctsm1.0.dev093 in cesm2_2_alpha05c

[ekluzek]

Brief summary of bug

This test fails with a floating point exception error...

SMS_D_Ln9.f09_f09_mg17.FC6XHIST.cheyenne_intel.cam-outfrq9s

General bug information

CTSM version you are using: ctsm2.0.dev093

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected:

WACCM-X starting in 1979 at f09

Details of bug

Important details of your setup / configuration so we can reproduce the bug

I tried with ctsm1.0.dev092 and it works. And it also works if I use a different finidat file to startup with.

It dies with a floating point exception, and it seems to be in CAM. But, this has been working until ctsm1.0.dev093. And just changing the finidat file seems to get it to work.

Important output or errors that show the problem

27:MPT ERROR: Rank 27(g:27) received signal SIGFPE(8).
27: Process ID: 28559, Host: r3i3n2, Program: /glade/scratch/erik/SMS_D_Ln9.f09_f09_mg17.FC6XHIST.cheyenne_intel.cam-outfrq9s.20200511_231110_3r6gn9/bld/cesm.exe
27: MPT Version: HPE MPT 2.21 11/28/19 04:21:40
27:
27:MPT: --------stack traceback-------
27:MPT: Attaching to program: /proc/28559/exe, process 28559
27:MPT: [New LWP 28569]
27:MPT: [New LWP 28585]

.
.
.

27:MPT: #19 0x00000000038d66e6 in edyn_esmf::edyn_esmf_update ()
27:MPT: at /glade/scratch/erik/cesm2_2_alpha05c/components/cam/src/ionosphere/waccmx/edyn_esmf.F90:326
27:MPT: #20 0x00000000037f4919 in dpie_coupling::d_pie_epotent (
27:MPT: highlat_potential_model=..., crit_out=...,
27:MPT: i0=<error reading variable: Cannot access memory at address 0x0>,
27:MPT: i1=<error reading variable: Cannot access memory at address 0x0>,
27:MPT: j0=<error reading variable: Cannot access memory at address 0x0>,
27:MPT: j1=<error reading variable: Cannot access memory at address 0x0>,
27:MPT: efxg=,
27:MPT: kevg=,
27:MPT: .tmp.HIGHLAT_POTENTIAL_MODEL.len_V$8f0=16)
27:MPT: at /glade/scratch/erik/cesm2_2_alpha05c/components/cam/src/ionosphere/waccmx/dpie_coupling.F90:136
27:MPT: #21 0x0000000000cc4390 in ionosphere_interface::ionosphere_run1 (
27:MPT: pbuf2d=0x2b0c38376980)
27:MPT: at /glade/scratch/erik/cesm2_2_alpha05c/components/cam/src/ionosphere/waccmx/ionosphere_interface.F90:478
27:MPT: #22 0x0000000000873420 in cam_comp::cam_run1 (
27:MPT: cam_in=<error reading variable: value requires 194016 bytes, which is more than max-value-size>,
27:MPT: cam_out=<error reading variable: value requires 203040 bytes, which is more than max-value-size>)
27:MPT: at /glade/scratch/erik/cesm2_2_alpha05c/components/cam/src/control/cam_comp.F90:247

[ekluzek]

The finidat file I can use that gets it to work is from an I compset...

finidat = '$DIN_LOC_ROOT/lnd/clm2/initdata_map/clmi.I2000Clm50BgcCrop.2011-01-01.1.9x2.5_gx1v7_gl4_simyr2000_c190312.nc'

so it isn't really the most appropriate one to use.

I've tried a bunch of recent coupled cases and get the same problem:

finidat = '$DIN_LOC_ROOT/cesm2_init/b.e21.BWHIST.f09_g17.CMIP6-historical-WACCM.001_v2/1950-01-01/b.e21.BWHIST.f09_g17.CMIP6-historical-WACCM.001_v2.clm2.r.1950-01-01-00000.nc'
finidat = '$DIN_LOC_ROOT/cesm2_init/b.e21.BWHIST.f09_g17.CMIP6-historical-WACCM.001_v2/2015-01-01/b.e21.BWHIST.f09_g17.CMIP6-historical-WACCM.001_v2.clm2.r.2015-01-01-00000.nc'
finidat = '$DIN_LOC_ROOT/cesm2_init/b.e21.BHIST.f09_g17.CMIP6-historical.010_v2/2015-01-01/b.e21.BHIST.f09_g17.CMIP6-historical.010_v2.clm2.r.2015-01-01-00000.nc'
finidat = '$DIN_LOC_ROOT/cesm2_init/f.e21.FHIST_BGC.f09_f09_mg17.CMIP6-AMIP.001/2000-01-01/f.e21.FHIST_BGC.f09_f09_mg17.CMIP6-AMIP.001.clm2.r.2000-01-01-00000.nc'

[ekluzek]

Chris found this test to fail in the same way as well:

SMS_D_Ln9.f09_f09_mg17.FC6XSD.cheyenne_intel.cam-outfrq9s

The only difference in the CLM namelist is the finidat that is being used.

[ekluzek]

Francis found that if you use an finidat file that has been interpolated -- it runs fine. I've verified that was well. I even let CTSM interpolate the file where the case dies and then used the finidat_interp_dest.nc it created to do a successful run (Francis found the same thing).

If I interpolate from that same file that was successful -- it aborts in the same manner. Note, we test to make sure answers are the same for files that don't need to be interpolated, but are. So it would be unlikely that the contents of the finidat file is the problem as much as just doing the interpolation itself. Note, I also tried to increase the memory by using more processors (using 540x3 rather than the default of 288x3), but it failed in a similar fashion as well.

[billsacks]

@ekluzek I'm having trouble following your previous comment. Are you saying this?:

1. Out-of-the-box initial conditions file, with use_init_interp = .true.: fails

2. finidat_interp_dest.nc from (1), with use_init_interp = .false.: runs

3. finidat_interp_dest.nc from (1), with use_init_interp = .true.: fails

If so, I'm starting to think that the memory issue you and Francis raised might actually be the explanation. You pointed out to me that there are some missing deallocate statements in the init_interp code. At the time, I said that allocatables don't need to be explicitly deallocated, but now that I look at the code, I see that many of the things allocated there are pointers, not allocatables. So those DO need to be deallocated, and I agree that we currently have memory leaks in the init_interp code. Do you want to see if adding appropriate deallocate statements solves this problem?

(Other things that could be good to confirm: (a) Is the output finidat_interp_dest.nc file in case (3) identical to the input finidat_interp_dest.nc file in that case? (b) If you hack the code in case (2) to run the interpolation but not do anything with the resulting file - i.e., still reading in from the input finidat rather than from the output finidat_interp_dest file - does it fail? If so, that points strongly to the memory issue.)

[ekluzek]

Hi @billsacks yes to all. And I came to the same conclusion by looking at the code. It looks like. there are a lot of pointers that are allocated, but not deallocated. So that's going to be a problem. There are also some allocatables, but. that doesn't cover everything.

[billsacks]

Okay, I'll open an issue for this. I'm happy to take this on since I have worked a lot with the init_interp code.

[ekluzek]

I guess I'm happy to keep working on this and then have you review the PR when it's ready. But, if you want to take it on that's fine too. I just don't want both of us to work on this at the same time.

I was planning on taking a stab over the weekend and see what happens. It might take a bit to figure out everything.

[ekluzek]

I was also thinking of not opening a separate issue, if indeed it's the underlying problem here.

[billsacks]

Okay, if you're happy to work on it, I won't object - especially since I probably won't get much done on it over the weekend. The main trickiness is that I think some of the derived types that I added, which use pointers, need final subroutines to deallocate their pointer components. I don't think I've done that before. I'm happy to help with that piece next week if you don't get to it before then.

[billsacks]

One other thought: where possible, rather than adding explicit deallocate statements, I think it would be better to change pointer variables to allocatables. Then they will automatically be deallocated. If they are pointers because something else is pointing to them, then they can be declared as allocatable, target. At a glance, it looks like this may be possible for the pointers declared in initInterp.F90, but it likely isn't possible for some / all of the pointers in derived types (because you can't use the target attribute for a pointer in a derived type).

My initial thought is that, for every pointer in the init_interp code, we need to either: (a) deallocate it (possibly with a final subroutine), or (b) change it to an allocatable.

[ekluzek]

OK, I did the simplest change to fix part of the known memory leak, and I still get the bad behavior in the F case. I changed the local arrays from pointer to allocatable, so FORTRAN will automatically deallocate them. That doesn't completely resolve the memory leak, but it takes care of a good portion of it. I plan to do more work and completely resolve the memory leak and be able to show that it makes a difference in memory. We thought the memory leak was the total problem behind this, but it's possible it's more involved than that. Or. it's possible that I just need to completely resolve the memory leak.