FATES Accelerated Decomposition Issues

[rgknox]

Brief summary of bug

At the very least, some spin-up flags are not being broadcast when FATES is on, and it is not clear if flags related to AD are being set and applied correctly when FATES is active. At a first order, I think we need to look closely at a FATES AD spinup and make sure that the correct rate-constants are being amplified during this phase, and returned to normal when starting the follow up phase.

General bug information

No specific version has been identified, but I don't think the code has changed since we first identified this issue and ctsm5.1.dev112 . This issue is the result of investigation by @olyson and I earlier this year. It is possible that there is no issue, as FATES does bypass some of the soil biogeochemical code, but not all. At the least, we need to verify that even if AD flags are not being set, they don't need to be set.

Details of bug

Appoligies in advance for how cloudy this report is, I'm reading throug old emails to reconstruct my understanding.

@olyson and I tried several ways of building the model and noticed an issue related to spinup_state and nitrogen supplementation, although things are still murkey. It appears that if we didn't actively set suplnitro=ALL, the spinup_state was not being set to 2 in AD model. It was odd, and it should be irrelevant anyway, as FATES is carbon only at the moment and shouldn't even work outside carbon only mode, which implies suplnitro=ALL.

@olyson also felt that even though we were able to get AD FATES to generate spinup_state = 2, he was suspicious that it was not actually setting amplified rate constatns during spinup.

@olyson did identify that when fates is active, we are not broadcasting flags, for instance:Although, for instance, suplnitro is not used when FATES is on anyway..

if (use_cn) then
       call mpi_bcast (suplnitro, len(suplnitro), MPI_CHARACTER, 0, mpicom, ier)
       call mpi_bcast (nfix_timeconst, 1, MPI_REAL8, 0, mpicom, ier)
       call mpi_bcast (spinup_state, 1, MPI_INTEGER, 0, mpicom, ier)
       call mpi_bcast (override_bgc_restart_mismatch_dump, 1, MPI_LOGICAL, 0, mpicom, ier)
    end if

Here is a branch we had been working with to resolve some of these issues:

https://github.com/rgknox/ctsm/tree/update_ADfates_histlist

Definition of Done:

• Bring in ctsm5.1.dev134 where FATES uses the standard BGC calling sequence.
• Add a test that exercises running FATES in AD decomp mode.
• Check that running FATES in AD decomp mode works as expected.

[olyson]

Thanks for the nice summary @rgknox . This prompted me to go back over my own notes and re-setup the simulation we had been working with.
What I have found is that with FATES on and spinup_state=2 (AD spinup), even if spinup_state is properly broadcast, the spinup_factor_deadwood is not initialized to:

spinup_factor_deadwood = spinup_factor = 10

However, this appears to be ok as the routines that use this factor (CNFireBaseMod, CNFireLi2021Mod, CNGapMortalityMod, CNVegStructUpdateMod) to accelerate the dead wood transfers (e.g., m_deadstemc_to_fire, etc.) are not used by FATES. I assume that is intended.

I also have now found that, if spinup_state is properly broadcast, the spinup factors for soil biogeochemistry are calculated and used when FATES is on. I'm running a normal (spinup_state=0) and a AD (spinup_state=2) simulation now to see if it has the spinup characteristics that we would expect.

[ekluzek]

@olyson are you saying there's a bug when NOT running with FATES so that spinup_factor_deadwood isn't initialized correctly? If so we should report that as an issue.

[olyson]

No bug when not running FATES.

[rgknox]

Re:

What I have found is that with FATES on and spinup_state=2 (AD spinup), even if spinup_state is properly broadcast, the spinup_factor_deadwood is not initialized to:

spinup_factor_deadwood = spinup_factor = 10

However, this appears to be ok as the routines that use this factor (CNFireBaseMod, CNFireLi2021Mod, CNGapMortalityMod, CNVegStructUpdateMod) to accelerate the dead wood transfers (e.g., m_deadstemc_to_fire, etc.) are not used by FATES. I assume that is intended.

This is how I understand it. coarse woody debris should not be tracked in CLM when FATES is on, it will only exist on the FATES side of the code. FATES won't pass the carbon from CWD to CLM until after it has been fragmented.

Thanks for re-running those tests, curious to see the results.

[jkshuman]

@rgknox thanks for posting this and @olyson for working through it. I did not see an associated issue on the FATES side. Should I open one, or just leave this on CTSM side?

[olyson]

I've run 420 years for both of these simulations. Below are spinup plots for these. I have more questions than answers but I see that TOTSOMC appears to be spunup or near spunup using AD mode (top plot), whereas in "normal" mode TOTSOMC is not yet spunup (bottom plot). So maybe that is working ok.
On the other hand, TOTECOSYSC is nearing equilibrium but is the same in both simulations, which I guess is expected since we are not accelerating any above ground stuff with FATES on. Note that in CLM without FATES, TOTECOSYSC includes TOTSOMC, but here for FATES we are using:

FATES_VEGC + FATES_SEED_BANK + FATES_LITTER_AG_FINE_EL[1] + FATES_LITTER_BG_FINE_EL[1] + FATES_LITTER_BG_CWD_EL[1] + FATES_LITTER_AG_CWD_EL[1]

I'm also not used to seeing so much noise in the "Delta" plots (TOTECOSYSC, TOTSOMC, GPP, TOTVEGC). We are sampling the same year of forcing in each 20-year period and we usually see a smooth curve in these plots.

[olyson]

Hi Ryan, looks like I need permissions set to push to your branch update_ADfates_histlist

[olyson]

Some spinup results here, we can discuss at some point:

https://docs.google.com/presentation/d/1K5Sw0DYYBpuyCmwuEIUBqVmmMNnavvI8/edit?usp=share_link&ouid=104026900087612515425&rtpof=true&sd=true

[ekluzek]

We believe this is fixed with ctsm5.1.dev134. But, we should really confirm that is the case. It would be good to confirm that this fixed at least in one tag, and figure out a test that can be done for that. It could be good to add a standard test, that at least exercises this, but it might require too much work to have a test that exercises it AND confirms that it's working. Those are good to have, but harder to put together.