st_archive issues in ctsm5.3.041 with our testing

[ekluzek]

Brief summary of bug

Baseline testing shows the st_archive tests as failing.

General bug information

CTSM version you are using: ctsm5.3.041

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: st_archive in test lists

Details of bug

Tests when run show the the ST_ARCHIVE fails. For example...

FAIL SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL SHORT_TERM_ARCHIVER exception

Important details of your setup / configuration so we can reproduce the bug

Normal aux_clm test suite.

Important output or errors that show the problem

 ---------------------------------------------------
2025-04-29 14:41:04: Exception during SHORT_TERM_ARCHIVER:
ERROR: Moved file rpointer.lnd_9999.1976-01-03-00000 with dispostion ignore to directory env_archive_test/archive/rest/1976-01-01-00000
Traceback (most recent call last):
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/SystemTests/system_tests_common.py", line 788, in _phase_modifying_call
    function()
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/SystemTests/system_tests_common.py", line 775, in _st_archive_case_test
    result = self._case.test_env_archive()
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/case/case_st_archive.py", line 1260, in test_env_archive
    _check_disposition(testdir)
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/case/case_st_archive.py", line 1309, in _check_disposition
    _file, root
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/utils.py", line 176, in expect
    raise exc_type(msg)
CIME.utils.CIMEError: ERROR: Moved file rpointer.lnd_9999.1976-01-03-00000 with dispostion ignore to directory env_archive_test/archive/rest/1976-01-01-00000

 ---------------------------------------------------
 ./xmlquery -p DOUT

Results in group run_data_archive
	DOUT_S: FALSE
	DOUT_S_SAVE_INTERIM_RESTART_FILES: FALSE

[ekluzek]

This must be due to #3067, but I swear we didn't see this before. Although it the ctsm5.3.041 baselines on both Derecho and Izumi do show it as a problem.

It seems to be doing something with the tests in config_archive.xml, which I didn't think it exercised.

[ekluzek]

OK, first I figured out an easy way to test this. In a test case you just do this

./case.st_archive --test-case

(and also try --test-all to test the config_archive.xml file)

Next it looks like that cime has to be updated to cime6.1.091 and cmeps to 1.0.47.

There also seems to be a problem with this line in the config_archive.xml file (in the file testing section).

 <tfile disposition="ignore">rpointer.lnd_9999.1976-01-03-00000</tfile>

This filename should NOT be copied, but seems to being copied anyway. CAM has some files like this that seem to work correctly. So I'm not clear on what's going on here. But, should be able to get something working with at least a workaround.

[ekluzek]

OK, so this is dependent on cmeps and cime externals update, which we aren't quite ready for on master. And I'm not sure the st_archive is working properly in that I thought it shouldn't move files that weren't for the date from the cpl files of 1976-01-01-00000, so there might be a problem there that we should resolved. And most importantly we need to get the tests working in config_archive.xml so that the st_archive part won't fail.

So we'll revert this both on master and b4b-dev. And then on b4b-dev we'll rebring it in with submodule updates needed as well as changes so that the CTSM config_archive.xml will PASS.

[ekluzek]

I was afraid that this bug meant that st_archive would also fail for production cases that use create_newcase rather than create_test.

So to check in ctsm5.3.041 I ran this test:

./create_newcase --compset I1850Clm60BgcCropCrujra --res f09_f09_mg17 --case st_archive_test_case --run-unsupported
./xmlchange STOP_N=1
./xmlchange STOP_OPTION=nmonths
./case.setup
vi user_nl_clm.     # Add use_init_interp = TRUE
./case.build
./case.submit

The st_archive batch log ends with "st_archive completed" and all of the files seem to be archived correctly.

ls -l /glade/derecho/scratch/erik/archive/st_archive_test_case/rest/0001-02-01-00000/
total 2179028
-rw-r--r-- 1 erik cgdtss        257 May  7 13:59 rpointer.atm.0001-02-01-00000
-rw-r--r-- 1 erik cgdtss         47 May  7 13:59 rpointer.cpl.0001-02-01-00000
-rw-r--r-- 1 erik cgdtss        513 May  7 13:57 rpointer.lnd.0001-01-01-01800
-rw-r--r-- 1 erik cgdtss        257 May  7 13:59 rpointer.lnd.0001-02-01-00000
-rw-r--r-- 1 erik cgdtss        257 May  7 13:59 rpointer.rof.0001-02-01-00000
-rw-r--r-- 1 erik cgdtss  327323912 May  7 13:59 st_archive_test_case.clm2.h0.0001-01.nc
-rw-r--r-- 1 erik cgdtss 1736211611 May  7 13:59 st_archive_test_case.clm2.r.0001-02-01-00000.nc
-rw-r--r-- 1 erik cgdtss     656140 May  7 13:59 st_archive_test_case.clm2.rh0.0001-02-01-00000.nc
-rw-r--r-- 1 erik cgdtss  103301164 May  7 13:59 st_archive_test_case.cpl.r.0001-02-01-00000.nc
-rw-r--r-- 1 erik cgdtss    2318584 May  7 13:59 st_archive_test_case.datm.r.0001-02-01-00000.nc
-rw-r--r-- 1 erik cgdtss   32157692 May  7 13:59 st_archive_test_case.mosart.h0.0001-01.nc
-rw-r--r-- 1 erik cgdtss   29037088 May  7 13:59 st_archive_test_case.mosart.r.0001-02-01-00000.nc
-rw-r--r-- 1 erik cgdtss     257292 May  7 13:59 st_archive_test_case.mosart.rh0.0001-02-01-00000.nc

I then continued the run setting CONTINUE_RUN=TRUE to verify it could continue on. And it was fine.