Cannot change RESUBMIT while model is running

[golaz]

In one of my recent simulation, I tried to increase the number of automatic resubmissions using:

./xmlchange -id RESUBMIT -val 6

Unfortunately, when the next job started, it stopped with the following error:

ERROR: File /compyfs/gola749/E3SM_simulations/20191216.alpha20.piControl.ne30_r05_oECv3_ICG.compy/case_scripts/env_run.xml appears to have changed without a corresponding invalidation, modtimes 1577796971.00 != 1577892633.00

Resubmitting the job manually afterwards with

./case.submit

worked and the job ran.

[rljacob]

I thought you can always edit env_run.xml without disrupting anything else in CIME. Is that not true?

[jedwards4b]

A feature implemented by @jgfouca - I think the issue is that there is an active object copy of env_run.xml in memory and writing to the file creates a discrepancy that is hard to reconcile.

[rljacob]

So to avoid the error, he'd have to wait for the current resubmissions to finish and then make the change?

[jedwards4b]

Yes, that's correct.

[golaz]

That seems like an inconvenience for the user. It's makes it more difficult to manage long simulations.

[billsacks]

In general, I really support what @jgfouca did to bring a bit more sanity and safety in the file locking. But I wonder if we should have a separate, very small file of variables that are safe to change in the midst of a run? I'm not sure what that would include other than RESUBMIT.... I'm also not sure if RESUBMIT is guaranteed to be safe to change in the midst of a run, but maybe it's a common enough desire to set that (and maybe a few other variables) that we can live with the small risk?

[billsacks]

Issues would arise if a cime process is trying to modify this file at the same time as a user. So actually, RESUBMIT is not completely safe to modify when the case is running, because the case will automatically change this variable at the end of the run, and if you get unlucky and run your xmlchange command at that same time, then you're in trouble.

It's all a matter of balancing risk vs. convenience....

[ekluzek]

The other way to do this would be to have the error show up when the xmlchange command was attempted -- rather than when the next submission happens. I'd rather see..

./xmlchange RESUBMIT=6

and have it tell me "You can't do that you idiot! You are currently running a simulation, and you can't change it until it finishes all of it's submissions. Or until you abort the current run. Idiot...", then have it abort the running simulation.

Would it be possible to have the error be reported in xmlchange? Or at least when check_case is run? I know that doesn't prevent you from hand editing the file, but you should expect problems then.

[golaz]

Personally, I have never been a fan of CIME's philosophy of protecting "stupid users from themselves"... I'm an optimist and prefer to think that users can be educated and should be empowered.

I can think of many legitimate reasons for wanting to change RESUBMIT, STOP_N, REST_N and possibly others for the next automatic resubmission.

[jgfouca]

@golaz , we are also protecting ourselves from from ourselves (cime developers). Prior to the additional error checking I put into the XML "database", it was extremely easy for CIME developers to introduce very subtle errors into CIME by changing XML files "behind-the-back" of the currently-running CIME process and then leaving the system in an inconsistent state.

See #2161 for the full details of that discussion.

There is definitely room for improvement in the error handling. With a sentinel file, xmlchange could know that the CIME case was actively running and immediately reject any changes as @ekluzek suggested.

Also relevant is a comment from me in another issue:

One thing that might be easier is if we further breakdown the env XML files in such a way that we know
it is safe to change values in file X during phase Y. For example, if we had env_submit.xml, we would
know it's safe to change env_submit.xml values during case_run.

This has come up at least 3 times in the last year, so we should probably discuss during tomorrow's telecon.

[jonbob]

For what it's worth, I agree with @golaz . E3SM is not a community project and is geared for users with a deeper understanding of the system. We need to provide capability for those users, not prioritize greater protection. I myself change RESUBMIT often during a run, depending on mid-run analysis, and find it very useful.

[billsacks]

I respectfully disagree with the sentiments raised by @jonbob and @golaz . I suspect that even most of the software engineers who work on E3SM and CESM aren't aware of many of the scenarios in which you could accidentally shoot yourself in the foot, particularly before the recent protections @jgfouca has added -- let alone more casual users. Even if most people read the documentation (which I suspect most people do not), educating people on all of the possible ways to shoot yourself in the foot would be a massive undertaking. I feel it is easier and far more robust to have protections in the software implementation.

That said, I'm hopeful that we can find a compromise for variables that are commonly modified during a run. (I wonder if there would be a way to have an xmlchange of RESUBMIT register the request for a change in some auxiliary file, and then this is reconciled with the actual env_run.xml at the end of a currently ongoing run.... details would need to be worked out.)