Current state of CIME data files

[jgfouca]

Hi all,

This is a follow-on discussion to the XML API discussion work that has been completed (not yet merged though as of this typing). Using the new API, I've confirmed that scanning for children is killing our performance. Note the 46% of runtime spent in scan_children (it would be a far higher % except for the fact that loading environment modules is crazy slow on this machine):

And the reason that we need to do so many scans is because it's hard to utilize assumptions about file structure to do vastly-more-efficient direct-child searches because our file formats are so inconsistent.

To help inform the discussion, here's a class diagram of the python classes in XML:

Black lines denote inheritance, red lines denote a "has-a" relationship with the critical Case class. Note that the case class splits it's has-a relationships to XML files between "entry-id" and "generic" files.

Diving deeper, here's an analysis of the files themselves:

Archive(GenericXML):
  * CIME/config/$MODEL/config_archive.xml
  * Format: custom

Batch(GenericXML):
  * CIME/config/$MODEL/machines/config_archive.xml
  * Format: custom

Compilers(GenericXML):
  * CIME/config/$MODEL/machines/config_compilers.xml
  * Format: custom, ACME and CESM have different formats (1.0 vs. 2.0)

Compsets(GenericXML):
  * CIME/config/acme/allactive/config_compsets.xml
  * CIME/src/drivers/mct/cime_config/config_compsets.xml
  * components can add more
  * Format: custom

EnvArchive(GenericXML):
  * $CASE/env_archive.xml : Note, this is the ONLY direct subclass of GenericXML that lives in the CASE directory
  * Format: custom, similar to Archive

Grids(GenericXML):
  * CIME/config/$MODEL/config_grids.xml
  * Format: custom, ACME and CESM have different formats (1.0 vs. 2.0)

Machines(GenericXML):
  * CIME/config/$MODEL/machines/config_machines.xml
  * Format: custom

Pes(GenericXML):
  * CIME/config/acme/allactive/config_pesall.xml
  * Format: custom

TestReporter(GenericXML):
  * ? (CESM only)
  * Format: custom

Tests(GenericXML):
  * CIME/config/config_tests.xml
  * Format: custom

TestSpec(GenericXML):
  * ? (CESM only)
  * Format: custom



Component(EntryId):
  * CIME/src/drivers/mct/cime_config/config_component.xml + config_component_$MODEL.xml
  * Format: entry_id->entry->(various entry info)

Headers(EntryId):
  * CIME/config/config_headers.xml
  * Format: custom, why is this a subclass of EntryId?

Files(EntryId):
  * CIME/config/config_headers.xml
  * Format: entry_id->entry->(various entry info)

NamelistDefinitions(EntryId):
  * Lots
  * Format: entry_id->entry->(various entry info)

PIO(EntryId):
  * CIME/config/$MODEL/machines/config_pio.xml
  * Format: entries->entry->(values) , why different root element tagname?



EnvBatch(EnvBase):
  * $CASE/env_batch.xml
  * Format: file->group(job)->entry + noncompliant batch_system block

EnvBuild(EnvBase):
  * $CASE/env_build.xml
  * Format: file->group->entry fully compliant file!

EnvCase(EnvBase):
  * $CASE/env_case.xml
  * Format: file->group->entry fully compliant file!

EnvMachPes(EnvBase):
  * $CASE/env_mach_pes.xml
  * Format: file->group->entry fully compliant file!

EnvMachSpecific(EnvBase):
  * $CASE/env_mach_specific.xml
  * Format: custom

EnvRun(EnvBase):
  * $CASE/env_run.xml
  * Format: file->group->entry fully compliant file!

EnvTest(EnvBase):
  * $CASE/env_test.xml
  * Format: file->group(job)->entry + noncompliant TEST block

The things that stick out to me:

• EnvArchive is strange. It's the only direct subclass of GenericXML that lives in the case directory, but it's totally non compliant with our standard env file formatting.
• Headers seems like is should be a direct subclass of GenericXML, not an EntryId, since it has no entries.
• The root element of PIO is inconsistent with its siblings
• EnvBatch is almost compliant except for one block
• EnvTest is almost compliant except for one block
• EnvMachPes makes no effort at all to be compliant

Short-term path forward:

• Re-work class hierarchy to differentiate compliant files from noncompliant
• The performance of all compliant Env*.xml files can be drastically increased by leveraging the consistent file structure they all share when retrieving entry id values
• This would most-likely involve a cached entry_name -> (group or entry node) map

Long-term path forward:

• It sure would be nice if there was more consistency all around
• break backwards compatibility by forcing more Env* files to be compliant
• more consistency between the direct subclasses of GenericXML would be nice (config_machines, config_grid), but is not a necessity since create_newcase is already pretty fast and the penalties for parsing these files is only incurred once (at create_newcase).

[jedwards4b]

I've fixed the issues in Headers and config_pio in my jpe/hide_xml_interface branch. I think we decided pretty early on that not all xml files would be compliant with the entryid format, that's why its a separate class from genericxml. The more compliance we have the easier it is to maintain, but I think that there are valid reasons for non compliant files or blocks within files.

[jgfouca]

Update: the scan_children cost has been significantly reduced (factor of 6 or so) by pre-caching entries in EnvBase, a PR that went in a while ago. The next hotspot in case.setup is in the buildnmls. Some poking around revealed another significant source of performance loss for us in CIME: we read and parse the same XML files over and over even though they aren't changing. We are pretty good about not doing this unless necessary for the env XML files managed under the Case class. The problem seems to be with other XML files.

I printed the XML file reads taking place during buildnml for a basic CIME case:

JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_files.xml
JGF read XML: config/acme/config_grids.xml
JGF read XML: src/components/data_comps/datm/cime_config/namelist_definition_datm.xml
JGF read XML: src/components/data_comps/dice/cime_config/namelist_definition_dice.xml
JGF read XML: src/components/data_comps/docn/cime_config/namelist_definition_docn.xml
JGF read XML: src/components/data_comps/drof/cime_config/namelist_definition_drof.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_drv.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml
JGF read XML: src/drivers/mct/cime_config/namelist_definition_modelio.xml

To summarize the above, we did 32 XML file reads, 24 of which were redundant.

I believe the next step in our quest for performance is to add a filename + hash cash to GenericXML. This will have the additional robustness benefit of multiple different GenericXML objects that wrap the same XML file will have the same ElementRoot, so the in-memory representation will be consistent across all of the objects.

[jgfouca]

Update: full-file caching of read-only XML files has been implemented. I think we can now cautiously say that CIME offers reasonable performance when used correctly.

That brings me to my next concern: the weak guarantees offered by the CIME data model (by that I mean the database of env XML values for a case). The value for an XML entry id for an active CIME process exists in at least 3 places:

1. storage in a local variable, e.g. foo = case.get_value("FOO")
2. The special entry-id cache in EnvBase
3. The in-memory XML representation in GenericXML
4. The value it has on-disk in its XML file

As a rough analogy, think of (1) as a CPU register, (2) as a CPU cache, (3) as main memory, and (4) as disk. All these things need to be coherent or things will get very weird fast, and the same is true for CIME.

There are two very obvious problems in our data model.

1. If you create two non-const GenericXML objects around the same XML file, these two instances will quickly fall out of sync:

obj1 = GenericXML("foo.xml")
obj2 = GenericXML("foo.xml")
obj2.add_child("bar")
obj1.get_child("bar") # None!

2. If the XML file is modified "behind the back" of GenericXML, things will be incoherent:

obj = GenericXML("foo.xml")
run_cmd("cp different_stuff.xml foo.xml")
obj.get_child("different_stuff") # None!

3. Unflushed writes are an issue when the client accesses the Env objects directly (not through Case). We should consider having all XML files be accessed with a context manager.

obj = GenericXML("foo.xml")
obj.add_child("bar")
* Process exists, client never called write *
obj = GenericXML("foo.xml")
obj.get_child("bar") # None!

CIME currently works because we almost never have multiple non-const GenericXML objects wrapping the same XML file at the same time. This is because we have nicely encapsulated the Env XML objects in the Case class. This mostly addresses (1), although not with much rigor because there were some parts of the code violating this assumption and getting lucky that it never caused a failure.

We addressed (2) by calling case.read_xml() every time we think any of the env_.xml files may have change behind our backs. This does a very expensive full re-read of all the env.xml files. Unfortunately, it's completely up to the CIME developer to remember to call this when needed. So, again, there's nothing to catch potential mistakes.

Thoughts?