Using old data files with Otoole

[vignesh1987]

Hi, I have a question on the otoole convert command. I see that the format for the simplicity data file (generated through Otoole) is slightly different from the older Gnumathprog formats. For example:
In the otoole format, the Operational life parameter is defined as in the example below

param default 1 : OperationalLife :=
SIMPLICITY ETHPLANT 30
SIMPLICITY GRID_EXP 50
;

In the older data file format, it used to be as in the example below. where the region index is defined as a group.

param OperationalLife default 1 : 
ETHPLANT GRID_EXP :=
SIMPLICITY 30 50
;

I get the following error when I want to use Otoole to convert old data files to CSV or Excel.

TypeError: object of type 'TabularRecord' has no len()

Is there an easier way to work around this issue?
Attached is the older data file I am referring to:
data_lao.txt

[willu47]

Hi @vignesh1987 - thanks for the bug report. Initially, I thought this is likely an issue with Amply, but I see that Amply does support this format for representing parameters (see tests).

Please could you share the full error report from otoole by running otoole with the -vvv flag. e.g. otoole -vvv convert datafile csv data_lao.txt data_lao_csv

[vignesh1987]

Hi @willu47,

Thanks for the pointers. I ran the code with the -vvv flag, and the error log is attached.
error_log_TabularRecord.txt

It refers to the first instance of the records list (line 308, Amply.py) and evaluates the length not equal to 1, if I am thinking correctly.

[vignesh1987]

Hi @willu47

I tried to debug by comparing the SIMPLICITY example. Solved the issue for this case, but I am not sure it is a fix for every data file.
Some pointers below

1. You are correct. amply.py can accept the older (bridged) format as well.
2. The Tabular record error was created for new parameters that have by mode definition (which were not part of the config.yml file on GitHub)
   TechnologyActivityByModeLowerLimit TechnologyActivityByModeUpperLimit TechnologyActivityDecreaseByModeLimit TechnologyActivityIncreaseByModeLimit
3. The new UNDESA interface and CLEWs models use COMMODITY instead of FUEL. I am not sure if there is a consensus on what should be used. It may be worthwhile thinking about streamlining. Maybe I am totally wrong!

[willu47]

Hi @vignesh1987 - thanks for reporting back.

So the issue is that if the list of parameters and sets in the config.yaml file does not match what is in the data file, we get a really cryptic error message?

For otoole, there's no issue with using fuel or commodity, as long as the set name is updated in the config.yaml file.

[trevorb1]

Hi @vignesh1987 and @willu47! I had a similar error message from Amply pop up before (see issue #151 which was replaced by issue #160).

Also, I think @willu47's good point of "So the issue is that if the list of parameters and sets in the config.yaml file does not match what is in the data file, we get a really cryptic error message?" is expanded upon in the (still open) issue of issue #169. In this issue, the error message of OtooleNameMismatch is explained to be a little to nondescript, so maybe moving to a more descriptive one (such as DatafileParseError as described by @willu47 in issue #182) would be better!

Thanks so much @vignesh1987 for reporting this issue!!

[vignesh1987]

Thanks, @trevorb1 and @willu47 : I wrote something simple to compare the datafile and the config.yml file. Basically, something that warns the user of parameters that are in the config file but not in the data file. Will that be useful here? Should it be made generic (excel, datafile and CSV folder VS the config file)? Also will make some effort to get involved more in contributing better (I have been wanting to do this for ages and never materialised).

[willu47]

Sounds useful. We have validation for the other formats. The data files are the tricky ones because of how the amply parser works. Technically amply could be modified to extract the parameter names, but it is a difficult library (build on pyparsing) to work with. Are you using a regular expression to search for the param and set keywords?