A new system to convert between UCI, CSV, JSON, and HDF5...

[timcera]

Working on a a new system that will convert between several representations and wanted to announce so that if someone is working on the UCI parser we could work together.

What I have so far:

• New UCI parser that supports the Parameter ESTimation supplemental file which would close Support Parameter Estimation (PEST) supplemental file #172
• Creates a dictionary of pandas DataFrames that can be written to HDF5, CSV, or JSON files which would close Propose a space delimited UCI #171. The JSON files are direct representations of the HSPF tables, so not exactly what was discussed in Propose a space delimited UCI #171, however the table based JSON is a one-line addition. Thought could pursue other JSON views later.

Current problem children:

• SPEC-ACTIONS table is complex and different from all other tables. Leaving until last. Need to work with @rburghol to make sure get in place what he needs.
• Using src/hsp2/hsp2tools/data/ParseTable.csv to shift between the different representations which has worked out really well. Right at the moment though I am not creating the table names for HDF5 correctly. I think a minor change either to the code or to ParseTable.csv will get that fixed.

[timcera]

Just looking at other issues and see @rburghol work on #90, #99, #105, #106, and #107. Quite a bit to take in.

With my new parser I can easily create UVQUAN and UVNAME pandas dataframes which can then be converted to CSV, HDF5, and JSON, however the conditional lines are a real problem to represent in a pandas dataframe and don't know what is needed.

[rburghol]

Hey @timcera -- appreciate the patience on my response time! I will try to address at least some of this at the meeting this Friday, or at least, the part of the conversion system that overlaps with my thoughts on data models in general and STATE in specific.

My foundation thought on this is that regardless of the specific implementation (pandas, h5, ...) the current draft h5 data model has the right idea in terms of how to store things in a easily traceable manner, i.e.:

• Example: The reach quantity VOL:
  • UCI Param: /PARAMS/RCHRES_R001/HYDR-INIT/VOL
    • Tho an argument could be made that this would be stored in something more generic (i.e. not tied to a UCI block), like:
    • /PARAMS/RCHRES_R001/VOL/INITIAL_VALUE
  • State value during any timestep: /STATE/RCHRES_R001/VOL
  • End of timestep data for logging/export: /TIMESERIES/RCHRES_R001/VOL
    So, in short, any method of parsing a UCI or modifying params, would ultimately use that hierarchical data structure to read and write data to, which makes the implementation toolkit agnostic (provided that long path names are supported) and straight-forward to support collaboration.

[rburghol]

@timcera -- see PR here with informational code on STATE #187