Script to collect metadata

[ryanmrichard]

We need a python script that will loop over submodules of the data lake and pull the metadata out into a single JSON file stored in the data lake.

[sadak2004]

1. What should the final single json file look like?

The single json file will be several schema 1.0 files

[ryanmrichard]

I was thinking something like:

{ 'name of submodule 0' : {JSON from submodule 0},
  'name of submodule 1' : {JSON from submodule 1},
  ...
}

The decision to use an associative array is so that it is easier to update later, e.g., if submodule 1 changes, you just have to delete the associated value. If it were just a list, it would be a bit hard to figure out which entry needs updating unless you somehow fixed the order of the submodules.

[sadak2004]

all_metadata.json

I have a script that compiles the all the metadata.json files into one ON A SEPARATE BRANCH called sidd3 Currently, it looks a list:

[
{
"dataset_name": "Iowa Lake Water Quality",
"description": "Measurements of nutrient concentrations in Iowa lakes from 2010 to 2020.",
"version": "1.1.0",
"created_on": "2023-02-10",
"updated_on": "2025-06-15",
"contributors": [
............................................
{
"dataset_name": "US River Discharge Measurements",
"description": "Hourly river discharge data for major US rivers from 2005 to 2022.",
"version": "2.0.3",
.................................................................

[sadak2004]

Here is a version that shows the submodular folder it was in ON THE sidd3 BRANCH

all_metadata2.json

:{
"sidd": {
"dataset_name": "Iowa Lake Water Quality",
"description": "Measurements of nutrient concentrations in Iowa lakes from 2010 to 2020.",
"version": "1.1.0",
"created_on": "2023-02-10",
..........................................................................
},
"test_ds": {
"dataset_name": "US River Discharge Measurements",
"description": "Hourly river discharge data for major US rivers from 2005 to 2022.",
"version": "2.0.3",
"created_on": "2022-11-05",
"updated_on": "2025-05-30",
"contributors": [

[ryanmrichard]

I'd vote for the latter format. Two submodules can not have the same name, and if we prohibit renaming submodules, we can then guarantee the keys are unique and persistent (relying on dataset_name could pose problems if the dataset is renamed or if people use the same name).