[FEATURE] Refactor and streamline metadata creation

[nesnoj]

Metadata creation should be streamlined and become more easy for users.
The current metadata files are located here.

Requirements:

• Automatic creation of one metadata.json for each DB table we want to publish, naming: db_schema.db_table.json
• Easy-to-fill-out template (e.g. yml) for users with examples on non-inferred/non-autofill fields
• Auto-validation of metadata via OMI
• Templates for common content such as licenses, context, ... (most are already there)
• OEMetadata v2 conformant Update to OEMetadata Standard v2 #1177
• Updated how-to for users in readme

Open questions:

• Where/how is defined which DB tables are to be published (and hence need metadata)? Central list?
• Do we want a metadata.json for each dataset (in the pipeline class sense)?
• Do we want a metadata.json for all results?

Depends on:

• Information on data i/o for a dataset file-based datasets
  [FEATURE] Unify setting source and target tables #1283
  Add structure to unify setting sources and targets of Datasets #1304

Further work/notes:

• Remove legacy code and other auto-created metadata (example 1, example 2 , just search for "add_metadata")

Please amend @jh-RLI if needed, feel free to create sub-issues.

[jh-RLI]

Automatic creation of one metadata.json for each DB table we want to publish, naming: db_schema.db_table.json

In oemetadata v2 we have introduced a new field for "topics". The pattern of 'schema.table' is quite custom to the OEP and was mainly implemented as the OEP uses the Topics to group data tables. Currently, the topics are also created on the OEDB as DB schemas, but this approach will be updated and database schemas will be harmonized to one "datasets" schema. The topics will be moved to the application, they will still exist and group data in the Webapp but they will no longer be part of the database structure.

As data tables might be part of multiple tables the oemetadata topics field is a list.

If the pattern of schema.table is not yet implemented in the egon data code, then i would vote for separating this information as suggested.

[nesnoj]

Related issues: #451 #457
(@jh-RLI please check and close if obsolete..)

[jh-RLI]

Is it possible to run eGon-data with python version > 3.8. For me, it would be helpful to be able to use at least python 3.10. I was able to install the environment using py3.10 but can't judge yet that everything still works.

Background is that we updated helpful packages like omi to use at least py 3.10.

Edit: Actually, the python version is blocking the "optimal" metadata integration.

[jh-RLI]

See #1344 (comment)