Proposal: Collaboration on ICAR Pydantic Models for ADEWG

[maximeleduc]

My Forage Systems (Mon Système Fourrager), based in Canada, is a startup developing innovative solutions for forage systems by seamlessly connecting soil to the animal.

In this context, we will begin working on the animal module in approximately 6 to 8 weeks. We are going to be working soon on python data classes for the ICAR standard using pydantic models. pydantic allows to validate the conformity of json data with the schema classes. We were also considering to make it available on pypi as well to simplify installation in python projects.

Are you interested in having this eventually as part of adewg? If so, is anyone interested to collaborate?

In the case you are interested, to facilitate the eventual transfer it would be best to discuss about some details.

Do you have any special requirements or recommendations regarding this? Ex: license to use, disclaimers to add, documentation generation tools you prefer, folder structure/data organisation, etc.

At first we would create a repository called ICAR-pydantic and when ready you could fork in the group adewg.
We could transfer the ownership of the pypi package at that moment as well.

[jrobichaud]

@cookeac heres the plan I have to create an auto-generated python package from icar specifications.

This plan works best if #509 is going to create a github release like I suggested.

First I will create a github actions pipeline on the repository icar-pydantic repository. When a github tag will be created, a pipeline will start.

The pipeline:

1. download an ICAR artifact with the same tag that hopefully will be generated by Bundle schemes for code generation #509, otherwise the pipeline will have to generate the files by itself.
2. extract combinedURLScheme.json from the artifact
3. use datamodel-code-generator to generate pydantic classes.

   $ datamodel-codegen --input combinedURLScheme.json  --input-file-type openapi --output icar.py

   it generates a huge python file with all the data models in it, you can see an example here
4. run a python program that splits icar.py into a clean python package using the code refactoring tool rope:
   1. restructure back the data models in modules with the same structure as in the repository, using the directories as reference

   icar/__init__.py
   icar/resources.py
   icar/types.py
   icar/enums.py
   icar/collections.py
   icar/<other files based what is declared only in url-scheme like "IcarBatchResult">.py

   2. rename the data models to remove the prefixes "Icar" Ex: IcarMedicineTransactionResource -> MedicineTransactionResource. This is optional but I highly recommended it.
5. use ruff and isort to make sure the code is clean
6. run some simple automated tests on the generated modules
7. publish a github release on the repository pydantic-icar
8. generate a pypi package

What do you think of this plan?

[broadbent]

I think this is a great idea @maximeleduc @jrobichaud.

I am currently implementing a similar set of ADE Pydantic models with FastAPI as part of a (as of yet, unreleased) open source digital twinning project for dairy farms. However, I am doing this manually at the moment.

An automated pipeline would reduce duplication and ensure consistency, especially as the ADE specification evolves over time.

I would be willing to contribute time and effort to this - let me know how I can be of assistance. @jrobichaud's proposal seems logical to me.

[jrobichaud]

@broadbent

Here's the first version of the code generation adewg/ICAR-pydantic#1 if you feel like taking a look.

There is a pre-release package available on pypi https://pypi.org/project/ICAR-pydantic/ if you want to try it. I havent used it myself in a project, only in a few ultra basic unit tests in the package.

It requires to be explicitly installed since it is a pre-release.
Ex:

pip install ICAR-pydantic==1.4.0.dev2

This would not work:

pip install ICAR-pydantic