Idea for Improving PET’s Portability and Structure

[matthewaltenburg]

I’ve been going through the PET codebase and thinking about how the pieces fit together. One thing that stood out is how tightly the platform-specific configs (Gadi, Monsoon, JASMIN) are tied into the main library. It works, but it also mixes HPC details with what is otherwise a clean flow:

📁 data → 🔧 pipeline → 🧠 model → 🚀 training

I wondered whether it might make sense to separate the platform bits into small add-on packages — for example:

🏷️ PET-gadi
🏷️ PET-monsoon
🏷️ PET-jasmin
🏷️ PET-local (for anyone running on their own systems)

PET itself would stay as the core set of tools for data, pipelines, models, and training, without the HPC assumptions baked in.

Doing it that way might make PET more portable, easier to package, and friendlier to container-based workflows where they’re supported. Scientists could use the full stack or just the pieces they need, and the platform plugins would simply supply archive paths and site-specific quirks.

Not suggesting this is the “right” direction — just sharing the thought and curious what you think.

[tennlee]

Conceptually I agree. To start with, I thought that putting them both in the main repo gave people a template to copy off, and simplified code access in the first instance. But if scaling to many sites, or if allowing the concept of the site to separate data science project spaces, then something more separable would make sense to me. I can imagine, for example, a data engineer supporting say 3 people on a specific project, with project-specific data which isn't suited for the open repo, by creating an installable package with the data accessors that everyone can then use. If this is something you'd like to work on, let's chat about what it might look like in practise. In terms of current user needs however, it's probably not the topmost of people's concerns, so it might not be prioritised for a while. But it's the right direction to look at taking things.