Add a basic schema for a pipeline

[surchs]

If I describe a pipeline to this catalogue so a computer can use its name as a controlled term, I want to know whether what I'm adding makes sense to a computer.

As a starting point, we can probably just model a pipeline as:

• a name (a string)
• versions (an array of strings)

TODO

• Agree on a definition of a pipeline
  • this might turn tricky. I.e. if I make a thing that wraps other existing and named pipelines into a sequence / workflow and I give that thing a name and version, is it a pipeline?
  • if I release a couple of helper tools with my "pipeline", do we consider the helper tools pipelines of their own (i.e. is a pipeline just anything that nipoppy can run?): https://github.com/nipoppy/nipoppy/blob/756fb3532cc201986300ae2b0095429e179e5a9a/nipoppy/data/descriptors/dcm2bids_helper-3.1.0.json
• Agree on the schema
• put the schema in a nice file. We could try linkML for its YAML prowess (and to get a better sense of it) and because it might be easy for people to edit/contribute
• Setup some basic validation for instances of pipelines

[michellewang]

if I make a thing that wraps other existing and named pipelines into a sequence / workflow and I give that thing a name and version, is it a pipeline?

I would say yes this counts as a pipeline

if I release a couple of helper tools with my "pipeline", do we consider the helper tools pipelines of their own (i.e. is a pipeline just anything that nipoppy can run?): https://github.com/nipoppy/nipoppy/blob/756fb3532cc201986300ae2b0095429e179e5a9a/nipoppy/data/descriptors/dcm2bids_helper-3.1.0.json

For reference, in the current Nipoppy configuration for dcm2bids, the pipeline name is dcm2bids but we have different steps and one of them is dcm2bids_helper

Agree on the schema

Mentioned to @alyssadai yesterday but eventually on the Nipoppy side we would want the ability to add additional information for specific versions of a pipeline. So then maybe versions wouldn't be a list of string. We don't need this now but just a heads up that in a few months we might need to come back to this.

put the schema in a nice file. We could try linkML for its YAML prowess (and to get a better sense of it) and because it might be easy for people to edit/contribute

Agreed that this would be a good opportunity to try LinkML!

[michellewang]

@surchs @alyssadai @nikhil153

@mathdugre and I are working on updating nipoppy to be able to download/upload pipeline configuration file bundles from Zenodo. In the ~near future, nipoppy init will not add any "default pipelines" to a new dataset -- instead, users will be expected to run something like nipoppy pipeline install (or nipoppy pipeline add) to download the files from Zenodo. We were thinking we could have the Zenodo IDs for the currently "known" pipelines/versions in the pipeline catalog directory.

So this will require a schema change. Currently we have this:

[
    {
        "name": "fmriprep",
        "versions": [
            "20.2.7",
            "23.1.3"
        ]
    }
]

We could maybe change it to something like this:

[
    {
        "name": "fmriprep",
        "versions": [
            {
                "version": "20.2.7",
                "zenodo_ids": ["zenodo.000000", "zenodo.111111"]
            },
            {
                "version": "23.1.3",
                "zenodo_ids": ["zenodo.222222", "zenodo.333333"]
            }
        ]
    }
]

Am not a fan of having versions and version so feel free to suggest alternatives!

[surchs]

Am not a fan of having versions and version

agreed.

How about this instead?

[
    {
        "name": "fmriprep",
        "versions": {
          "20.2.7":
            {
                "zenodo_ids": ["zenodo.000000", "zenodo.111111"]
            },
          "23.1.3":
            {
                "zenodo_ids": ["zenodo.222222", "zenodo.333333"]
            }
          }
    }
]

[michellewang]

Yeah that could work. I was trying to avoid having versions directly as keys but we might end up having to do that. Thanks!

[alyssadai]

Revised current proposal, including an optional url

[
    {
        "name": "fmriprep",
        "url": "https://fmriprep.org/en/stable/",
        "versions": [
            {
                "tag": "20.2.7",
                "zenodo_ids": ["zenodo.000000", "zenodo.111111"]
            },
            {
                "tag": "23.1.3",
                "zenodo_ids": ["zenodo.222222", "zenodo.333333"]
            }
        ]
    }
]

Other potential additions:

• If Zenodo IDs will be automatically extracted to populate zenodo_ids, to make it easier to determine which invocation to use, or which is a good default (on the Nipoppy side), we could have a more detailed value like:

"zenodo_records": [
  {
     "id": "zenodo.222222",
     "description": "ENIGMA-PD fMRIPrep"
     "nipoppy_default": true,
  }
]

[michellewang]

@mathdugre thoughts

Should we implement a Zenodo search instead of a pipeline catalog search in Nipoppy? Similar to bosh search in Boutiques

• This way users can always get up-to-date information from Zenodo
• So the pipeline catalog would still only be used by the Neurobagel CLI, and it wouldn't need to know information about Zenodo IDs. This also means the schema doesn't have to change (though @mathdugre points out that not having pipeline names/versions as keys makes searching less efficient)
• We would have some CI workflow to crawl Zenodo and open a PR to update the pipeline catalog with new pipeline names/versions
  • This would include some validation that the files in the Zenodo records work with the latest version of Nipoppy