Request for guide/guidance for adding new Atlas / CoordinateSpace / Parcellation to openMINDS (specifically Marmoset)

[baishi]

Hi,

I'm tasked to investigate how to include our Marmoset atlas into the Siibra suit. I noticed there are multiple references to the OpenMinds/ParcellationEntities etc. So I wish to craft the needed metadata.

I searched around and didn't find a holistic document or step by step guide about this task.

My understanding is that to add a whole new set of Atlas / CoordinateSpace / Parcellation of a new Species, I need to craft the BrainAtlas, BrainAtlasVersion, commonCoordinateSpaces, commonCoordinateSpaceVersion, parcellationEntity, parcellationEntityVersion. So is there any script to ensure everything mandatory is populated and there is no typo etc or I need to copy an existing one as template and fill in accordingly?

Also I noticed there are overlapping attributes in say BrainAtlas and BrainAtlasVersions. So with the same attribute, does the BrainAtlas's value should be the value of the latest version? Is there any overriding happening in the resolving process?

When I get the metadata ready, do I create a PR for the files?

Best Regards,
Shi

[baishi]

Hi @Raphael-Gazzotti, any input please?

[apdavison]

Hi @baishi - I don't think we have a step-by-step guide yet. We'll discuss this in our next developers' meeting and get back to you.

[UlrikeS91]

Hi @baishi,

@apdavison is correct. We do not have a step-by-step guide yet, but I can help you getting started.

1. Overview

Since the Marmoset atlas does not exist in openMINDS yet, we will have to register all atlas elements for the atlas. These include:

1. brain atlas (with nested parcellation terminology)
2. brain atlas version (with nested parcellation terminology version)
3. common coordinate space
4. common coordinate space version (which is linked by one or multiple brain atlas version that used it)
5. parcellation entities (which make up the terminology across all versions of the atlas)
6. parcellation entity versions (which make up the specific terminology used within a specific version of the atlas; with nested information about the atlas annotations)

2. Version entities vs. version-independent entities

Version-independent entities: brain atlas (& parcellation terminology), common coordinate space, parcellation entity
Version entities: brain atlas version (& parcellation terminology version), common coordinate space version, parcellation entity version (& atlas annotation)
As you noticed, there are several overlapping attributes between them. We typically register the same value for both unless there are any reasons for having something different (e.g., a new name and abbreviation is assigned to a parcellation entity version in the latest version which has a different name in the version before). Even though the value for the version entity could be inherited from the version-independent one, we try to be as explicit as possible (i.e., duplication is prefer for the openMINDS instances). The "versionIdentifier" specifically identifies the version and is combined with, e.g., abbreviation or short name to create specific identifiers for the different entities (e.g., WHSSDatlas vs. WHSSDatlas_v4). Here is an overview of relevant properties and how we typically handle them for integrating atlas-related instances into the openMINDS_instances repository:

AtlasEntity(Version)	property	duplication?
BrainAtlas(Version)	abbreviation	yes
BrainAtlas(Version)	description	yes ("versionInnovation" of version should be used to provide version specific descriptions)
BrainAtlas(Version)	digitalIdentifier	no; must be specific, leave blank if none exists
BrainAtlas(Version)	fullName	yes
BrainAtlas(Version)	homepage	yes/no; preferably specific, but one webpage may describe both
BrainAtlas(Version)	howToCite	typically specific or BrainAtlas has none
BrainAtlas(Version)	ontologyIdentifier	no; must be specific, leave blank if none exists
BrainAtlas(Version)	shortName	yes
CommonCoordinateSpace(Version)	abbreviation	yes
CommonCoordinateSpace(Version)	description	yes ("versionInnovation" of version should be used to provide version specific descriptions)
CommonCoordinateSpace(Version)	digitalIdentifier	no; must be specific, leave blank if none exists
CommonCoordinateSpace(Version)	fullName	yes
CommonCoordinateSpace(Version)	homepage	yes/no; preferably specific, but one webpage may describe both (or may be the same as for the BrainAtlas(Version) that used/created it)
CommonCoordinateSpace(Version)	howToCite	typically specific or CommonCoordinateSpace has none
CommonCoordinateSpace(Version)	ontologyIdentifier	no; must be specific, leave blank if none exists
CommonCoordinateSpace(Version)	shortName	yes
ParcellationEntity(Version)	abbreviation	reflects atlas documentation; ParcellationEntity typically corresponds to latest version if changes have been made between versions
ParcellationEntity(Version)	alternateName	reflects atlas documentation (i.e., if multiple names have been used for the same entity, these may be stated here); we have e.g., used it to add older versions of the name to reflect their relationship
ParcellationEntity(Version)	hasParent	can point to same entity, reflects atlas documentation
ParcellationEntity(Version)	lookupLabel	no; must be specific and name/abbreviation + atlas abbreviation + versionIdentifier
ParcellationEntity(Version)	name	reflects atlas documentation; often yes
ParcellationEntity(Version)	ontologyIdentifier	no; must be specific, leave blank if none exists

For ParcellationEntity(Version) it is important to note that we only register delineated brain regions as ParcellationEntityVersions. Other regions used to create a hierarchy (i.e., the name assigned to a combination of several delineations) or that are included in the overall terminology but are not delineated yet are registered as ParcellationEntities.

You may use existing atlas representations as template/inspiration but please be aware that atlas representations can vary a lot and that our collection is still a work in progress. The most complete representations are the Julich Brain Atlas (JBA) and Waxholm Space rat atlas (WHSSDatlas).

3. Programmatic population

You can use one of our code packages to help you create the necessary JSON-LDs (e.g., https://github.com/openMetadataInitiative/openMINDS_python or https://github.com/openMetadataInitiative/openMINDS_matlab).

4. Integration into openMINDS

Yes, once you have created the necessary JSON-LDS, you can create a PR. Feel free to create a draft PR if you'd like my input on your work in progress. This way I review along the way and hel you reduce the amount of work to put into creating the atlas representation.

Let me know if anything is unclear or if you need more information about something.

[baishi]

Hi @UlrikeS91,

Thank you for the detailed guide. Now I know the direction and I will fork the repo and create a branch for you to review my addition.

In the meanwhile, do you have any docs about openMINDS_python?

I tried something like

>>> from openminds import Collection
>>> c = Collection()
>>> c.load('WHSSDatlas.jsonld')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/baishi/miniconda3/envs/python3/lib/python3.10/site-packages/openminds/collection.py", line 169, in load
    cls = lookup_type(data["@type"])
  File "/home/baishi/miniconda3/envs/python3/lib/python3.10/site-packages/openminds/registry.py", line 54, in lookup_type
    raise ValueError(f"Type '{class_type}' was not found in the registry for version {version}.")
ValueError: Type 'https://openminds.om-i.org/types/BrainAtlas' was not found in the registry for version latest.

I still don't know the basic usage of the module.

I wish to use the module to validate my jsonld file crafted.

Shi

[UlrikeS91]

@apdavison , could you help @baishi with the openMINDS_python code package?

[apdavison]

Hi @baishi

Before loading the collection, you need to import the relevant schemas, e.g.:

import openminds.latest as om

We're aware this is not intuitive, and plan to fix this in future (see ticket here).

You may also need to upgrade openMINDS Python to v0.3.0 (pip install -U openminds).

[baishi]

Hi @apdavison,

I'm trying to use this one to validate the metadata

import openminds.latest as om
pe = om.sands.ParcellationEntity()
pe.load('instances/latest/parcellationEntities/WHSSDatlas/WHSSDatlas_zonaIncertaDorsalPart.jsonld')
pe.validate()
defaultdict(list, {'required': ['name is required, but was not provided']})

But the file indeed has "name" attribute.

On the other hand

ba = om.sands.BrainAtlas()
ba.load('instances/latest/brainAtlases/WHSSDatlas.jsonld')
ba.validate()
defaultdict(list,
            {'required': ['authors is required, but was not provided',
              'description is required, but was not provided',
              'full_name is required, but was not provided',
              'has_terminology is required, but was not provided',
              'has_versions is required, but was not provided',
              'short_name is required, but was not provided']})

However, authors, description fullName, hasTerminology hasVersion are defined in the metadata file.

Is that a mismatch or something I used wrongly?

Best Regards,
Shi

[apdavison]

Hi @baishi

Loading metadata should be done like this:

import openminds.latest as om

pe = om.sands.ParcellationEntity.load('instances/latest/parcellationEntities/WHSSDatlas/WHSSDatlas_zonaIncertaDorsalPart.jsonld')

ba = om.sands.BrainAtlas.load('instances/latest/brainAtlases/WHSSDatlas.jsonld')

[baishi]

Hi @apdavison

It didn't work for me:

In [7]: import openminds

In [8]: print (openminds.__version__)
0.3.0

In [1]: import openminds.latest as om
In [3]: pe = om.sands.ParcellationEntity.load('instances/v3.0/terminologies/UBERONParcellation/cerebellarNuclearComplex.jsonld')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 pe = om.sands.ParcellationEntity.load('instances/v3.0/terminologies/UBERONParcellation/cerebellarNuclearComplex.jsonld')

File ~/miniconda3/envs/python3/lib/python3.10/site-packages/openminds/base.py:218, in LinkedMetadata.load(cls, file_path)
    216 with open(file_path, "r") as input_file:
    217     data = json.load(input_file)
--> 218 return cls.from_jsonld(data)

File ~/miniconda3/envs/python3/lib/python3.10/site-packages/openminds/base.py:112, in Node.from_jsonld(cls, data)
    110 type_ = data_copy.pop("@type")
    111 if type_ and type_ != cls.type_:
--> 112     raise TypeError(f"Mismatched types. Data has '{type_}' but trying to create '{cls.type_}'")
    113 deserialized_data = {}
    114 if issubclass(cls, LinkedMetadata):

TypeError: Mismatched types. Data has 'https://openminds.ebrains.eu/controlledTerms/UBERONParcellation' but trying to create 'https://openminds.om-i.org/types/ParcellationEntity'

In [4]: pe = om.sands.ParcellationEntity.load('instances/latest/parcellationEntities/WHSSDatlas/WHSSDatlas_zonaIncertaDorsalPart.jsonld')

In [5]: pe.validate()
Out[5]: 
defaultdict(list,
            {'type': ["has_versions: Expected ParcellationEntityVersion, value contains <class 'openminds.base.Link'>"]})

I ran the script against main/HEAD

commit 4b3f8206f5e70476ec345b6669cd463d544f2091 (HEAD -> main, origin/main, origin/HEAD)
Merge: 11bd7ab2 b50e012d
Author: Lyuba Zehl <lyuba.zehl@ebrains.eu>
Date:   Wed Apr 16 10:11:07 2025 +0200

    Merge pull request #263 from Raphael-Gazzotti/instance-validation
    
    Reusable validation workflow from openMINDS_actions.

[apdavison]

There are a few issues here:

1. You need to match the openMINDS version of the schemas and the instances. You were trying to load an instance from "instances/v3.0", which means you need version 3 of the schemas:

import openminds.v3 as omv3

cnc = omv3.controlled_terms.UBERONParcellation.load('instances/v3.0/terminologies/UBERONParcellation/cerebellarNuclearComplex.jsonld')

2. You need to use the correct class for the instance (you were trying to load an UBERONParcellation instance using the ParcellationEntity class).

3. In your final example, the validation error is because "has_versions" contains just a link to another instance, so the validator cannot confirm the linked instance is of the correct type. For a full validation you would need to download all the linked instances and load everything at once into a Collection. To suppress this error, use pe.validate(ignore=["type"])

[lzehl]

@baishi just a small note from my side here before we hopefully meet for further discussions :) It is not enough for an instance to list the attribute with a null value to fulfill requirement. It actually has to provide a value. However, we are diverging a bit from openMINDS for atlases and common coordinate spaces in the openMINDS registration (to not collide with systems like EBRAINS and their registration, as for e.g. Persons). Fixing this is the required properties is on our TODO list for the next version.

Moreover, there are other conventions from for controlled openMINDS instances that are not validated. For example: Underscore is used as a specific separator in atlas instances (between certain components in the atids). We do not allow this to be used to replace white spaces.

I'm looking forward to meeting you and discuss the Marmoset Nencki-Monash template :)

[lzehl]

@baishi since I cannot raise issues on your fork I will provide first feedback here on the instances you've created for the NM Template and the NM Atlas.

Let's start with the ParcellationEntityVersions (PEVs):

We follow an internal convention for PEV "name", "@id", "lookupLabel", "abbreviation", and "alternateName":

• "name": Stick to the name and capitalization actually used in the atlas data. I saw in your case that you do not capitalize in the atlas labels file (e.g. "areas 1 and 2 of cortex" not "Areas 1 and 2 of cortex"). Please see also my comment below about parent structures.
• "abbreviation": Stick to the abbreviation you use in your atlas data (e.g. in file names). I think you are not providing them at all at this moment.
• "lookupLabel": Should use the following convention ATLAS-ABBREVIATION_VERSION_AREA-NAME-LOWER-CAMEL-CASE (MNMA_v1.0_areas1And2OfCortex) or ATLAS-ABBREVIATION_VERSION_AREA-ABBREVIATION (MNMA_v1.0_A1-2). Alternatives for the atlas abbreviation: "MNMA", "MNMAtlas", "MNM-atlas", "MarmosetNMAtlas", etc.
• "@id": Should be composed of "https://openminds.om-i.org/instances/parcellationEntityVersion/" + whatever you decide for the lookupLabel.
• "alternateName": Has to be null or a list, even if this list only contains one alternate name. Alternate name can contain whatever name you have used for that PEV in any publication. In your current case this could be ["A1/2", "area 1 and 2"] or ["A1/2", "area 1 and 2 of cortex"] (depending on what you use as name).

PEVs should potentially be hierarchically structured by identifying their anatomical parent structure. Parent structures if not separately provided as annotation should be conceptually defined as ParcellationEntities (PEs). Identifying parent structures could also simplify naming of PEVs (not sure how flexible you are here in changing them still). For example: PEV "MNMA_v1.0_areas1And2" could identify PE "MNMA_cortex" as parent instead of implicitly mentioning the parent only in the PEV name as done right now ("MNMA_v1.0_areas1And2OfCortex").

As AtlasAnnotations (AAs) you do provide multiple as far as I know in two different spaces:

1. probabilistic map (criteria type probabilistic; NM-volume-template)
2. maximum probability projection (criteria type deterministic; NM-volume-template)
3. surface map (at least for the cortical areas I assume) (criteria type deterministic ?; NM-surface-template)

You need one PEV for each space. And in the PEV of the NM-volume-template should have at least 2 annotations. It might be more if the area annotation in the left and right hemisphere have different identifiers or are stored in different files. As suggested we can discuss this further in our meeting.

Version identifier: you've stated "v1" so far which is fine if you only consider major updates in future. If minor updates are possible I would suggest to use "v1.0".

More feedback for the other instance types (parcellation entities, brain atlas, brain atlas version, etc) will follow shortly. I hope this helps already adapting your scripts :)