OME Metadata Support

[joshmoore]

This issue captures the requirements as well as the possible implementation choices for a first integration of OME metadata into the OME-NGFF container. The UoD team is targeting the specification as well as implementations, including within OMERO, by mid-2022.

Goal

NGFF specifications up to version 0.4 contain only minimal metadata fields that cover the existing OME model (e.g. physical pixel size). As a result, converting data into OME-NGFF, e.g. with bioformats2raw, loses more metadata than the equivalent conversion to OME-TIFF. The goal of this issue is to achieve a parity between the two formats in terms of capturing metadata contained in the OME model (2016-06).

Here we would like to discuss, plan, and specify an initial integration of the OME model into OME-NGFF. As with other specifications, this initial work will likely be followed by multiple, possibly breaking, changes to expand the scope. Where possible, we will also try to capture that roadmap here.

In-scope requirements

• The primary requirement is the ability to fully convert an OME-TIFF in its entirety into an OME-NGFF dataset.

• As a corollary, it should be also possible to capture what Bio-Formats knows about proprietary files formats (PFFs). This should cover minimally the intermediate Zarr output of bioformats2raw and be readable by all readers (not just raw2ometiff).

• The specification should make clear how the combination of the OME model metadata and NGFF metadata is to be interpreted and what readers should do in the case of a conflict.

Out-of-scope requirements

• This need not be the final mechanism used for OME-NGFF to store metadata. It is more important to capture the metadata that exists today.

• It is not necessary, at least initially, that an OME-NGFF be fully convertible back into an OME-TIFF since, e.g., there is no location for labels, transformations, or file annotations in OME-TIFF.

---

Design decision #1: Location of metadata

The current NGFF specification solely uses the formats custom-attributes (.zattrs) for storing metadata. Several other locations are conceivable, though some are more or less within the bounds of the Zarr specification (See zarr-developers/zarr-specs#112 for more discussion.)

Option a) .zattrs

The status quo at the moment is that all metadata should be represented as JSON in .zattrs. The benefit is that no new mechanism needs to be introduced. A downside is that metadata is spread across multiple zgroups and zarrays (See related comments in #102). Projects such as xarray store metadata in “well-known” keys within the .zattrs like _ARRAY_DIMENSIONS (docs).

Option b) Custom files

An alternative is to introduce new files outside the scope of the Zarr spec, which only defines .zattrs, .zarray, .zgroup, and chunk files. bioformats2raw currently stores metadata in a file named METADATA.ome.xml. Other projects like netcdf-c store custom files (e.g. .nczarr; docs) with their own proprietary customizations. The benefit of this strategy is maximum flexibility since no key conflicts can occur. Implementations may need to be aware that such files are essentially 1-dimensional byte arrays.

Option c) Arrays

Metadata files can be encoded as Zarr arrays, which is similar to option b) but does not require introducing any new Zarr behavior. Additionally, the files themselves can carry metadata in their own .zattrs and be chunked. However, all tools that wish to consume them must be Zarr-aware.

Option d) String

Metadata can be encoded as a single (albeit large) string within .zattrs. Depending on Design Decision #2, storing a single string with the metadata has the advantage of working with existing formats as well as consolidating the metadata, but it does require escaping, etc.

---

Design decision #2: Format of metadata

Similarly to #1, currently all metadata is stored as JSON within .zattrs.

Option a) Design a JSON format

The option closest to the current NGFF process would consist of specifying a new JSON format to capture all of the information in OME-XML. This process would likely be extended and would need to be maintained for some time. One route to achieving it would be to generate json-schema from the XSD using ome-types.

Option b) Use the JSON-LD syntax of OME-OWL

Using JSON-LD would keep the metadata in JSON but would make use of the existing work on OME-OWL, and therefore not create another format that needs supporting. Additionally, the JSON-LD model provides an extensibility that is needed within the community. The downside is increased complexity in the programming model.

Option c) Store the OME-XML directly.

Finally, if the first goal is to support the existing model, using the OME-XML model is likely the fastest route. Downsides include the general aversion felt towards XML as well as the need to map between XML elements/identities and objections specified within the JSON. There will also not be an extensibility (beyond the standard annotations) in the first instance.

---

Implementation reports

Below we enumerate possible implementations and (eventually) the status of investigations into each of them. If anyone else is interested in proposing (or especially prototyping) an implementation, please mention so below.

1b2c: standardize the current bioformats2raw format

Standardizing the bioformats2raw output would require:

• permitting multiple images in a single Zarr fileset which would likely be initially achieved by adopting the OME-XML collections as opposed to those discussed in Collections Specification #31
• reading the OME-XML from the well-known location (OME/METADATA.ome.xml) or potentially adding metadata to find that metadata
• deciding how duplicated metadata fields are to be handled (and/or synchronized)

An additional benefit of this implementation is that the current bioformats2raw code can be adopted as the official .

Related issues:

• Enumerate series in zarr metadata? glencoesoftware/bioformats2raw#126
• Document that axes/transformations should be consistent with METADATA.ome.xml glencoesoftware/raw2ometiff#71

[joshmoore]

Quick update that with the script below as well as ome/ome-zarr-py#174, it's possible to build a Python reader of the bioformats2raw output. The next step would likely be an implicit convention or explicit metadata for mapping between the IDs in the OME-XML and the groups in the Implicit spec. (Currently raw2ometiff assumes the values are the offsets into the array.)

ome_types parser

#!/usr/bin/env python
import ome_types
import os
import re
import sys
import tempfile
import xml
import zarr

from ome_zarr.io import parse_url
from ome_zarr.reader import Reader
from xml.etree import ElementTree as ET


def fix_xml(ns, elem):
    """
    Note: elem.insert() was not updating the object correctly.
    """
    if elem.tag == f"{ns}Pixels":
        elem.append(ET.Element(f"{ns}MetadataOnly"))


def parse_xml(filename):
    # Parse the file and find the current schema
    root = ET.parse(filename)
    m = re.match(r'\{.*\}', root.getroot().tag)
    ns = m.group(0) if m else ''

    # Update the XML to include MetadataOnly
    for child in list(root.iter()):
        fix_xml(ns, child)
    fixed = ET.tostring(root.getroot()).decode()

    # Write file out for ome_types
    with tempfile.NamedTemporaryFile() as t:
        t.write(fixed.encode())
        t.flush()
        return ome_types.from_xml(t.name)



def handle_ome_zarr(filename):
    reader = Reader(parse_url(filename))
    import pdb; pdb.set_trace()
    for node in reader():
        print("Found node", node)
        metadata = node.zarr.subpath("OME/METADATA.ome.xml")
        print("Looking for metadata in ", metadata)
        if os.path.exists(metadata):
            data = parse_xml(metadata)
            print(data)


if __name__ == "__main__":
    import argparse
    import logging

    parser = argparse.ArgumentParser()
    parser.add_argument("-v", "--verbose", action="store_true")
    parser.add_argument("filename", nargs="+")

    ns = parser.parse_args()
    if ns.verbose:
        logging.basicConfig(level=logging.DEBUG)

    for x in ns.filename:
        print(f"Handling {x}")
        handle_ome_zarr(x)

[sbesson]

Possibly in scope for this work - see the discussion happening in #107 around the confusion created bv the usage of integers for group names.

Except in the HCS case, the current bioformats2raw layout heavily relies on this integer-based naming scheme as it allows to map the individual Zarr multiscales groups via name to the corresponding Image element in the OME-XML metadata.
Relaxing this constraint would likely mean storing this relationship in the metadata e.g. using indices similarly to what has been done in #24. I suspect that also overlaps with the considerations in glencoesoftware/bioformats2raw#126.

[chris-allan]

Obviously, we're closest to 1b2c being usable but it's the least comfortable; like a jacket that's functional but just doesn't fit right. Given this uncomfortable position and certainly the community desire for more thorough use of Zarr group/array metadata and JSON markup I'd suggest that if we decide to formalize it maybe we call it OME-NGFF Transitional or something to that effect?

What follows is some background for how bioformats2raw got to where it currently is.

History

For the benefit of everyone let me first outline the history of how bioformats2raw got to where it is and hopefully that will help people understand the motivations behind its current structure. I'll just state for the record that the initial design decisions behind both bioformats2raw and raw2ometiff were made solely by Glencoe Software and that the primary domains we operate in are whole slide imaging and high content screening. The use cases we strive to support closely reflect our customer base and don't necessarily completely overlap with those of the OME community as a whole.

When work on bioformats2raw started back in October 2019 the repository was called mrxs2ometiff. Like the wider OME community ¹ we were frustrated with the usability and performance problems associated with a growing number of proprietary file formats. So were our customers. The Glencoe Software and University of Dundee teams wrote ² about this as it applies to digital pathology in May 2019. Given the OME tooling available and the difficulties involved in performing real time translation of the MRXS file format converting to OME-TIFF made sense. However, adding an MRXS reader to core Bio-Formats was out of the question and bfconvert had serious performance and scalability problems. Consequently, the project was born.

To date, bioformats2raw remains the only place in the OME ecosystem where support for formats like MRXS and BioTek Cytation are present. mrxs2ometiff and now the combination of bioformats2raw and raw2ometiff are responsible for converting 10s of TBs of our customers' whole slide imaging data into OME-TIFF every week for use with OMERO, QuPath, Vitessce and numerous other open source tools as well as commercial visualization and analysis software like HALO and Visiopharm.

Around the same time, we were commissioned by several of our customers to develop tooling to convert Philips' iSyntax file format into OME-TIFF. Due to the nature of Philips' file format at the time the only option was to depend on Philips' SDK to support this conversion and the only programming language the SDK was available in was Python. Rewriting the entirety of high performance TIFF writing and OME-XML support present in Bio-Formats and mrxs2ometiff in Python was again out of the question. "Two stage" conversion was born, mrxs2ometiff was split into bioformats2raw and raw2ometiff and the isyntax2raw project started. We wrote about this ³ in December 2019.

All of this work predates the formalization of OME-NGFF and as stated in the aforementioned blog post "...converts to a temporary N5 or Zarr structure" and "The current N5 or Zarr intermediate format should be considered temporary at this stage of development as it is likely to undergo several changes over the coming months." The initial default intermediate format used was initially N5. Zarr support was poor at best and completely broken at worst.

isyntax2raw and raw2ometiff remain the only place in the OME ecosystem where support for formats like iSyntax are present and these tools are also responsibly for converting 10s of TBs of our customers' data into OME-TIFF every week.

bioformats2raw was not compatible with OME-NGFF until version 0.3.0, nearly 4 months after the OME-NGFF 0.2.0 specification was published.

I hope this helps everyone understand that as things stand today, the primary use case in the wild for bioformats2raw is as a vehicle for conversion into OME-TIFF via raw2ometiff. This does not mean we think this use case deserves some kind of preferential treatment but rather that it be considered equally. Furthermore, the individuals who perform this use case are not going to be watching GitHub or posting on image.sc.

Rationale for some design decisions

1. What is OME/METADATA.ome.xml and why does it exist?

The content of OME/METADATA.ome.xml is the full OME-XML document produced by Bio-Formats reflecting the OME data model metadata for all images Bio-Formats recognizes in the specified input file. That is, at a fundamental level bioformats2raw does not convert images it converts filesets. We need this metadata to deliver on the conversion to OME-TIFF use case. It is also expected that this metadata be easily resolvable against the OME-NGFF structure to simplify the job of raw2ometiff.

2. Why does bioformats2raw extend the hierarchy?

As aforementioned, bioformats2raw converts filesets. In Bio-Formats and OMERO parlance, each image of a fileset is identified by its "series". "series" is an ascending number starting from 0 and the order is tighly controlled between Bio-Formats versions ⁴. Within the Bio-Formats API, this is the only way of identifying a particular image. Consequently, it makes most sense for each image in the bioformats2raw output to reflect its series. Otherwise, at least with the currently available tooling and without a great deal of additional effort, it is both impossible to reproduce the correct ordering as far as Bio-Formats is concerned and impossible to relate each image to its corresponding <Image> block within OME/METADATA.ome.xml.

These criteria are essential for being able to drive raw2ometiff to produce correct OME-TIFF output and why we state emphatically in the documentation that fiddling with --scale-format-string may break compatibility. We probably should go further than that in saying that it may also break compatibility and resolution against OME/METADATA.ome.xml as well. Furthermore, these criteria reflect the way in which OMERO initializes Bio-Formats within the PixelBuffer infrastructure to get access to the correct pixel data corresponding to a particular Image. We use the output of bioformats2raw in combination with OMERO and OMERO microservices extensively.

3. What is this top level group and what does the metadata (ex. bioformats2raw.layout) mean?

Given the bioformats2raw legacy as well as the current lack of formalization surrounding the concept of a fileset we needed a way to differentiate the historical bioformats2raw output layouts (there have been several) to downstream tooling like raw2ometiff.

Footnotes

1. https://www.openmicroscopy.org/2019/06/25/formats.html ↩

2. https://www.glencoesoftware.com/blog/2019/05/31/open-data-for-digital-pathology.html ↩

3. https://www.glencoesoftware.com/blog/2019/12/09/converting-whole-slide-images-to-OME-TIFF.html ↩

4. https://docs.openmicroscopy.org/bio-formats/latest/about/index.html#versioning-policy ↩

[will-moore]

I seem to remember that Vitessce made use of the METADATA.ome.xml in the Zarr from bioformats2raw, but I don't see any mention of that now http://vitessce.io/docs/data-file-types/#rasterome-zarr. @keller-mark @ilan-gold Am I wrong or has something changed?

[keller-mark]

Vitessce currently uses the metadata from the .zattrs to support loading OME-NGFF via Zarr. Vitessce also supports a (pre-OME-NGFF) Bioformats-Zarr format, which I believe makes use of METADATA.ome.xml (@ilan-gold or @manzt would know better) https://github.com/hms-dbmi/viv/blob/master/src/loaders/zarr/index.ts#L24 (in the Vitessce docs, this corresponds to raster.json with "type": "zarr")

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/collections-in-ome-ngff/63656/6

[joshmoore]

@keller-mark, thanks! @ilan-gold / @manzt: any thoughts on the value (or perhaps the cost) of trying to focus on Vitessce as the client for this work as opposed to updating vizarr to make use of OME-XML?

[ilan-gold]

If you were to officially focus on OMEXML as the fastest route to full metadata support, we would probably just add this to the core Viv library and then add a loader to Vitessce for it, neither of which would take too long. I can't comment on Vizarr though.