Iterator for object detection workflows?

[jo-mueller]

Hi @lorenzocerrone ,

Supercool project! LEveraging ome-zarr is the next-big thing to think about. This is 50% thinking out loud and 50% feature request: Is it possible to build the iterator so that it can also spit out ROIs instead of label images? I mean the spec doesn't currently support ROIs but things like object detection (i.e., maximum detection, YOLO classifiers) are not super uncommon. The output would have to be handled without outside ome-zarr but it would be supercool to be able to do it in the first place.

[lorenzocerrone]

Hey @jo-mueller,

Thanks a lot for looking into the project!

We don't yet have a specific iterator for image detection (but I agree it would be very cool, and it would fit nicely with the current architecture)

In ngio, we do a lot with ROIs (and we already have a lot of APIs to handle them). We also have a mini "spec" on how we serialize them into the ome-zarr (either as Anndata, CSV, or parquet, see https://biovisioncenter.github.io/ngio/stable/table_specs/overview/).

The detection case is somewhat similar to the feature extraction case one. You can see an example here https://biovisioncenter.github.io/ngio/stable/tutorials/feature_extraction/.

[jluethi]

Hey @jo-mueller Thanks for engaging here! Just to reiterate what @lorenzocerrone just said, such feature requests are very welcome here and a detection iterator is totally in scope for ngio. I'd say it should already be doable as a custom feature extraction iterator: Use it to extract the image, run processing (e.g. Yolo), convert the resulting detection into a ngio ROI object (& handle the fact that the iterator runs on potentially different ROIs already => conversion from array coordinates to world coordiantes) and add it to a list.
At the end of the iterator, save the list of ROIs as a ROI table :)

Would be fun to try this and learn whether a more specialised iterator is needed for this or this is fully covered by the feature extraction pattern :)

And if nobody has time for this in October: can I suggest a hackathon side project 😇

[jo-mueller]

Hi @jluethi @lorenzocerrone ,

Thanks for the lightning-fast replies! I may have some time on my hand next week or later that I could use to go exploring a bit. Those tips of yours should go a long way to set me up in the right direction. I'll post here if I get some cool results or more questions 👍

[jo-mueller]

Just for to clarify: ngio.open_image doesn't open ome-zarr versions 0.5/zarr v3? I just tried to point it at an image of mine I had created with ngff-zarr and that yields an error.

[lorenzocerrone]

Hey @jo-mueller, we still only support 0.4 and zarr v2. The transition to v3 will be the next step, but we still have a lot of legacy code to port before we can move to zarr-python v3.

[jo-mueller]

Also: Seems to work :)

The remaining question would be how the regions are handled at the edges of chunks. If the chunks are sufficiently larger than the rois one is interested in it's probably negligible but otherwise some sort of overlap would be the next thing to ask for for the iterator (with a bunch of follow-up questions directly in tow 😬 )

[lorenzocerrone]

Nice!

I am not sure I am following the chunks issue, the rois in the iterator are completely independent from the zarr chunking. Am I missing somenthing? 😅

[jo-mueller]

I am not sure I am following the chunks issue, the rois in the iterator are completely independent from the zarr chunking. Am I missing somenthing? 😅

Ah, that's already one piece of information relevant to me. Do the ROIs in the iterator overlap? If they don't then there may be an issue with detecting objects that sit right on top of the edge between two ROIs pulled out by the iterator. Some tools (i.e., monai) have a sliding-window inference function, which introduces an overlap between the pulled ROIs. This of course brings the problem of resolving detections in the overlapping areas.

[lorenzocerrone]

Ah! I see what you mean :)

So yes, ROIs can overlap, and yes, this is undefined behaviour. For now, I would say to avoid overlaps unless intended, the iterators have a method "check_if_chunks_overlap" to detect potential overlaps.

In the long term, the idea is to define stereotypical behaviours for handling the overlaps. This is one of the reasons to have "typed" iterators. For example, automatically fuse segmentations from overlapping ROIs in the segmentation iterator, and reduction functions for image processing iterators. It's still early, but the roadmap is definitely to support iterator overlap.

[jluethi]

For some extra context: We currently only handle overlapping ROIs in the Fractal context when we have masking ROI tables, e.g. the bounding boxes overlap, but each ROI is associated to a distinct object from a label image => each pixel is only associated to a single object, even if ROI bounding boxes overlap.

The whole logic of handling sliding windows or other ways of splitting a large image into pieces that are processed, but still getting detection results as if the whole array was processed is of course very interesting, but not something we'd have existing answers for here :)

[jo-mueller]

Thanks for the insights. I think for my purpose setting the overlap to a sufficiently large values and then handling the potential ROI duplicates separately would do the trick. I would have to implement a non-maximum-supression but that shouldn't be too hard. Would be happy to contribute that down the road once it's relevant 👍