int16 data sets not read properly

[ercius]

I work for a large microscopy center which is starting to generate data written in HDF5 format (FEI's EMD format). Your HDF5 vibez plugin for Fiji is extremely useful for users who are not able to read the data using any other GUI based software. However, signed integer 16-bit data is not being properly read. It looks like there is an issue between converting from HDF5 int16 and ImageJ's int16 format.

I had a similar issue writing a Jython plugin for Fiji. The solution was to read unsigned and signed data as signed. Then everything worked when creating a ImagePlus.

The issue is probably in this part of the Vibez plugin

As per information from my issue posted to the ImageJ Forum linked above:

short in Java is always signed (-32,768 to 32,767)
a ShortProcessor in ImageJ1 is always unsigned

I am not a Java developer and not setup to compile/test this in ImageJ. Do you have anytime to look into this issue?

I can send a small test H5 file with int16 and uint16 datasets to show the problem. It's easy to create this using the following Python code

import h5py
import numpy as np

int1 = np.linspace(-32768,32767,100,dtype=np.int16)
uint1 = np.linspace(0,65535,100,dtype=np.uint16)

YY,XX = np.meshgrid(int1,int1)
YYu,XXu = np.meshgrid(uint1,uint1)

with h5py.File('integers.h5','w') as f1:
    f1.create_dataset('int16',data=XX.astype(np.int16),compression='gzip')
    f1.create_dataset('uint16',data=XXu.astype(np.uint16),compression='gzip')
    f1.create_dataset('XX',data=XX.astype(np.float32),compression='gzip')
    f1.create_dataset('XXu',data=XXu.astype(np.float32),compression='gzip')

Here is the data read into Fiji using your plugin:
uint16

int16

PS Thank you very much for all your work on this plugin. Its extremely useful and will see a lot more use as HDF5 data sets become more ubiquitous.

[rejsmont]

Hi,

legacy ImageJ does not have an internal data structure to handle int16 images. While it is possible to convert int16 to unit16, by simple addition, the meaning of negative values would get lost. A lossless solution would be to open int16 datasets as float32 images in ImageJ.

All this, of course, will not be necessary in the ImageJ2 world, but converting this plugin to SciFIO is currently not high on my priority list.

[ctrueden]

legacy ImageJ does not have an internal data structure to handle int16 images. While it is possible to convert int16 to unit16, by simple addition, the meaning of negative values would get lost.

There is a way to "calibrate" images such that signed 16-bit integers can be handled. This is what ImageJ1 does for the limited selection of formats it supports with this data type.

However, I agree that opening such data as float32 ends up being conceptually simpler, and less prone to accidental math problems, as long as you can afford to pay the 2x space penalty.

converting this plugin to SciFIO is currently not high on my priority list.

I think there are two separate things: 1) using ImgLib2 data structures under the hood; and 2) recasting the I/O logic as a SCIFIO plugin. I think (1) is worth dipping into sooner than (2), since you'd be able to start overcoming data type limitations, and gain other benefits from ImgLib2, while still being able to treat the images as ImageJ1 images thanks to the ImageJ Legacy layer.