Explore and implement parsing encapsulated image bytes into a common native representation

[suyashkumar]

We should explore the process of parsing encapsulated image data (that may be encoded using many different transfer syntaxes) into a native representation (like a simple integer slice, like exists for NativePixel data already). It will be beneficial to have one in-transit representation of a dicom image, if it is feasible to safely and reliably convert an encapsulated image to a native pixel data format. To determine this, the pipelines for viewing native and encapsulated data need to be inspected to see if there are points where the data has a common representation.

Ultimately the goal is to represent both native and encapsulated pixel data in a common data structure without compromising the display pipeline for either image.

[suyashkumar]

pydicom uses a collection of image handlers to convert encapsulated pixel data bytes to a numpy array. Many different image handlers exist in master that have support for different transfer syntaxes...
https://github.com/pydicom/pydicom/tree/master/pydicom/pixel_data_handlers
From pydicom docs:

Image handlers for converting pixel data. This is an ordered list that the dataset._get_pixel_array() method will try to extract a correctly sized numpy array from the PixelData attribute.

The first handler that both announces that it supports the transfer syntax and does not throw an exception, either in getting the data or when the data is reshaped to the correct dimensions, is the handler that will provide the data.
If they all fail, the last one to throw an exception gets to see its exception thrown up.
If no one throws an exception, but they all refuse to support the transfer syntax, then this fact is announced in a NotImplementedError exception.

Assuming that the handlers extract the data exactly as expected for each transfer syntax, presumably whatever image viewing pipeline exists for encapsulated images can be resumed from that data structure.

However, it is unclear if the image processing pipeline from here on out for encapsulated pixel data is the same as for native pixel data. Post-rescaling of the native pixel data I think that the pipelines may be the same, but further research is needed.

[suyashkumar]

I've checked into the options for parsing lossless JPEG (one of the possible transfer syntaxes) in go, and it appears that the standard library image/jpeg does not support it. Further, there seems to be an open issue about this from another individual who mentioned medical imagery.

From the golang source

        // As per section 4.5, there are four modes of operation (selected by the
	// SOF? markers): sequential DCT, progressive DCT, lossless and
	// hierarchical, although this implementation does not support the latter
	// two non-DCT modes. Sequential DCT is further split into baseline and
	// extended, as per section 4.11.

It is possible for us to tackle this if needed, but as mentioned previously it'll require expenditure of time that may not be needed for an MVP if we reuse parsing tooling that exists in cornerstone JS for initial viewing purposes. Long-term we should support parsing of these images on the backend though.

Further investigation will continue.

[suyashkumar]

We may be able to patch into a C or C++ library as well for the purposes of lossless JPEG: https://github.com/team-charls/charls

[suyashkumar]

The relevant transfer syntaxes we must support for encapsulated images are here. There are 11 possible transfer syntaxes from what I can see here.

pydicom supports the following subset of transfer syntaxes for encapsulated images:

JPEGBaseLineLossy8bit = UID('1.2.840.10008.1.2.4.50')
JPEGBaseLineLossy12bit = UID('1.2.840.10008.1.2.4.51')
JPEGLossless = UID('1.2.840.10008.1.2.4.70')
JPEGLSLossless = UID('1.2.840.10008.1.2.4.80')
JPEGLSLossy = UID('1.2.840.10008.1.2.4.81')
JPEG2000Lossless = UID('1.2.840.10008.1.2.4.90')
JPEG2000Lossy = UID('1.2.840.10008.1.2.4.91')
RLELossless = UID('1.2.840.10008.1.2.5')

https://github.com/pydicom/pydicom/blob/f0e8e14ae1019cacaecf7c5f20fee879640f55e9/pydicom/uid.py#L243-L250

[suyashkumar]

It appears that cornerstone's WADO imageLoader only supports a key subset of transfer syntaxes as well, which seems in line with pydicom. Ultimately we will need a plan to support all relevant transfer syntaxes (and regulatory oversight here needs to be looked into) but initially we can support a subset of them with the transfer syntaxes mentioned in this issue being the first priority.

import decodeLittleEndian from './decoders/decodeLittleEndian.js';
import decodeBigEndian from './decoders/decodeBigEndian.js';
import decodeRLE from './decoders/decodeRLE.js';
import decodeJPEGBaseline from './decoders/decodeJPEGBaseline.js';
import decodeJPEGLossless from './decoders/decodeJPEGLossless.js';
import decodeJPEGLS from './decoders/decodeJPEGLS.js';
import decodeJPEG2000 from './decoders/decodeJPEG2000.js';