Extend SIF metadata

[mruijzendaal]

Hi,

It would be great if we could extend the current metadata to incorporate more information on e.g. the gate time.
For reference, I include here the "Acquisition info" that Andor Solis lists for a measurement I took, and the 66-line header for the same file:

measurement_info.txt:

Filename:                     measurement.sif
Date and Time:                Fri Jul 28 13:51:04.409 2023
Software Version:             4.32.30000.0
Temperature (C):              -25
Model:                        DH334T-18F-63
Data Type:                    Counts
Acquisition Mode:             Kinetics
Trigger Mode:                 External
Exposure Time (secs):         3
Number of Accumulations:      1
Number in Kinetics Series:    20
Readout Mode:                 Full Vertical Binning
Horizontal binning:           1
Extended Data Range:          off
Horizontally flipped:         false
Vertical Shift Speed (usecs): 6.5
Pixel Readout Rate (MHz):     1
Clock Amplitude:              Normal
Serial Number:                ICCD-06390
Pre-Amplifier Gain:           4x
Trigger level (V):            1
Termination:                  50 ohms
Inverted:                     false
Intelligate:                  off
Insertion delay:              normal
Gain level:                   2500
Integrate on chip:            on
IOC frequency (Hz):           5e+05
IOC number of pulses:         1
PIV Mode:                     off
Output A Width (nsecs):       10
Output A Delay (nsecs):       0
Output A Polarity:            Negative
Gate Mode:                    DDG on
Gate Width (nsecs):           10
Gate Delay (nsecs):           0
Gate Delay Step (nsecs):      0
Gate Width Step (nsecs):      0
Spurious Noise Filter Mode:   No Filter
Photon counted:               false
Data Averaging Filter Mode:   No Filter
KY328i:
Serial Number:                KY-4231
Wavelength (nm):              561.47
Grating Groove Density (l/mm):599.566
Grating Blaze:                650NM
Output Flipper Port:          Direct
Input Side Slit Width (um):   50
Side Iris Steps:              50

measurement.sif header:

Andor Technology Multi-Channel File
65538 1
65567 0 0 1 1690545259 -25 � � � � � 0 3 3.0221 3.0221 1 � 3.0221 1e-06 0 1 2500 0 0 1e-08 0 0 � � 1 0 5 0 0 0 0 0 0 1 500000 0 6.5e-06 0 4 6390 1 1 -999 0 0 0 0 0 5 4 32 30000 0 0 1 1000 0 0 0 550 0 0 0 0 0 0 0 0 0 1 1 13
DH334T-18F-63 
 1024 1024 69
C:\asdasdasdasd\measurement.sif 
65538 2048
<line 8 with non-ascii characters omitted>
<line 9 with non-ascii characters omitted>
<line 10 with non-ascii characters omitted>
<line 11 with non-ascii characters omitted>
<line 12 with non-ascii characters omitted>
<line 13 with non-ascii characters omitted>
65538 � � � � 0.027 0
65540 1 1 561.47 1 1 599.566 650NM
1 50
1 0 0 0 0  0 0 0 0 0 0 10
0 KY328i
0 10
0 10
0 0 
65537 0 500 1200
0 999 999

65539 1 5 2500 0 0 0 10000 1 2000000 1 100 0 0 100 0 0
1 1 0 10000
0 1 0 2000
0 1 0 2000
1 0 0

-1
65538 0 1024 1024 1024 1024 1 1 1 1
65540 � � � � � �
529.93812442523 0.061715845778342 -2.28349748230931e-07 -5.07163560661353e-11
0 1 0 0
0 1 0 0
422
13
13
1
10
Wavelength6
Counts12
Pixel number65541 1 1024 1024 1 20 1 20480 1024
65538 1 1024 1024 1 1024 1 0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
         0
0

measurement.sif.zip
measurement_info.txt

[mruijzendaal]

#34 implements changes that would partially fix the current incomplete metadata. @fujiisoup Could we create a new release with the proposed changes?

[fujiisoup]

Could we create a new release with the proposed changes?

Yes, I will. Let's finalize the PR.

[mruijzendaal]

I found the following XML-structured metadata at the end of the SIF file:

<?xml version="1.0" ?>
<Signals>
    <Signal>
        <TInstaImageData>
            <PreAmpGainText>4x</PreAmpGainText>
            <SpectrographSerial>KY-4231</SpectrographSerial>
            <mSec>616</mSec>
            <DIRECT_IRIS_PRESENT>0</DIRECT_IRIS_PRESENT>
            <DIRECT_IRIS_POSITION>-1</DIRECT_IRIS_POSITION>
            <SIDE_IRIS_PRESENT>1</SIDE_IRIS_PRESENT>
            <SIDE_IRIS_POSITION>50</SIDE_IRIS_POSITION>
            <FullSerialNumber>ICCD-06390</FullSerialNumber>
            <DARKTHRESHOLD>-999</DARKTHRESHOLD>
        </TInstaImageData>
        <TInstaImage>
            <IRIGAvailable>0</IRIGAvailable>
        </TInstaImage>
        <TCalibImage>
            <TCalibImageData />
            <TAndorImage>
                <TAndorImageData>
                    <ImageFormat>0</ImageFormat>
                    <ReadoutPort>-1</ReadoutPort>
                    <ExposureWindowHeight>0</ExposureWindowHeight>
                    <LineScanSpeed>0</LineScanSpeed>
                    <AlternatingReadoutDirection>0</AlternatingReadoutDirection>
                    <ScanSpeedControl>0</ScanSpeedControl>
                    <ReadoutDirection>-1</ReadoutDirection>
                    <FastKineticsStorageMode>0</FastKineticsStorageMode>
                    <FastKineticsTimeScanMode>-1</FastKineticsTimeScanMode>
                    <TickInterval>10</TickInterval>
                    <PIVMode>0</PIVMode>
                    <ExtendedDynamicRange>0</ExtendedDynamicRange>
                </TAndorImageData>
            </TAndorImage>
        </TCalibImage>
    </Signal>
</Signals>
<References />
<Backgrounds>
    <Background>
        <TInstaImageData>
            <PreAmpGainText>4x</PreAmpGainText>
            <SpectrographSerial>KY-4231</SpectrographSerial>
            <mSec>0</mSec>
            <DIRECT_IRIS_PRESENT>0</DIRECT_IRIS_PRESENT>
            <DIRECT_IRIS_POSITION>-1</DIRECT_IRIS_POSITION>
            <SIDE_IRIS_PRESENT>1</SIDE_IRIS_PRESENT>
            <SIDE_IRIS_POSITION>50</SIDE_IRIS_POSITION>
            <FullSerialNumber>ICCD-06390</FullSerialNumber>
            <DARKTHRESHOLD>-999</DARKTHRESHOLD>
        </TInstaImageData>
        <TInstaImage>
            <IRIGAvailable>0</IRIGAvailable>
        </TInstaImage>
        <TCalibImage>
            <TCalibImageData />
            <TAndorImage>
                <TAndorImageData>
                    <ImageFormat>0</ImageFormat>
                    <ReadoutPort>-1</ReadoutPort>
                    <ExposureWindowHeight>0</ExposureWindowHeight>
                    <LineScanSpeed>0</LineScanSpeed>
                    <AlternatingReadoutDirection>0</AlternatingReadoutDirection>
                    <ScanSpeedControl>0</ScanSpeedControl>
                    <ReadoutDirection>-1</ReadoutDirection>
                    <FastKineticsStorageMode>0</FastKineticsStorageMode>
                    <FastKineticsTimeScanMode>-1</FastKineticsTimeScanMode>
                    <TickInterval>10</TickInterval>
                    <PIVMode>0</PIVMode>
                    <ExtendedDynamicRange>0</ExtendedDynamicRange>
                </TAndorImageData>
            </TAndorImage>
        </TCalibImage>
    </Background>
</Backgrounds>
<Lives />
<Sources />
�������SIFX

This seems to also be useful metadata.

[mruijzendaal]

As for the XML metadata, I'm not sure how many edge cases there are (multiple signals, references or backgrounds?). I think there are two options:

• Flattening this metadata (e.g. keys such as Signal_0_PreAmpGainText)
• Convert it to a dict structure.

Do you have any preference?

[fujiisoup]

Hi @mruijzendaal

I like this option:

Flattening this metadata (e.g. keys such as Signal_0_PreAmpGainText)

I think we can just ignore edge cases. If someone really needs these metadata, they may implement it.

[hakunamatatarr]

Hi!
First of all thanks a lot for maintaining sif_parser!
While trying to use it for transient photoluminescence measurements with an Andor iCCD (iStar DH320T-18U-73), I was missing mainly the gate delay step from the parser. I compared some .sif files with the corresponding Andor-exported .asc files and I think I found the relevant parts in the .sif file. I used the function sif_open.np_open and in the function _open for the case info['SifVersion'] > 65565, there are some blocks that are skipped.

The gate delay step is encoded in the first line of the skipped block that appears after the intensifier gate delay / width values. The line looks for example like
1 10000000000 1 1 36000 0.1 100 0 0
where the 36000 seems to be the Gate Delay Step [ps] and the 0.1 the corresponding exponent. Not sure what the other numbers mean.

The trigger level (V) seems to be encoded a few lines later in the same skipped block as a three-number line, e.g.
0.5 0 0 or 1 0 0,
where the first number matches the Andor .asc field Trigger level (V).

I’ll attach some paired .sif and .asc files so this can be checked easily and maybe more connection can be found. I know this is not much but I thought I share it here.

Thanks again!

Andor sif and asc files.zip

[fujiisoup]

Sorry for my late reply. Thank you for raising an issue, and suggestion.

I may need a little more days to have time to work for this, but I will try to.
In the meantime, it would be great if you could propose a change and send a pull request. While we need to run some compatibility test, but I think this way we may be able to implement this did more quickly