Multiple arsinh transforms?

[chillerb]

Is your feature request related to a problem? Please describe.
Hi Scott, I hope you are doing alright. While flowkit already provides an Asinh transform, it does seem to be somewhat different from the asinh transform that is offered in other popular packages like pytometry, cf. also here. Their arsinh transform only requires a single "co-factor" parameter - as far as I know, the transform was initially proposed by the flowVS paper:

Azad, A., Rajwa, B. & Pothen, A. flowVS: channel-specific variance stabilization in flow cytometry. BMC Bioinformatics 17, 291 (2016). https://doi.org/10.1186/s12859-016-1083-9

Describe the solution you'd like
Here, I propose a prototype implementation. For custom transforms, I think it would also be a good idea to expose the base transform class (I had to cheat a little bit).

"""
Extens flowkit by providing the popular flowVS arsinh transform with a co-factor parameter.
"""


import numpy as np
import flowkit as fk


class ArsinhTransform(fk.transforms._transforms.Transform):
    """
    Nonlinear transformation proposed by:

    > Azad, A., Rajwa, B. & Pothen, A. flowVS: channel-specific variance stabilization in flow cytometry. BMC Bioinformatics 17, 291 (2016). https://doi.org/10.1186/s12859-016-1083-9

    f(x) = arsinh(x / cofactor)

    :param param_cofactor: recommended value is 5 for CyTOF and 150 for Flow Cytometry
    """
    def __init__(self, param_cofactor=5):
        super().__init__()
        self.param_cofactor = param_cofactor


    def __repr__(self):
        return f"{self.__class__.__name__}(c: {self.param_cofactor})"

    def apply(self, events):
        """
        Apply transform to given events.

        :param events: NumPy array of FCS event data
        :return: NumPy array of transformed events
        """
        new_events = np.arcsinh(events / self.param_cofactor)

        return new_events

    def inverse(self, events):
        """
        Apply the inverse transform to given events.

        :param events: NumPy array of FCS event data
        :return: NumPy array of inversely transformed events
        """
        new_events = np.sinh(events) * self.param_cofactor

        return new_events

Additional context
I'm coming from a Computer Science PoV, so I don't know if what I'm proposing makes sense from a cytometry PoV.

[whitews]

Hey Bjarne,

Thanks for the proposal, I find this quite interesting as I've been down this rabbit hole before. As you're probably aware, the transform implementations live in the underlying FlowUtils library. Quite a while back, the FlowUtils asinh function was very similar to what you've written, with a single pre_scale argument (though it was the mathematical inverse of what you call the cofactor). You can find that old implementation here. I had been experimenting with simpler alternatives to the favored Logicle transform, which I thought at the time was overly complex.

Turns out, people smarter than me put a lot more thought into all this ;). There's always a trade-off. For the data sets I was testing at the time, my simple asinh seemed fine. But as more data came through, I needed to adjust my pre_scale and I didn't always know what the best value should be. I appreciate you linking to the flowVS paper, as it goes into much more detail regarding the selection of the optimum cofactor value.

The trade-off for the simplicity of this implementation is to push the determination of the best cofactor value onto the end user. That would be fine except I could never formalize the definition of the pre_scale argument. From the end user's perspective, the question will always be: What value should I use for the cofactor?

The answer from the flowVS paper is that you just find the minimum solution for Bartlett's statistic. And Bartlett's statistic couldn't be more simple:

Of course that's sarcasm, but this is what I found. I needed endless tweaking to find the best value. I'd love for it to be true that such a simple transform would replace the multi-parametric ones used in the flow community, but it really just kicks the complexity can down the road. The current AsinhTransform in FlowKit is based on the GatingML 2.0 specification, presumably arrived at by a group of folks that put a lot of time into thinking about all of this.

The question is: what are you proposing? If you are proposing just to expose the base Transform class in the FlowKit API, then I agree. FlowKit should be extensible in this way. On the other hand, if you propose to include this simplified ArsinhTransform in FlowKit, then I'd need more convincing based on the above explanation.

Kind regards,
Scott

P.S. If you'd like to help with FlowKit development, let me know. I have limited time to work on these libraries these days and would welcome any help.

[chillerb]

Hello Scott,

thanks for your response which provided a lot of helpful context to me. I completely agree with you on the importance of the question on how to select the transform parameters, which sometimes seems to be a somewhat opaque process mostly based on years of domain experience and trial and error.

While the flowVS paper used a more elaborate optimization scheme using Bartlett's statistic, it appears to me that in practice, people often just directly use a co-factor of 5 for CyTOF, and 150 for flow cytometry data. Just to provide an example, the first time I encountered the transform was in this paper:

Haidong Yi and Natalie Stanley. 2021. CytoSet: predicting clinical outcomes via set-modeling of cytometry data. In Proceedings of the 12th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (BCB '21). Association for Computing Machinery, New York, NY, USA, Article 47, 1–8. https://doi.org/10.1145/3459930.3469529

I like that flowkit is firmly based on the GatingML 2.0 specification (by the way, while checking, I noticed that the GatingML link in flowkit's README seems to be dead / temporarily unavailable?) and while I keep encountering the same cofactor based arsinh transform in different articles and implementations, it is somewhat challenging to pin down it's origins - I don't think there is a "newer specification" yet, that actually includes it.

Thank you for the offer to work on flowkit - however, I don't really feel qualified for that, since working with flow cytometry data is more of a side project to me at the moment and I'm lacking experience.

[haoqichen20]

Hi Scott and Bjarne,

what a timely discussion you've had! I was trying to automate some CyTOF data analysis on Cytobank using Flowkit, and I had a lot of trouble reproducing the Cytobank plot due to the difference in Arcsinh implementation.

I do not have much insights to contribute here but just like to confirm what Scott said about the Cytof field, that a co-factor of 5 is often the default value for the simpler version of Arcsinh function, and was used in Cytobank. For example, the value is used in this paper from my host group that did a lot of Cytof: https://www.sciencedirect.com/science/article/pii/S0092867423012205

For Scott, did you manage to find a combination of parameter that behave similarly to cofactor = 5? Thanks!

Best,
Haoqi