Timur / Magma: Standardize flow population names

[coleshaw]

As @graft mentioned, users that run into data integrity issues within Timur will implicate Timur as having bad data, instead of upstream processes. Perhaps for the Flow population data, that means we should standardize the population names, to make them more searchable? I think we had discussed that at one point, but I forget how we reached the conclusion that we wanted to keep the population names consistent with the WSP (un-standardized).

Perhaps we should revisit that discussion, and see if it now makes sense to standardize population names according to Lenny's pipeline code, that he used for gating comparisons? We could just copy it into Airflow at this point, since it's all Python.

[dtm2451]

Yea, this seems important after our discussion yesterday! I was either not here or not part of the discussions, so idk past reasoning, but if a purpose of not updating the names was to keep the reproducibility link for anyone actually going back to the .fcs/.wsp files, then we could keep current names as an extra attribute of the model! name_in_wsp maybe?

[graft]

First - is there a better place for these issues? this is really a DS/integrity issue not a DL issue.

The workflow I would favor here is having the original wsp at the patient, clean_wsp broken out into the flow record, and population as usual built from clean_wsp. In airflow we take the original WSP, extract a gate tree for a given sample into a new WSP, clean the gate names, extract to population table. This way the original data is available, we have WSPs at the assay level instead of way up at the patient, and we can clean up names.

[coleshaw]

The workflow I would favor here is having the original wsp at the patient, clean_wsp broken out into the flow record, and population as usual built from clean_wsp. In airflow we take the original WSP, extract a gate tree for a given sample into a new WSP, clean the gate names, extract to population table. This way the original data is available, we have WSPs at the assay level instead of way up at the patient, and we can clean up names.

I'm not sure this is possible? The reason that Lenny didn't do this in the original pipeline is, he said changing the gate names in the WSP breaks something else, elsewhere -- so there is some metadata that has to also be changed. I'm not sure, so we would have to do some investigation to see if this is possible, or exactly how to make the change. Might just be messy.

A small issue with the above investigation is that they don't make FlowJo for Linux anymore. So either we need a Windows laptop / license (and then I could dual boot my machine ... but would have to repartition the harddrive), or is there alternative WSP software that runs on Linux, that you're aware of?

[coleshaw]

First - is there a better place for these issues? this is really a DS/integrity issue not a DL issue.

Well, if we decide to provide a mitigation for the data integrity issue, then it becomes a DL issue. I guess the conversation is really an IPI data integrity issue, so perhaps the Slack data-integrity channel, or someone needs to bring it up at the bi-weekly IPI meetings?

[coleshaw]

Ooh, or perhaps we should start GH discussions around stuff like this, that aren't specific technical issues...