Anndata format output

[rob-p]

Following up on COMBINE-lab/alevin-fry#151 raised by @ilan-gold, we are planning to first add AnnData output as an option in simpleaf, and then to potentially port it "upstream" directly into alevin-fry (possibly deprecating and dropping the old output format).

This commit 20ab78e includes the first draft implementation of this idea. It creates an anndata format version of the output quant.h5ad that lives directly alongside the existing CSR matrix. Creation of this matrix is enabled by passing --anndata-out as a flag to the simpleaf quant command. I may also add an additional flag to remove the CSR matrix as well to avoid the redundancy, but for now, they live side-by-side.

There are a few dependencies that I am waiting to stabilize (e.g. ureq 3.0 and a fix for this bug in flate2), but after that, we'll be ready to cut a simpleaf release with this feature!

P.S. @an-altosian / @DongzeHE : We should decide about the conversion in the case of visium data. Since we have extra information in the row labels, we should decide if we ignore that, put it in the anndata annotation, etc.

[DongzeHE]

I think we should definitely add the aux data into the anndata obs data. Also, we should make sure we save spliced, unspliced and ambiguous count matrices into three different layers.

[ilan-gold]

Yes! You should definitely take full advantage of the data model to store all the metadata, summary statistics, and quality info you can. Happy to hop on a call to run down with you where would be the best place for this stuff if you need.

[rob-p]

Agreed! @DongzeHE : the tip of dev currently properly splits the information into spliced/unspliced and ambiguous (if alevin-fry was run in USA mode). What other auxiliary data do you have in mind?

[an-altosian]

Mainly the optional aux information in the cell barcode file and the gene_id_to_name file. Currently there are nothing else in my mind that are used in the downstream analysis.

…

On Jan 14, 2025, at 9:27 PM, Rob Patro ***@***.***> wrote: Agreed! @DongzeHE <https://github.com/DongzeHE> : the tip of dev currently properly splits the information into spliced/unspliced and ambiguous (if alevin-fry was run in USA mode). What other auxiliary data do you have in mind? — Reply to this email directly, view it on GitHub <#177 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BI7FTBUWNY43D6VSW6TCEKD2KXWTDAVCNFSM6AAAAABU65BQQKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBTHA4TIMI>. You are receiving this because you were mentioned.

[ilan-gold]

@rob-p why would the spot locations be in var?

[rob-p]

I misspoke, they are in obs!

[ilan-gold]

Sorry, just got a chance to look at the email:

1. Maybe use all CamelCase or snake_case for the column names?
2. gene_id and gene_symbol look identical - is that supposed to be the case?
3. Definitely be sure to document what the terms mean!
4. Maybe unrelated, but why have the full cmd instead of just the args in uns so something like the following for quant_info (modulo quant, which seems implied) quant -i simpleaf_quant/af_quant -o simpleaf_quant/af_quant -t 16 -m t2g.tsv -r cr-like instead of the full /usr/conda/envs/simpleaf/bin/alevin-fry quant -i simpleaf_quant/af_quant -o simpleaf_quant/af_quant -t 16 -m t2g.tsv -r cr-like

Other than that, this all looks amazing. Thanks!

[ilan-gold]

(I think number 4 is perhaps out of scope, but could be the beginning of a nice discussion)

[an-altosian]

For point number 2,

The generation of the gene id to symbol mapping file (gene_id_to_name.tsv) in simpleaf is tricky -- it depends on the input type.

• When the input reference set are a genome FASTA file + a gene annotation GTF file, we find the gene_id and gene_name columns in the GTF records, and use that to generate the file.
• If we start with a transcriptomic FASTA file, we do not generate this mapping as it is hard to parse FASTA record headers to find this information.
• If we start with a probe set CSV file or a feature barcode set CSV file, we create a pseudo mapping file the IDs and symbols are identical.

Because this is from a visium dataset generated from a probe set, the ids and symbols in the id to name mapping file are identical.

@rob-p Do you think I should just avoid generating the id to name mapping file when taking CSV files, instead of generating one with identical IDs and names?