Spidna

[andrewkern]

PR for the SPIDNA network and associated simulator / processor for genotypes. Also includes testing.

Here are representative results from a 3 epoch model where I trained the embedder and normalizing flow separately.

Posterior at prior high:

Posterior at prior low:

Posterior at prior mean:

Calibration (looks great!):

Concentration:

Posterior Expectation:

Overall-- I think this is ready to go! We need to decide on parameterization for a 'production' run still, but this run was done with the parameters specified in workflow/config/variable_popnSize_spidna.yaml

[andrewkern]

one decision I punted on here: what to do about the summary stats embeddings that @ningyuxin1999 did

[fbaumdicker]

I just discussed this with @ningyuxin1999 today. We can write the summary stats processor for the summary statistics we used, a combination of the SFS and mean LD in different distance bins.

[andrewkern]

the big question to me here is do we do mean aggregation of those summaries as you were doing originally, and if so how given our workflow.

a few alternatives to consider:

1. we could not do mean aggregation and instead just get one summary stat vector (LD+SFS stats)
2. we could simulate a very long stretch of sequence, chop it in to windows, calculate stats from each window, and then collapse to means
3. we could write a specialized simulator and associate processor that for a given parameter set simulates multiple reps (as you are doing) and then does summary stats / mean aggregation

[nspope]

LGTM!

[nspope]

This is detecting ploidy, right? It's a little messy to do this on the fly (e.g. would fail if there were no homozygous derived alleles, which probably won't happen in practice, but still ...)

A better way is to do this directly from the individuals in the ts, e.g.

ploidy = [ind.nodes.size for ind in ts.individuals()]

or if we only need the number of haploids

samples = ts.num_samples

[andrewkern]

this MAF filtering step was part of Yuxin's pipeline that I just copied over. I agree it's a bit messy to do it at this step. let me push a change to use samples from ts.num_samples.

[andrewkern]

okay pushed a change @nspope and it still passes tests