Benchmarking

[tirohia]

Mōrena.

I haven't actually had a chance to run anything as of yet, but this looks .... promising. A python implementations of differential expression/methylation are something that's sorely lacking in the bioinformatics world.

Have you had a chance to benchmark the results of the python implementation of LImma against the R implementation? My linear algebra isn't up to scratch, so every time I've tried to do something similar to this, I've run up against the base implementations of logistic regressions in python and R are subtly different, and never produce the same output.

Being able to demonstrate similar results across the two implementations would be quite useful I think.

Thanks for getting this underway though.

[dare-afolabi]

Kia ora 🙋‍♂️

Great question; and you’re absolutely right. Consistency with established workflows is essential, especially for someone looking to trust a Python-first pipeline.

I pushed v0.2.0 earlier today, which fixes a few bugs, refactors some internals, and adds a TCGA-based demo. I haven’t benchmarked dmeth directly against R yet, but that’s now my top priority. My plan is to run a full side-by-side comparison on TCGA methylation data using the standard minfi/limma pipeline as the reference.

If you’re interested in helping validate or benchmark, I’d genuinely welcome the collaboration. Even a quick comparison on a dataset you already use and are familiar with would be incredibly valuable.

Out of curiosity, what methylation platform are you working with?

[tirohia]

We have a combination of Illumina EPIC v1 and v2 array data. We have some nanopore, but we're only really beginning to dip our toes into that. Primarily it's the EPIC data. At the moment we've got a differential methylation step as part of feature selection before we head onto building models for classification. I've cobbled together a workflow that passes a filter to an Rscript via rpy2. It's not a pretty solution, and if I can validate a python only workflow, it would make the process so much cleaner.

I'm hoping to at least be able to do a test run of dmeth sometime in the next week or so, I'll compare to limma/R results and let you know how I get on.

[dare-afolabi]

Great, your EPIC data should slot right in. I created a notebook specifically to address your question. It runs a side-by-side comparison of dmeth (Python) and limma (via rpy2) differential methylation analysis using paired TCGA-LUAD tumor-normal data.

Summary:

• Both analyses use the same design matrix (~ patient_id + group) for maximum parity.
• The notebook generates logFC, p-values, adjusted p-values, and posterior variances, then compares them across the two methods.
• Concordance is extremely high: logFC Pearson r = 1, logFC Spearman ρ = 1, and p-value Spearman ρ ≈ 0.999.
• Diagnostic plots (scatterplots, Bland–Altman, residuals, posterior variance comparisons) show that dmeth reproduces limma’s results almost perfectly, with negligible numerical differences.

The notebook also shows that dmeth’s chunked implementation scales cleanly while maintaining numerical equivalence to the canonical R pipeline.

You can check it out here:
https://github.com/dare-afolabi/dmeth/blob/77ac692d2f09d529a370917c9150ad82d18caee3/demo/dmeth_limma_benchmark_2025-12-01.ipynb

If you try it on your full Illumina EPIC data, it would be fantastic to see how well dmeth matches your current limma/R workflow. Even a small test could help validate it and guide further development.

Would love to hear your thoughts once you’ve had a chance to run it.