Species-wise EM within vs across samples

[lauramason326]

Hi
I am working on a taxonomic profile of 215 metagenomes, and I have a question about the species-wise EM used in the condense function. From my reading, it seems like the EM works to revise taxonomic inconsistencies across markers based on the coverage of the sequences with differing taxonomic assignments. Does that mean that the EM might result in different identities being assigned to the same sequence across samples? Should singleM condense be run across all samples and not just within a sample? Thanks!
Laura

[wwood]

Hi Laura,

Excellent question.

Condense is done per-sample, there is no "sharing" of information between samples. In part this is don so each sample is processed independently, which helps scaling to large numbers of samples, but also because SingleM isn't told of the relationship between samples - should they contain the same species?

It is possible that the EM (either the genus or the species one) gives different results for different samples that can be assumed to have the same species. In the extreme case, the two metagenomes might be from the same sample - in that case the data should be merged before condense happens (e.g. with summarise --collapse-to-sample-name).

In the general case, it might be possible to process two or more samples at once and then output multiple condensed profiles. Do you have an example that you are thinking of specifically? If so can you please send me (either here or over email) the archive OTU tables? If you do merge them before running condense, do the totals change relative to the sum of the individually processed samples for lineages you particular care about?

Thanks, ben

[lauramason326]

Hi Ben,
Thanks for the feedback. I ran a test using the default settings and with a modification where I made archive OTU tables and condensed them in 2 steps. Then I looked for specific taxa to see if the coverages changed.

Code:

Default

source /home/opt/Miniconda3/miniconda3/bin/activate singlem_0.18.3

singlem pipe -1 sample1_R1.fastq.gz  -2 sample1_R2.fastq.gz  -p sample1_profile.tsv -o sample1_otuTable.csv

singlem pipe -1 sample2_R1.fastq.gz  -2 sample2_R2.fastq.gz   -p sample1_profile.tsv -o sample2_otuTable.csv

Two step:

source /home/opt/Miniconda3/miniconda3/bin/activate singlem_0.18.3

singlem pipe -1 sample1_R1.fastq.gz  -2 sample1_R2.fastq.gz  --otu-table sample1_OTU-table.csv --archive-otu-table sample1_archive.json.zb

singlem pipe -1 sample2_R1.fastq.gz  -2 sample2_R2.fastq.gz  --otu-table sample2_OTU-table.csv --archive-otu-table sample2_archive.json.zb

singlem condense --input-archive-otu-tables sample1_archive.json.zb sample2_archive.json.zb --taxonomic-profile test_profile2.csv

Results:

test lineage: s__Nitrospira_C sp029240595

Sample 1, default: 2.17
Sample 2, default: 2.96
Sample 1, 2 step: 2.17
Sample 2, 2 step: 2.96

test linage: c__Actinomycetes

Sample 1, default: 12.59
Sample 2, default: 13.82
Sample 1, 2 step: 12.59
Sample 2, 2 step: 13.82

test lineage: f__Quadrisphaeraceae

Sample 1, default: N/A
Sample 2, default: 0.54
Sample 1, 2 step: N/A
Sample 2, 2 step: 0.54

So, no change in coverage for "common" bugs at 2 taxonomic levels, and one "uncommon."

I am assuming this means it's fine to run the defaults and combine the resulting coverage tables? Downstream, I am planning to use the data in some basic ecological stats (alpha and beta div, maybe sPLS) and in machine learning to link taxonomy to soil health parameters. Is there anything else you think I should consider?

Thanks!
Laura

[wwood]

Thanks Laura - that makes sense, but what I meant actually was to merge the archive OTU tables into one first (using summarise --collapse-to-sample-name) and then condense just that one sample.

Running condense on 2 samples as you've done is no different than condensing each separately, so I'm not sure we've tested what we wanted to here.

That said, what you are suggesting is rather advanced, and I imagine you are fine to move downstream without messing around here. It was just a good question..

fwiw, I use .json not .json.zb for the archive tables, but that is a matter of convention, it doesn't change the results.

[lauramason326]

Hi again
So I tried to run the summarise command as above, but this is as close as I can get to running it and then condensing that file

(singlem_0.18.3) [lmason@zenith raw_reads]$ singlem summarise \
 --input-taxonomic-profiles singleM_otuTables/110_ID_Aberdeen_WPM_p1_r1_0_OTU-table.tsv singleM_otuTables/106_ID_Aberdeen_SPM_p1_r1_0_OTU-table.tsv \
--output-otu-table summarized_otu_table.tsv

• for this, I cannot output the archived OTU table to use in condense

singlem summarise --input-taxonomic-profiles  singleM_otuTables/106_ID_Aberdeen_SPM_p1_r1_0_OTU-table.tsv singleM_otuTables/110_ID_Aberdeen_WPM_p1_r1_0_OTU-table.tsv \
    --output-species-by-site-relative-abundance bySite_otu.csv \
    --output-species-by-site-level species

I'm not sure if this is what you meant, but the coverages given for my key taxa (above) are vastly different. Do you have any advice?

Thank,
Laura

[wwood]

Would you mind emailing me the two files? Maybe I need to check things work as expected, then I can provide more detailed instructions.

[lauramason326]

Sure - thank you Ben! I have attached the outputs for: singlem summarise --input-taxonomic-profiles singleM_otuTables/106_ID_Aberdeen_SPM_p1_r1_0_OTU-table.tsv singleM_otuTables/110_ID_Aberdeen_WPM_p1_r1_0_OTU-table.tsv \ --output-species-by-site-relative-abundance bySite_otu.csv \ --output-species-by-site-level species and singlem summarise \--input-taxonomic-profiles singleM_otuTables/110_ID_Aberdeen_WPM_p1_r1_0_OTU-table.tsv singleM_otuTables/106_ID_Aberdeen_SPM_p1_r1_0_OTU-table.tsv \ --output-otu-table summarized_otu_table.tsv As well as the original OTU tables and the archived OTU tables for each of the two samples, as well as the condensed profile (two samples together). Do you need anything else? I don't think I can email the raw read files, but might be able to arrange something else 106_aberdeen_OTU-table.csv <https://drive.google.com/file/d/15VfMTdpQCcMZmZDO1qO3Scgsg6yiUvdq/view?usp=drive_web> 106_archive.json.zb <https://drive.google.com/file/d/1hf-4gieUEpCZwUG0k_rue9OFwgPwtL7I/view?usp=drive_web> 110_aberdeen_OTU-table.csv <https://drive.google.com/file/d/1h9QfVhvWiyvQJevSEWoa8WqQtUgXYhGZ/view?usp=drive_web> 110_archive.json.zb <https://drive.google.com/file/d/1kmDKjK11FPsZTOuwl3tlPcQB_FFiFwfa/view?usp=drive_web> Cheers Laura

…

On Mon, May 19, 2025 at 8:12 PM Ben J Woodcroft ***@***.***> wrote: *wwood* left a comment (wwood/singlem#222) <#222 (comment)> Would you mind emailing me the two files? Maybe I need to check things work as expected, then I can provide more detailed instructions. — Reply to this email directly, view it on GitHub <#222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWZYJIES2Q6556IDKVL6DG327KFR5AVCNFSM6AAAAAB5BE57DCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQOJSG4YDONZQGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wwood]

Hi Laura,

I got this to work using v0.18.3

singlem summarise --input-archive-otu-tables 106_archive.json.zb 110_archive.json.zb --output-archive-otu-table combined.json --collapse-to-sample-name combined
singlem condense --input-archive-otu-table combined.json -p combined.profile.csv

I hope that works for you? I haven't had a chance to investigate whether it makes any difference to the EM part of the algorithm - WDYT?

[lauramason326]

Hey Ben
Just gave that a try - thanks for putting the code together! It look like it combines the samples together and calculates the total coverage. Is there a way to do this while retaining sample IDs? If not, no worries. I agree that the best course of action might be to run singlem pipe to generate the taxonomic profiles individually.
Thanks!
Laura

[lauramason326]

Sorry - as a follow up, our main concern was that the EM algorithm would result in slightly different final taxonomic IDs for the same sequence in different samples given the coverage of other reads in the sample, but this might not be something to worry too much over?

[wwood]

yes, that is just treating them as if they are the same sample.

There is currently no way to simultaneously (1) consider both samples together and (2) keep the coverages separate for each sample. So in short, no way to do what you are (quite sensibly) asking. It isn't just a coding issue either - I cannot immediately think of a straightforward way to do this conceptually.

I suppose I was curious whether the combined profile is actually much different than simply adding up the two independent profiles - this will give you a sense of whether this is really an issue for you to worry about. Alternately, you could simply proceed knowing that you are have reached the technical edge of SingleM.

In these samples not that much of the profile is assigned to the species level anyway, of course - they seem to be quite copmlex samples.

HTH, even if a bit messy

[lauramason326]

Ah gotcha - no prob. And it looks like, at least for the ID I tested, the combined profile does not equal the the sum of the coverages for the two samples.

test lineage: s__Nitrospira_C sp029240595
110 together: 2.96
106 together: 2.17
Combined profile: 5.63

[wwood]

Thanks. It makes sense to me that the coverages are slightly different, given the vagaries of the condense algorithm. That doesn't look so different to me though - not sure if you have a different opinoin.