SingleM Pipe error: Unexpected line format for DIAMOND output

[magicprotoss]

Hi, I've encountered another issue, I think it has sth. to do with handling diamond output...

SingleM version: singlem-0.20.3 (bioconda)

Run log:

2026/01/15 04:16:46 PM INFO: SingleM v0.20.3
2026/01/15 04:16:46 PM INFO: Retrieval successful. Location of backpack is: /mnt/hdd/bio_db/singlem-db/
2026/01/15 04:16:47 PM INFO: Loaded 59 SingleM packages
2026/01/15 04:16:47 PM INFO: Using as input 42 different pairs of sequence files e.g. ../1.valid_subset_150bp/CK_7_1_1.fastq.gz & ../1.valid_subset_150bp/CK_7_1_2.fastq.gz
2026/01/15 04:16:47 PM INFO: Filtering sequence files through DIAMOND blastx
Traceback (most recent call last):
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 100, in _prefilter
qseqid, full_qseq, sseqid, qstart = line.strip().split('\t')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 4, got 3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/miniconda3/envs/singlem-0.20.3/bin/singlem", line 9, in
sys.exit(main())
^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/main.py", line 776, in main
singlem.pipe.SearchPipe().run(
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/pipe.py", line 73, in run
otu_table_object = self.run_to_otu_table(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/pipe.py", line 370, in run_to_otu_table
self._num_threads, self._working_directory).run_diamond(
^^^^^^^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 37, in run_diamond
revs = self._prefilter(dmnd, reverse_read_files, True, performance_parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 102, in _prefilter
raise Exception(f"Unexpected line format for DIAMOND output line '{line.strip()}'")
Exception: Unexpected line format for DIAMOND output line 'ML150004291L1C042R04006838127/2 GACCGACTGGAGAGCTTCGACATATACGCCCACCTGGACGGCGGCGGGCCGTCCGGTCAGGCTGGTGCATTGCGTCTGGCCATCGCGCGGGCGCTGATCATCGTGCAGCCCGAGGACCGGCCGGCACTGAAGAAAGCGGGGTTCCTCACA ribosomal_S9~RS_GCF_027886475.'

[wwood]

Hi, Thanks for the report I came across this one the other day myself, but wasn't able to reproduce. Does the input seqs happen to have tabs in the ID line? Can you try again with the newest diamond installed? Diamond didn't run out of ram did it? -\------------- Ben Woodcroft Group leader, Centre for Microbiome Research, QUT

…

________________________________ From: magicprotoss ***@***.***> Sent: Saturday, January 17, 2026 5:40:42 PM To: wwood/singlem ***@***.***> Cc: Subscribed ***@***.***> Subject: [wwood/singlem] SingleM Pipe error: Unexpected line format for DIAMOND output (Issue #281) [https://avatars.githubusercontent.com/u/117822705?s=20&v=4]magicprotoss created an issue (wwood/singlem#281)<#281> Hi, I've encountered another issue, I think it has sth. to do with handling diamond output... SingleM version: singlem-0.20.3 (bioconda) Run log: 2026/01/15 04:16:46 PM INFO: SingleM v0.20.3 2026/01/15 04:16:46 PM INFO: Retrieval successful. Location of backpack is: /mnt/hdd/bio_db/singlem-db/ 2026/01/15 04:16:47 PM INFO: Loaded 59 SingleM packages 2026/01/15 04:16:47 PM INFO: Using as input 42 different pairs of sequence files e.g. ../1.valid_subset_150bp/CK_7_1_1.fastq.gz & ../1.valid_subset_150bp/CK_7_1_2.fastq.gz 2026/01/15 04:16:47 PM INFO: Filtering sequence files through DIAMOND blastx Traceback (most recent call last): File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 100, in _prefilter qseqid, full_qseq, sseqid, qstart = line.strip().split('\t') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: not enough values to unpack (expected 4, got 3) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/miniconda3/envs/singlem-0.20.3/bin/singlem", line 9, in sys.exit(main()) ^^^^^^ File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/main.py", line 776, in main singlem.pipe.SearchPipe().run( File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/pipe.py", line 73, in run otu_table_object = self.run_to_otu_table(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/pipe.py", line 370, in run_to_otu_table self._num_threads, self._working_directory).run_diamond( ^^^^^^^^^^^^ File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 37, in run_diamond revs = self._prefilter(dmnd, reverse_read_files, True, performance_parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 102, in _prefilter raise Exception(f"Unexpected line format for DIAMOND output line '{line.strip()}'") Exception: Unexpected line format for DIAMOND output line 'ML150004291L1C042R04006838127/2 GACCGACTGGAGAGCTTCGACATATACGCCCACCTGGACGGCGGCGGGCCGTCCGGTCAGGCTGGTGCATTGCGTCTGGCCATCGCGCGGGCGCTGATCATCGTGCAGCCCGAGGACCGGCCGGCACTGAAGAAAGCGGGGTTCCTCACA ribosomal_S9~RS_GCF_027886475.' ― Reply to this email directly, view it on GitHub<#281>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAADX5CJG35P35ZWQBIQ3YD4HHRPVAVCNFSM6AAAAACR7XELYGVHI2DSMVQWIX3LMV43ASLTON2WKOZTHAZDINRTGI3DIOI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[magicprotoss]

Hi,

Thanks for the report

I came across this one the other day myself, but wasn't able to reproduce. Does the input seqs happen to have tabs in the ID line? Can you try again with the newest diamond installed? Diamond didn't run out of ram did it?

--------------
Ben Woodcroft
Group leader, Centre for Microbiome Research, QUT
…

Does the input seqs happen to have tabs in the ID line?

I don't think it has something to do with the ids though, the "/" in the ids are defaults for MGI T7 platform I think.
I ran singlem 0.19.0 on another dataset from the same platform and it worked great.

Can you try again with the newest diamond installed?

I'll try with singlem 0.19.0 to see if I can reproduce the issue first, since it ran successfully last month.

Diamond didn't run out of ram did it?

That's not the case either, the system had 1.5T physical and 1.5T swap configured

[magicprotoss]

The problem disappeared with singlem version 0.19.0, if nothing changed in the singlem codebase, I suppose it's a diamond version problem...

navi@navi-virtual-machine:~$ conda list -n singlem-0.19.0 | grep diamond
diamond 2.1.10 h43eeafb_2 bioconda

administrator@zhanglab-epyc9000:~$ conda list -n singlem-0.20.3 | grep diamond
diamond 2.1.15 h13889ed_0 bioconda

Here's the run log

singlem_pipe.log

[wwood]

OK, hmm. I just pushed an update to singlem 0.20.3 conda which uses 2.1.19 (i.e. build number 1) - does that solve it?

[magicprotoss]

OK, hmm. I just pushed an update to singlem 0.20.3 conda which uses 2.1.19 (i.e. build number 1) - does that solve it?

Still the same issue, maybe some recent changes caused this?

2026/01/25 03:03:34 PM INFO: SingleM v0.20.3
2026/01/25 03:03:34 PM INFO: Retrieval successful. Location of backpack is: /mnt/hdd/bio_db/singlem-db/
2026/01/25 03:03:34 PM INFO: Loaded 59 SingleM packages
2026/01/25 03:03:35 PM INFO: Using as input 42 different pairs of sequence files e.g. ../1.valid_subset_150bp/CK_7_1_1.fastq.gz & ../1.valid_subset_150bp/CK_7_1_2.fastq.gz
2026/01/25 03:03:35 PM INFO: Filtering sequence files through DIAMOND blastx
Traceback (most recent call last):
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 100, in _prefilter
qseqid, full_qseq, sseqid, qstart = line.strip().split('\t')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 4, got 3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/miniconda3/envs/singlem-0.20.3/bin/singlem", line 9, in
sys.exit(main())
^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/main.py", line 776, in main
singlem.pipe.SearchPipe().run(
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/pipe.py", line 73, in run
otu_table_object = self.run_to_otu_table(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/pipe.py", line 370, in run_to_otu_table
self._num_threads, self._working_directory).run_diamond(
^^^^^^^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 34, in run_diamond
fwds = self._prefilter(dmnd, forward_read_files, False, performance_parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/miniconda3/envs/singlem-0.20.3/lib/python3.12/site-packages/singlem/diamond_spkg_searcher.py", line 102, in _prefilter
raise Exception(f"Unexpected line format for DIAMOND output line '{line.strip()}'")
Exception: Unexpected line format for DIAMOND output line 'ML150004291L1C037R00101255044/1 CAAAACGTCTTCTTGCATCAATCAAGCTATGTGGAGTTGCCCAACATCCTGGAAGAACTCGCAATCGACGGTGTCGACCGAGTGCTCGCTGATCTTGGTCTCTCCTCAGACCAATTGGCGGACAACACTCGCGGCTTTAGCTTCGATGCC TIGR00006~GB_GCA_02042632'

(singlem-0.20.3) administrator@zhanglab-epyc9000:/mnt/hdd/ghp/xyj/25ZS_RBDX_MBBR/2.singlem$ conda list | grep diamond
diamond 2.1.19 h13889ed_0 bioconda

[wwood]

It looks like it is the same error, but it is actually a different read (and maybe different readset? I cannot tell)

Can you try just running that individual sequence i.e. feeding just the one file with these contents in them?

>ML150004291L1C037R00101255044/1
CAAAACGTCTTCTTGCATCAATCAAGCTATGTGGAGTTGCCCAACATCCTGGAAGAACTCGCAATCGACGGTGTCGACCGAGTGCTCGCTGATCTTGGTCTCTCCTCAGACCAATTGGCGGACAACACTCGCGGCTTTAGCTTCGATGCC TIGR00006~GB_GCA_02042632

When I do that, I have no problems. However, it also gave not OTUs, so maybe diamond hits but it doesn't match the window?

Anyway if I add the original sequence in the issue as a reverse

(conda)cpu1n001:20260127:~/git/singlem/test/data$ cat issue281.fna
>ML150004291L1C037R00101255044/1
CAAAACGTCTTCTTGCATCAATCAAGCTATGTGGAGTTGCCCAACATCCTGGAAGAACTCGCAATCGACGGTGTCGACCGAGTGCTCGCTGATCTTGGTCTCTCCTCAGACCAATTGGCGGACAACACTCGCGGCTTTAGCTTCGATGCC
(conda)cpu1n001:20260127:~/git/singlem/test/data$ cat issue281.2.fna
>ML150004291L1C042R04006838127/2
GACCGACTGGAGAGCTTCGACATATACGCCCACCTGGACGGCGGCGGGCCGTCCGGTCAGGCTGGTGCATTGCGTCTGGCCATCGCGCGGGCGCTGATCATCGTGCAGCCCGAGGACCGGCCGGCACTGAAGAAAGCGGGGTTCCTCACA

I get an OTU. Do you?

singlem pipe --forward issue281.fna --otu-table /dev/stdout --metapackage ~/m/db/singlem/S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb --reverse issue281.2.fna
2026/01/25 09:20:54 PM INFO: SingleM v0.20.3
2026/01/25 09:20:54 PM INFO: Acquiring SingleM metapackage from Zenodo backpack directory specified ..
2026/01/25 09:20:54 PM INFO: Retrieval successful. Location of backpack is: /home/woodcrob/m/db/singlem/S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb
2026/01/25 09:20:54 PM INFO: Loaded 59 SingleM packages
2026/01/25 09:20:59 PM INFO: Using as input 1 different pairs of sequence files e.g. issue281.fna & issue281.2.fna
2026/01/25 09:20:59 PM INFO: Filtering sequence files through DIAMOND blastx
2026/01/25 09:21:00 PM INFO: Finished DIAMOND prefilter phase
2026/01/25 09:21:00 PM INFO: Assigning sequences to SingleM packages with DIAMOND ..
2026/01/25 09:21:06 PM INFO: Running taxonomic assignment ..
2026/01/25 09:21:06 PM INFO: Assigning taxonomy by singlem query ..
2026/01/25 09:21:06 PM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2
2026/01/25 09:21:06 PM INFO: Searching with SMAFA NAIVE by nucleotide sequence ..
2026/01/25 09:21:06 PM INFO: Querying index for S3.13.ribosomal_S9
2026/01/25 09:21:07 PM INFO: Finished running singlem query-based taxonomic assignment, now running diamond ..
2026/01/25 09:21:07 PM INFO: Assigning taxonomy with DIAMOND for 0 and 1 out of 0 and 1 sequences (n/a% and 100.0%) for sample issue281, package S3.13.ribosomal_S9.spkg
2026/01/25 09:21:09 PM INFO: Finished running taxonomic assignment
2026/01/25 09:21:09 PM INFO: Finished
2026/01/25 09:21:09 PM INFO: Writing /dev/stdout
gene    sample  sequence        num_hits        coverage        taxonomy
S3.13.ribosomal_S9      issue281        GGCGGCGGGCCGTCCGGTCAGGCTGGTGCATTGCGTCTGGCCATCGCGCGGGCGCTGATC    1       1.65    Root; d__Bacteria; p__Actinomycetota; c__Actinomycetes; o__Mycobacteriales; f__Mycobacteriaceae; g__Mycobacterium

btw, I assume

$ diamond --version

says something recent? FWIW though, I don't see this issue with diamond 2.1.18

Also, there have been changes to this part of the code base since 0.19.0, but there you are simply doing a split on a diamond output line, which doesn't leave a lot of room for introduced errors.

If that doesn't work, can you try just on a single run of the 42, and if that fails, possibly transfer me the file? Annoying one to track down.

[wwood]

Actually, sorry, it seems to run from my pixi checkout from the main branch, but not a 0.20.3 conda checkout. But it fails in a different place, later on during taxonomy assignment. Maybe a memory issue on my side? I'll update the diamond dep even further as there's 2.1.21 now.