--star-aligner fails with Segmentation Fault during StarIndexBuilder on WSL2

[CD656]

Hello ShapeMapper Team,

I'm encountering a consistent Segmentation fault when trying to use the --star-aligner option on WSL2 (Ubuntu 22.04.4 LTS on Windows 11).

The analysis runs perfectly fine with the default Bowtie2 aligner on the exact same dataset, which suggests the issue is specific to the STAR integration or its interaction with the WSL2 environment.

Environment Details:

OS: Windows 11
WSL Version: WSL2
Linux Distribution: Ubuntu 22.04.4 LTS
ShapeMapper Version: v2.3.0

Command Used:

./shapemapper \
    --name Ecoli_TPP_analysis_STAR \
    --target ~/Ecoli/data/TPP_ref.fa \
    --out ~/Ecoli/results_star \
    --modified --R1 ~/Ecoli/data/N_R1.fastq.gz --R2 ~/Ecoli/data/N_R2.fastq.gz \
    --untreated --R1 ~/Ecoli/data/D_R1.fastq.gz --R2 ~/Ecoli/data/D_R2.fastq.gz \
    --star-aligner \
    --nproc 14 \
    --overwrite

Log Output and Error:
The pipeline fails very quickly during the STAR index building step. Here is the complete log output:

Started ShapeMapper v2.3.0 at 2025-08-28 15:29:56
Output will be logged to Ecoli_TPP_analysis_STAR_shapemapper_log.txt
Running from directory: /home/adm_wh/shapemapper2-2.3
args:  --name Ecoli_TPP_analysis_STAR --target /home/adm_wh/Ecoli/data/TPP_ref.fa --out /home/adm_wh/Ecoli/results_star --modified --R1 /home/adm_wh/Ecoli/data/N_R1.fastq.gz --R2 /home/adm_wh/Ecoli/data/N_R2.fastq.gz --untreated --R1 /home/adm_wh/Ecoli/data/D_R1.fastq.gz --R2 /home/adm_wh/Ecoli/data/D_R2.fastq.gz --star-aligner --nproc 14 --overwrite
Serial mode currently required for STAR aligner. Switching.
Created pipeline at 2025-08-28 15:29:56
Running FastaFormatChecker at 2025-08-28 15:29:56 . . .
. . . done at 2025-08-28 15:29:56
Running StarIndexBuilder at 2025-08-28 15:29:56 . . .

ERROR: Component "StarIndexBuilder" failed, giving the following error message:=======================

Segmentation fault
=======================================================================================================
ShapeMapper run failed at 2025--08-28 15:29:58.

I have tried using both the system-installed STAR (apt-get install rna-star) and the official pre-compiled STAR binary (v2.7.11a), but the result is identical in both cases. The Bowtie2 pipeline continues to work flawlessly.

Please let me know if you need any more information or if there are other tests I can run to help diagnose this.

Thank you for your time and for developing this great tool!

[Psirving]

ShapeMapper2 comes packaged with its dependencies, so your local installations of STAR should not effect the version that ShapeMapper2 calls. Anyways, I don't suspect the STAR version is the issue. However, it might be informative to try calling those other versions directly to try to build the TPP_ref genome index (not using ShapeMapper2). If that returns a segfault, then it is surely the interaction between STAR and WSL2 (assuming TPP_ref is a small, simple genome). Let me know what happens if you try this!

[CD656]

Hi Psirving,

Thank you so much for your quick and insightful reply; it really helped point me in the right direction for testing. Based on your suggestions, I've run a series of tests, and it seems the issue might be more fundamental than we thought. I'm still a bit puzzled.

First off, my apologies for the large font in my initial post; I'm not sure why it formatted that way when pasting from my terminal.

Second, I'd like to provide some context for my use case, which might explain why I was initially testing STAR with the small TPP_ref.fa genome. Due to funding constraints, I must fully familiarize myself with your SHAPE analysis pipeline before I can propose a project and secure support from my supervisor. As a result, I don't have my own sequencing data yet. I apologize for this, but I was using the example_data bundled with your software to simulate third-party data for learning purposes. Specifically, I used the cat command to merge the TPPminus_R1 files to simulate a DMSO control (R1/R2) and did the same for the TPPplus files to simulate an NAI-modified sample. It was during this learning process with this simulated data that I encountered the Segmentation fault when using the STAR option.

As you correctly pointed out, TPP_ref is a very small sequence, and I did overlook the note in the README about STAR's performance with small fragments. That was my oversight.

Following your valuable advice, I've conducted more systematic tests over the past few days, incorporating a more appropriately sized target: the West Nile Virus (WNV) genome (~11kb). This data comes from a published paper where the authors explicitly mention using ShapeMapper2 for their analysis. However, the outcome was consistent regardless of which genome I used.

Here are my key findings:

1. Direct calls to the bundled STAR for index generation consistently result in a segfault:
This confirms your suspicion that the issue likely lies in the interaction between STAR and my WSL2 environment, and it appears to be independent of the genome size.

Test ①: With the E.coli TPP genome (TPP_ref.fa)

/home/admin/shapemapper2-2.3/internals/thirdparty/miniconda/envs/shapemapper_make/bin/STAR \
    --runMode genomeGenerate \
    --genomeDir ~/Ecoli/TPP_STAR_index \
    --genomeFastaFiles ~/Ecoli/data/TPP_ref.fa \
    --runThreadN 14 \
    --genomeSAindexNbases 4
Segmentation fault (core dumped)

Test ②: With the West Nile Virus genome (West_Nile_virus_complete_genome.fa)

/home/admin/shapemapper2-2.3/internals/thirdparty/miniconda/envs/shapemapper_make/bin/STAR \
    --runMode genomeGenerate \
    --genomeDir ~/West_Nile_Virus_C6-36_Analysis/WNV_STAR_index \
    --genomeFastaFiles ~/West_Nile_Virus_C6-36_Analysis/data/West_Nile_virus_complete_genome.fa \
    --runThreadN 14 \
    --genomeSAindexNbases 10
Segmentation fault (core dumped)

2. The full ShapeMapper pipeline using the STAR option also fails at the indexing step:
Whether using my simulated E.coli data or the new WNV data, the program fails at the StarIndexBuilder stage due to a Segmentation fault.

Test ③: Running the full pipeline with West Nile Virus data

./shapemapper \
    --name WNV_C636_analysis_rep1_STAR \
    --target ~/West_Nile_Virus_C6-36_Analysis/data/West_Nile_virus_complete_genome.fa \
    --out ~/West_Nile_Virus_C6-36_Analysis/results_star_rep1 \
    --star-aligner \
    --overwrite \
    --untreated --U ~/West_Nile_Virus_C6-36_Analysis/data/C636-DMSO-1.fastq.gz \
    --modified --U ~/West_Nile_Virus_C6-36_Analysis/data/C636-NAI-1.fastq.gz \
    --nproc 14

Log Output Summary:

...
Running FastaFormatChecker at 2025-08-31 01:12:15 . . .
. . . done at 2025-08-31 01:12:15
Running StarIndexBuilder at 2025-08-31 01:12:15 . . .

ERROR: Component "StarIndexBuilder" failed, giving the following error message:========================

Segmentation fault
========================================================================================================
ShapeMapper run failed at 2025-08-31 01:12:17.

3. The software's own run_all_tests.sh script also reports STAR-related failures:
This might be the most telling piece of evidence. When I ran internals/test/run_all_tests.sh, I noticed that all STAR-related tests failed, while the Bowtie2 tests passed. This seems to further confirm the issue is with the STAR executable bundled with your software.

Error Summary:

[       OK ] ROC test 0: 2-sample rRNAs, bowtie2
...
[  FAILED  ] ROC test 1: 2-sample rRNAs, STAR
...
0 / 98  c++ unit test(s) failed.
8 / 32  end-to-end success test(s) failed.
0 / 39  module failure detection test(s) failed.
0 / 13  sequence variant correction test(s) failed.
1 / 2   area under ROC curve test(s) failed.
----------
9 / 184 total test(s) failed.
FAILURE

Just to confirm, I assume run_all_tests.sh calls the internally packaged STAR and isn't affected by any other STAR versions on my system, is that correct?

In summary, all evidence points to a compatibility issue between the STAR binary bundled with ShapeMapper2 and my WSL2 (Ubuntu 22.04 on Windows 11) environment. Meanwhile, the Bowtie2 pipeline works perfectly in all scenarios.

I hope this detailed information is helpful for diagnosing the issue. Please let me know if there's anything else I can test on my end.

Thanks again for your time and for this excellent tool!

[Psirving]

As far as I can tell, your tests have all called ShapeMapper2's internal STAR. Have you tried running an independently installed version of STAR to build the index?

Another consideration for you is the actual performance gain of using STAR vs. the time invested in getting it to work here. As one of the maintainers of this software, I'm invested in figuring out what's going on here. But for you, while STAR is certainly more efficient and quicker than running Bowtie2, Bowtie2 should be usable depending on your read depth and computation resources.

[CD656]

Hi Psirving,

Thank you for your previous suggestion! It led me to a decisive discovery.

I can confirm that directly calling the STAR executable bundled within ShapeMapper2 consistently results in an immediate segmentation fault.

However, the key finding is that when I downloaded an independent, official build of STAR (the exact same version, 2.7.11a) and ran it with identical parameters, the genome index generation completed perfectly. Here is the full terminal output:

admin@DESKTOP-LDCV0PT:~$ ~/STAR-2.7.11a/bin/Linux_x86_64_static/STAR --runMode genomeGenerate --genomeDir ~/STAR_independent_test_index --genomeFastaFiles ~/Ecoli/data/TPP_ref.fa --runThreadN 14 --genomeSAindexNbases 5
	/home/admin/STAR-2.7.11a/bin/Linux_x86_64_static/STAR --runMode genomeGenerate --genomeDir /home/admin/STAR_independent_test_index --genomeFastaFiles /home/admin/Ecoli/data/TPP_ref.fa --runThreadN 14 --genomeSAindexNbases 5
	STAR version: 2.7.11a   compiled: 2023-08-15T11:38:34-04:00 :/home/dobin/data/STAR/STARcode/STAR.master/source
Sep 02 15:25:18 ..... started STAR run
Sep 02 15:25:18 ... starting to generate Genome files
!!!!! WARNING: --genomeSAindexNbases 5 is too large for the genome size=137, which may cause seg-fault at the mapping step. Re-run genome generation with recommended --genomeSAindexNbases 2
Sep 02 15:25:18 ... starting to sort Suffix Array. This may take a long time...
Sep 02 15:25:18 ... sorting Suffix Array chunks and saving them to disk...
Sep 02 15:25:18 ... loading chunks from disk, packing SA...
Sep 02 15:25:18 ... finished generating suffix array
Sep 02 15:25:18 ... generating Suffix Array index
Sep 02 15:25:18 ... completed Suffix Array index
Sep 02 15:25:18 ... writing Genome to disk ...
Sep 02 15:25:18 ... writing Suffix Array to disk ...
Sep 02 15:25:18 ... writing SAindex to disk
Sep 02 15:25:18 ..... finished successfully

This strongly suggests that the issue is likely related to a library conflict between the Miniconda environment that ShapeMapper2 uses to run STAR and my host WSL2 system environment.

Meanwhile, the Bowtie2 pipeline continues to work flawlessly.

I hope this detailed comparison provides the final clue needed to pinpoint the issue. Please let me know if there's anything else I can test.

[Psirving]

This is very helpful. Thank you for running that test. I will try to recreate the issue under the same conditions. I won't be able to get to this right away, but I'll keep you updated.

-Patrick