generate_multihetsep.py generates no output

[MatteoSebastianelli]

Hello,

I am trying to generate the input files for msmc2 with the generate_multihetsep.py. The script runs for some time and SLURM marks the job as completed. However, the output files that are generated are empty. I hope you can help me to figure out what am I doing wrong here.

I paste below the command I used (notice it is a slurm array, one job for each separate chromosome):

python $msmc_tools/generate_multihetsep.py --mask $mapp_mask/darwin_finch_"$CHR".mask.bed.gz --mask $mask_folder/acutirostris_Genovesa_101/acutirostris_Genovesa_101_"$CHR"mask.bed.gz --mask $mask_folder/noctis_Barbados_1/noctis_Barbados_1"$CHR"mask.bed.gz --mask $mask_folder/bicolor_Barbados_1/bicolor_Barbados_1"$CHR"mask.bed.gz --mask $mask_folder/BB_Daphne_19127/BB_Daphne_19127"$CHR"mask.bed.gz --mask $mask_folder/conirostris_Espanola_3/conirostris_Espanola_3"$CHR"mask.bed.gz --mask $mask_folder/crassirostris_SantaCruz_4/crassirostris_SantaCruz_4"$CHR"mask.bed.gz --mask $mask_folder/difficilis_Pinta_7/difficilis_Pinta_7"$CHR"mask.bed.gz --mask $mask_folder/fortis_Isabela_106/fortis_Isabela_106"$CHR"mask.bed.gz --mask $mask_folder/fuliginosa_Santiago_16/fuliginosa_Santiago_16"$CHR"mask.bed.gz --mask $mask_folder/fusca_SanCristobal_7/fusca_SanCristobal_7"$CHR"mask.bed.gz --mask $mask_folder/heliobates_Isabela_2/heliobates_Isabela_2"$CHR"mask.bed.gz --mask $mask_folder/inornata_Cocos_7/inornata_Cocos_7"$CHR"mask.bed.gz --mask $mask_folder/02Mac1231/02Mac1231"$CHR"mask.bed.gz --mask $mask_folder/31717/31717"$CHR"mask.bed.gz --mask $mask_folder/olivacea_Santiago_14/olivacea_Santiago_14"$CHR"mask.bed.gz --mask $mask_folder/P16602_1020_S20/P16602_1020_S20"$CHR"mask.bed.gz --mask $mask_folder/parvulus_Isabela_4/parvulus_Isabela_4"$CHR"mask.bed.gz --mask $mask_folder/pauper_Floreana_11/pauper_Floreana_11"$CHR"mask.bed.gz --mask $mask_folder/propinqua_Genovesa_1/propinqua_Genovesa_1"$CHR"mask.bed.gz --mask $mask_folder/psittacula_Pinta_1/psittacula_Pinta_1"$CHR"mask.bed.gz --mask $mask_folder/scandens_Santiago_3/scandens_Santiago_3"$CHR"mask.bed.gz --mask $mask_folder/septentrionalis_Wolf_2/septentrionalis_Wolf_2"$CHR"mask.bed.gz $mask_folder/acutirostris_Genovesa_101/acutirostris_Genovesa_101"$CHR".vcf.gz $mask_folder/noctis_Barbados_1/noctis_Barbados_1_"$CHR".vcf.gz $mask_folder/bicolor_Barbados_1/bicolor_Barbados_1_"$CHR".vcf.gz $mask_folder/BB_Daphne_19127/BB_Daphne_19127_"$CHR".vcf.gz $mask_folder/conirostris_Espanola_3/conirostris_Espanola_3_"$CHR".vcf.gz $mask_folder/crassirostris_SantaCruz_4/crassirostris_SantaCruz_4_"$CHR".vcf.gz $mask_folder/difficilis_Pinta_7/difficilis_Pinta_7_"$CHR".vcf.gz $mask_folder/fortis_Isabela_106/fortis_Isabela_106_"$CHR".vcf.gz $mask_folder/fuliginosa_Santiago_16/fuliginosa_Santiago_16_"$CHR".vcf.gz $mask_folder/fusca_SanCristobal_7/fusca_SanCristobal_7_"$CHR".vcf.gz $mask_folder/heliobates_Isabela_2/heliobates_Isabela_2_"$CHR".vcf.gz $mask_folder/inornata_Cocos_7/inornata_Cocos_7_"$CHR".vcf.gz $mask_folder/02Mac1231/02Mac1231_"$CHR".vcf.gz $mask_folder/31717/31717_"$CHR".vcf.gz $mask_folder/olivacea_Santiago_14/olivacea_Santiago_14_"$CHR".vcf.gz $mask_folder/P16602_1020_S20/P16602_1020_S20_"$CHR".vcf.gz $mask_folder/parvulus_Isabela_4/parvulus_Isabela_4_"$CHR".vcf.gz $mask_folder/pauper_Floreana_11/pauper_Floreana_11_"$CHR".vcf.gz $mask_folder/propinqua_Genovesa_1/propinqua_Genovesa_1_"$CHR".vcf.gz $mask_folder/psittacula_Pinta_1/psittacula_Pinta_1_"$CHR".vcf.gz $mask_folder/scandens_Santiago_3/scandens_Santiago_3_"$CHR".vcf.gz $mask_folder/septentrionalis_Wolf_2/septentrionalis_Wolf_2_"$CHR".vcf.gz > ./finches/multihetsep_files/finches_"$CHR"_multihetsep.txt

And here is an example of what I get in the slurm log file (from a small chromosome)

generating msmc input file with 44 haplotypes
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/masks/darwin_finch_chr22.mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/acutirostris_Genovesa_101/acutirostris_Genovesa_101_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/noctis_Barbados_1/noctis_Barbados_1_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/bicolor_Barbados_1/bicolor_Barbados_1_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/BB_Daphne_19127/BB_Daphne_19127_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/conirostris_Espanola_3/conirostris_Espanola_3_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/crassirostris_SantaCruz_4/crassirostris_SantaCruz_4_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/difficilis_Pinta_7/difficilis_Pinta_7_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/fortis_Isabela_106/fortis_Isabela_106_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/fuliginosa_Santiago_16/fuliginosa_Santiago_16_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/fusca_SanCristobal_7/fusca_SanCristobal_7_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/heliobates_Isabela_2/heliobates_Isabela_2_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/inornata_Cocos_7/inornata_Cocos_7_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/02Mac1231/02Mac1231_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/31717/31717_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/olivacea_Santiago_14/olivacea_Santiago_14_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/P16602_1020_S20/P16602_1020_S20_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/parvulus_Isabela_4/parvulus_Isabela_4_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/pauper_Floreana_11/pauper_Floreana_11_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/propinqua_Genovesa_1/propinqua_Genovesa_1_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/psittacula_Pinta_1/psittacula_Pinta_1_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/scandens_Santiago_3/scandens_Santiago_3_chr22_mask.bed.gz
adding mask: /crex/proj/snic2020-2-19/private/darwins_finches/users/seba/finches463/msmc2/coverage_masks/septentrionalis_Wolf_2/septentrionalis_Wolf_2_chr22_mask.bed.gz
processing pos 1000000
processing pos 2000000
processing pos 3000000
processing pos 4000000

Thanks in advance for the help!

Matteo

[stschiff]

I don't know what you're up to here, but I can think of no scenario where it would be useful to merge so many haplotypes. I don't know why the output is empty, but you should check that there is enough overlap of sites present in all of these individual haplotypes and bed files. If you want to trouble-shoot, maybe start with just merging pairs of haplotypes and first analysing those. You can then progressively build up combinations of more haplotypes.