WGS pipeline parallelization

[kamyshova]

Issue

WGS pipeline can be noticeably time-consuming due to deep sequencing over the entire genome (~ 2 billion reads). It would be great to parallelize the post-alignment and variant calling process where it might be.

Approach

The parallelization can be done for the post-alignment and variant calling processes:

• Post-alignment step: multi-threading approach - Markduplicate + BaseRecal + ApplyBaseRecal spark versions of GATK tools
• Variant calling step: scatter-gather approach - the splitting of reference into pieces.
  E.g. Mutect2 tool can be run with lists of intervals to restrict operating on a subset of genomic regions.

Spark-enabled GATK tools

• MarkDuplicatesSpark

gatk MarkDuplicatesSpark \
   -I sorted_with_readgroup.bam \
   -O output_marked_duplicates.bam \
   -M marked_dup_metrics.txt \ # optional ?
   --spark-runner SPARK \
   --spark-master MASTER_URL

• BaseRecalibratorSpark

gatk BaseRecalibratorSpark \
   -I output_marked_duplicates.bam \
   -R reference.fasta \
   --known-sites sites_of_variation.vcf \
   --known-sites setOfSitesToMask.vcf \
   -O output_recal.table \
   --spark-runner SPARK \
   --spark-master MASTER_URL

• ApplyBQSRSpark

gatk ApplyBQSRSpark \
   -I output_marked_duplicates.bam \
   -bqsr output_recal.table \
   -O output_bqsr.bam \
   --spark-runner SPARK \
   --spark-master MASTER_URL