ConcurrentModificationException when executing multithreaded

[ddemaeyer]

Annotating vcf file using 32-cores results in multiple ConcurrentModificationExceptions. These Exceptions are all thrown from Transcript in get5primeUtrs

java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at org.snpeff.interval.Transcript.get5primeUtrs(Transcript.java:1171)
at org.snpeff.interval.Transcript.cds(Transcript.java:524)
at org.snpeff.snpEffect.VariantEffect.getCdsLength(VariantEffect.java:416)
at org.snpeff.vcf.VcfEffect.set(VcfEffect.java:1070)
at org.snpeff.vcf.VcfEffect.(VcfEffect.java:147)
at org.snpeff.outputFormatter.VcfOutputFormatter.addInfo(VcfOutputFormatter.java:97)
at org.snpeff.outputFormatter.VcfOutputFormatter.toString(VcfOutputFormatter.java:285)
at org.snpeff.outputFormatter.OutputFormatter.endSection(OutputFormatter.java:112)
at org.snpeff.outputFormatter.VcfOutputFormatter.endSection(VcfOutputFormatter.java:229)
at org.snpeff.snpEffect.commandLine.eff.WorkerEff.calculate(WorkerEff.java:66)
at org.snpeff.snpEffect.commandLine.eff.WorkerEff.calculate(WorkerEff.java:18)
at org.snpeff.akka.Worker.calculate(Worker.java:52)
at org.snpeff.akka.Worker.onReceive(Worker.java:63)
at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:154)
at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:153)
at akka.actor.Actor$class.apply(Actor.scala:311)
at akka.actor.UntypedActor.apply(UntypedActor.scala:93)
at akka.actor.ActorCell.invoke(ActorCell.scala:619)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:196)
at akka.dispatch.Mailbox.run(Mailbox.scala:178)
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:505)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

[pcingola]

Could you please send more details to replicate the issue?
(SnpEff version, Input file, command line, etc.)

[ddemaeyer]

SnpEff version 4.3b, running on data from 1000 genomes (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/) on a 32-core machine

Command:
java -Xmx45g -jar snpEff.jar -v -t -stats ex1.html GRCh38.86 ../ALL.wgs.phase3_shapeit2_mvncall_integrated_v5b.20130502.sites.vcf.gz > ../ALL.wgs.phase3_shapeit2_mvncall_integrated_v5b.20130502.sites_annotated_snp_eff.vcf.gz

[bbimber]

Hello - we're seeing the same issue as well, same SnpEff version, though a different genome.

[slagelwa]

As are we.

[garrettjstevens]

I've been seeing this issue as well. I'm using version 4.3t of SnpEff with the standard GRCh37.87 genome. The VCF and config file I'm using are attached: test.zip. This is being run on an AWS c5.2xlarge instance (8 cores 16GB mem). Command line is

java -Xmx11604m -jar ./snpEff/snpEff.jar -config ./snpeff/SnpEff.config -t -verbose GRCh37.87 ./input/chr22.vcf.gz > chr22_snpeff.vcf 2> chr22_snpeff.log

The error isn't always exactly the same, but it's usually like:

java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        at java.util.ArrayList$Itr.next(ArrayList.java:851)
        at org.snpeff.interval.Transcript.get5primeUtrs(Transcript.java:1087)
        at org.snpeff.interval.Transcript.cds(Transcript.java:524)
        at org.snpeff.snpEffect.VariantEffect.getCdsLength(VariantEffect.java:416)
        at org.snpeff.vcf.VcfEffect.set(VcfEffect.java:1046)
        at org.snpeff.vcf.VcfEffect.<init>(VcfEffect.java:147)
        at org.snpeff.outputFormatter.VcfOutputFormatter.addInfo(VcfOutputFormatter.java:98)
        at org.snpeff.outputFormatter.VcfOutputFormatter.toString(VcfOutputFormatter.java:286)
        at org.snpeff.outputFormatter.OutputFormatter.endSection(OutputFormatter.java:112)
        at org.snpeff.outputFormatter.VcfOutputFormatter.endSection(VcfOutputFormatter.java:230)
        at org.snpeff.snpEffect.commandLine.eff.WorkerEff.calculate(WorkerEff.java:66)
        at org.snpeff.snpEffect.commandLine.eff.WorkerEff.calculate(WorkerEff.java:18)
        at org.snpeff.akka.Worker.calculate(Worker.java:52)
        at org.snpeff.akka.Worker.onReceive(Worker.java:63)
        at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:154)
        at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:153)
        at akka.actor.Actor$class.apply(Actor.scala:311)
        at akka.actor.UntypedActor.apply(UntypedActor.scala:93)
        at akka.actor.ActorCell.invoke(ActorCell.scala:619)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:196)
        at akka.dispatch.Mailbox.run(Mailbox.scala:178)
        at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:505)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Error: Error while processing VCF entry (line 405997) :
        10      12939449        .       AGAG    A       .       .       .
java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        at java.util.ArrayList$Itr.next(ArrayList.java:851)
        at org.snpeff.interval.Transcript.get5primeUtrs(Transcript.java:1087)
        at org.snpeff.interval.Transcript.cds(Transcript.java:524)
        at org.snpeff.snpEffect.VariantEffect.getCdsLength(VariantEffect.java:416)
        at org.snpeff.vcf.VcfEffect.set(VcfEffect.java:1046)
        at org.snpeff.vcf.VcfEffect.<init>(VcfEffect.java:147)
        at org.snpeff.outputFormatter.VcfOutputFormatter.addInfo(VcfOutputFormatter.java:98)
        at org.snpeff.outputFormatter.VcfOutputFormatter.toString(VcfOutputFormatter.java:286)
        at org.snpeff.outputFormatter.OutputFormatter.endSection(OutputFormatter.java:112)
        at org.snpeff.outputFormatter.VcfOutputFormatter.endSection(VcfOutputFormatter.java:230)
        at org.snpeff.snpEffect.commandLine.eff.WorkerEff.calculate(WorkerEff.java:66)
        at org.snpeff.snpEffect.commandLine.eff.WorkerEff.calculate(WorkerEff.java:18)
        at org.snpeff.akka.Worker.calculate(Worker.java:52)
        at org.snpeff.akka.Worker.onReceive(Worker.java:63)
        at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:154)
        at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:153)
        at akka.actor.Actor$class.apply(Actor.scala:311)
        at akka.actor.UntypedActor.apply(UntypedActor.scala:93)
        at akka.actor.ActorCell.invoke(ActorCell.scala:619)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:196)
        at akka.dispatch.Mailbox.run(Mailbox.scala:178)
        at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:505)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

I've run the VCF above five times and four of the five times this error occurred, but for a different line in the VCF each time.

[eszekeres]

Not sure if ever was identified reason for this, but we too just got this message processing a sample with 4.3t (reference build GRCh37.75...first I have seen it....

Error: Error while processing VCF entry (line 212416) :
16 29973395 . C T . clustered_events;t_lod_fstar DP=24;MQ=60.00;TLOD=5.00;FractionInformativeReads=1.000 GT:AD:AF:F1R2:F2R1:DP:SB:MB 0/1:22,2:0.083:9,0:13,2:24:20,2,2,0:11,11,0,2
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.snpeff.interval.Transcript.get5primeUtrs(Transcript.java:1087)
at org.snpeff.interval.Utr5prime.get5primeUtrs(Utr5prime.java:40)

[csw]

I was also running into this, and put together #236 as a fix. It's working for me so far, but please give it more testing!

[pcingola]

Closing old issues.