[ADAM-909] Refactoring variation RDDs. by fnothaft · Pull Request #1015 · bigdatagenomics/adam

fnothaft · 2016-04-24T20:58:53Z

Resolves #909:

Refactors org.bdgenomics.adam.rdd.variation to add GenomicRDDs for Genotype, Variant, and VariantContext. These classes write sequence and sample metadata to disk.
Refactors ADAMRDDFunctions to an abstract class in preparation for further refactoring in Factor out *RDDFunctions classes #1011.
Added AvroGenomicRDD trait which consolidates Parquet + Avro metadata writing code across all Avro data models.

AmplabJenkins · 2016-04-24T21:15:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1176/

Build result: FAILURE

[...truncated 24 lines...]Triggering ADAM-prb ? 2.3.0,2.10,1.6.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2016-04-25T13:52:49Z

Jenkins, retest this please.

AmplabJenkins · 2016-04-25T14:04:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1177/
Test PASSed.

heuermh · 2016-04-26T17:25:15Z

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

    val path = resourcePath("small.vcf")

-    val vcs = sc.loadGenotypes(path).toVariantContext.collect.sortBy(_.position)
+    val vcs = sc.loadGenotypes(path).toVariantContextRDD.rdd.collect.sortBy(_.position)


Since we have this

object ADAMContext { implicit def genomicRDDToRDD[T](gRdd: GenomicRDD[T]): RDD[T] = gRdd.rdd

https://github.com/bigdatagenomics/adam/blob/master/adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala#L114

why is the .rdd here and elsewhere necessary? Implicits work fine except for when they don't?

Oh man. I zoned on the implicit. That's my bad. I'll fix this.

Sorry, what I mean was that sometimes the genomicRDDToRDD implicit works for me and sometimes it doesn't. I assume because you are using .rdd here that they are necessary. The scala compiler and I are still getting to know each other.

Nah I was using them because I forgot that we had the implicit.

fnothaft · 2016-05-20T16:33:30Z

Just pushed an update with upgrades to bdg-formats 0.8.0 and cleanup of the .rdd bits. This fails unit tests right now, which I'll patch up this AM. CC @erictu @heuermh

The 0.8.0 upgrade is split out into a separate commit. This will need to be squashed down before merge.

AmplabJenkins · 2016-05-20T16:36:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1239/

Build result: FAILURE

GitHub pull request #1015 of commit 0ecfae9 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1015/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 8a08bb5f7a926e05f9627de97e383f281c935d51 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1015/merge^{commit} # timeout=10Checking out Revision 8a08bb5f7a926e05f9627de97e383f281c935d51 (origin/pr/1015/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8a08bb5f7a926e05f9627de97e383f281c935d51First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2016-05-20T17:23:14Z

Just pushed another commit to address the failing unit tests. Please review @heuermh @erictu.

@heuermh you may be interested to find that GZIPed VCF loading doesn't work. Alas, the VCFHeaderReader code from Hadoop-BAM seems to fail when trying to load GZIPed VCF. I think there's a fall-through case that wasn't considered properly. I'm going to open an issue here to test more, and then if I can isolate/test/resolve it, I'll go upstream to Hadoop-BAM.

heuermh · 2016-05-20T17:29:54Z

you may be interested to find that GZIPed VCF loading doesn't work.

It was working for me, and there is a unit test to verify it.
https://github.com/bigdatagenomics/adam/blame/master/adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala#L284

fnothaft · 2016-05-20T17:32:42Z

you may be interested to find that GZIPed VCF loading doesn't work.

It was working for me, and there is a unit test to verify it.
https://github.com/bigdatagenomics/adam/blame/master/adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala#L284

Oh, yeah; didn't mean to imply that it was currently misimplemented. The issue comes up because we now rely on reading the VCF header separately to pull out sequence/sample metadata:

https://github.com/bigdatagenomics/adam/pull/1015/files#diff-d36ea7d0742decd0b040a73a96af06e9R136

The VCFHeaderReader in Hadoop-BAM appears to mishandle GZIP/BGZIPed VCFs. As far as I can tell, it falls through and tries to treat them as BCF files.

AmplabJenkins · 2016-05-20T17:40:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1240/
Test PASSed.

heuermh · 2016-05-20T17:44:35Z

Yep, in agreement. There's some duplicated code in Hadoop-BAM that should be shared, and things probably diverged between the VCF input format and VCF header reader in this case.

heuermh · 2016-05-23T16:59:27Z

adam-cli/src/main/scala/org/bdgenomics/adam/cli/Vcf2ADAM.scala

+    val variantContextRdd = sc.loadVcf(args.vcfPath, sdOpt = dictionary)
+    var variantContextsToSave = if (args.coalesce > 0) {
+      if (args.coalesce > variantContextRdd.partitions.size || args.forceShuffle) {
+        variantContextRdd.transform(_.coalesce(args.coalesce, shuffle = true))


could you explain what the .transform(_. part is doing here? is it not possible to call variantContextRdd.coalesce directly?

.transform(_.) transforms the RDD that underlies the VariantContextRDD and emits a new VariantContextRDD.

If you did variantContextRdd.coalesce that would emit a RDD[VariantContext] and you'd lose the .saveAsVcf/etc functions, which we use later.

I'm liking this pattern less as time goes on. Perhaps we should drop RDD from the VariantContextRDD class names for something more generic and remove the implicit conversion. Or at least remove the implicit conversion. What the user gets back from an ADAMContext load method should has_a RDD and not pretend to is_a RDD.

Perhaps we should drop RDD from the VariantContextRDD

I agree there may be some nomenclature concern that our many *RDD container objects for rdd+dicts are not RDDs in same true sense as IntervalRDD and IndexedRDD which actually extend RDD. If I am reading correctly, there is not an implicit conversion that would even allow these to be treated as an RDD directly. The meaning seems more like Holder ofRDDofVariantContextWithDicts though I don't advocate that name. What do you think @fnothaft ? At this point though we have already gone down this path a ways using the suffix "RDD" in a broad sense, so a renaming should perhaps come in the future after this PR.

There's this implicit conversion from GenomicRDD (which VariantContextRDD and others extend from) to RDD
https://github.com/fnothaft/adam/blob/822c64e3530d7ed8bb86daeb7b900bd7bf2f7854/adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala#L116

Created a new issue #1040, I'm ok with this for now.

heuermh · 2016-05-24T19:04:34Z

Created HadoopGenomics/Hadoop-BAM#96 to document missing support for gzipped and BGZF VCF formats.

erictu · 2016-05-24T20:55:05Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/variation/VariationRDDFunctions.scala

-      .map { case (v: RichVariant, g) => new VariantContext(ReferencePosition(v), v, g, None) }
-  }

  def filterByOverlappingRegion(query: ReferenceRegion): RDD[Genotype] = {


Can we get this operation supported over the GenotypeRDD class?

fnothaft · 2016-05-25T23:37:50Z

Just pushed 0f90302 to address the VCF.GZ issues. I will merge @jpdna's PR as well. @heuermh I'm still working on the BCF issue. There's something more going on here. I will ping back with more of an error trace.

AmplabJenkins · 2016-05-25T23:51:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1243/
Test PASSed.

heuermh · 2016-05-26T15:52:04Z

I'm still working on the BCF issue.

Which BCF issue? BCF files probably won't work in ADAM due to issue(s) upstream in HTSJDK, where they don't follow the latest BCF specification.

fnothaft · 2016-05-26T16:05:25Z

I'm still working on the BCF issue.

Which BCF issue? BCF files probably won't work in ADAM due to issue(s) upstream in HTSJDK, where they don't follow the latest BCF specification.

Ah, OK! That is the exact issue I was running into. I won't hassle myself trying to fix that then.

(I tried regenerating BCF using Picard, which amusingly enough, doesn't seem to work)

AmplabJenkins · 2016-05-26T16:53:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1244/
Test PASSed.

fnothaft · 2016-05-26T20:12:41Z

Just pushed a commit that (I think) addresses the remaining review comments. Can I get a final pass on this? I will then squash this down to two commits.

AmplabJenkins · 2016-05-26T20:27:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1245/
Test PASSed.

heuermh · 2016-05-26T21:11:03Z

+1, thanks!

jpdna · 2016-05-26T21:15:54Z

+1 - I have no outstanding concerns

erictu · 2016-05-26T21:34:12Z

+1, seems good to me!

fnothaft · 2016-05-26T21:40:05Z

Squashed down into three commits and cleaned up the history.

fnothaft · 2016-05-26T21:44:40Z

Hold on the merge; I see a unit test failure in one of the intermediate commits when building locally.

AmplabJenkins · 2016-05-26T21:55:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1246/
Test PASSed.

fnothaft · 2016-05-26T22:43:16Z

Fixed and pushed.

AmplabJenkins · 2016-05-26T22:57:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1247/

Build result: FAILURE

[...truncated 24 lines...]Triggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2016-05-26T23:16:32Z

Jenkins, retest this please.

heuermh · 2016-05-26T23:18:08Z

Do you think our logging related changes might have anything to do with that failure? We removed some of the logging configuration initialization bits in #1028

AmplabJenkins · 2016-05-26T23:32:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1248/
Test PASSed.

erictu · 2016-05-27T01:16:33Z

Can we add in the corresponding changes of schema to org.bdgenomics.adam.projections.GenotypeField?

heuermh · 2016-06-01T20:14:24Z

@fnothaft do you want to push another commit for GenotypeField and then re-squash? I could pr against this one but it probably isn't worth it for such a small change.

heuermh · 2016-06-01T20:14:52Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+        // due to a bug upstream in Hadoop-BAM, the VCFHeaderReader class errors when reading
+        // headers from .vcf.gz files
+        //


add link to HadoopGenomics/Hadoop-BAM#96?

Resolves bigdatagenomics#909: * Refactors `org.bdgenomics.adam.rdd.variation` to add `GenomicRDD`s for `Genotype`, `Variant`, and `VariantContext`. These classes write sequence and sample metadata to disk. * Refactors `ADAMRDDFunctions` to an abstract class in preparation for further refactoring in bigdatagenomics#1011. * Added `AvroGenomicRDD` trait which consolidates Parquet + Avro metadata writing code across all Avro data models.

fnothaft · 2016-06-01T21:25:10Z

@fnothaft do you want to push another commit for GenotypeField and then re-squash? I could pr against this one but it probably isn't worth it for such a small change.

Sorry about the delay on this @heuermh @erictu! Just pushed the change.

AmplabJenkins · 2016-06-01T21:46:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1249/
Test PASSed.

heuermh · 2016-06-01T21:52:46Z

+1

erictu · 2016-06-02T16:46:11Z

+1

heuermh · 2016-06-03T04:27:59Z

Merged manually as 449a882, 2c07382 and 525fda8. Thanks!

fnothaft mentioned this pull request Apr 24, 2016

Factor out *RDDFunctions classes #1011

Closed

heuermh reviewed Apr 26, 2016
View reviewed changes

fnothaft mentioned this pull request May 10, 2016

Replace contig with contigName in Variant. Resolves #75. bigdatagenomics/bdg-formats#76

Merged

fnothaft mentioned this pull request May 18, 2016

Add fragmentEndPosition to NucleotideContigFragment #1030

Closed

fnothaft force-pushed the genotypes-rdd branch from 36ba552 to 0ecfae9 Compare May 20, 2016 16:32

heuermh reviewed May 23, 2016
View reviewed changes

erictu reviewed May 24, 2016
View reviewed changes

fnothaft force-pushed the genotypes-rdd branch from 4476c1c to 0f90302 Compare May 25, 2016 23:36

jpdna mentioned this pull request May 26, 2016

VCF sample metadata - proposal for a GenotypedSampleMetadata object #1039

Closed

heuermh mentioned this pull request May 26, 2016

Remove implicit GenomicRDD to RDD conversion #1040

Closed

fnothaft force-pushed the genotypes-rdd branch from 1355a6e to bb9acc4 Compare May 26, 2016 21:38

fnothaft force-pushed the genotypes-rdd branch from bb9acc4 to 4c52f92 Compare May 26, 2016 22:42

heuermh reviewed Jun 1, 2016
View reviewed changes

fnothaft added 3 commits June 1, 2016 14:24

Add WAR to broken header reading path in HTSJDK.

2c07382

Upgrading to bdg-formats 0.8.0

525fda8

fnothaft force-pushed the genotypes-rdd branch from 4c52f92 to 525fda8 Compare June 1, 2016 21:24

heuermh merged commit 525fda8 into bigdatagenomics:master Jun 3, 2016

heuermh mentioned this pull request Jun 7, 2016

Release ADAM version 0.20.0 #1048

Closed

61 tasks

fnothaft mentioned this pull request Jul 6, 2016

Explore if SeqDict data can be factored out more aggressively #983

Closed

Conversation

fnothaft commented Apr 24, 2016

Uh oh!

AmplabJenkins commented Apr 24, 2016

Build result: FAILURE

Uh oh!

fnothaft commented Apr 25, 2016

Uh oh!

AmplabJenkins commented Apr 25, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fnothaft commented May 20, 2016

Uh oh!

AmplabJenkins commented May 20, 2016

Build result: FAILURE

Uh oh!

fnothaft commented May 20, 2016

Uh oh!

heuermh commented May 20, 2016

Uh oh!

fnothaft commented May 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented May 20, 2016

Uh oh!

heuermh commented May 20, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

heuermh May 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

heuermh commented May 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fnothaft commented May 25, 2016

Uh oh!

AmplabJenkins commented May 25, 2016

Uh oh!

heuermh commented May 26, 2016

Uh oh!

fnothaft commented May 26, 2016

Uh oh!

AmplabJenkins commented May 26, 2016

Uh oh!

fnothaft commented May 26, 2016

Uh oh!

AmplabJenkins commented May 26, 2016

Uh oh!

heuermh commented May 26, 2016

Uh oh!

jpdna commented May 26, 2016

Uh oh!

erictu commented May 26, 2016

Uh oh!

fnothaft commented May 26, 2016

Uh oh!

fnothaft commented May 26, 2016

Uh oh!

fnothaft commented May 20, 2016 •

edited

Loading

heuermh May 26, 2016 •

edited

Loading

erictu commented May 27, 2016 •

edited

Loading