Creating relese with shaded dependencies

[kristoffSC]

Hi,
great library :)

Would it be possible to release a new version with all dependencies shaded?
Especially those:

• org.apache.calcite:calcite-core
• org.apache.calcite:calcite-druid
• org.apache.calcite:calcite-linq4j
• org.apache.calcite:calcite-avatica
• org.apache.parquet", "parquet-column"
• org.apache.parquet", "parquet-hadoop-bundle"
• org.apache.hive", "hive-exec"
• org.apache.hadoop", "hadoop-common"

I would like to use this library for creating tests for one of the Apache Flink connectors but I have very nasty conflicts on those libraries, since Flink also have those but with different versions that are not binary compatible with eachother.

Having this library with shaded dependencies would solve the problem

[patduin]

Hi and thanks,

Probably easier to exclude those from your pom.
I think for this library we should make a new release where we make all those deps set to provided.
What do you think @abhimanyugupta07 ?

[kristoffSC]

Hi @patduin
thanks for replay.

we make all those deps set to provided

This will not help in my case at all. There is a binary incompatibility between dependencies versions that you are using and that Flink [1] does.

Probably easier to exclude those from your pom.

Actually that was very hard, what I ended up with is using pure hive-exec and hive-metastore and reusing high level classes from your library. In fact i had to exclude a bunch of stuff from those two libraries, at the end I have:

"org.apache.hive" % "hive-metastore" % "3.1.2" % "test" excludeAll(
        ExclusionRule("org.apache.avro", "avro"),
        ExclusionRule("org.slf4j", "slf4j-log4j12"),
        ExclusionRule("org.pentaho"),
        ExclusionRule("org.apache.hbase"),
        ExclusionRule("org.apache.hbase"),
        ExclusionRule("co.cask.tephra"),
        ExclusionRule("com.google.code.findbugs", "jsr305"),
        ExclusionRule("org.eclipse.jetty.aggregate", "module: 'jetty-all"),
        ExclusionRule("org.eclipse.jetty.orbit", "javax.servlet"),
        ExclusionRule("org.apache.parquet", "parquet-hadoop-bundle"),
        ExclusionRule("com.tdunning", "json"),
        ExclusionRule("javax.transaction", "transaction-api"),
        ExclusionRule("'com.zaxxer", "HikariCP"),
      ),
      // Exclusions due to conflicts with Flink's libraries from table planer, hive, calcite etc.
      "org.apache.hive" % "hive-exec" % "3.1.2" % "test" classifier "core" excludeAll(
        ExclusionRule("'org.apache.avro", "avro"),
        ExclusionRule("org.slf4j", "slf4j-log4j12"),
        ExclusionRule("org.pentaho"),
        ExclusionRule("com.google.code.findbugs", "jsr305"),
        ExclusionRule("org.apache.calcite.avatica"),
        ExclusionRule("org.apache.calcite"),
        ExclusionRule("org.apache.hive", "hive-llap-tez"),
        ExclusionRule("org.apache.logging.log4j"),
        ExclusionRule("com.google.protobuf", "protobuf-java"),
      ),

When I tried same exclusions on your library I still had issues with binary incompatibility.
From my perspective would be the best if library would be self contained and have its all dependencies shared but I can understand if you would not want to do it.

Maybe having two artifacts? Shaded and not?

[1] https://nightlies.apache.org/flink/flink-docs-release-1.16/

[patduin]

Thanks for the detailed reply! I see what you mean. We'll have to look into this a bit.