CDAP-15266 Revive mongodb plugins by vladhlinskiy · Pull Request #14 · vladhlinskiy/database-plugins

vladhlinskiy · 2019-07-23T13:28:30Z

JIRA: https://issues.cask.co/browse/CDAP-15266
WIKI: https://wiki.cask.co/display/CE/MongoDB+database+plugin

In the scope of this PR:

MongoDB Plugin moved from 'hydrator-plugins'
Config properties changed to match current plugin standards
Field-level lineage support added

Plugins tested with current MongoDB release 4.0.10.

…tandards

…ndards

mlozbin-cybervisiontech · 2019-07-24T11:44:20Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/source/MongoDBBatchSource.java

+     */
+    public Schema getSchema() {
+      if (schema == null) {
+        throw new IllegalArgumentException("Schema cannot be null.");


this error type should not be thrown during validation phase. Should be InvalidConfigPropertyException. Also IAE usually thrown on illegal method param, in cases like this it looks more like IllegalStateException

Changed to:

/** * @return the schema of the dataset */ @Nullable public Schema getSchema() { try { return null == schema ? null : Schema.parseJson(schema); } catch (IOException e) { throw new InvalidConfigPropertyException("Invalid schema", e, MongoDBConstants.SCHEMA); } }

mlozbin-cybervisiontech · 2019-07-24T11:45:49Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/source/MongoDBBatchSource.java

+  @Override
+  public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
+    super.configurePipeline(pipelineConfigurer);
+    Schema schema = config.getSchema();


Schema may be null on this stage

Changed to validate the config first:

@Override public void configurePipeline(PipelineConfigurer pipelineConfigurer) { super.configurePipeline(pipelineConfigurer); config.validate(); Schema schema = config.getSchema(); ...

mlozbin-cybervisiontech · 2019-07-24T11:46:51Z

mongodb-plugin/src/main/java/io/cdap/plugin/BSONConverter.java

+    return builder.build();
+  }
+
+  public static void validateSchema(Schema schema) {


this class should not be responsible for validation

Moved to MongoDBSourceConfig#validate.

mlozbin-cybervisiontech · 2019-07-24T11:47:35Z

mongodb-plugin/src/main/java/io/cdap/plugin/BSONConverter.java

+      case BOOLEAN:
+      case STRING:
+        return object;
+      case NULL:


I don't think this case is possible

Removed, thanks.

mlozbin-cybervisiontech · 2019-07-24T11:48:43Z

mongodb-plugin/src/main/java/io/cdap/plugin/MongoDBConfig.java

+  @Description("User to use to connect to the specified database. Required for databases that " +
+    "need authentication. Optional for databases that do not require authentication.")
+  @Nullable
+  public String user;


why no fields support macro?

mlozbin-cybervisiontech · 2019-07-24T11:49:55Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/sink/MongoDBBatchSink.java

+  @Override
+  public void transform(StructuredRecord input, Emitter<KeyValue<NullWritable, BSONWritable>> emitter)
+    throws Exception {
+    BasicDBObjectBuilder bsonBuilder = BasicDBObjectBuilder.start();


this transformation should also be excluded to separate class, usually with Transformer suffix

RecordToBSONWritableTransformer class created.

mlozbin-cybervisiontech · 2019-07-24T11:50:32Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/sink/MongoDBBatchSink.java

+      lineageRecorder.createExternalDataset(context.getInputSchema());
+      List<Schema.Field> fields = context.getInputSchema().getFields();
+      if (fields != null && !fields.isEmpty()) {
+        lineageRecorder.recordWrite("Write", "Wrote to MongoDB collection.",


we could add collection name here

Collection name added.

mlozbin-cybervisiontech · 2019-07-24T14:49:17Z

mongodb-plugin/src/main/java/io/cdap/plugin/MongoDBConfig.java

+      throw new InvalidConfigPropertyException("Host must be specified", MongoDBConstants.HOST);
+    }
+    if (!containsMacro(MongoDBConstants.PORT)) {
+      if (null == port) {


null should be on the right side

mlozbin-cybervisiontech · 2019-07-24T14:50:14Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/sink/MongoDBBatchSink.java

+  @Override
+  public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
+    super.configurePipeline(pipelineConfigurer);
+    config.validate();


should we have input schema validation?

MongoDBBatchSink#validateInputSchema method created to check for supported types:

private void validateInputSchema(Schema inputSchema) { List<Schema.Field> fields = inputSchema.getFields(); if (fields == null || fields.isEmpty()) { throw new InvalidStageException("Input schema should contain fields"); } for (Schema.Field field : fields) { Schema.Type fieldType = field.getSchema().isNullable() ? field.getSchema().getNonNullable().getType() : field.getSchema().getType(); if (!SUPPORTED_FIELD_TYPES.contains(fieldType)) { String supportedTypes = SUPPORTED_FIELD_TYPES.stream() .map(Enum::name) .map(String::toLowerCase) .collect(Collectors.joining(", ")); String errorMessage = String.format("Field '%s' is of unsupported type '%s'. Supported types are: %s.", field.getName(), fieldType, supportedTypes); throw new InvalidStageException(errorMessage); } } }

mlozbin-cybervisiontech · 2019-07-24T14:51:12Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/source/MongoDBBatchSource.java

+      super.validate();
+      if (!containsMacro(MongoDBConstants.SCHEMA)) {
+        Schema parsedSchema = getSchema();
+        if (null == parsedSchema) {


null should be on the right side

mlozbin-cybervisiontech · 2019-07-24T14:51:50Z

mongodb-plugin/src/main/java/io/cdap/plugin/batch/source/BSONObjectToRecordTransformer.java

  private final Schema schema;

-  public BSONConverter(Schema schema) throws IOException {
+  public BSONObjectToRecordTransformer(Schema schema) {


please add unit tests for both transformers

Added:

RecordToBSONWritableTransformerTest

BSONObjectToRecordTransformerTest

mlozbin-cybervisiontech · 2019-07-24T14:53:19Z

mongodb-plugin/src/main/java/io/cdap/plugin/MongoDBConfig.java

@@ -18,9 +18,12 @@

 import com.google.common.base.Strings;


please add unit tests for both configs

Added:

MongoDBConfigTest

MongoDBSourceConfigTest

vhlinskyi added 4 commits July 23, 2019 16:20

CDAP-15266 MongoDB Plugin moved from 'hydrator-plugins'

56d1279

CDAP-15266 Changed Source config properties to match current plugin s…

6ffec59

…tandards

CDAP-15266 Changed Sink config properties to match current plugin sta…

9fadf5a

…ndards

CDAP-15266 Add field lineage support

bd8e229

mlozbin-cybervisiontech suggested changes Jul 24, 2019

View reviewed changes

vhlinskyi added 2 commits July 24, 2019 16:50

CDAP-15266 Address comments

f1148e5

CDAP-15266 Changed to use 'json-editor' widget

17afb4e

mlozbin-cybervisiontech suggested changes Jul 24, 2019

View reviewed changes

CDAP-15266 Address comments

59dcaab

mlozbin-cybervisiontech approved these changes Jul 25, 2019

View reviewed changes

CDAP-15266 Update MongoDB Connector for Hadoop version

ccf09c9

Conversation

vladhlinskiy commented Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vladhlinskiy Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vladhlinskiy commented Jul 23, 2019 •

edited

Loading

vladhlinskiy Jul 24, 2019 •

edited

Loading