DRILL-7903: Update mongo driver from 3.12 to 4.2 #2201

vvysotskyi · 2021-04-17T15:08:45Z

DRILL-7903: Update mongo driver from 3.12 to 4.2

Description

Replaced old mongo-java-driver with the latest version of mongodb-driver-sync.
Replaced de.flapdoodle.embed.mongo usage with testcontainers to avoid dependencies on old mongo-java-driver.
Removed dependency on mongo from the java-exec module (and moved Bson reader).

Documentation

Yes.

Testing

All unit tests pass (now unit tests use docker containers with real mongo).

luocooong

@vvysotskyi Nice work. Thanks for your PR.

luocooong · 2021-04-18T02:33:10Z

contrib/storage-mongo/pom.xml

@@ -64,9 +64,9 @@
      <scope>test</scope>
    </dependency>
    <dependency>
-      <groupId>de.flapdoodle.embed</groupId>


The de.flapdoodle.embed is a good embedded mongo framework. It is a pity that it cannot support our test now. Can you explain that why to using the based on docker framework to instead this?

de.flapdoodle.embed uses mongo-java-driver and is incompatible with mongodb-driver-sync, so we have to move out to another solution for testing.

luocooong · 2021-04-18T02:47:29Z

contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoGroupScan.java

-      return list1.size() - list2.size();
-    }
-  };
+  private static final Comparator<List<MongoSubScanSpec>> LIST_SIZE_COMPARATOR = Comparator.comparingInt(List::size);


More simple (with Java Lambda), but not easy to read and debug. (Not a problem)

luocooong · 2021-04-18T03:14:17Z

contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoGroupScan.java

@@ -530,8 +503,7 @@ public ScanStats getScanStats() {
      if (recordCount != 0) {
        //toJson should use client's codec, otherwise toJson could fail on
        // some types not known to DocumentCodec, e.g. DBRef.
-        final DocumentCodec codec =
-            new DocumentCodec(client.getMongoClientOptions().getCodecRegistry(), new BsonTypeClassMap());
+        DocumentCodec codec = new DocumentCodec(db.getCodecRegistry(), new BsonTypeClassMap());


Is this change due to a version requirement (driver 4.2) or exist an issue before?

Yes, API was changed, there is no method for MongoClientOptions anymore.

luocooong · 2021-04-18T03:41:05Z

contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoGroupScan.java

    MongoDatabase database = storagePlugin.getClient().getDatabase(CONFIG);
    //Identify the primary shard of the queried database.
    MongoCollection<Document> collection = database.getCollection(DATABASES);
    Bson filter = new Document(ID, this.scanSpec.getDbName());
    Bson projection = new Document(PRIMARY, select);
-    Document document = collection.find(filter).projection(projection).first();
-    Preconditions.checkNotNull(document);
+    Document document = Objects.requireNonNull(collection.find(filter).projection(projection).first());


An user had reported a problem on our Slack channel (some days ago):

I am trying to query a Sharded cluster of MongoDB using embedded mode and distributed mode. But both is giving an error of:
query: select * from mongo.db.collection limit 1;
warning UserRemoteException : SYSTEM ERROR: ClassCastException: org.bson.types.ObjectId cannot be cast to java.lang.String
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: ClassCastException: org.bson.types.ObjectId cannot be cast to java.lang.String
Please, refer to logs for more information.
[Error Id: 43733103-4cac-4aef-a09f-275df4b4ee4b on ubuntu:31010]
store.mongo.all_text_mode : true
store.mongo.bson.record.reader: false
store.mongo.read_numbers_as_double: true
It throws an error only when collection is sharded, on non-sharded collection, it is working fine.

Could you please check that the failed not an issues? thanks

Drill 1.18
MongoDB 4.4 Sharded (Not a Replica Sets)

I have reproduced it locally and fixed it.

luocooong · 2021-04-18T03:41:19Z

contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuite.java

-          .version(Version.Main.V3_4)
-          .net(new Net(LOCALHOST, MONGOS_PORT, Network.localhostIsIPv6()))
-          .cmdOptions(cmdOptions).build();
+      try {


There is a new practice to apply the same opeation (In my production environment):

..... rs.initiate(cfg); while(db.isMaster().ismaster == false) { } db = db.getSiblingDB('admin'); db.createUser({user:"admin", pwd:"0000", roles:["root"]}); db.createUser({user:"users", pwd:"0000", roles:["userAdminAnyDatabase"]}); .....

mongo --quiet init.js

Sorry, I didn't understand what you proposed and how to use it...
This code initializes replica set as it is recommended in official mongo instructions: https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
And after that, waits until it is initialized.

Code part related to creating users defined in the createMongoUser() method and it is quite flexible.

It's not a problem. It's only a new practice to initial cluster that using the script to execute once instead of calling the execInContainer many times.

vvysotskyi

@luocooong, thanks for the review, I have addressed your comments, could you please look again?

vvysotskyi · 2021-04-18T08:51:23Z

contrib/storage-mongo/pom.xml

@@ -64,9 +64,9 @@
      <scope>test</scope>
    </dependency>
    <dependency>
-      <groupId>de.flapdoodle.embed</groupId>


de.flapdoodle.embed uses mongo-java-driver and is incompatible with mongodb-driver-sync, so we have to move out to another solution for testing.

vvysotskyi · 2021-04-18T10:22:32Z

contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoGroupScan.java

@@ -530,8 +503,7 @@ public ScanStats getScanStats() {
      if (recordCount != 0) {
        //toJson should use client's codec, otherwise toJson could fail on
        // some types not known to DocumentCodec, e.g. DBRef.
-        final DocumentCodec codec =
-            new DocumentCodec(client.getMongoClientOptions().getCodecRegistry(), new BsonTypeClassMap());
+        DocumentCodec codec = new DocumentCodec(db.getCodecRegistry(), new BsonTypeClassMap());


Yes, API was changed, there is no method for MongoClientOptions anymore.

vvysotskyi · 2021-04-20T15:03:41Z

contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestSuite.java

-          .version(Version.Main.V3_4)
-          .net(new Net(LOCALHOST, MONGOS_PORT, Network.localhostIsIPv6()))
-          .cmdOptions(cmdOptions).build();
+      try {


Sorry, I didn't understand what you proposed and how to use it...
This code initializes replica set as it is recommended in official mongo instructions: https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
And after that, waits until it is initialized.

Code part related to creating users defined in the createMongoUser() method and it is quite flexible.

vvysotskyi · 2021-04-27T20:29:43Z

contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoGroupScan.java

    MongoDatabase database = storagePlugin.getClient().getDatabase(CONFIG);
    //Identify the primary shard of the queried database.
    MongoCollection<Document> collection = database.getCollection(DATABASES);
    Bson filter = new Document(ID, this.scanSpec.getDbName());
    Bson projection = new Document(PRIMARY, select);
-    Document document = collection.find(filter).projection(projection).first();
-    Preconditions.checkNotNull(document);
+    Document document = Objects.requireNonNull(collection.find(filter).projection(projection).first());


I have reproduced it locally and fixed it.

luocooong

@vvysotskyi +1
Thanks for the revision.

luocooong assigned vvysotskyi Apr 17, 2021

luocooong added minor-update code-cleanup labels Apr 17, 2021

luocooong self-requested a review April 17, 2021 23:24

luocooong reviewed Apr 18, 2021

View reviewed changes

vvysotskyi added 2 commits April 27, 2021 22:08

DRILL-7903: Update mongo driver from 3.12 to 4.2

0a2b002

DRILL-7903: Fix ClassCastException when querying sharded cluster

9068397

vvysotskyi force-pushed the DRILL-7903 branch from 79b7182 to 9068397 Compare April 27, 2021 19:17

vvysotskyi added updates and removed minor-update labels Apr 27, 2021

vvysotskyi commented Apr 27, 2021

View reviewed changes

luocooong approved these changes Apr 28, 2021

View reviewed changes

luocooong merged commit ed0c304 into apache:master Apr 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRILL-7903: Update mongo driver from 3.12 to 4.2 #2201

DRILL-7903: Update mongo driver from 3.12 to 4.2 #2201

vvysotskyi commented Apr 17, 2021

luocooong left a comment

luocooong Apr 18, 2021

vvysotskyi Apr 18, 2021

luocooong Apr 18, 2021

luocooong Apr 18, 2021

vvysotskyi Apr 18, 2021

luocooong Apr 18, 2021

vvysotskyi Apr 27, 2021

luocooong Apr 18, 2021

vvysotskyi Apr 20, 2021

luocooong Apr 28, 2021

vvysotskyi left a comment

vvysotskyi Apr 18, 2021

vvysotskyi Apr 18, 2021

vvysotskyi Apr 20, 2021

vvysotskyi Apr 27, 2021

luocooong left a comment

DRILL-7903: Update mongo driver from 3.12 to 4.2 #2201

DRILL-7903: Update mongo driver from 3.12 to 4.2 #2201

Conversation

vvysotskyi commented Apr 17, 2021

DRILL-7903: Update mongo driver from 3.12 to 4.2

Description

Documentation

Testing

luocooong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvysotskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luocooong left a comment

Choose a reason for hiding this comment