DRILL-7901: Bump junit from 4.12 to 4.13.2 #2200

cgivre · 2021-04-14T16:23:33Z

DRILL-7901: Bump junit from 4.12 to 4.13.2

Description

Release notes

JUnit 4.13.1

Please refer to the release notes for details.

JUnit 4.13

Please refer to the release notes for details.

JUnit 4.13 RC 2

Please refer to the release notes for details.

JUnit 4.13 RC 1

Please refer to the release notes for details.

JUnit 4.13 Beta 3

Please refer to the release notes for details.

JUnit 4.13 Beta 2

Please refer to the release notes for details.

JUnit 4.13 Beta 1

Please refer to the release notes for details.

Documentation

N/A

Testing

N/A

UPDATE: This PR involves rewriting the unit tests for the Kafka plugin. All seem to use the deprecated PlanTestBase class which has been deprecated and also causes an issue with the newer version of junit.

vvysotskyi · 2021-04-14T18:31:45Z

I recall some tests were hanging when updating the JUnit version to 4.13.1. Have you tried running all unit tests locally since CI is not working?

cgivre · 2021-04-14T19:31:54Z

I recall some tests were hanging when updating the JUnit version to 4.13.1. Have you tried running all unit tests locally since CI is not working?

I don't think this is related, but I'm getting out of memory errors on my local machine. Is there anything I can do to my environment to fix this?

Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure allocating buffer.
Caused by: java.lang.Error: failed to allocate 16777216 byte(s) of direct memory (used: 2617245696, max: 2621440000)

[INFO] Running org.apache.drill.TestDirScanToValuesConversion
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.417 s - in org.apache.drill.TestDirScanToValuesConversion
[WARNING] Tests run: 66, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 349.714 s - in org.apache.drill.exec.sql.TestMetastoreCommands
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   TestBugFixes.testDRILL1337_LocalRightFilterLeftOutJoin » UserRemote RESOURCE E...
[ERROR]   TestBugFixes.testDRILL4884:231->BaseTestQuery.testRunAndReturn:346 » Rpc org.a...
[ERROR]   TestExampleQueries.testParquetComplex » UserRemote RESOURCE ERROR: One or more...
[ERROR]   TestExampleQueries.testWindowFunAndStarCol:1109->BaseTestQuery.testRunAndReturn:346 » Rpc
[ERROR]   TestTpchDistributed.tpch01 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch03 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch04 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch05 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch07 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch08 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch09 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch10 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch11 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch12 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch13 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch15 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch16 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch17 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch18 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch19_1 » UserRemote RESOURCE ERROR: One or more nodes ra...
[ERROR]   TestTpchDistributed.tpch20 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestTpchDistributed.tpch21 » UserRemote RESOURCE ERROR: One or more nodes ran ...
[ERROR]   TestUnionAll.testDrill4147_2:1075->BaseTestQuery.testRunAndReturn:346 » Rpc or...
[ERROR]   TestUnionDistinct.testUnionWithManyColumns » UserRemote RESOURCE ERROR: One or...
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testEXTERNAL_SORT
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testGROUP_BY
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testHashJoin
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testJDKHugeStringConstantCompilation
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR]   TestLargeFileCompilation.testMergeJoin » UserRemote SYSTEM ERROR: OutOfMemoryE...
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testPARQUET_WRITER
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testProject
[ERROR]   Run 1: TestLargeFileCompilation.testProject » User CONNECTION ERROR: Connection /127....
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testTEXT_WRITER
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR] org.apache.drill.exec.compile.TestLargeFileCompilation.testTOP_N_SORT
[ERROR]   Run 1: TestLargeFileCompilation.setJDK:174 » IllegalState org.apache.drill.common.exc...
[ERROR]   Run 2: TestLargeFileCompilation.resetJDK:179 » IllegalState org.apache.drill.common.e...
[INFO]
[ERROR]   TestAggregateFunctions.testDRILLNestedGBWithSubsetKeys:223 »  org.apache.drill...
[ERROR]   TestAggregateFunctions.testDrill2170:177 »  org.apache.drill.common.exceptions...
[ERROR]   TestNewSimpleRepeatedFunctions.testRepeatedCountVarBinary:259 »  org.apache.dr...
[ERROR]   TestNewSimpleRepeatedFunctions.testRepeatedCountVarChar:150 »  org.apache.dril...
[ERROR]   TestNewSimpleRepeatedFunctions.testRepeatedCountVarDecimal:249 »  org.apache.d...
[ERROR]   TestDummyWriter>SubOperatorTest.classSetup:34 » OutOfMemory Failure allocating...
[ERROR]   TestFillEmpties>SubOperatorTest.classSetup:34 » OutOfMemory Failure allocating...
[ERROR] org.apache.drill.exec.physical.rowSet.TestOffsetVectorWriter.null
[ERROR]   Run 1: TestOffsetVectorWriter.setup:62 » OutOfMemory Failure allocating buffer.
[ERROR]   Run 2: TestOffsetVectorWriter>SubOperatorTest.classTeardown:39 » IllegalState Allocat...
[INFO]
[ERROR]   TestScalarAccessors>SubOperatorTest.classSetup:34 » OutOfMemory Failure alloca...
[ERROR]   TestAnalyze.join » UserRemote RESOURCE ERROR: One or more nodes ran out of mem...
[ERROR]   TestAnalyze.testAnalyzeSupportedFormats » UserRemote RESOURCE ERROR: One or mo...
[ERROR]   TestAnalyze.testUseStatistics » UserRemote RESOURCE ERROR: One or more nodes r...
[ERROR]   TestMockPlugin.testSizeLimit » UserRemote EXECUTION_ERROR ERROR: Failure alloc...
[ERROR] org.apache.drill.exec.work.filter.BloomFilterTest.null
[ERROR]   Run 1: BloomFilterTest>SubOperatorTest.classSetup:34 » OutOfMemory Failure allocating...
[ERROR]   Run 2: BloomFilterTest>SubOperatorTest.classTeardown:39 » IllegalState Allocator[ROOT...
[INFO]
[ERROR]   TestFillEmpties>SubOperatorTest.classSetup:34 » OutOfMemory Failure allocating...
[ERROR] org.apache.drill.vector.TestToNullable.null
[ERROR]   Run 1: TestToNullable>SubOperatorTest.classSetup:34 » OutOfMemory Failure allocating ...
[ERROR]   Run 2: TestToNullable>SubOperatorTest.classTeardown:39 » IllegalState Allocator[ROOT]...
[INFO]
[INFO]
[ERROR] Tests run: 4921, Failures: 0, Errors: 49, Skipped: 149

vdiravka · 2021-04-16T13:55:16Z

I had Junit 5 migration code for Drill in my shelf. Let me check it how tests will pass on it.

cgivre · 2021-04-28T14:11:36Z

I had Junit 5 migration code for Drill in my shelf. Let me check it how tests will pass on it.

Hi @vdiravka
Would you recommend using Junit 5 rather than 4.13?

laurentgo · 2021-05-04T04:29:14Z

If the change is ready, I would recommend merging it asap. Moving to JUnit5 is likely to involve much more code change, and why beneficial, I'm not sure it warrants delaying the dependency update.

luocooong · 2021-05-04T05:35:49Z

@cgivre Would you please rebase on the master? and then re-run the CI on the basis of the #2203.
Let us merge this PR once the CI successfully. Create a new PR for the Bump the version to JUnit 5?

martin-g · 2021-05-13T06:42:25Z

I don't think this is related, but I'm getting out of memory errors on my local machine. Is there anything I can do to my environment to fix this?

@cgivre You can overcome this by increasing the value of directMemoryMb system property. This is what I did with #2219

martin-g · 2021-05-13T06:44:54Z

Would you recommend using Junit 5 rather than 4.13?

I'd recommend JUnit 5 too!
The initial upgrade would be very easy by using JUnit Vintage Engine. It is fully backwards compatible with JUnit 4.
Later you could migrate the tests one by one to JUnit 5 Jupiter Engine.

martin-g · 2021-05-13T06:46:32Z

But if you decide to stay with 4.x for some more then maybe you should upgrade to 4.13.2 (https://search.maven.org/artifact/junit/junit/4.13.2/jar). It has been released 3 months ago.

cgivre · 2021-05-16T14:36:44Z

I'm a little stuck on this PR. If I run the unit tests in KafkaQueriesTest individually on my local machine or from the CLI they all work. However, if I run the entire test suite in my IDE, either the testPartitionMaxOff or testPartitiionMinOff fails with an IllegalThreadException.

Is this my environment, or does anyone have any suggestions as to how I can get this to work? I'm not sure if this is a blocking issue or not.

cgivre · 2021-05-16T21:34:46Z

@vdiravka Could you do a quick review of this?
Thx!

contrib/storage-kafka/pom.xml

vvysotskyi · 2021-05-17T05:37:57Z

...b/storage-kafka/src/test/java/org/apache/drill/exec/store/kafka/KafkaFilterPushdownTest.java

  }
-}
+}


Please revert formatting changes here and in other places. It is fine to have 4 spaces for the case when moving builder methods or arguments on the new line and having an empty line at the end of the class.

@vvysotskyi
I put the version of this file from master back in this PR, so this should be what was originally there. I didn't modify this file at all, so I'm not sure why the spacing got all messed up in the first place. I hope that's ok.

vvysotskyi

+1

* Initial Commit * Updated Kafka Library * Still not working... * WIP * Kafka tests working * Bump version to 4.13.2 * Added Fix Order * Removed unnecessary dependencies * Fixed spacing on KafkaFilterPushdownTest

cgivre added code-cleanup security labels Apr 14, 2021

cgivre self-assigned this Apr 14, 2021

cgivre force-pushed the Update_junit branch from bcf052c to 27f878b Compare April 26, 2021 03:20

cgivre force-pushed the Update_junit branch 2 times, most recently from 6fa7c3a to 3f7fc3b Compare May 7, 2021 15:42

cgivre force-pushed the Update_junit branch from 3f7fc3b to 85ec98f Compare May 16, 2021 13:00

cgivre requested a review from vdiravka May 16, 2021 21:43

vvysotskyi requested changes May 17, 2021

View reviewed changes

cgivre added 9 commits May 19, 2021 09:17

Initial Commit

ced9b81

Updated Kafka Library

1745304

Still not working...

e40380a

WIP

16cd11b

Kafka tests working

090edec

Bump version to 4.13.2

463c6b4

Added Fix Order

ee6bdff

Removed unnecessary dependencies

fa84d42

Fixed spacing on KafkaFilterPushdownTest

9478f63

cgivre force-pushed the Update_junit branch from 5c0ce5d to 9478f63 Compare May 19, 2021 13:18

vvysotskyi approved these changes May 19, 2021

View reviewed changes

cgivre changed the title ~~DRILL-7901: Bump junit from 4.12 to 4.13.1~~ DRILL-7901: Bump junit from 4.12 to 4.13.2 May 19, 2021

cgivre merged commit 3bbfb7e into apache:master May 19, 2021

cgivre deleted the Update_junit branch May 19, 2021 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRILL-7901: Bump junit from 4.12 to 4.13.2 #2200

DRILL-7901: Bump junit from 4.12 to 4.13.2 #2200

cgivre commented Apr 14, 2021 •

edited

Loading

vvysotskyi commented Apr 14, 2021

cgivre commented Apr 14, 2021

vdiravka commented Apr 16, 2021

cgivre commented Apr 28, 2021

laurentgo commented May 4, 2021

luocooong commented May 4, 2021

martin-g commented May 13, 2021

martin-g commented May 13, 2021

martin-g commented May 13, 2021

cgivre commented May 16, 2021

cgivre commented May 16, 2021

vvysotskyi May 17, 2021

cgivre May 19, 2021

vvysotskyi left a comment

DRILL-7901: Bump junit from 4.12 to 4.13.2 #2200

DRILL-7901: Bump junit from 4.12 to 4.13.2 #2200

Conversation

cgivre commented Apr 14, 2021 • edited Loading

DRILL-7901: Bump junit from 4.12 to 4.13.2

Description

JUnit 4.13.1

JUnit 4.13

JUnit 4.13 RC 2

JUnit 4.13 RC 1

JUnit 4.13 Beta 3

JUnit 4.13 Beta 2

JUnit 4.13 Beta 1

Documentation

Testing

vvysotskyi commented Apr 14, 2021

cgivre commented Apr 14, 2021

vdiravka commented Apr 16, 2021

cgivre commented Apr 28, 2021

laurentgo commented May 4, 2021

luocooong commented May 4, 2021

martin-g commented May 13, 2021

martin-g commented May 13, 2021

martin-g commented May 13, 2021

cgivre commented May 16, 2021

cgivre commented May 16, 2021

vvysotskyi May 17, 2021

Choose a reason for hiding this comment

cgivre May 19, 2021

Choose a reason for hiding this comment

vvysotskyi left a comment

Choose a reason for hiding this comment

cgivre commented Apr 14, 2021 •

edited

Loading