Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Running Scala tests against Spark 3.3 results in lots of Spark logging output #5145

Closed
jlowe opened this issue Apr 5, 2022 · 0 comments · Fixed by #5822
Closed

[BUG] Running Scala tests against Spark 3.3 results in lots of Spark logging output #5145

jlowe opened this issue Apr 5, 2022 · 0 comments · Fixed by #5822
Assignees
Labels
test Only impacts tests

Comments

@jlowe
Copy link
Member

jlowe commented Apr 5, 2022

Describe the bug
Running the Scala tests against Spark 3.3 will result in a lot of logging messages to the console.

Steps/Code to reproduce bug
mvn clean package -Dbuildver=330

When the build gets to the point of running the Scala tests with a Spark session, note how many Spark logging messages appear, e.g.:

[INFO] --- scalatest-maven-plugin:2.0.2:test (test) @ rapids-4-spark-udf_2.12 ---
Discovery starting.
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
22/04/05 15:15:28 INFO SparkContext: Running Spark version 3.3.0-SNAPSHOT
22/04/05 15:15:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/04/05 15:15:28 INFO ResourceUtils: ==============================================================
22/04/05 15:15:28 INFO ResourceUtils: No custom resources configured for spark.driver.
22/04/05 15:15:28 INFO ResourceUtils: ==============================================================
22/04/05 15:15:28 INFO SparkContext: Submitted application: OpcodeSuite
22/04/05 15:15:28 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/04/05 15:15:28 INFO ResourceProfile: Limiting resource is cpu
22/04/05 15:15:28 INFO ResourceProfileManager: Added ResourceProfile id: 0
[...]

Expected behavior
Logging messages should be redirected to a file in surefire-reports rather than sent to the console when running the Scala tests (i.e.: the behavior when running with Spark 3.2.x or earlier)

Environment details (please complete the following information)
Spark 3.3.0

@jlowe jlowe added bug Something isn't working ? - Needs Triage Need team to review and classify labels Apr 5, 2022
@mattahrens mattahrens added test Only impacts tests and removed ? - Needs Triage Need team to review and classify bug Something isn't working labels Apr 5, 2022
@res-life res-life self-assigned this Jun 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Only impacts tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants