Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] NPE when removing shuffle #556

Closed
rongou opened this issue Aug 13, 2020 · 1 comment · Fixed by #562
Closed

[BUG] NPE when removing shuffle #556

rongou opened this issue Aug 13, 2020 · 1 comment · Fixed by #562
Assignees
@abellina
Labels
bug Something isn't working P0 Must have for release
Milestone
Aug 3 - Aug 14

Comments

@rongou
Copy link
Collaborator

rongou commented Aug 13, 2020

Describe the bug
When running a query with shuffle manager enabled, getting an NPE after the first run.

Steps/Code to reproduce bug
Enable shuffle manager by setting

--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark300.RapidsShuffleManager

Expected behavior
The query still finishes, but probably shouldn't throw an NPE.

Environment details (please complete the following information)

  • Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
    Standalone
  • Spark configuration settings related to the issue
--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark300.RapidsShuffleManager

Additional context
Full stack trace:

*** RUNNING TPCx-BB QUERY 16
*** Start cold run 0:
*** Cold run 0 took 8955 msec.                                                  
*** Start hot run 0:
20/08/13 11:11:41 ERROR BlockManagerSlaveEndpoint: Error in removing shuffle 0
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.getCatalog(GpuShuffleEnv.scala:138)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.catalog$lzycompute(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.org$apache$spark$sql$rapids$RapidsShuffleInternalManagerBase$$catalog(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterGpuShuffle(RapidsShuffleInternalManager.scala:344)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterShuffle(RapidsShuffleInternalManager.scala:351)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerSlaveEndpoint.scala:61)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint.$anonfun$doAsync$1(BlockManagerSlaveEndpoint.scala:86)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/08/13 11:11:41 WARN BlockManagerMaster: Failed to remove shuffle 0 - null
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.getCatalog(GpuShuffleEnv.scala:138)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.catalog$lzycompute(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.org$apache$spark$sql$rapids$RapidsShuffleInternalManagerBase$$catalog(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterGpuShuffle(RapidsShuffleInternalManager.scala:344)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterShuffle(RapidsShuffleInternalManager.scala:351)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerSlaveEndpoint.scala:61)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint.$anonfun$doAsync$1(BlockManagerSlaveEndpoint.scala:86)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/08/13 11:11:41 ERROR BlockManagerSlaveEndpoint: Error in removing shuffle 2
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.getCatalog(GpuShuffleEnv.scala:138)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.catalog$lzycompute(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.org$apache$spark$sql$rapids$RapidsShuffleInternalManagerBase$$catalog(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterGpuShuffle(RapidsShuffleInternalManager.scala:344)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterShuffle(RapidsShuffleInternalManager.scala:351)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerSlaveEndpoint.scala:61)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint.$anonfun$doAsync$1(BlockManagerSlaveEndpoint.scala:86)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/08/13 11:11:41 WARN BlockManagerMaster: Failed to remove shuffle 2 - null
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.getCatalog(GpuShuffleEnv.scala:138)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.catalog$lzycompute(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.org$apache$spark$sql$rapids$RapidsShuffleInternalManagerBase$$catalog(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterGpuShuffle(RapidsShuffleInternalManager.scala:344)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterShuffle(RapidsShuffleInternalManager.scala:351)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerSlaveEndpoint.scala:61)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint.$anonfun$doAsync$1(BlockManagerSlaveEndpoint.scala:86)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/08/13 11:11:41 ERROR BlockManagerSlaveEndpoint: Error in removing shuffle 1
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.getCatalog(GpuShuffleEnv.scala:138)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.catalog$lzycompute(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.org$apache$spark$sql$rapids$RapidsShuffleInternalManagerBase$$catalog(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterGpuShuffle(RapidsShuffleInternalManager.scala:344)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterShuffle(RapidsShuffleInternalManager.scala:351)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerSlaveEndpoint.scala:61)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint.$anonfun$doAsync$1(BlockManagerSlaveEndpoint.scala:86)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/08/13 11:11:41 WARN BlockManagerMaster: Failed to remove shuffle 1 - null
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.getCatalog(GpuShuffleEnv.scala:138)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.catalog$lzycompute(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.org$apache$spark$sql$rapids$RapidsShuffleInternalManagerBase$$catalog(RapidsShuffleInternalManager.scala:205)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterGpuShuffle(RapidsShuffleInternalManager.scala:344)
	at org.apache.spark.sql.rapids.RapidsShuffleInternalManagerBase.unregisterShuffle(RapidsShuffleInternalManager.scala:351)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerSlaveEndpoint.scala:61)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.spark.storage.BlockManagerSlaveEndpoint.$anonfun$doAsync$1(BlockManagerSlaveEndpoint.scala:86)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
*** Hot run 0 took 5063 msec.

@jlowe
@rongou rongou added bug Something isn't working ? - Needs Triage Need team to review and classify labels Aug 13, 2020
@jlowe jlowe self-assigned this Aug 13, 2020
@jlowe jlowe added P0 Must have for release and removed ? - Needs Triage Need team to review and classify labels Aug 13, 2020
@jlowe jlowe added this to the Aug 3 - Aug 14 milestone Aug 13, 2020
@abellina abellina assigned abellina and unassigned jlowe Aug 14, 2020
@abellina abellina mentioned this issue Aug 14, 2020
Merged
@abellina
Copy link
Collaborator

@rongou thanks for reporting. I was able to reproduce and fix in #562

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abellina abellina

Labels
bug Something isn't working P0 Must have for release
Projects
None yet
Milestone
Aug 3 - Aug 14
Development

Successfully merging a pull request may close this issue.

getCatalog can be called from the driver, and can return null abellina/spark-rapids
3 participants
@rongou @jlowe @abellina