[FEA] Please consider to support spark.sql.extensions=com.nvidia.spark.rapids.SQLExecPlugin

[wbo4958]

Is your feature request related to a problem? Please describe.
This issue is regarding to #460, when user sets spark.sql.extensitions=com.nvidia.spark.rapids.SQLExecPlugin instead of spark.plugins=com.nvidia.spark.SQLPlugin, It will result Null Pointer exception

20/07/29 15:51:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.19.183.93, executor 0): java.lang.NullPointerException
	at org.apache.spark.sql.rapids.GpuShuffleEnv$.isRapidsShuffleEnabled(GpuShuffleEnv.scala:128)
	at com.nvidia.spark.rapids.GpuPartitioning.sliceInternalGpuOrCpu(GpuPartitioning.scala:99)
	at com.nvidia.spark.rapids.GpuPartitioning.sliceInternalGpuOrCpu$(GpuPartitioning.scala:97)
	at com.nvidia.spark.rapids.GpuRoundRobinPartitioning.sliceInternalGpuOrCpu(GpuRoundRobinPartitioning.scala:34)
	at com.nvidia.spark.rapids.GpuRoundRobinPartitioning.columnarEval(GpuRoundRobinPartitioning.scala:81)
	at com.nvidia.spark.rapids.GpuShuffleExchangeExec$.$anonfun$prepareBatchShuffleDependency$2(GpuShuffleExchangeExec.scala:156)
	at com.nvidia.spark.rapids.GpuShuffleExchangeExec$$anon$1.partNextBatch(GpuShuffleExchangeExec.scala:177)
	at com.nvidia.spark.rapids.GpuShuffleExchangeExec$$anon$1.hasNext(GpuShuffleExchangeExec.scala:188)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:132)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

That's because RapidsExecutorPlugin will not be initialized, So GpuShuffleEnv will also not be initialized. If any code calls the method of GpuShuffleEnv, it may suffer NullPointerException, because the code assumes that the GpuShuffleEnv must have been initialized.

Describe the solution you'd like
Better to support spark.sql.extensitions=com.nvidia.spark.rapids.SQLExecPlugin

[abellina]

Sorry I though this issue was to have a friendlier exception thrown here. I am going to file something different for that, it is a nuisance and different to the main issue discussed here.

[abellina]

@wbo4958 fyi, we are changing this exception to be an IllegalStateException for now.

We need, specifically, to have the correct GPU acquired in the plugin. One approach as of today for you to be able to run is to a) acquire a gpu, and b) initialize GpuShuffleEnv with the mem info required there from the device. This would mean as far as I can tell that spark.sql.extensions would need to be prefixed with one of your own to initialize things.

How do solve for the correct GPU in your case? That seems to be the main issue we have concerns about. Let us know what you think, perhaps there's more we can do to make it easier.

[wbo4958]

Thx @abellina.

[revans2]

@wbo4958 is what @abellina described an acceptable long term solution? If not then we should reopen this. We can probably make it work with lazy initialization, but it is a matter of time and prioritization.