Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Benchmark runner fails to produce report if benchmark fails due to an invalid query plan #1270

Closed
andygrove opened this issue Dec 4, 2020 · 2 comments · Fixed by #1299
Closed

[BUG] Benchmark runner fails to produce report if benchmark fails due to an invalid query plan #1270

andygrove opened this issue Dec 4, 2020 · 2 comments · Fixed by #1299
Assignees
@andygrove
Labels
benchmark Benchmarking, benchmarking tools bug Something isn't working
Milestone
Dec 7 - Dec 18

Comments

@andygrove
Copy link
Contributor

Describe the bug
The benchmark runner relies on BenchmarkListener.onFailure to capture error information about failed queries so that errors can be written to the summary output file. It also attempts to convert the query plan to JSON. If the plan is not valid then this code can trigger an exception, causing the report to not be written.

Steps/Code to reproduce bug
This bug was discovered due to #1269

Expected behavior
The report should still be written, but without the query plan if it cannot be converted to JSON.
@andygrove andygrove added bug Something isn't working ? - Needs Triage Need team to review and classify benchmark Benchmarking, benchmarking tools labels Dec 4, 2020
@andygrove andygrove added this to the Nov 23 - Dec 4 milestone Dec 4, 2020
@andygrove andygrove self-assigned this Dec 4, 2020
@andygrove
Copy link
Contributor Author

stack trace:

20/12/04 16:19:48 ERROR ExecutionListenerBus: Listener BenchmarkListener threw an exception
java.lang.IllegalStateException: class org.apache.spark.sql.rapids.GpuSubstring is not expected to be a part of a SortOrder
	at com.nvidia.spark.rapids.GpuOverrides$.canonicalizeToCpuForSortOrder(GpuOverrides.scala:417)
	at com.nvidia.spark.rapids.GpuOverrides$.gpuOrderingSemanticEquals(GpuOverrides.scala:423)
	at com.nvidia.spark.rapids.GpuOverrides$.$anonfun$orderingSatisfies$1(GpuOverrides.scala:428)
	at com.nvidia.spark.rapids.GpuOverrides$.$anonfun$orderingSatisfies$1$adapted(GpuOverrides.scala:428)
	at scala.collection.immutable.Set$Set1.exists(Set.scala:100)
	at com.nvidia.spark.rapids.GpuOverrides$.orderingSatisfies(GpuOverrides.scala:428)
	at com.nvidia.spark.rapids.GpuOverrides$.$anonfun$orderingSatisfies$2(GpuOverrides.scala:444)
	at com.nvidia.spark.rapids.GpuOverrides$.$anonfun$orderingSatisfies$2$adapted(GpuOverrides.scala:443)
	at scala.collection.Iterator.forall(Iterator.scala:953)
	at scala.collection.Iterator.forall$(Iterator.scala:951)
	at scala.collection.AbstractIterator.forall(Iterator.scala:1429)
	at scala.collection.IterableLike.forall(IterableLike.scala:77)
	at scala.collection.IterableLike.forall$(IterableLike.scala:76)
	at scala.collection.AbstractIterable.forall(Iterable.scala:56)
	at com.nvidia.spark.rapids.GpuOverrides$.com$nvidia$spark$rapids$GpuOverrides$$orderingSatisfies(GpuOverrides.scala:443)
	at com.nvidia.spark.rapids.GpuOverrides.$anonfun$ensureOrdering$1(GpuOverrides.scala:2287)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike.map(TraversableLike.scala:238)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
	at scala.collection.immutable.List.map(List.scala:298)
	at com.nvidia.spark.rapids.GpuOverrides.com$nvidia$spark$rapids$GpuOverrides$$ensureOrdering(GpuOverrides.scala:2285)
	at com.nvidia.spark.rapids.GpuOverrides$$anonfun$addSortsIfNeeded$1.applyOrElse(GpuOverrides.scala:2308)
	at com.nvidia.spark.rapids.GpuOverrides$$anonfun$addSortsIfNeeded$1.applyOrElse(GpuOverrides.scala:2306)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:333)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:333)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:330)
	at com.nvidia.spark.rapids.GpuOverrides.addSortsIfNeeded(GpuOverrides.scala:2306)
	at com.nvidia.spark.rapids.GpuOverrides.apply(GpuOverrides.scala:2266)
	at com.nvidia.spark.rapids.GpuOverrides.apply(GpuOverrides.scala:2251)
	at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.$anonfun$apply$1(Columnar.scala:514)
	at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.$anonfun$apply$1$adapted(Columnar.scala:513)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.apply(Columnar.scala:513)
	at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.apply(Columnar.scala:482)
	at org.apache.spark.sql.execution.QueryExecution$.$anonfun$prepareForExecution$1(QueryExecution.scala:316)
	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	at scala.collection.immutable.List.foldLeft(List.scala:89)
	at org.apache.spark.sql.execution.QueryExecution$.prepareForExecution(QueryExecution.scala:316)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executedPlan$1(QueryExecution.scala:107)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:133)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
	at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:133)
	at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:107)
	at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:100)
	at com.nvidia.spark.rapids.tests.common.BenchmarkListener.onFailure(BenchUtils.scala:683)
	at org.apache.spark.sql.util.ExecutionListenerBus.doPostEvent(QueryExecutionListener.scala:151)
	at org.apache.spark.sql.util.ExecutionListenerBus.doPostEvent(QueryExecutionListener.scala:127)
	at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:115)
	at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:99)
	at org.apache.spark.sql.util.ExecutionListenerBus.postToAll(QueryExecutionListener.scala:127)
	at org.apache.spark.sql.util.ExecutionListenerBus.onOtherEvent(QueryExecutionListener.scala:133)
	at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:82)
	at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
	at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
	at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
	at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:115)
	at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:99)
	at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
	at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

This was referenced Dec 7, 2020
Closed
Merged
@andygrove andygrove linked a pull request Dec 7, 2020 that will close this issue
Benchmark runner improved error handling #1299
Merged
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Dec 8, 2020
@sameerz
Copy link
Collaborator

sameerz commented Dec 8, 2020

Closed by #1299

@sameerz sameerz closed this as completed Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygrove andygrove

Labels
benchmark Benchmarking, benchmarking tools bug Something isn't working
Projects
None yet
Milestone
Dec 7 - Dec 18
Development

Successfully merging a pull request may close this issue.

Benchmark runner improved error handling andygrove/spark-rapids
2 participants
@andygrove @sameerz