Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Addressing the Named Parameter change in Spark 4.0.0 [databricks] #10992

Merged
merged 1 commit into from
Jun 8, 2024
Merged

Addressing the Named Parameter change in Spark 4.0.0 [databricks] #10992

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions tests/src/test/spark330/scala/org/apache/spark/sql/rapids/GpuInSubqueryExecSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,24 +65,27 @@ class GpuInSubqueryExecSuite extends SparkQueryCompareTestSuite {

private def buildCpuInSubqueryPlan(
spark: SparkSession,
shouldBroadcast: Boolean): SparkPlan = {
shouldBroadcastOrDpp: Boolean): SparkPlan = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference is to minimize the change. I would leave the name as is, then we could have the whole patch a one-line change

val df1ReadExec = readToPhysicalPlan(nullableStringsIntsDf(spark))
val df2ReadExec = readToPhysicalPlan(subqueryTable(spark))
val inSubquery = InSubqueryExec(
df1ReadExec.output.head,
SubqueryExec("sbe",
ProjectExec(Seq(df2ReadExec.output.head), df2ReadExec)),
ExprId(7),
shouldBroadcast=shouldBroadcast)
shouldBroadcastOrDpp)
FilterExec(DynamicPruningExpression(inSubquery), df1ReadExec)
}

for (shouldBroadcast <- Seq(false, true)) {
test(s"InSubqueryExec shouldBroadcast=$shouldBroadcast") {
/**
* The named parameter shouldBroadcast was renamed to isDynamicPruning in Spark 4.0.0+
*/
for (shouldBroadcastOrDpp <- Seq(false, true)) {
test(s"InSubqueryExec shouldBroadcastOrDpp=$shouldBroadcastOrDpp") {
val gpuResults = withGpuSparkSession({ spark =>
val overrides = new GpuOverrides()
val transitionOverrides = new GpuTransitionOverrides()
val cpuPlan = buildCpuInSubqueryPlan(spark, shouldBroadcast)
val cpuPlan = buildCpuInSubqueryPlan(spark, shouldBroadcastOrDpp)
val gpuPlan = transitionOverrides(overrides(cpuPlan))
gpuPlan.execute().collect()
})
Expand Down