Fix bug where AQE does not respect entirePlanWillNotWork

[andygrove]

Closes #6230

While testing some changes related to Delta Lake fallback, I discovered that entirePlanWillNotWork is not always respected by AQE because we weren't tagging the underlying plan, so during the planning for individual query stages, we could decide to run on GPU, and this can lead to invalid plans with mixed CPU/GPU usage that then fail to run.

For the example that I hit, the query failed with the following error due to an invalid transition being inserted. See the issue for more details.

java.lang.AssertionError: assertion failed
  at scala.Predef$.assert(Predef.scala:208)
  at org.apache.spark.sql.execution.ColumnarToRowExec.<init>(Columnar.scala:69)

This issue can only happen when spark.rapids.sql.detectDeltaLogQueries is enabled (because that is the only time we call entirePlanWillNotWork). This setting is currently enabled by default.

I have tried some different approaches to writing unit or integration tests for this change, but it is proving to be challenging, so I am open to suggestions on how to do this. I hit the error with some changes on another branch that I am no longer planning on merging.

[andygrove]

build

[andygrove]

build

[tgravescs]

Can we file a followup to try to find a test for this? I think we need to find better ways to test AQE in general

[andygrove]

Can we file a followup to try to find a test for this? I think we need to find better ways to test AQE in general

Filed #6253

[andygrove]

One downside to this fix is that it results in noisy logging

22/08/08 14:13:05 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU
22/08/08 14:13:05 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU
22/08/08 14:13:05 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU
22/08/08 14:13:05 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU
22/08/08 14:13:05 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU
22/08/08 14:13:05 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU,Delta Lake metadata queries are not efficient on GPU

[tgravescs]

can we do more to limit it to when AQE is on? Or perhaps we can make the logging smarter. Since this seems to be needed for 22.08, I think the logging is ok being noisy but we can file a followup to improve in 22.10. Unless others have other ideas.

[andygrove]

can we do more to limit it to when AQE is on? Or perhaps we can make the logging smarter. Since this seems to be needed for 22.08, I think the logging is ok being noisy but we can file a followup to improve in 22.10. Unless others have other ideas.

Yes, I think we can change getReasonsNotToReplaceEntirePlan to return a Set instead of a Seq. Testing this now.

[andygrove]

Using Set instead of Seq makes the logging much more reasonable:

22/08/08 14:31:43 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU
22/08/08 14:31:43 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU
22/08/08 14:31:43 WARN GpuOverrides: Can't replace any part of this plan due to: Delta Lake metadata queries are not efficient on GPU

[andygrove]

build