[SPARK-4693] [SQL] PruningPredicates may be wrong if predicates contains an empty AttributeSet() references #3556
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #24042 has finished for PR 3556 at commit
|
|
Can you please add a test to |
|
Test build #24071 has finished for PR 3556 at commit
|
|
Test build #24079 has finished for PR 3556 at commit
|
| val (pruningPredicates, otherPredicates) = predicates.partition { | ||
| _.references.subsetOf(partitionKeyIds) | ||
| val (pruningPredicates, otherPredicates) = predicates.partition { x => | ||
| x.references.baseSet != null && |
|
ping, do you think you will have time to address the comments? Otherwise I might do it as this would be a good bug to fix. Thanks! |
|
@marmbrus Thank you for your comments. I will do it right away. |
…ins an empty AttributeSet() references
…ins an empty AttributeSet() references
|
Test build #24581 has finished for PR 3556 at commit
|
…ins an empty AttributeSet() references
|
Test build #24582 has finished for PR 3556 at commit
|
|
@marmbrus Please review again. Thanks. |
|
Thanks! Merged to master. |
| val (pruningPredicates, otherPredicates) = predicates.partition { | ||
| _.references.subsetOf(partitionKeyIds) | ||
| val (pruningPredicates, otherPredicates) = predicates.partition { predicate => | ||
| !predicate.references.isEmpty && |
There was a problem hiding this comment.
This line sounds useless in Spark 2.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The sql "select * from spark_test::for_test where abs(20141202) is not null" has predicates=List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)) and
partitionKeyIds=AttributeSet(). PruningPredicates is List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)). Then the exception "java.lang.IllegalArgumentException: requirement failed: Partition pruning predicates only supported for partitioned tables." is thrown.
The sql "select * from spark_test::for_test_partitioned_table where abs(20141202) is not null and type_id=11 and platform = 3" with partitioned key insert_date has predicates=List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202), (type_id#12 = 11), (platform#8 = 3)) and partitionKeyIds=AttributeSet(insert_date#24). PruningPredicates is List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)).