-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-39857][SQL][3.3] V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate #37324
Conversation
…ue data type for In predicate
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, @huaxingao .
The PR description will be a commit message.
Please copy all the PR description to the backport PR and adds additional message like the following
back port #37271 to 3.3
"`a.b`.cint", // two level nested field, and top level contains `dot` | ||
"`a.b`.`c.int`", // two level nested field, and both levels contain `dot` | ||
"a.b.cint" // three level nested field | ||
)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit, to be clear, this is a backport of SPARK-39784 technically.
test("translate simple expression") { attrInts.zip(attrStrs) | ||
.foreach { case ((attrInt, intColName), (attrStr, strColName)) => | ||
testTranslateFilter(EqualTo(attrInt, 1), | ||
Some(new Predicate("=", Array(FieldReference(intColName), LiteralValue(1, IntegerType))))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto. This is a backport of SPARK-39784 technically and is irrelevant to this PR's InSet
contribution.
…ue data type for In predicate ### What changes were proposed in this pull request? When building V2 In `Predicate` in `V2ExpressionBuilder`, `InSet.dataType` (which is BooleanType) is used to build the `LiteralValue`, `InSet.child.dataType `should be used instead. back port #37271 to 3.3 ### Why are the changes needed? bug fix ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? new test Closes #37324 from huaxingao/backport. Authored-by: huaxingao <huaxin_gao@apple.com> Signed-off-by: huaxingao <huaxin_gao@apple.com>
The GA status is unfinished, but the tests have been finished a while ago, so I will merge. |
Thank you very much @dongjoon-hyun for reviewing! |
What changes were proposed in this pull request?
When building V2 In
Predicate
inV2ExpressionBuilder
,InSet.dataType
(which is BooleanType) is used to build theLiteralValue
,InSet.child.dataType
should be used instead.back port #37271 to 3.3
Why are the changes needed?
bug fix
Does this PR introduce any user-facing change?
no
How was this patch tested?
new test