[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #28860

AngersZhuuuu · 2020-06-18T14:32:15Z

What changes were proposed in this pull request?

For data nest.json

{"a": [{"b": [{"c": [1,2]}]}]}
{"a": [{"b": [{"c": [1]}, {"c": [2]}]}]}

run with

val df: DataFrame = spark.read.json(testFile("nest-data.json"))
df.createTempView("nest_table")
sql("select a.b.c from nest_table").show()

will got error

org.apache.spark.sql.AnalysisException: cannot resolve 'nest_table.`a`.`b`['c']' due to data type mismatch: argument 2 requires integral type, however, ''c'' is of string type.; line 1 pos 7;
'Project [a#6.b[c] AS c#8|#6.b[c] AS c#8]
+- SubqueryAlias `nest_table`
+- Relationa#6 json

This is because after resolve a.b, it construct GetArrayStructField from a#
result with type ArrayType(ArrayType(StructType)), then it can't be extract through current ExtractValue method.

Support extract StructField form nested Array-Struct combination

Why are the changes needed?

Support more use case

Does this PR introduce any user-facing change?

Yes, users will be able to select from nested array and structs.

How was this patch tested?

Added UT

SparkQA · 2020-06-18T14:37:42Z

Test build #124217 has finished for PR 28860 at commit da9a3d5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-18T15:28:57Z

Test build #124220 has finished for PR 28860 at commit b8f4a69.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-19T07:05:02Z

Test build #124252 has finished for PR 28860 at commit 400ad7d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-06-19T07:13:45Z

retest this please

SparkQA · 2020-06-19T11:40:55Z

Test build #124268 has finished for PR 28860 at commit 400ad7d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

gatorsmile · 2020-06-20T04:42:22Z

cc @MaxGekk @HyukjinKwon

AngersZhuuuu · 2020-06-20T06:43:30Z

Seems this way will change the schema, maybe we should add a new class to get multidimensional array form GetArrayStruct

Co-authored-by: Ruslan Dautkhanov <Tagar@users.noreply.github.com>

SparkQA · 2020-06-20T07:05:02Z

Test build #124314 has finished for PR 28860 at commit b6e92c0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-06-26T09:50:25Z

retest this please

HyukjinKwon · 2020-06-26T10:40:17Z

cc @viirya too FYI

HyukjinKwon · 2020-06-26T10:41:25Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

@@ -59,6 +59,23 @@ object ExtractValue {
        GetArrayStructFields(child, fields(ordinal).copy(name = fieldName),
          ordinal, fields.length, containsNull)

+      case (ExtractNestedArray(StructType(fields), containsNull, containsNullSeq),
+      NonNullLiteral(v, StringType)) =>


nit:

case (ExtractNestedArray( StructType(fields), containsNull, containsNullSeq), NonNullLiteral(v, StringType)) =>

Let's also update the documentation and table above.

HyukjinKwon · 2020-06-26T10:44:37Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

@@ -95,6 +112,50 @@ object ExtractValue {

 trait ExtractValue extends Expression

+object ExtractNestedArray {


Let's add documentation here.

HyukjinKwon · 2020-06-26T10:44:53Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

@@ -95,6 +112,50 @@ object ExtractValue {

 trait ExtractValue extends Expression

+object ExtractNestedArray {
+
+  type ReturnType = Option[(DataType, Boolean, Seq[Boolean])]


Let's also add some comments for what this type means.

HyukjinKwon · 2020-06-26T10:46:13Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

+
+  def extractArrayStruct(expr: Expression): ReturnType = {
+    expr match {
+      case gas @ GetArrayStructFields(child, _, _, _, _) =>


Let's avoid arguments matching. This is actually an anti pattern - https://github.com/databricks/scala-style-guide#pattern-matching

HyukjinKwon · 2020-06-26T10:47:25Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

+    expr match {
+      case gas @ GetArrayStructFields(child, _, _, _, _) =>
+        extractArrayStruct(child) match {
+          case Some((e, deep)) => Some(e, deep + 1)


deep -> depth?

HyukjinKwon · 2020-06-26T10:50:25Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

+     extractArrayType(dataType)
+  }
+
+  def extractArrayType(dataType: DataType): ReturnType = {


Can we combine this and unapply?

HyukjinKwon · 2020-06-26T10:52:37Z

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

+
+      checkAnswer(sql(
+        """
+          |SELECT a.b.c FROM nest


Can we add deeper cases? Also, you can simplify the test cases, for example, as below:

val df = spark.range(10).select(array(struct(array(struct("id")).alias("col1"))).alias("col0")) df.selectExpr("col0.col1.id")

HyukjinKwon · 2020-06-26T10:57:42Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala

@@ -95,6 +112,50 @@ object ExtractValue {

 trait ExtractValue extends Expression

+object ExtractNestedArray {


Let's also name it something like ExtractNestedArrayType

SparkQA · 2020-06-26T14:31:17Z

Test build #124544 has finished for PR 28860 at commit b6e92c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

guiyanakuang · 2020-07-03T13:17:14Z

org.apache.spark.sql.execution.ProjectionOverSchema.scala. A Scala extractor that projects an expression over a given schema. I think this code also needs to have matching rules added to it. To support nested ArrayStruct. Otherwise, ParquetSchemaPruning won't work either.

AngersZhuuuu · 2020-07-04T04:14:37Z

org.apache.spark.sql.execution.ProjectionOverSchema.scala. A Scala extractor that projects an expression over a given schema. I think this code also needs to have matching rules added to it. To support nested ArrayStruct. Otherwise, ParquetSchemaPruning won't work either.

Yea, right, current way change the schema and won't work well for all case, I am thinking about add a new class to handle nested ArrayStruct type

github-actions · 2020-10-13T00:52:17Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

AngersZhuuuu added 2 commits June 18, 2020 22:25

save

4339b81

Update SQLQuerySuite.scala

da9a3d5

probot-autolabeler bot added the SQL label Jun 18, 2020

Update SQLQuerySuite.scala

b8f4a69

Update SelectedFieldSuite.scala

400ad7d

Tagar reviewed Jun 19, 2020

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala Outdated Show resolved Hide resolved

maropu changed the title ~~[SPARK-32002][SQL]Support Extract valve from nested ArrayStruct~~ [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct Jun 20, 2020

Update sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

b6e92c0

Co-authored-by: Ruslan Dautkhanov <Tagar@users.noreply.github.com>

HyukjinKwon reviewed Jun 26, 2020

View reviewed changes

github-actions bot added the Stale label Oct 13, 2020

github-actions bot closed this Oct 14, 2020

HyukjinKwon mentioned this pull request Nov 24, 2020

[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #30467

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #28860

[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #28860

AngersZhuuuu commented Jun 18, 2020 •

edited by HyukjinKwon

Loading

SparkQA commented Jun 18, 2020

SparkQA commented Jun 18, 2020

SparkQA commented Jun 19, 2020

AngersZhuuuu commented Jun 19, 2020

SparkQA commented Jun 19, 2020

gatorsmile commented Jun 20, 2020

AngersZhuuuu commented Jun 20, 2020

SparkQA commented Jun 20, 2020

HyukjinKwon commented Jun 26, 2020

HyukjinKwon commented Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020

HyukjinKwon Jun 26, 2020 •

edited

Loading

HyukjinKwon Jun 26, 2020 •

edited

Loading

SparkQA commented Jun 26, 2020

guiyanakuang commented Jul 3, 2020

AngersZhuuuu commented Jul 4, 2020

github-actions bot commented Oct 13, 2020

		@@ -95,6 +112,50 @@ object ExtractValue {

		trait ExtractValue extends Expression

		object ExtractNestedArray {

[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #28860

[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #28860

Conversation

AngersZhuuuu commented Jun 18, 2020 • edited by HyukjinKwon Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jun 18, 2020

SparkQA commented Jun 18, 2020

SparkQA commented Jun 19, 2020

AngersZhuuuu commented Jun 19, 2020

SparkQA commented Jun 19, 2020

gatorsmile commented Jun 20, 2020

AngersZhuuuu commented Jun 20, 2020

SparkQA commented Jun 20, 2020

HyukjinKwon commented Jun 26, 2020

HyukjinKwon commented Jun 26, 2020

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020 • edited Loading

Choose a reason for hiding this comment

HyukjinKwon Jun 26, 2020 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Jun 26, 2020

guiyanakuang commented Jul 3, 2020

AngersZhuuuu commented Jul 4, 2020

github-actions bot commented Oct 13, 2020

AngersZhuuuu commented Jun 18, 2020 •

edited by HyukjinKwon

Loading

HyukjinKwon Jun 26, 2020 •

edited

Loading

HyukjinKwon Jun 26, 2020 •

edited

Loading