[SPARK-33818][SQL][DOC] Add descriptions about `spark.sql.parser.quotedRegexColumnNames` in the SQL documents #30816

AngersZhuuuu · 2020-12-17T04:37:03Z

What changes were proposed in this pull request?

According to #30805 (comment),
doc spark.sql.parser.quotedRegexColumnNames since we need user know about this in doc and it's useful.

Why are the changes needed?

Complete doc

Does this PR introduce any user-facing change?

No

How was this patch tested?

Not need

SparkQA · 2020-12-17T05:25:23Z

Test build #132929 has finished for PR 30816 at commit 93e1c1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
CREATE TABLE person (id INT, name STRING, age INT, class INT, address STRING);

SparkQA · 2020-12-17T06:01:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37532/

SparkQA · 2020-12-17T06:06:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37532/

SparkQA · 2020-12-17T07:26:23Z

Test build #132938 has finished for PR 30816 at commit 9df29ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-12-18T01:57:47Z

cc @maropu @huaxingao @dilipbiswal FYI

maropu · 2020-12-18T02:20:47Z

docs/sql-ref-syntax-qry-select-regex-column.md

+    (400, 'Dan', 50, 4, 'Street 4');
+SET spark.sql.parser.quotedRegexColumnNames=true;
+
+SELECT `(class|address)?+.+` FROM person;


We need a dedicated page for this functionality? How about adding some simple SELECT examples in the top SELECT page? IMHO its okay to add them in https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select.html cc: @HyukjinKwon @gatorsmile

Wait for their advise and I will update later.

Yeah, I would prefer to don't have a dedicated page.

maropu · 2020-12-18T02:22:25Z

docs/sql-ref-syntax-qry-select.md

@@ -175,3 +180,4 @@ SELECT [ hints , ... ] [ ALL | DISTINCT ] { named_expression [ , ... ] }
 * [CASE Clause](sql-ref-syntax-qry-select-case.html)
 * [PIVOT Clause](sql-ref-syntax-qry-select-pivot.html)
 * [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)
+* [Regex Column Names](sql-ref-syntax-qry-select-regex-column.html)


I think its better to avoid abbreviation forms (e.g., Regex) in user's docs.

maropu · 2020-12-18T02:22:51Z

docs/sql-ref-syntax-qry-select-regex-column.md

+* [DISTRIBUTE BY Clause](sql-ref-syntax-qry-select-distribute-by.html)
+* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
+* [CASE Clause](sql-ref-syntax-qry-select-case.html)
+* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)


All the pages above are related to this option?

AngersZhuuuu · 2021-01-05T05:10:01Z

@maropu Have updated as what you said. Can you review again? thanks.

SparkQA · 2021-01-05T05:27:03Z

Test build #133650 has finished for PR 30816 at commit 5f7ca84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

docs/sql-ref-syntax-qry-select.md

SparkQA · 2021-01-05T06:28:18Z

Test build #133653 has finished for PR 30816 at commit aa52dd9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-01-05T07:18:39Z

Test build #133656 has finished for PR 30816 at commit 1c042d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-01-05T08:02:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38242/

SparkQA · 2021-01-05T08:06:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38239/

SparkQA · 2021-01-05T08:08:42Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38242/

SparkQA · 2021-01-05T08:56:37Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38239/

SparkQA · 2021-01-05T09:03:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38245/

SparkQA · 2021-01-05T09:08:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38245/

maropu · 2021-01-05T13:35:37Z

docs/sql-ref-syntax-qry-select.md

+     ```sql
+     SELECT `(a|b)?+.+` FROM (
+       SELECT 1 as a, 2 as b, 3 as c
+     ) src


How about moving this example into the bottom of this page just like the other SQL references?

How about moving this example into the bottom of this page just like the other SQL references?

Here we just show a example usage and look good. Only move this one SQL to bottom is strange and reader may ignore it(To be honest, for example, me...).

maropu · 2021-01-05T13:56:07Z

For reviews, could you put the screenshot of the updated doc in the PR description?

AngersZhuuuu · 2021-01-05T14:20:19Z

For reviews, could you put the screenshot of the updated doc in the PR description?

Yea, add whole page snapshot about changed place, looks pretty fine.

AngersZhuuuu · 2021-01-08T02:43:04Z

gentle ping @HyukjinKwon @dongjoon-hyun

dongjoon-hyun · 2021-01-08T02:44:41Z

Thank you for pinging me, @AngersZhuuuu .

dongjoon-hyun · 2021-01-08T02:47:49Z

docs/sql-ref-syntax-qry-select.md

+     ```sql
+     SELECT `(a|b)?+.+` FROM (
+       SELECT 1 as a, 2 as b, 3 as c
+     ) src


Do we need this src? I think we can omit this.

spark-sql> SELECT a FROM (SELECT 1 as a, 2 as b, 3 as c); 1

Do we need this src? I think we can omit this.

Ur, not need, Omit this.

dongjoon-hyun

+1, LGTM. Thank you, @AngersZhuuuu , @maropu , @HyukjinKwon and all.
Merged to master/3.1.

…edRegexColumnNames` in the SQL documents ### What changes were proposed in this pull request? According to #30805 (comment), doc `spark.sql.parser.quotedRegexColumnNames` since we need user know about this in doc and it's useful. ![image](https://user-images.githubusercontent.com/46485123/103656543-afa4aa80-4fa3-11eb-8cd3-a9d1b87a3489.png) ![image](https://user-images.githubusercontent.com/46485123/103656551-b2070480-4fa3-11eb-9ce7-95cc424242a6.png) ### Why are the changes needed? Complete doc ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not need Closes #30816 from AngersZhuuuu/SPARK-33818. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 9b54da4) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…edRegexColumnNames` in the SQL documents ### What changes were proposed in this pull request? According to apache#30805 (comment), doc `spark.sql.parser.quotedRegexColumnNames` since we need user know about this in doc and it's useful. ![image](https://user-images.githubusercontent.com/46485123/103656543-afa4aa80-4fa3-11eb-8cd3-a9d1b87a3489.png) ![image](https://user-images.githubusercontent.com/46485123/103656551-b2070480-4fa3-11eb-9ce7-95cc424242a6.png) ### Why are the changes needed? Complete doc ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not need Closes apache#30816 from AngersZhuuuu/SPARK-33818. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-33818][DOC] Doc spark.sql.parser.quotedRegexColumnNames

93e1c1f

github-actions bot added the DOCS label Dec 17, 2020

Update sql-ref-syntax-qry-select-regex-column.md

8dd84d0

save

9df29ca

maropu reviewed Dec 18, 2020

View reviewed changes

maropu changed the title ~~[SPARK-33818][DOC] Doc spark.sql.parser.quotedRegexColumnNames~~ [SPARK-33818][SQL][DOC] Add descriptions about spark.sql.parser.quotedRegexColumnNames in the SQL documents Dec 18, 2020

follow comment

5f7ca84

HyukjinKwon reviewed Jan 5, 2021

View reviewed changes

docs/sql-ref-syntax-qry-select.md Outdated Show resolved Hide resolved

HyukjinKwon reviewed Jan 5, 2021

View reviewed changes

docs/sql-ref-syntax-qry-select.md Outdated Show resolved Hide resolved

Update sql-ref-syntax-qry-select.md

aa52dd9

HyukjinKwon reviewed Jan 5, 2021

View reviewed changes

docs/sql-ref-syntax-qry-select.md Outdated Show resolved Hide resolved

Update sql-ref-syntax-qry-select.md

1c042d1

maropu reviewed Jan 5, 2021

View reviewed changes

maropu approved these changes Jan 5, 2021

View reviewed changes

dongjoon-hyun reviewed Jan 8, 2021

View reviewed changes

Update sql-ref-syntax-qry-select.md

ce42a51

dongjoon-hyun approved these changes Jan 8, 2021

View reviewed changes

dongjoon-hyun closed this in 9b54da4 Jan 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-33818][SQL][DOC] Add descriptions about `spark.sql.parser.quotedRegexColumnNames` in the SQL documents #30816

[SPARK-33818][SQL][DOC] Add descriptions about `spark.sql.parser.quotedRegexColumnNames` in the SQL documents #30816

AngersZhuuuu commented Dec 17, 2020 •

edited

Loading

SparkQA commented Dec 17, 2020

SparkQA commented Dec 17, 2020

SparkQA commented Dec 17, 2020

SparkQA commented Dec 17, 2020

HyukjinKwon commented Dec 18, 2020

maropu Dec 18, 2020

AngersZhuuuu Dec 18, 2020

HyukjinKwon Jan 5, 2021

maropu Dec 18, 2020

maropu Dec 18, 2020

AngersZhuuuu commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

maropu Jan 5, 2021

AngersZhuuuu Jan 5, 2021 •

edited

Loading

maropu commented Jan 5, 2021

AngersZhuuuu commented Jan 5, 2021

AngersZhuuuu commented Jan 8, 2021

dongjoon-hyun commented Jan 8, 2021

dongjoon-hyun Jan 8, 2021

dongjoon-hyun Jan 8, 2021

AngersZhuuuu Jan 8, 2021

dongjoon-hyun left a comment

[SPARK-33818][SQL][DOC] Add descriptions about spark.sql.parser.quotedRegexColumnNames in the SQL documents #30816

[SPARK-33818][SQL][DOC] Add descriptions about spark.sql.parser.quotedRegexColumnNames in the SQL documents #30816

Conversation

AngersZhuuuu commented Dec 17, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Dec 17, 2020

SparkQA commented Dec 17, 2020

SparkQA commented Dec 17, 2020

SparkQA commented Dec 17, 2020

HyukjinKwon commented Dec 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AngersZhuuuu commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

SparkQA commented Jan 5, 2021

Choose a reason for hiding this comment

AngersZhuuuu Jan 5, 2021 • edited Loading

Choose a reason for hiding this comment

maropu commented Jan 5, 2021

AngersZhuuuu commented Jan 5, 2021

AngersZhuuuu commented Jan 8, 2021

dongjoon-hyun commented Jan 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

[SPARK-33818][SQL][DOC] Add descriptions about `spark.sql.parser.quotedRegexColumnNames` in the SQL documents #30816

[SPARK-33818][SQL][DOC] Add descriptions about `spark.sql.parser.quotedRegexColumnNames` in the SQL documents #30816

AngersZhuuuu commented Dec 17, 2020 •

edited

Loading

AngersZhuuuu Jan 5, 2021 •

edited

Loading