-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-20380][SQL] Unable to set/unset table comment property using ALTER TABLE SET/UNSET TBLPROPERTIES ddl #17649
Conversation
cc @wzhfy |
test this please |
also cc @gatorsmile |
@sujith71955 Can you change the title to be more specific, like "Support altering table comment"? |
val newTable = table.copy(properties = table.properties ++ properties) | ||
val newTable = table.copy( | ||
properties = table.properties ++ properties, | ||
comment = properties.get("comment")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add some comment to explain why this is necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will add a comment
ok to test |
This is a separate issue. Could you open a new JIRA and use a different JIRA number? |
DESC formatted table_comment; | ||
|
||
-- DROP TEST TABLE | ||
DROP TABLE table_comment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The basic function has been tested in the other suites. How about changing this test suite to describe-table-after-alter-table
?
CREATE TABLE table_comment (a STRING, b INT)
-- ALTER TABLE BY ADDING COMMENT
ALTER TABLE table_comment set tblproperties(comment = "added comment");
DESC formatted table_comment;
-- ALTER TABLE BY MODIFYING COMMENT
ALTER TABLE table_comment set tblproperties(comment = "modified comment");
DESC formatted table_comment;
-- DROP TEST TABLE
DROP TABLE table_comment;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds we also have a bug in the ALTER TABLE UNSET.
Please also add another test case
-- ALTER TABLE BY UNSETTING COMMENT
ALTER TABLE table_comment unset tblproperties('comment');
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, i will add a new jira for this problem and i will update the test suite name as per the suggestion, as you suggested i will also verify ALTER TABLE UNSET .
Test build #75840 has finished for PR 17649 at commit
|
Any update? ping @sujith71955 @wzhfy |
467a694
to
19d80c6
Compare
@gatorsmile @wzhfy updated the PR based on the review comments, please let me know for any suggestions. |
Test build #76002 has finished for PR 17649 at commit
|
Test build #76004 has finished for PR 17649 at commit
|
Type MANAGED | ||
Provider parquet | ||
Comment modified comment | ||
Properties [comment=modified comment] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment string i.e. "modified comment" is in both Properties
and Comment
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should remove comment
from Properties
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, i saw for hive tables already its been taken care in HiveClientImpl.scala class where "comment" is getting filtered from table properties, also for parquet table same has to be taken care.
DESC formatted table_with_comment; | ||
|
||
-- ALTER TABLE BY MODIFYING COMMENT | ||
ALTER TABLE table_with_comment set tblproperties(comment = "modified comment"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be better to also have other properties with comment in this test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename describe-table-after-alter-table.sql
to alter-table-comment.sql
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nvm I missed your previous discussion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
modified the testcase as per comment, added one more proeprty while altering the table.
@gatorsmile Should we care about case sensitivity of comment? Hive allows both "comment" and "Comment" in table parameters, while we are extracting comment as a field of |
None | ||
} else { | ||
table.properties.get("comment") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit:
val comment = if (propKeys.contains("comment")) None else table.properties.get("comment")
@wzhfy Could you check the behavior of Hive? |
19d80c6
to
50deed9
Compare
Test build #76085 has finished for PR 17649 at commit
|
@gatorsmile Hive treats comment simply as a key in the string-string parameter map, while spark extracts comment from the map as a field in Here's the results of hive:
|
50deed9
to
1156483
Compare
Test build #76139 has finished for PR 17649 at commit
|
1156483
to
99ba5be
Compare
Test build #76142 has finished for PR 17649 at commit
|
@gatorsmile @wzhfy updated the PR by removing the 'comment' from table properties . |
The changes look good to me if we don't care about the case sensitivity issue. |
@wzhfy In HiveClientImpl.scala, we follow Hive and consider the case sensitivity of property key @sujith71955 Could you resolve the following comment? |
@@ -267,8 +271,11 @@ case class AlterTableUnsetPropertiesCommand( | |||
} | |||
} | |||
} | |||
// if 'comment' key is present in the seq of keys which needs to be unset then reset the table | |||
// level comment with none. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: How about?
If
comment
is in the table property, we reset it to None
@@ -222,7 +222,6 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat | |||
} else { | |||
tableDefinition.storage.locationUri | |||
} | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please revert it back.
@@ -0,0 +1,29 @@ | |||
CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'added'; | |||
|
|||
DESC formatted table_with_comment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: formatted
-> FORMATTED
-- ALTER TABLE BY MODIFYING COMMENT | ||
ALTER TABLE table_with_comment set tblproperties("comment"= "modified comment", "type"= "parquet"); | ||
|
||
DESC formatted table_with_comment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: formatted
-> FORMATTED
-- ALTER TABLE BY ADDING COMMENT | ||
ALTER TABLE table_comment set tblproperties(comment = "added comment"); | ||
|
||
DESC formatted table_comment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: formatted
-> FORMATTED
-- ALTER UNSET PROPERTIES COMMENT | ||
ALTER TABLE table_comment UNSET TBLPROPERTIES IF EXISTS ('comment'); | ||
|
||
DESC formatted table_comment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: formatted
-> FORMATTED
DESC formatted table_comment; | ||
|
||
-- ALTER TABLE BY ADDING COMMENT | ||
ALTER TABLE table_comment set tblproperties(comment = "added comment"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: set tblproperties
-> SET TBLPROPERTIES
DESC formatted table_with_comment; | ||
|
||
-- ALTER TABLE BY MODIFYING COMMENT | ||
ALTER TABLE table_with_comment set tblproperties("comment"= "modified comment", "type"= "parquet"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: set tblproperties
-> SET TBLPROPERTIES
-- CREATE TABLE WITHOUT COMMENT | ||
CREATE TABLE table_comment (a STRING, b INT) USING parquet; | ||
|
||
DESC formatted table_comment; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: formatted
-> FORMATTED
LGTM except a few minor comments. |
…ding/modifying table comment using Alter TableSetPropertiesCommand and UNSET TBLPROPERTIES query. ### What changes were proposed in this pull request? This PR is a follow up PR to JIRA issue 20023 where table comment was not displayed in describe formatted command after user creates a table with table comment. But still was issue when user alter the table properties and adds/updates table comment. table comment which is directly part of CatalogTable instance where not getting updated and old table comment was shown, to handle this issue while updating the table properties map with newly added/modified properties in CatalogTable instance also update the comment parameter with newly added/modified comment as part of AlterTableSetPropertiesCommand which is also now exist in CatalogTable instance level not only in table properties map of CatalogTable instance. this pr has also taken care of unsetting the table comment when user executes AlterTableUnsetPropertiesCommand. ### How was this patch tested? Added test cases as part of SQLQueryTestSuite for verifying table comment using desc formatted table query after adding or modifying table comment as part of AlterTableSetPropertiesCommand and AlterTableUnsetPropertiesCommand.
433ac2b
to
4f02ada
Compare
Test build #76496 has finished for PR 17649 at commit
|
@wzhfy @gatorsmile fixed all the comments, thanks for reviewing the changes and providing me valuable sharings and comments. thanks. |
@sujith71955 Could you update the PR description and title? This PR is to fix the issue in modify table comments using ALTER TABLE SET/UNSET TBLPROPERTIES, instead of fixing the DESC FORMATTED, right? |
@gatorsmile you are right, pr should address the issue which is handled in alter table set/unset properties ddls, updated the description and title, let me know for any clarifications. thanks. |
retest this please |
Test build #76551 has finished for PR 17649 at commit
|
Thanks! Merging to master. |
…LTER TABLE SET/UNSET TBLPROPERTIES ddl ### What changes were proposed in this pull request? Table comment was not getting set/unset using **ALTER TABLE SET/UNSET TBLPROPERTIES** query eg: ALTER TABLE table_with_comment SET TBLPROPERTIES("comment"= "modified comment) when user alter the table properties and adds/updates table comment,table comment which is a field of **CatalogTable** instance is not getting updated and old table comment if exists was shown to user, inorder to handle this issue, update the comment field value in **CatalogTable** with the newly added/modified comment along with other table level properties when user executes **ALTER TABLE SET TBLPROPERTIES** query. This pr has also taken care of unsetting the table comment when user executes query **ALTER TABLE UNSET TBLPROPERTIES** inorder to unset or remove table comment. eg: ALTER TABLE table_comment UNSET TBLPROPERTIES IF EXISTS ('comment') ### How was this patch tested? Added test cases as part of **SQLQueryTestSuite** for verifying table comment using desc formatted table query after adding/modifying table comment as part of **AlterTableSetPropertiesCommand** and unsetting the table comment using **AlterTableUnsetPropertiesCommand**. Author: sujith71955 <sujithchacko.2010@gmail.com> Closes apache#17649 from sujith71955/alter_table_comment.
What changes were proposed in this pull request?
Table comment was not getting set/unset using ALTER TABLE SET/UNSET TBLPROPERTIES query
eg: ALTER TABLE table_with_comment SET TBLPROPERTIES("comment"= "modified comment)
when user alter the table properties and adds/updates table comment,table comment which is a field of CatalogTable instance is not getting updated and old table comment if exists was shown to user, inorder to handle this issue, update the comment field value in CatalogTable with the newly added/modified comment along with other table level properties when user executes ALTER TABLE SET TBLPROPERTIES query.
This pr has also taken care of unsetting the table comment when user executes query ALTER TABLE UNSET TBLPROPERTIES inorder to unset or remove table comment.
eg: ALTER TABLE table_comment UNSET TBLPROPERTIES IF EXISTS ('comment')
How was this patch tested?
Added test cases as part of SQLQueryTestSuite for verifying table comment using desc formatted table query after adding/modifying table comment as part of AlterTableSetPropertiesCommand and unsetting the table comment using AlterTableUnsetPropertiesCommand.