Add new table format presets. Add support for custom format #23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mmateja
reviewed
Dec 15, 2021
docs/CHANGELOG.md
Outdated
| @@ -3,6 +3,11 @@ | |||
|
|
|||
| ## 1.0 | |||
There was a problem hiding this comment.
That could use an update as well :)
mmateja
reviewed
Dec 15, 2021
docs/CHANGELOG.md
Outdated
| @@ -3,6 +3,11 @@ | |||
|
|
|||
| ## 1.0 | |||
|
|
|||
| ### 2.0.0 | |||
|
|
|||
| - **[breaking change]** Make `:orc` table format to have `orc.column.index.access` disabled by default. `:orc_legacy` preset replaces the previous one | |||
There was a problem hiding this comment.
Suggested change
| - **[breaking change]** Make `:orc` table format to have `orc.column.index.access` disabled by default. `:orc_legacy` preset replaces the previous one | |
| - **[breaking change]** Make `:orc` table format use column names instead of indexes by default. `:orc_legacy` preset preserves existing behavior. |
mmateja
approved these changes
Dec 15, 2021
mkrawc
reviewed
Dec 16, 2021
lib/egis/table_ddl_generator.rb
Outdated
| STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' | ||
| OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' | ||
| SQL | ||
| when :orc_legacy |
There was a problem hiding this comment.
legacy has a negative connotation. How about we call it explicitly like orc_with_index_access?
How about we also support json format, we use it in Socialguide A+ query
when :json
'ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
lib/egis/table_ddl_generator.rb
Outdated
|
|
||
| def serde_properties(format) | ||
| return '' unless format.key?(:serde_properties) | ||
| return "ROW FORMAT #{format}" if format.is_a?(String) |
There was a problem hiding this comment.
As discussed the String option should also allow to define STORED AS option. I'd remove the "ROW FORMAT" from here and allow user to fill it in or not, for example if he just want to use "STORED AS PARQUET" option
Ciebiada
changed the title
|
@mkrawc I've merged this. However. If you have any further comments please let me know and I will get back to it |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
lib/egis/version.rb