Add --use-decimals flag to TPC-DS ConvertFiles #1506
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Andy Grove <andygrove@nvidia.com>
andygrove
force-pushed
the
benchmark-convert-files-use-decimal
branch
from
084153b
to
528b53a
Compare
|
build |
nartal1
reviewed
Jan 13, 2021
integration_tests/src/main/scala/com/nvidia/spark/rapids/tests/tpcds/TpcdsLikeSpark.scala
Show resolved
Hide resolved
Signed-off-by: Andy Grove <andygrove@nvidia.com>
gerashegalov
previously approved these changes
Jan 13, 2021
| conf.coalesce, | ||
| conf.repartition, | ||
| conf.withPartitioning()) | ||
| baseInput = conf.input(), |
There was a problem hiding this comment.
nit: I think it's better to use named parameters consistently, so spark should be spark = spark, or is it too redundant?
There was a problem hiding this comment.
My main goal was to make sure the configuration options were being passed to the correct arguments, but it makes sense to use named args for all of them so I have updated this.
nartal1
previously approved these changes
Jan 13, 2021
|
build |
|
build |
nartal1
approved these changes
Jan 14, 2021
gerashegalov
approved these changes
Jan 14, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Add --use-decimals flag to TPC-DS ConvertFiles Signed-off-by: Andy Grove <andygrove@nvidia.com> * update copyright Signed-off-by: Andy Grove <andygrove@nvidia.com> * make named args consistent
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Add --use-decimals flag to TPC-DS ConvertFiles Signed-off-by: Andy Grove <andygrove@nvidia.com> * update copyright Signed-off-by: Andy Grove <andygrove@nvidia.com> * make named args consistent
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…VIDIA#1506) * Prevent optimization * Add comment Signed-off-by: Nghia Truong <nghiat@nvidia.com> * Revert "Add comment" This reverts commit 025fad50bf317b8608e9f41ea637d3131181eadc. --------- Signed-off-by: Nghia Truong <nghiat@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the ability to specify to use decimal types when converting TPC-DS data to Parquet when using spark-submit with the ConvertFiles utility,