Use multi-threaded parquet read with small files #677
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@apache.org>
|
build |
tgravescs
commented
Sep 8, 2020
| "parquet files in parallel. This can not be changed at runtime after the executor has" + | ||
| "started.") | ||
| .integerConf | ||
| .createWithDefault(20) |
There was a problem hiding this comment.
this worked well on databricks with hosts with 4 to 6 cores. If we have executors with lots of cores this might not be ideal. I could try to adjust this based on executor cores but with standalone mode I don't know what that is. Any thoughts on this? I figure this is good start and we can adjust later.
There was a problem hiding this comment.
+1 for putting something in that we can adjust later on if needed.
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
build |
revans2
previously approved these changes
Sep 8, 2020
jlowe
reviewed
Sep 8, 2020
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuInputFileBlock.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@apache.org>
jlowe
approved these changes
Sep 8, 2020
|
build |
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Try multi-threaded read with parquet with small files Signed-off-by: Thomas Graves <tgraves@nvidia.com> * cleanup and comments * comment config Signed-off-by: Thomas Graves <tgraves@apache.org> * Add note about TaskContext not being set in the threadpool Signed-off-by: Thomas Graves <tgraves@nvidia.com> * remove extra import and use closeOnExcept * try just using future throw * let future throw Signed-off-by: Thomas Graves <tgraves@apache.org> * Use safeclose() Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Try multi-threaded read with parquet with small files Signed-off-by: Thomas Graves <tgraves@nvidia.com> * cleanup and comments * comment config Signed-off-by: Thomas Graves <tgraves@apache.org> * Add note about TaskContext not being set in the threadpool Signed-off-by: Thomas Graves <tgraves@nvidia.com> * remove extra import and use closeOnExcept * try just using future throw * let future throw Signed-off-by: Thomas Graves <tgraves@apache.org> * Use safeclose() Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com>
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…IDIA#677) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #627
closes #608
This changes the parquet reader small file improvements to use multi-threaded read instead of aggregating the files together. I changed the config names to match this implementation. There is 1 config to turn it on and off, 1 to control the number of threads used per executor (its shared across all tasks) and then 1 config to control the max number of files to be processed per task in parallel before it gets copied to the GPU. The last config allows you to somewhat control the host memory being used if you are limited.
Essentially what happens now is that we do everything for each file in a separate thread. So we get the list of files for each task and launch multiple threads in parallel and each thread does everything from reading the footer, to filtering the blocks down, and then copying the data into host memory buffers. This makes things pretty straight forward as to what is going on and allows us to support the input_file_name and mergeSchema options that we didn't before.
The code launches the files to be processed in the order in which they come in to match the CPU side and they run in parallel up to the number of free threads.
I tested on 2 queries which both have partitioning and this showed great improvement. one went from about 11 minutes down to 4.5 minutes. the other went from 7-8 minutes down to 2.4 minutes. Those were using executors with 4 or 6 cores (on databricks - so standalone mode) each and 20 threads worked very well there. The downside to 20 is that if you have executors with lots of cores 20 might not be enough.
I tested with consolidated files on the first query without partitioning where each task only got a part of a file and the results were the same or slightly better than previous implementation.
Signed-off-by: Thomas Graves tgraves@nvidia.com