Profile/qualification tool error handling improvements and support spark < 3.1.1 #2604
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <tgraves@apache.org>
This reverts commit 6f5271c.
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
tgravescs
added
the
task
|
build |
|
build |
This reverts commit 828b302.
|
build |
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
build |
nartal1
previously approved these changes
Jun 6, 2021
|
build |
|
missed test files on upmerge |
|
build |
nartal1
approved these changes
Jun 6, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
…ark < 3.1.1 (NVIDIA#2604) * Qualification tool Signed-off-by: Thomas Graves <tgraves@apache.org> * remove unused func * Add missing files * Add checks for format option * cast columsn to string to write to text * Revert "Add checks for format option" This reverts commit 6f5271c. * cleanup Signed-off-by: Thomas Graves <tgraves@nvidia.com> * update output dir * formating * Update help messages * update app name * cleanup * put test functions back * fix typo * add printSQLPlanMetrics and printRapidsJar * use opt * Add Analysis * format output * more tests Signed-off-by: Thomas Graves <tgraves@apache.org> * tests working * test rearrange utils * move test file * move test file right location * add Analysis Suite * update test analysis * add rapids-4-spark-tools/src/test/resources/ProfilingExpectations/rapids_join_eventlog_jobandstagemetrics_expectation.csv * add more tests * more tests Signed-off-by: Thomas Graves <tgraves@nvidia.com> * remove unneeded expectation file * Add more analysis tests * comment * cleanup * Start handling ResourceProfile with reflection Signed-off-by: Thomas Graves <tgraves@nvidia.com> * allow spark 3.0 and 3.1.1 to parse logs and fix bug with missing table * Add more test files * Add more tests and error handling * improve error handling to skip bad event logs * update readme * remove unneeded temp dir * Revert "remove unneeded temp dir" This reverts commit 828b302. * close file writer on exception Signed-off-by: Thomas Graves <tgraves@nvidia.com> * move test files * fix move of test files
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
…ark < 3.1.1 (NVIDIA#2604) * Qualification tool Signed-off-by: Thomas Graves <tgraves@apache.org> * remove unused func * Add missing files * Add checks for format option * cast columsn to string to write to text * Revert "Add checks for format option" This reverts commit 6f5271c. * cleanup Signed-off-by: Thomas Graves <tgraves@nvidia.com> * update output dir * formating * Update help messages * update app name * cleanup * put test functions back * fix typo * add printSQLPlanMetrics and printRapidsJar * use opt * Add Analysis * format output * more tests Signed-off-by: Thomas Graves <tgraves@apache.org> * tests working * test rearrange utils * move test file * move test file right location * add Analysis Suite * update test analysis * add rapids-4-spark-tools/src/test/resources/ProfilingExpectations/rapids_join_eventlog_jobandstagemetrics_expectation.csv * add more tests * more tests Signed-off-by: Thomas Graves <tgraves@nvidia.com> * remove unneeded expectation file * Add more analysis tests * comment * cleanup * Start handling ResourceProfile with reflection Signed-off-by: Thomas Graves <tgraves@nvidia.com> * allow spark 3.0 and 3.1.1 to parse logs and fix bug with missing table * Add more test files * Add more tests and error handling * improve error handling to skip bad event logs * update readme * remove unneeded temp dir * Revert "remove unneeded temp dir" This reverts commit 828b302. * close file writer on exception Signed-off-by: Thomas Graves <tgraves@nvidia.com> * move test files * fix move of test files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I found a few issues with bad event logs, added more error handling and tests.
I also added support for running with Spark versions < 3.1.1, this was due to the ResourceProfile event only in 3.1.
Renamed Rank to be Score based on feedback.