Qualification tool hook up final output based on per exec analysis #5550
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
Signed-off-by: Thomas Graves <tgraves@apache.org>
|
build |
tgravescs
requested review from
amahussein and
nartal1
and removed request for
amahussein
nartal1
reviewed
May 20, 2022
tools/src/main/scala/com/nvidia/spark/rapids/tool/planparser/SQLPlanParser.scala
Show resolved
Hide resolved
tgravescs
changed the title
tgravescs
commented
May 20, 2022
|
|
||
| // TODO - do we want to use this and rely on stages, but some SQL don't have stages | ||
| // so this is less than SQL DF real | ||
| val sqlDFWallClockDuration = |
There was a problem hiding this comment.
there is a bug here with the df duration because its looking at supported vs all. I will fix in a followup #5570
amahussein
previously approved these changes
May 20, 2022
There was a problem hiding this comment.
I checked the UI works fine with this PR.
I approve this PR as long as we can followup with PRs addressing the issues listed in #3792
|
build |
nartal1
previously approved these changes
May 20, 2022
|
build |
|
not sure why a test was hanging due to this, but fixed. |
|
build |
nartal1
approved these changes
May 23, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #5512 and fixes #5364
Lots of changes here.
So we can try to estimate based on the SQL task times, which is really estimated with the stages task times added up, and we use job overhead To do this, we line up the SQLID to stages and then which operators were in each Stage based on the accumulators. Note not all operators have a mapping to stages.
We take the task time calculations and get a ratio and then try to apply them to the wall clock times to display to the user.
change some of the UDF based operators to have score of 1.2 as we only accelerate the data transfer and udfs still run on cpu.
Each exec checks for udfs and dataset individually to say those aren't supported vs doing the entire SQL operation.
We have 4 levels of recommendation, strongly, recommended, not and not applicable. not applicable is when we have stage or job failures. Need to followup and update UI with good way to display this.
Updated the output to be what we want. Focus on wall clock for the user readable side, the csv file has the task level details. I think we will want to put more in the csv file but for now this works. Outputs the exec level in csv file and outputs the stages used in SQL operations in a csv file.
Need to do more testing on this but would like to get this in and be able to get others to test. A few tests were commented out to update later.
There is also more code cleanup and refactoring I would like to do to make this more readable but that can wait.