Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Spark properties table to profiling tool output #4781

Merged
merged 8 commits into from
Feb 15, 2022

Conversation

tgravescs
Copy link
Collaborator

fixes #4753

The profiling tool now prints all the Spark properties into a separate table. It still prints just the rapids related ones into one table so you can easily see those and then all the other Spark properties into a separate table. The spark properties are very useful for comparing applications.

This also has a random change to change a log info message when we see events we don't handle to a log debug message.

examples:

rapids properties no change:

Spark Rapids parameters set explicitly:
+-----------------------------------+----------+----------+
|propertyName                       |appIndex_2|appIndex_1|
+-----------------------------------+----------+----------+
|spark.rapids.memory.pinnedPool.size|2G        |2G        |
|spark.rapids.sql.concurrentGpuTasks|2         |2         |
|spark.rapids.sql.explain           |NOT_ON_GPU|NOT_ON_GPU|
+-----------------------------------+----------+----------+

Added spark properties:

Spark Properties:
+---------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
|propertyName                     |appIndex_2                                                                                |appIndex_1                                                                                |
+---------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
|spark.app.id                     |local-1622821994212                                                                       |local-1622814619968                                                                       |
|spark.app.name                   |Spark shell                                                                               |Spark shell                                                                               |
....

@tgravescs tgravescs added this to the Feb 14 - Feb 25 milestone Feb 14, 2022
@tgravescs tgravescs self-assigned this Feb 14, 2022
@nartal1
Copy link
Collaborator

nartal1 commented Feb 14, 2022

LGTM. Do we need to update in spark-profiling-tool.md that spark properties are also covered along with rapids properties?

@tgravescs
Copy link
Collaborator Author

yes, thanks for pointing that out, I added it to list. I didn't put example as it seems like its pretty self explanatory.

@tgravescs
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit 6a096e5 into NVIDIA:branch-22.04 Feb 15, 2022
@tgravescs tgravescs deleted the sparkpropsProfile branch February 15, 2022 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] Profiling tool should report all spark configs
2 participants