-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement test for qualification tool sql metric aggregates #2591
Conversation
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
@@ -97,4 +99,73 @@ class QualificationSuite extends FunSuite with Logging { | |||
val logFiles = Array(s"$logDir/nds_q86_test") | |||
runQualificationTest(logFiles, "nds_q86_test_expectation.csv") | |||
} | |||
|
|||
test("sql metric agg") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we could do this separately and if you don't have time this week just let me know, but it would be nice to also validate the sql time aggregation across multiple SQL jobs. Add in one with dataset for bonus.
build |
|
||
def generateEventLog(eventLogDir: File, appName: String) | ||
(fun: SparkSession => DataFrame): String = { | ||
val spark = SparkSession |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make this lazy
@andygrove Test seems to be failing. |
Signed-off-by: Andy Grove <andygrove@nvidia.com>
build |
build |
) * Fix package name Signed-off-by: Andy Grove <andygrove@nvidia.com> * Rename to ToolTestUtils Signed-off-by: Andy Grove <andygrove@nvidia.com> * Implement basic structure for test Signed-off-by: Andy Grove <andygrove@nvidia.com> * Test checks that number of tasks is correct Signed-off-by: Andy Grove <andygrove@nvidia.com> * Add more tests Signed-off-by: Andy Grove <andygrove@nvidia.com> * Add cpuTime test, commented out Signed-off-by: Andy Grove <andygrove@nvidia.com> * Change executorCpuTime logic to match qualification tool Signed-off-by: Andy Grove <andygrove@nvidia.com> * use dedicated spark session in test * create new SparkSession before each test Signed-off-by: Andy Grove <andygrove@nvidia.com>
) * Fix package name Signed-off-by: Andy Grove <andygrove@nvidia.com> * Rename to ToolTestUtils Signed-off-by: Andy Grove <andygrove@nvidia.com> * Implement basic structure for test Signed-off-by: Andy Grove <andygrove@nvidia.com> * Test checks that number of tasks is correct Signed-off-by: Andy Grove <andygrove@nvidia.com> * Add more tests Signed-off-by: Andy Grove <andygrove@nvidia.com> * Add cpuTime test, commented out Signed-off-by: Andy Grove <andygrove@nvidia.com> * Change executorCpuTime logic to match qualification tool Signed-off-by: Andy Grove <andygrove@nvidia.com> * use dedicated spark session in test * create new SparkSession before each test Signed-off-by: Andy Grove <andygrove@nvidia.com>
Adds tests to ensure that we are correctly aggregating SQL metrics in the qualification tool.