Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Scala tests should support running with Adaptive Query Execution enabled #145

Closed
andygrove opened this issue Jun 10, 2020 · 0 comments · Fixed by #258
Closed

[FEA] Scala tests should support running with Adaptive Query Execution enabled #145

andygrove opened this issue Jun 10, 2020 · 0 comments · Fixed by #258
Assignees
Labels
feature request New feature or request test Only impacts tests

Comments

@andygrove
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Many of the current Scala unit and integration tests check that queries (or subsets of queries) are going to run on the GPU by inspecting the execution plan made available from the DataFrame API. This approach does not work when Adaptive Query Execution (AQE) is enabled, because the execution plan is determined during query execution. This means we cannot currently test the plugin with AQE to see what issues need to be resolved to fully support it.

Describe the solution you'd like
The tests should be updated to register a listener with the Spark context so that the listenener can receive the executed plan after query execution has been completed and then use that to determine which parts of the query ran on CPU or GPU.

Describe alternatives you've considered
N/A

Additional context
N/A

@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify test Only impacts tests labels Jun 10, 2020
@revans2 revans2 removed the ? - Needs Triage Need team to review and classify label Jun 23, 2020
@sameerz sameerz added this to the Jul 6 - Jul 17 milestone Jul 17, 2020
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request test Only impacts tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants