-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle minimum GPU architecture supported [databricks] #10540
Handle minimum GPU architecture supported [databricks] #10540
Conversation
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
This reverts commit 7b8eaea.
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
…hitectures Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Converted to draft since this
Need to wait for a new artefact for |
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala
Outdated
Show resolved
Hide resolved
Co-authored-by: Jason Lowe <jlowe@nvidia.com>
build |
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking good, but agree with @gerashegalov that this should have at least some test. Refactoring checkGpuArchitecture
to take the property set and the GPU major/minor architectures makes it easier to mock and test various scenarios of the core logic for this.
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thank you @parthosa for working through all the issues.
tests/src/test/scala/com/nvidia/spark/rapids/GpuArchitectureTestSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thank you @gerashegalov and @jlowe |
Fixes #10430. This PR ensures that Spark RAPIDS jobs are executed on supported GPU architectures without relying on manual configuration.
Changes:
gpu_architectures
property from the*version-info.properties
file generated by the native builds.Testing
Tested on a Dataproc VM running on Nvidia P4 (GPU Architecture 6.1)
Related PR