forked from NVIDIA/spark-rapids
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add dynamic Spark configuration for Databricks (NVIDIA#2116)
* Add daynamic spark confs for the Databricks We need a way to set spark confs dynamically for the Databricks, e.g., when we test cuDF sonatype release jars, we need to disable cudf-rapids version match by adding "--conf spark.rapids.cudfVersionOverride=true", or enable/disable AQE, or anything else. By adding the parameter spark_conf="--conf spark.xxx.xxx=xxx --conf ......" for the script 'run-tests.py', we can dynamically add whatever confs for the Databricks cluster. Signed-off-by: Tim Liu <timl@nvidia.com> * Comma separated list of spark configurations Signed-off-by: Tim Liu <timl@nvidia.com> * Add a comment to make the '-f' format clear * Add a comment to make the '-f' format clear * Fix typo * Add '--conf' if the SPARK_CONF is not empty
- Loading branch information
Showing
3 changed files
with
28 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters