Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add dynamic Spark configuration for Databricks #2116

Merged
merged 6 commits into from
Apr 15, 2021
Merged

Add dynamic Spark configuration for Databricks #2116

merged 6 commits into from
Apr 15, 2021

Commits on Apr 10, 2021

  1. Add daynamic spark confs for the Databricks 

    We need a way to set spark confs dynamically for the Databricks,
e.g., when we test cuDF sonatype release jars, we need to disable cudf-rapids version match by adding
"--conf spark.rapids.cudfVersionOverride=true", or enable/disable AQE, or anything else.

By adding the parameter spark_conf="--conf spark.xxx.xxx=xxx --conf ......" for the script 'run-tests.py',
we can dynamically add whatever confs for the Databricks cluster.

Signed-off-by: Tim Liu <timl@nvidia.com>
    @NvTimLiu
    NvTimLiu committed Apr 10, 2021
    Configuration menu
    Copy the full SHA
    244ef65 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2021

  1. Comma separated list of spark configurations 

    Signed-off-by: Tim Liu <timl@nvidia.com>
    @NvTimLiu
    NvTimLiu committed Apr 13, 2021
    Configuration menu
    Copy the full SHA
    4631bb9 View commit details
    Browse the repository at this point in the history

  2. Add a comment to make the '-f' format clear

    @NvTimLiu
    NvTimLiu committed Apr 13, 2021
    Configuration menu
    Copy the full SHA
    0367a2b View commit details
    Browse the repository at this point in the history

  3. Add a comment to make the '-f' format clear

    @NvTimLiu
    NvTimLiu committed Apr 13, 2021
    Configuration menu
    Copy the full SHA
    0331f3a View commit details
    Browse the repository at this point in the history

  4. Fix typo

    @NvTimLiu
    NvTimLiu committed Apr 13, 2021
    Configuration menu
    Copy the full SHA
    1353df7 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2021

  1. Add '--conf' if the SPARK_CONF is not empty

    @NvTimLiu
    NvTimLiu committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    48f136d View commit details
    Browse the repository at this point in the history