Skip to content

Commit

Permalink
Add dynamic configs for the spark-rapids IT pipelines (NVIDIA#1450)
Browse files Browse the repository at this point in the history
Add dynamic spark-rapids configs for the IT pipelines via parameter:
'BASE_SPARK_SUBMIT_ARGS=--conf abc=xxx --conf xyz=xxx --conf def=xxx ...'.

e.g. We can enable spark 'AQE' for spark-rapids integration tests pipeline by:
    'BASE_SPARK_SUBMIT_ARGS=--conf spark.sql.adaptive.enabled=true'

The default value 'BASE_SPARK_SUBMIT_ARGS' is unset

Signed-off-by: Tim Liu <timl@nvidia.com>
  • Loading branch information
NvTimLiu authored Jan 7, 2021
1 parent e16493b commit 53442d5
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion jenkins/spark-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@ OUTPUT="$WORKSPACE/output"
# spark.sql.cache.serializer conf is ignored for versions prior to 3.1.0
SERIALIZER="--conf spark.sql.cache.serializer=com.nvidia.spark.rapids.shims.spark310.ParquetCachedBatchSerializer"

BASE_SPARK_SUBMIT_ARGS="--master spark://$HOSTNAME:7077 \
BASE_SPARK_SUBMIT_ARGS="$BASE_SPARK_SUBMIT_ARGS \
--master spark://$HOSTNAME:7077 \
--executor-memory 12G \
--total-executor-cores 6 \
--conf spark.sql.shuffle.partitions=12 \
Expand Down

0 comments on commit 53442d5

Please sign in to comment.