Skip to content

Commit

Permalink
Add some more checks to databricks build scripts (#721)
Browse files Browse the repository at this point in the history
* Add some more checks to databricks build scripts

Signed-off-by: Thomas Graves <tgraves@nvidia.com>

* remove extra newline

* use the right -gt for bash
  • Loading branch information
tgravescs authored Sep 10, 2020
1 parent e137226 commit 9582bc8
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions jenkins/databricks/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,9 @@ mvn -B '-Pdatabricks,!snapshot-shims' clean package -DskipTests

# Copy so we pick up new built jar and latesty CuDF jar. Note that the jar names has to be
# exactly what is in the staticly setup Databricks cluster we use.
echo "Copying rapids jars: dist/target/$RAPIDS_BUILT_JAR $DB_RAPIDS_JAR_LOC"
sudo cp dist/target/$RAPIDS_BUILT_JAR $DB_RAPIDS_JAR_LOC
echo "Copying cudf jars: $CUDF_JAR $DB_CUDF_JAR_LOC"
sudo cp $CUDF_JAR $DB_CUDF_JAR_LOC

# tests
Expand All @@ -106,6 +108,16 @@ sudo ln -s /databricks/jars/ $SPARK_HOME/jars || true
sudo chmod 777 /databricks/data/logs/
sudo chmod 777 /databricks/data/logs/*
echo { \"port\":\"15002\" } > ~/.databricks-connect
if [ `ls $DB_JAR_LOC/rapids* | wc -l` -gt 1 ]; then
echo "ERROR: Too many rapids jars in $DB_JAR_LOC"
ls $DB_JAR_LOC/rapids*
exit 1
fi
if [ `ls $DB_JAR_LOC/cudf* | wc -l` -gt 1 ]; then
echo "ERROR: Too many cudf jars in $DB_JAR_LOC"
ls $DB_JAR_LOC/cudf*
exit 1
fi
$SPARK_HOME/bin/spark-submit ./runtests.py --runtime_env="databricks"
cd /home/ubuntu
tar -zcvf spark-rapids-built.tgz spark-rapids

0 comments on commit 9582bc8

Please sign in to comment.