Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add some more checks to databricks build scripts #721

Merged
merged 3 commits into from
Sep 10, 2020
Merged

Add some more checks to databricks build scripts #721

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
275a2c4
Add some more checks to databricks build scripts
tgravescs Sep 10, 2020
ddbd1fa
remove extra newline
tgravescs Sep 10, 2020
72fac13
use the right -gt for bash
tgravescs Sep 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions jenkins/databricks/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,9 @@ mvn -B '-Pdatabricks,!snapshot-shims' clean package -DskipTests

# Copy so we pick up new built jar and latesty CuDF jar. Note that the jar names has to be
# exactly what is in the staticly setup Databricks cluster we use.
echo "Copying rapids jars: dist/target/$RAPIDS_BUILT_JAR $DB_RAPIDS_JAR_LOC"
sudo cp dist/target/$RAPIDS_BUILT_JAR $DB_RAPIDS_JAR_LOC
echo "Copying cudf jars: $CUDF_JAR $DB_CUDF_JAR_LOC"
sudo cp $CUDF_JAR $DB_CUDF_JAR_LOC

# tests
Expand All @@ -106,6 +108,16 @@ sudo ln -s /databricks/jars/ $SPARK_HOME/jars || true
sudo chmod 777 /databricks/data/logs/
sudo chmod 777 /databricks/data/logs/*
echo { \"port\":\"15002\" } > ~/.databricks-connect
if [ `ls $DB_JAR_LOC/rapids* | wc -l` -gt 1 ]; then
echo "ERROR: Too many rapids jars in $DB_JAR_LOC"
ls $DB_JAR_LOC/rapids*
exit 1
fi
if [ `ls $DB_JAR_LOC/cudf* | wc -l` -gt 1 ]; then
echo "ERROR: Too many cudf jars in $DB_JAR_LOC"
ls $DB_JAR_LOC/cudf*
exit 1
fi
$SPARK_HOME/bin/spark-submit ./runtests.py --runtime_env="databricks"
cd /home/ubuntu
tar -zcvf spark-rapids-built.tgz spark-rapids