Skip to content

Commit

Permalink
Initiate arm64 CI support
Browse files Browse the repository at this point in the history
Signed-off-by: Peixin Li <pxli@nyu.edu>
  • Loading branch information
pxLi committed Sep 27, 2023
1 parent 2dc4d5b commit de1a6d2
Show file tree
Hide file tree
Showing 2 changed files with 84 additions and 0 deletions.
10 changes: 10 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,16 @@ flag if cross-compilation is required.
mvn clean verify -Dbuildver=330 -P<jdk11|jdk17>
```
### Building and Testing with ARM
To build our project on ARM platform, please add `-Parm64` to your maven commands.
NOTE: Build process does not require arm machine, so if you want to build the artifacts only
on X86 machine, please also add `-DskipTests` in commands.
```bash
mvn clean verify -Dbuildver=311 -Parm64
```
### Iterative development during local testing
When iterating on changes impacting the `dist` module artifact directly or via
Expand Down
74 changes: 74 additions & 0 deletions jenkins/Dockerfile-blossom.multi
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

###
#
# Dockerfile for multi-platform build
#
# Arguments Example:
# CUDA_VER=11.8.0
# UBUNTU_VER=20.04
# UCX_VER=1.15.0-rc6
###

ARG CUDA_VER=11.8.0
ARG UBUNTU_VER=20.04
ARG UCX_VER=1.15.0-rc6
# multi-platform build with: docker buildx build --platform linux/arm64,linux/amd64 <ARGS> on either amd64 or arm64 host
# check available offcial arm-based docker images at https://hub.docker.com/r/nvidia/cuda/tags (OS/ARCH)
FROM --platform=$TARGETPLATFORM nvidia/cuda:${CUDA_VER}-runtime-ubuntu${UBUNTU_VER}
ARG CUDA_VER
ARG UBUNTU_VER
ARG UCX_VER
ARG ARCH=arm64
ARG UCX_ARCH=aarch64

# Install jdk-8, jdk-11, maven, docker image
RUN apt-get update -y && \
apt-get install -y software-properties-common rsync

RUN add-apt-repository ppa:deadsnakes/ppa && \
apt-get update -y && \
DEBIAN_FRONTEND="noninteractive" apt-get install -y maven \
openjdk-8-jdk openjdk-11-jdk python3.9 python3.9-distutils python3-setuptools tzdata git zip unzip wget

# apt python3-pip would install pip for OS default python3 version only
# like for ubuntu 18.04, it would only install pip for python3.6
# so we install pip for specific python version explicitly
RUN wget https://bootstrap.pypa.io/get-pip.py && python3.9 get-pip.py

# Set default jdk as 1.8.0
RUN update-java-alternatives --set /usr/lib/jvm/java-1.8.0-openjdk-${ARCH}

RUN ln -sfn /usr/bin/python3.9 /usr/bin/python
RUN ln -sfn /usr/bin/python3.9 /usr/bin/python3
RUN python -m pip install pytest sre_yield requests pandas pyarrow findspark pytest-xdist pre-commit pytest-order

# libnuma1 and libgomp1 are required by ucx packaging
RUN apt install -y inetutils-ping expect wget libnuma1 libgomp1

RUN UCX_CUDA_VER=`echo ${CUDA_VER} | cut -d '.' -f1` && \
mkdir -p /tmp/ucx && cd /tmp/ucx && \
wget https://github.com/openucx/ucx/releases/download/v${UCX_VER}/ucx-${UCX_VER}-ubuntu${UBUNTU_VER}-mofed5-cuda${UCX_CUDA_VER}-${UCX_ARCH}.tar.bz2 && \
tar -xvf *.bz2 && dpkg -i *.deb && \
rm -rf /tmp/ucx

# export JAVA_HOME for mvn option 'source-javadoc'
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-${ARCH}

# install locale and generate UTF-8, but don't set it as default, the test cases should specify it somehow
RUN apt-get install -y locales
RUN locale-gen en_US.UTF-8

0 comments on commit de1a6d2

Please sign in to comment.