Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arm64 profile to build arm artifacts #9273

Merged
merged 1 commit into from
Sep 21, 2023
Merged

Conversation

pxLi
Copy link
Collaborator

@pxLi pxLi commented Sep 20, 2023

follow-up of NVIDIA/spark-rapids-jni#1385 to add arm64 profile to support build and test against arm experimentally

This change would,

  1. Add arm64 profile to build new plugin artifacts with arm-based JNI artifact (currently only available on URM internally)
  2. Do not change the deployment process for now to avoid affecting existing development and release (x86).

When we decide to officially support arm in plugin, and we will have some updates for build and deploy setup to include arm artifacts as part of regular deploy&release. Filed a ticket to track in 23.12 or later #9274

e.g. command like

mvn -Dmaven.wagon.http.retryHandler.count=3 -U -B -s jenkins/settings.xml -P mirror-apache-to-urm,arm64 -Dbuildver=311 package -Drat.skip=true -Dskip -Dmaven.scalastyle.skip=true -Dcuda.version=cuda11 -DskipTests

would produce

 % ll dist/target/*.jar
-rw-rw-r-- 1 peixinl peixinl 449M Sep 20 05:27 dist/target/rapids-4-spark_2.12-23.10.0-SNAPSHOT-cuda11-arm64.jar

in the artifact it would include the arm64-based JNI objects

% ll -R aarch64
aarch64:
total 4.0K
drwxrwxr-x 2 peixinl peixinl 4.0K Sep 20 05:50 Linux

aarch64/Linux:
total 690M
-rw-r--r-- 1 peixinl peixinl  68K Sep 20 05:26 libcudfjni.so
-rw-r--r-- 1 peixinl peixinl 646M Sep 20 05:26 libcudf.so
-rw-r--r-- 1 peixinl peixinl  21M Sep 20 05:26 libnvcomp_bitcomp.so
-rw-r--r-- 1 peixinl peixinl  11M Sep 20 05:26 libnvcomp_gdeflate.so
-rw-r--r-- 1 peixinl peixinl  14M Sep 20 05:26 libnvcomp.so

@pxLi pxLi added the build Related to CI / CD or cleanly building label Sep 20, 2023
@pxLi pxLi marked this pull request as draft September 20, 2023 07:18
Signed-off-by: Peixin Li <pxli@nyu.edu>
@pxLi
Copy link
Collaborator Author

pxLi commented Sep 20, 2023

build

@pxLi pxLi marked this pull request as ready for review September 20, 2023 07:51
@tgravescs
Copy link
Collaborator

contributing.md doc should be updated with new profile and instructions

@pxLi
Copy link
Collaborator Author

pxLi commented Sep 20, 2023

contributing.md doc should be updated with new profile and instructions

Sure, I will add some simple instructions later but since arm support is still in experimental stage internally,
this change targets firstly providing support for internal usage to try expose problems earlier.
Currently we are not planning to deploy arm-based jni to sonatype repo regularly, so external development support is limited in 23.10, I will add more for the cases after #9274

@pxLi
Copy link
Collaborator Author

pxLi commented Sep 21, 2023

I will merge this one to setup the internal arm build and test pipelines (JNI+Plugin build + sanity tests) first

Updated ticket #9274 to keep tracking requirements to add deployment to sonatype for arm-based JNI nightly artifacts and update instructions of arm build in contributing.md

@pxLi pxLi merged commit 01fce6c into NVIDIA:branch-23.10 Sep 21, 2023
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants