Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use specific Android NDK version in CI builds. #12350

Merged
merged 3 commits into from
Jul 28, 2022
Merged

Use specific Android NDK version in CI builds. #12350

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a1ae347
Add installation of specific NDK version to Android CI build.
edgchen1 Jul 27, 2022
8e411fe
Add default value for version parameter, rename template, use it in p…
edgchen1 Jul 27, 2022
023067d
Add NDK template to packaging pipeline testing job.
edgchen1 Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions tools/ci_build/github/azure-pipelines/android-x86_64-crosscompile-ci-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ jobs:
- script: sudo apt-get update -y && sudo apt-get install -y coreutils ninja-build
displayName: Install coreutils and ninja

- template: "templates/use-android-ndk.yml"

- script: /bin/bash tools/ci_build/github/android/setup_gradle_wrapper.sh $(pwd)
displayName: Setup gradle wrapper to use gradle 6.8.3

Expand All @@ -35,12 +37,8 @@ jobs:
displayName: Build Host Protoc

- script: |
export ANDROID_SDK_ROOT=/usr/local/lib/android/sdk
export ANDROID_HOME=/usr/local/lib/android/sdk
export ANDROID_NDK_HOME=/usr/local/lib/android/sdk/ndk-bundle
export ANDROID_NDK_ROOT=/usr/local/lib/android/sdk/ndk-bundle
env | grep ANDROID
displayName: Set Android ENVs
displayName: View Android ENVs

# Start switching to jdk 11 after the Android Emulator is started
# since Android SDK manager requires java 8
Expand Down Expand Up @@ -116,6 +114,13 @@ jobs:
artifact: 'CPUBuildOutput'
path: $(Build.SourcesDirectory)

- task: UsePythonVersion@0
displayName: Use Python $(pythonVersion)
inputs:
versionSpec: $(pythonVersion)

- template: "templates/use-android-ndk.yml"

- script: /bin/bash tools/ci_build/github/android/setup_gradle_wrapper.sh $(pwd)
displayName: Setup gradle wrapper to use gradle 6.8.3

Expand Down Expand Up @@ -173,6 +178,8 @@ jobs:
- script: sudo apt-get update -y && sudo apt-get install -y coreutils ninja-build
displayName: Install coreutils and ninja

- template: "templates/use-android-ndk.yml"

- script: /bin/bash tools/ci_build/github/android/setup_gradle_wrapper.sh $(pwd)
displayName: Setup gradle wrapper to use gradle 6.8.3

Expand All @@ -185,12 +192,8 @@ jobs:
displayName: Build Host Protoc

- script: |
export ANDROID_SDK_ROOT=/usr/local/lib/android/sdk
export ANDROID_HOME=/usr/local/lib/android/sdk
export ANDROID_NDK_HOME=/usr/local/lib/android/sdk/ndk-bundle
export ANDROID_NDK_ROOT=/usr/local/lib/android/sdk/ndk-bundle
env | grep ANDROID
displayName: set Android ENVs
displayName: View Android ENVs

# Start switching to jdk 11 after the Android Emulator is started since Android SDK manager requires java 8
- task: JavaToolInstaller@0
Expand Down Expand Up @@ -272,6 +275,8 @@ jobs:
inputs:
versionSpec: $(pythonVersion)

- template: "templates/use-android-ndk.yml"

- script: /bin/bash tools/ci_build/github/android/setup_gradle_wrapper.sh $(pwd)
displayName: Setup gradle wrapper to use gradle 6.8.3

Expand Down Expand Up @@ -350,6 +355,8 @@ jobs:
inputs:
versionSpec: $(pythonVersion)

- template: "templates/use-android-ndk.yml"

- script: /bin/bash tools/ci_build/github/android/setup_gradle_wrapper.sh $(pwd)
displayName: Setup gradle wrapper to use gradle 6.8.3

Expand Down
2 changes: 2 additions & 0 deletions tools/ci_build/github/azure-pipelines/templates/android-java-api-aar.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ jobs:
DockerBuildArgs: "--build-arg BUILD_UID=$( id -u )"
Repository: onnxruntimecpubuild

- template: use-android-ndk.yml

- task: CmdLine@2
displayName: Build Android AAR Packages
inputs:
Expand Down
30 changes: 30 additions & 0 deletions tools/ci_build/github/azure-pipelines/templates/use-android-ndk.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Installs the Android NDK and sets environment variables ANDROID_NDK_HOME and ANDROID_NDK_ROOT to refer to it.

parameters:
- name: AndroidNdkVersion
type: string
default: "25.0.8775105" # LTS version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand: in my test change, when it failed I got an error message with "No version of NDK matched the requested version 21.0.6113669. Versions available locally: 21.4.7075529, 23.2.8568313, 24.0.8215888". So, 25.0 isn't in the list. How does it work for you?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we install NDK 25.0 and set the relevant environment variables in this template. it's done for both Linux and Mac jobs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're using a version that is not already available in the env (which I'm guessing might be the case based on Changming's list) is there any concern about network usage? The zipped windows NDK is 446MB, so a lot to download if every time the CI runs it has to do that once.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, it's not available in the build machine env at the moment. in the Android CI pipeline, it takes about 20 seconds to install on the Linux machines and around a minute on the Mac machines.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a hot fix, it looks good to me. I still hope we don't need to download the SDK every time. It's not only about network spendings, the more we download, the more likely network errors could occur.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can pick another version too, as long as we use the same one consistently. I just picked 25.0.8775105 because it sounds like it is supported.
https://developer.android.com/ndk/downloads

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actions/runner-images#5930
looks like 25 will be added soon to the hosted agents


steps:
- bash: |
set -e

"${ANDROID_SDK_ROOT}/cmdline-tools/latest/bin/sdkmanager" --install "ndk;${{ parameters.AndroidNdkVersion }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the sdk is already installed? Would it cause any error?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested locally, was able to run it multiple times without error

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to check if the version is already installed? This way when version 25 is added to the hosted agents we dont need to make any changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the installation is a hot fix. It should be removed while the upstream image become stable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may not need to make changes when 25 is already installed on the hosted agent if running the install command again succeeds.
we can update it to not attempt to install anything at all eventually if that is desired, but I would like to have the NDK version that we actually use be more visible.


NDK_PATH="${ANDROID_SDK_ROOT}/ndk/${{ parameters.AndroidNdkVersion }}"
if [[ ! -d "${NDK_PATH}" ]]; then
echo "NDK directory is not in expected location: ${NDK_PATH}"
exit 1
fi

set_var() {
local VAR_NAME=${1:?}
local VAR_VALUE=${2:?}
echo "##vso[task.setvariable variable=${VAR_NAME}]${VAR_VALUE}"
echo "${VAR_NAME}: ${VAR_VALUE}"
}

set_var "ANDROID_NDK_HOME" "${NDK_PATH}"
set_var "ANDROID_NDK_ROOT" "${NDK_PATH}"

displayName: Use Android NDK ${{ parameters.AndroidNdkVersion }}