Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publish nightly wheels to NVIDIA index instead of PyPI #1294

Merged
merged 7 commits into from
Dec 6, 2023
Merged

Publish nightly wheels to NVIDIA index instead of PyPI #1294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
72072c9
Publish nightly wheels to NVIDIA index instead of PyPI
pentschev Dec 5, 2023
936ca9c
Update PR workflow
pentschev Dec 5, 2023
33634a9
Fix build wheel path
pentschev Dec 5, 2023
67dfdc9
Update `VERSION` and `dask_cuda/_version.py` as before
pentschev Dec 5, 2023
d0d2d00
Add missing `commit` entry
pentschev Dec 5, 2023
1af8e50
Inline `package_name` variable
pentschev Dec 5, 2023
1b9c928
Pull alphas when building nightlies
pentschev Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,22 +58,23 @@ jobs:
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
wheel-build:
runs-on: ubuntu-latest
container:
image: rapidsai/ci-conda:latest
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Build wheel
run: ci/build_python_pypi.sh
env:
GH_TOKEN: ${{ github.token }}
- name: Publish distribution 📦 to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.RAPIDSAI_PYPI_TOKEN }}
skip-existing: true
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.02
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
sha: ${{ inputs.sha }}
date: ${{ inputs.date }}
script: ci/build_wheel.sh
# Package is pure Python and only ever requires one build.
matrix_filter: map(select(.ARCH == "amd64" and .PY_VER == "3.10" and .CUDA_VER == "12.0.1"))
Comment on lines +69 to +70
Copy link
Member

@ajschmidt8 ajschmidt8 Dec 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pentschev, you can generalize this with the following:

matrix_filter: map(select(.ARCH=="amd64")) | sort_by(.CUDA_VER, (.PY_VER | split(".") | map(tonumber))) | [.[-1]]

That will filter the matrix to only include amd64 machines, sort those machines by CUDA_VER and PY_VER, and then select the last element (which will have the latest CUDA and Python version).

You can optionally take this a step further to switch from a centos7/rockylinux image to an Ubuntu image with this minor amendment:

matrix_filter: map(select(.ARCH=="amd64")) | sort_by(.CUDA_VER, (.PY_VER | split(".") | map(tonumber))) | [.[-1] + {LINUX_VER: "ubuntu22.04"}]

Though this will require the Ubuntu version to be updated periodically as RAPIDS rotates CI image versions.

Copy link
Member

@ajschmidt8 ajschmidt8 Dec 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if you don't care about the CUDA_VER, then this can be simplified to:

matrix_filter: map(select(.ARCH=="amd64")) | sort_by((.PY_VER | split(".") | map(tonumber))) | [.[-1]]

and

matrix_filter: map(select(.ARCH=="amd64")) | sort_by((.PY_VER | split(".") | map(tonumber))) | [.[-1] + {LINUX_VER: "ubuntu22.04"}]

Here's one final variation that can be used too, which sorts the entries by PY_VER, gets the last entry, and then overrides the ARCH and LINUX_VER values:

matrix_filter: sort_by((.PY_VER | split(".") | map(tonumber))) | [.[-1] + {ARCH: "amd64", LINUX_VER: "ubuntu22.04"}]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Dask-CUDA is a pure Python package (agnostic to OS, Python version, CUDA version, etc.), think this is just selecting one build arbitrarily

Is the idea of these suggestions just to avoid hard-coding version information in the selector?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. the hardcoded Python and CUDA versions will eventually be invalid as we update to newer versions. these generalized snippets prevent us from having to upgrade these versions in the future

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed as issue: https://github.com/rapidsai/dask-cuda/issues/1296

wheel-publish:
needs: wheel-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.02
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
sha: ${{ inputs.sha }}
date: ${{ inputs.date }}
package-name: dask-cuda
22 changes: 7 additions & 15 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,18 +45,10 @@ jobs:
container_image: "rapidsai/ci-conda:latest"
run_script: "ci/build_docs.sh"
wheel-build:
needs: checks
runs-on: ubuntu-latest
container:
image: rapidsai/ci-conda:latest
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Build wheel
run: ci/build_python_pypi.sh
env:
GH_TOKEN: ${{ github.token }}
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.02
with:
build_type: pull-request
# Package is pure Python and only ever requires one build.
matrix_filter: map(select(.ARCH == "amd64" and .PY_VER == "3.10" and .CUDA_VER == "12.0.1"))
script: "ci/build_wheel.sh"
42 changes: 0 additions & 42 deletions ci/build_python_pypi.sh
View file
Open in desktop

This file was deleted.

28 changes: 28 additions & 0 deletions ci/build_wheel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash
# Copyright (c) 2023, NVIDIA CORPORATION.

set -euo pipefail

source rapids-configure-sccache
source rapids-date-string

version=$(rapids-generate-version)
commit=$(git rev-parse HEAD)

echo "${version}" | tr -d '"' > VERSION
sed -i "/^__git_commit__/ s/= .*/= \"${commit}\"/g" "dask_cuda/_version.py"

# For nightlies we want to ensure that we're pulling in alphas as well. The
# easiest way to do so is to augment the spec with a constraint containing a
# min alpha version that doesn't affect the version bounds but does allow usage
# of alpha versions for that dependency without --pre
alpha_spec=''
if ! rapids-is-release-build; then
alpha_spec=',>=0.0.0a0'
fi

sed -r -i "s/rapids-dask-dependency==(.*)\"/rapids-dask-dependency==\1${alpha_spec}\"/g" pyproject.toml

python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check

RAPIDS_PY_WHEEL_NAME="dask-cuda" rapids-upload-wheels-to-s3 dist