Publish nightly wheels to NVIDIA index instead of PyPI #1294
Conversation
pentschev
added
bug
pentschev
force-pushed
the
update-nightly-wheels-publishing
branch
from
d4cb930
to
72072c9
Compare
There was a problem hiding this comment.
Looks like we also need to switch to build_wheel.sh in the PR workflow?
dask-cuda/.github/workflows/pr.yaml
Line 60 in 6de222f
|
Good catch @charlesbluca , updated. |
There was a problem hiding this comment.
Looks like the wheel build is still failing - my guess is that this is because the provided nightly version of 24.02.00a22 isn't PEP440 compliant? Perhaps we need to do something like #1279 here?
You're right, but honestly I don't know what's the "right way" to fix this. The version being generated |
|
On second thought, looks like maybe what we want to do here is bump the version in dask_cuda/VERSION rather than in the pyproject file, similar to what's happening here: |
|
/merge |
| # Package is pure Python and only ever requires one build. | ||
| matrix_filter: map(select(.ARCH == "amd64" and .PY_VER == "3.10" and .CUDA_VER == "12.0.1")) |
There was a problem hiding this comment.
@pentschev, you can generalize this with the following:
matrix_filter: map(select(.ARCH=="amd64")) | sort_by(.CUDA_VER, (.PY_VER | split(".") | map(tonumber))) | [.[-1]]That will filter the matrix to only include amd64 machines, sort those machines by CUDA_VER and PY_VER, and then select the last element (which will have the latest CUDA and Python version).
You can optionally take this a step further to switch from a centos7/rockylinux image to an Ubuntu image with this minor amendment:
matrix_filter: map(select(.ARCH=="amd64")) | sort_by(.CUDA_VER, (.PY_VER | split(".") | map(tonumber))) | [.[-1] + {LINUX_VER: "ubuntu22.04"}]Though this will require the Ubuntu version to be updated periodically as RAPIDS rotates CI image versions.
There was a problem hiding this comment.
Also, if you don't care about the CUDA_VER, then this can be simplified to:
matrix_filter: map(select(.ARCH=="amd64")) | sort_by((.PY_VER | split(".") | map(tonumber))) | [.[-1]]and
matrix_filter: map(select(.ARCH=="amd64")) | sort_by((.PY_VER | split(".") | map(tonumber))) | [.[-1] + {LINUX_VER: "ubuntu22.04"}]Here's one final variation that can be used too, which sorts the entries by PY_VER, gets the last entry, and then overrides the ARCH and LINUX_VER values:
matrix_filter: sort_by((.PY_VER | split(".") | map(tonumber))) | [.[-1] + {ARCH: "amd64", LINUX_VER: "ubuntu22.04"}]
There was a problem hiding this comment.
As Dask-CUDA is a pure Python package (agnostic to OS, Python version, CUDA version, etc.), think this is just selecting one build arbitrarily
Is the idea of these suggestions just to avoid hard-coding version information in the selector?
There was a problem hiding this comment.
yes. the hardcoded Python and CUDA versions will eventually be invalid as we update to newer versions. these generalized snippets prevent us from having to upgrade these versions in the future
There was a problem hiding this comment.
Filed as issue: https://github.com/rapidsai/dask-cuda/issues/1296
Nightly wheels also require `rapids-dask-dependency` which is only available in NVIDIA's PyPI index and cannot be published to PyPI as it installs Dask/Distributed from GitHub, which is forbidden by PyPI. Therefore, we're switching to publishing nightlies only to NVIDIA index as it doesn't seem external projects currently rely on nightlies. Release packages will continue to be published to PyPI. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Charles Blackmon-Luca (https://github.com/charlesbluca) - Ray Douglass (https://github.com/raydouglass) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#1294
Nightly wheels also require
rapids-dask-dependencywhich is only available in NVIDIA's PyPI index and cannot be published to PyPI as it installs Dask/Distributed from GitHub, which is forbidden by PyPI. Therefore, we're switching to publishing nightlies only to NVIDIA index as it doesn't seem external projects currently rely on nightlies. Release packages will continue to be published to PyPI.