Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move creation of env.yaml outside the current directory #14476

Merged
merged 10 commits into from
Dec 12, 2023
Merged

Move creation of env.yaml outside the current directory #14476

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ad6b513
Move creation of env.yaml outside local repo directory
davidwendt Nov 22, 2023
84e7bab
replace PWD with temp-dir for test-results
davidwendt Nov 29, 2023
4f618f6
Merge branch 'branch-24.02' into move-env-yaml
davidwendt Nov 29, 2023
8821590
Merge branch 'branch-24.02' into move-env-yaml
davidwendt Nov 30, 2023
a2f4e67
Merge branch 'branch-24.02' into move-env-yaml
davidwendt Dec 1, 2023
1b9f4fb
remove PWD from test_python_common.sh
davidwendt Dec 1, 2023
0380249
Merge branch 'branch-24.02' into move-env-yaml
davidwendt Dec 5, 2023
f169eaa
Merge branch 'move-env-yaml' of github.com:davidwendt/cudf into move-…
davidwendt Dec 7, 2023
0282a1d
Merge branch 'branch-24.02' into move-env-yaml
davidwendt Dec 7, 2023
3a88e95
add double-quotes around dir var
davidwendt Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ci/build_docs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@ set -euo pipefail
rapids-logger "Create test conda environment"
. /opt/conda/etc/profile.d/conda.sh

ENV_YAML_DIR="$(mktemp -d)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not strictly necessary since these scripts almost always run in a container, but cleaning up temp directories isn't a bad idea.

Suggested change
ENV_YAML_DIR="$(mktemp -d)"
ENV_YAML_DIR="$(mktemp -d)"
trap 'rm -rf -- "$ENV_YAML_DIR"' EXIT

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't temp directories cleared on boot automatically? I'd vote to omit this extra line to keep things as succinct as possible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plus like you mentioned, 99% of the time, these scripts are run in containers.


rapids-dependency-file-generator \
--output conda \
--file_key docs \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --force -f env.yaml -n docs
rapids-mamba-retry env create --force -f "${ENV_YAML_DIR}/env.yaml" -n docs
conda activate docs

rapids-print-env
Expand Down
6 changes: 4 additions & 2 deletions ci/check_style.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@ set -euo pipefail
rapids-logger "Create checks conda environment"
. /opt/conda/etc/profile.d/conda.sh

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file_key checks \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --force -f env.yaml -n checks
rapids-mamba-retry env create --force -f "${ENV_YAML_DIR}/env.yaml" -n checks
conda activate checks

FORMAT_FILE_URL=https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-24.02/cmake-format-rapids-cmake.json
Expand Down
10 changes: 7 additions & 3 deletions ci/test_cpp_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,24 @@ set -euo pipefail
. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Generate C++ testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file_key test_cpp \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --force -f env.yaml -n test
rapids-mamba-retry env create --force -f "${ENV_YAML_DIR}/env.yaml" -n test

# Temporarily allow unbound variables for conda activation.
set +u
conda activate test
set -u

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}/
RESULTS_DIR=${RAPIDS_TESTS_DIR:-"$(mktemp -d)"}
RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${RESULTS_DIR}/test-results"}/
mkdir -p "${RAPIDS_TESTS_DIR}"

rapids-print-env
Expand Down
7 changes: 5 additions & 2 deletions ci/test_java.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ set -euo pipefail
. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Generate Java testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file_key test_java \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --force -f env.yaml -n test
rapids-mamba-retry env create --force -f "${ENV_YAML_DIR}/env.yaml" -n test

export CMAKE_GENERATOR=Ninja

Expand Down
7 changes: 5 additions & 2 deletions ci/test_notebooks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ set -euo pipefail
. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Generate notebook testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file_key test_notebooks \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --force -f env.yaml -n test
rapids-mamba-retry env create --force -f "${ENV_YAML_DIR}/env.yaml" -n test

# Temporarily allow unbound variables for conda activation.
set +u
Expand Down
12 changes: 8 additions & 4 deletions ci/test_python_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,15 @@ set -euo pipefail
. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Generate Python testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file_key test_python \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --force -f env.yaml -n test
rapids-mamba-retry env create --force -f "${ENV_YAML_DIR}/env.yaml" -n test

# Temporarily allow unbound variables for conda activation.
set +u
Expand All @@ -24,8 +27,9 @@ rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}
RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${PWD}/coverage-results"}
RESULTS_DIR=${RAPIDS_TESTS_DIR:-"$(mktemp -d)"}
RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${RESULTS_DIR}/test-results"}/
RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${RESULTS_DIR}/coverage-results"}
mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}"

rapids-print-env
Expand Down