From 49c169fe930eef526b8c55da938da77995bd7fa3 Mon Sep 17 00:00:00 2001 From: Devon Ryan Date: Thu, 17 Feb 2022 09:31:29 +0100 Subject: [PATCH 1/6] Update local building instructions. Xref #736 --- docs/source/contributor/build-system.rst | 19 ++--- docs/source/contributor/building-locally.rst | 80 ++++---------------- 2 files changed, 24 insertions(+), 75 deletions(-) diff --git a/docs/source/contributor/build-system.rst b/docs/source/contributor/build-system.rst index 235fa886ea..9caa4a0313 100644 --- a/docs/source/contributor/build-system.rst +++ b/docs/source/contributor/build-system.rst @@ -3,7 +3,7 @@ Build system The build system for Bioconda takes recipes and converts them into conda packages that are uploaded to anaconda.org as well as Docker containers that -are uploaded to quay.io. All of this happens in a transparent way, with all +are uploaded to quay.io as part of the Biocontainers project. All of this happens in a transparent way, with all build logs available for inspection. The code for the build system can be found in `bioconda-utils `_, but parts are also with the ``bioconda-recipes`` repo. This document serves as @@ -11,7 +11,7 @@ a high-level overview; the code remains the authoritative source on exactly what happens during a build. Why so complicated? We have to work within the constraints of conda-build, -Docker, and CircleCI, while simultaneously supporting the same build system on +Docker, and Azure, while simultaneously supporting the same build system on a local machine so contributors can test. We also have isolated bioconda-utils from bioconda-recipes to better facilitate testing of the infrastructure, and to (one day!) make it general enough that others can use the framework for @@ -22,28 +22,29 @@ simplify, and do so where we can. Stages of a bioconda build -------------------------- A GitHub pull request, or any pushed changes to a GitHub pull request, triggers -a new build on CircleCI. One build can contain mulitple recipes, limited only -by the time limit imposed by CircleCI (they have generously given us an -extended build time). Each build on CircleCI starts with a fresh VM, so we +a new build on Azure DevOps. One build can contain mulitple recipes, limited only +by the time limit imposed by Azure. +Each build on Azure starts with a fresh VM, so we need to create the entire bioconda-build system environment from scratch for each build. -When testing locally with ``circleci build``, we use the +When testing locally, we use the ``quay.io/bioconda/bioconda-utils-build-env-cos7`` Docker container to avoid changing the local system. This container is defined by `this Dockerfile `_. -Otherwise, when running on CircleCI, a new Linux or MacOS VM is created for +Otherwise, when running on Azure, a new Linux or MacOS VM is created for each build. -The steps are orchestrated by the `circleci config file -`_. +The steps are orchestrated by the `Azure config file `_. Configure the environment ~~~~~~~~~~~~~~~~~~~~~~~~~ +N.B., due to transitioning to Azure, the remainder of this section is no longer relevant and requires rewritting. + - Configure the CI environment: - ``bioconda-recipes: .circleci/config.yml`` is the primary configuration file for the steps that are run. See https://circleci.com/docs/2.0/ for diff --git a/docs/source/contributor/building-locally.rst b/docs/source/contributor/building-locally.rst index c2d9f40f47..e674dd8f30 100644 --- a/docs/source/contributor/building-locally.rst +++ b/docs/source/contributor/building-locally.rst @@ -1,7 +1,7 @@ Testing Recipes Locally ======================= -Queue times on CircleCI may sometimes make it more convenient and +Queue times on Azure DevOps may sometimes make it more convenient and faster to work on complex packages locally. There are several ways to do so, each with their own caveats. @@ -9,61 +9,20 @@ do so, each with their own caveats. :local: -.. _circleci-client: +.. _bioconda_utils: -Using the Circle CI Client -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Using bioconda-utils +~~~~~~~~~~~~~~~~~~~~ -You can execute an almost exact copy of our Linux build pipeline by -`installing the CircleCI client locally -`_ and running it from the -folder where your copy of ``bioconda-recipes`` resides:: +Whether on a CI node or locally, Bioconda packages are built and tested using ``bioconda-utils``. +You can install ``bioconda-utils`` locally by creating a new conda environment: - # Ensure the build container is up-to-date - docker pull quay.io/bioconda/bioconda-utils-build-env:latest +.. code-block:: - # Run the build locally - circleci build + # You can use "conda create" here instead, if you don't have mamba installed + mamba create -n bioconda -c conda-forge -c bioconda bioconda-utils -You will have to have "commited" some changes as described above. The -command will run our build system just as it is run online in a docker -container. It will detect which recipes where modified using the -``git`` history and proceed with linting and building those. - -Please note that this will only run the Linux build steps. - -You can use Docker volume bind-mounts to capture the local package -channel with the newly built packages:: - - mkdir /tmp/conda-bld - circleci build --volume /tmp/conda-bld:/opt/conda/conda-bld - -After a successful build, you can then install from the local channel by -providing the path to it:: - - conda install -c file:///tmp/conda-bld your-package - -Note that if the package also exists with the same version or a newer -version in remote channels, then you may need to specify the exact local -build string in your package spec to ensure your local build is -installed, e.g. ``your-package ==x.y.z build_string``. See -`conda-build:resources/package-spec` for more details on how to specify -packages. - - -.. _bootstrap: - -Using the "Bootstrap" Method -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Due to technical limitations of the Circle CI client, the above test does -**not** run the more stringent ``mulled-build`` tests. To do so, use the -following commands: - -.. code-block:: bash - - ./bootstrap.py /tmp/miniconda - source ~/.config/bioconda/activate + conda activate bioconda # optional linting bioconda-utils lint --git-range master @@ -73,14 +32,9 @@ following commands: The above commands do the following: -- Install a separate miniconda installation in a temporary directory: - - Set up bioconda channels, - - install bioconda-utils dependencies into the root environment of - that installation, - - and write the file ``~/.config/bioconda/activate`` -- Source that new file to specifically activate the root environment - of the new installation -- Run ``bioconda-utils`` in the new installation: +- Creates a new environment with bioconda-utils. +- Activates the new environment. You can later just start at ``conda activate bioconda`` +- Run ``bioconda-utils`` in the new environment: - The ``lint`` command will run the lint checks on your recipes - The ``build`` command will run the build pipeline @@ -99,13 +53,7 @@ The above commands do the following: to be included in the build container. If you do not have access to Docker, you can still run the basic test by -telling the bootstrap setup to not use docker, and by excluding the -``--docker`` and ``--mulled-test`` arguments in the last command:: - - ./bootstrap.py --no-docker /tmp/miniconda - source ~/.config/bioconda/activate - bioconda-utils build --git-range master - +omitting the ``--docker`` and ``--mulled-test`` options. Using the "Debug" Method ~~~~~~~~~~~~~~~~~~~~~~~~ From c178eefdc425faf2b51570993a335c2451e3abc8 Mon Sep 17 00:00:00 2001 From: Devon Ryan Date: Thu, 17 Feb 2022 09:51:27 +0100 Subject: [PATCH 2/6] Base image has been updated to python 3.9 it seems --- .circleci/config.yml | 3 ++- Dockerfile | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8399b0c062..907887756b 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -32,7 +32,8 @@ variables: macos: xcode: "9.4.1" linux: &linux - machine: true + machine: + image: ubuntu-2004:202201-02 install_bioconda_utils: &install_bioconda_utils run: name: Install bioconda-utils diff --git a/Dockerfile b/Dockerfile index 5a77726be6..e8e9484bac 100644 --- a/Dockerfile +++ b/Dockerfile @@ -28,6 +28,7 @@ RUN . /opt/conda/etc/profile.d/conda.sh && \ FROM base as build WORKDIR /tmp/repo COPY . ./ +RUN conda list RUN . /opt/conda/etc/profile.d/conda.sh && conda activate base && \ pip wheel . && \ mkdir - /opt/bioconda-utils && \ From 549bd4acceaccb96160ece20907b6d927a75f74d Mon Sep 17 00:00:00 2001 From: Devon Ryan Date: Thu, 17 Feb 2022 09:57:21 +0100 Subject: [PATCH 3/6] oops --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index e8e9484bac..251af127dd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -28,7 +28,7 @@ RUN . /opt/conda/etc/profile.d/conda.sh && \ FROM base as build WORKDIR /tmp/repo COPY . ./ -RUN conda list +RUN . /opt/conda/etc/profile.d/conda.sh && conda list RUN . /opt/conda/etc/profile.d/conda.sh && conda activate base && \ pip wheel . && \ mkdir - /opt/bioconda-utils && \ From 38e929309a03f9980502c4c40e1ed4c78da7a48d Mon Sep 17 00:00:00 2001 From: Devon Ryan Date: Thu, 17 Feb 2022 10:04:23 +0100 Subject: [PATCH 4/6] needs with --- .github/workflows/build-image.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-image.yml b/.github/workflows/build-image.yml index 2c64c56b7d..f831d2ace7 100644 --- a/.github/workflows/build-image.yml +++ b/.github/workflows/build-image.yml @@ -13,7 +13,8 @@ jobs: steps: - uses: actions/checkout@v2 - fetch-depth: 0 + with: + fetch-depth: 0 - id: get-tag run: | @@ -36,4 +37,4 @@ jobs: image='${{ steps.buildah-build.outputs.image }}' for tag in ${{ steps.buildah-build.outputs.tags }} ; do podman run --rm "${image}:${tag}" bioconda-utils --version - done \ No newline at end of file + done From 1a2e4033796d86125111e3443e8dee18ce783778 Mon Sep 17 00:00:00 2001 From: Devon Ryan Date: Thu, 17 Feb 2022 10:22:03 +0100 Subject: [PATCH 5/6] Update conda for support of newer python versions --- bioconda_utils/bioconda_utils-requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bioconda_utils/bioconda_utils-requirements.txt b/bioconda_utils/bioconda_utils-requirements.txt index 698769d90f..bf0c6a9ed9 100644 --- a/bioconda_utils/bioconda_utils-requirements.txt +++ b/bioconda_utils/bioconda_utils-requirements.txt @@ -4,7 +4,7 @@ # basics python>=3.7 -conda=4.8.4 +conda=4.11.0 conda-build=3.21.4 conda-verify=3.1.* argh=0.26.* # CLI From ef78a5700b5f5173e1c63ddd1ca38d9e1ee2d8e2 Mon Sep 17 00:00:00 2001 From: Devon Ryan Date: Fri, 18 Feb 2022 21:19:39 +0100 Subject: [PATCH 6/6] Update bioconda_utils-requirements.txt --- bioconda_utils/bioconda_utils-requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bioconda_utils/bioconda_utils-requirements.txt b/bioconda_utils/bioconda_utils-requirements.txt index bf0c6a9ed9..61bffc6a8e 100644 --- a/bioconda_utils/bioconda_utils-requirements.txt +++ b/bioconda_utils/bioconda_utils-requirements.txt @@ -20,7 +20,7 @@ boltons=18.* jsonschema=2.6.* # JSON schema verification # pinnings -conda-forge-pinning=2022.01.13.01.57.35 +conda-forge-pinning=2022.02.18.13.24.38 # tools anaconda-client=1.6.* # anaconda_upload