Publish RC container images (#7556)

* WIP * Add missing checkout * Add debuggin * Fix VAR name * Bug fix * Rework jobs * Revert "Rework jobs" This reverts commit 2bfa79f. * Add cache * Add temp default for testing * Add missing checkout * Fix patch * Comment out the GPG check for now * Rename polkadot_injected_release into a more appropriate polkadot_injected_debian * Refactoring / renaming * Introduce a generic image for binary injection * Flag files to be deleted and changes to be done * WIP * Fix multi binaries images * Add test build scripts * Remove old file, add polkadot build-injected script * Fix doc * Fix tagging * Add build of the injected container * Fix for docker * Remove the need for TTY * Handling container publishing * Fix owner and registry * Fix vars * Fix repo * Fix var naming * Fix case when there is no tag * Fix case with no tag * Handle error * Fix spacings * Fix tags * Remove unnecessary grep that may fail * Add final check * Clean up and introduce GPG check * Add doc * Add doc * Update doc/docker.md Co-authored-by: Mira Ressel <mira@parity.io> * type Co-authored-by: Mira Ressel <mira@parity.io> * Fix used VAR * Improve doc * ci: Update .build-push-image jobs to use the new build-injected.sh * ci: fix path to build-injected.sh script * Rename the release artifacts folder to prevent confusion due to a similar folder in the gitlab CI * ci: check out polkadot repo in .build-push-image This seems far cleaner than copying the entire scripts/ folder into our job artifacts. * feat(build-injected.sh): make PROJECT_ROOT configurable This lets us avoid a dependency on git in our CI image. * ci: build injected images with buildah * ci: pass full image names to zombienet * Add missing ignore --------- Co-authored-by: Mira Ressel <mira@parity.io>
paritytech · Aug 11, 2023 · b420bad · b420bad
1 parent c2efce0
commit b420bad
Show file tree

Hide file tree

Showing 35 changed files with 661 additions and 323 deletions.
diff --git a/.github/workflows/check-licenses.yml b/.github/workflows/check-licenses.yml
@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-22.04
     steps:
       - name: Checkout sources
-        uses: actions/checkout@v3.3.0
+        uses: actions/checkout@v3
       - uses: actions/setup-node@v3.7.0
         with:
           node-version: '18.x'

diff --git a/.github/workflows/release-40_publish-rc-image.yml b/.github/workflows/release-40_publish-rc-image.yml
@@ -0,0 +1,132 @@
+name: Release - Publish RC Container image
+# see https://github.com/paritytech/release-engineering/issues/97#issuecomment-1651372277
+
+on:
+  workflow_dispatch:
+    inputs:
+      release_id:
+        description: |
+          Release ID.
+          You can find it using the command:
+            curl -s \
+              -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/repos/$OWNER/$REPO/releases | \
+              jq '.[] | { name: .name, id: .id }'
+        required: true
+        type: string
+      registry:
+        description: "Container registry"
+        required: true
+        type: string
+        default: docker.io
+      owner:
+        description: Owner of the container image repo
+        required: true
+        type: string
+        default: parity
+
+env:
+  RELEASE_ID: ${{ inputs.release_id }}
+  ENGINE: docker
+  REGISTRY: ${{ inputs.registry }}
+  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  DOCKER_OWNER: ${{ inputs.owner || github.repository_owner }}
+  REPO: ${{ github.repository }}
+  ARTIFACT_FOLDER: release-artifacts
+
+jobs:
+  fetch-artifacts:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v3
+
+      - name: Fetch all artifacts
+        run: |
+          . ./scripts/ci/common/lib.sh
+          fetch_release_artifacts
+
+      - name: Cache the artifacts
+        uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1
+        with:
+          key: artifacts-${{ github.sha }}
+          path: |
+            ${ARTIFACT_FOLDER}/**/*
+
+  build-container:
+    runs-on: ubuntu-latest
+    needs: fetch-artifacts
+
+    strategy:
+      matrix:
+        binary: ["polkadot", "staking-miner"]
+
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v3
+
+      - name: Get artifacts from cache
+        uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1
+        with:
+          key: artifacts-${{ github.sha }}
+          path: |
+            ${ARTIFACT_FOLDER}/**/*
+
+      - name: Check sha256 ${{ matrix.binary }}
+        working-directory: ${ARTIFACT_FOLDER}
+        run: |
+          . ../scripts/ci/common/lib.sh
+
+          echo "Checking binary ${{ matrix.binary }}"
+          check_sha256 ${{ matrix.binary }} && echo "OK" || echo "ERR"
+
+      - name: Check GPG ${{ matrix.binary }}
+        working-directory: ${ARTIFACT_FOLDER}
+        run: |
+          . ../scripts/ci/common/lib.sh
+          import_gpg_keys
+          check_gpg ${{ matrix.binary }}
+
+      - name: Fetch commit and tag
+        id: fetch_refs
+        run: |
+          release=release-${{ inputs.release_id }} && \
+          echo "release=${release}" >> $GITHUB_OUTPUT
+
+          commit=$(git rev-parse --short HEAD) && \
+          echo "commit=${commit}" >> $GITHUB_OUTPUT
+
+          tag=$(git name-rev --tags --name-only $(git rev-parse HEAD)) && \
+          [ "${tag}" != "undefined" ] && echo "tag=${tag}" >> $GITHUB_OUTPUT || \
+          echo "No tag, doing without"
+
+      - name: Build Injected Container image for ${{ matrix.binary }}
+        env:
+            BIN_FOLDER: ${ARTIFACT_FOLDER}
+            BINARY: ${{ matrix.binary }}
+            TAGS: ${{join(steps.fetch_refs.outputs.*, ',')}}
+        run: |
+          echo "Building container for ${{ matrix.binary }}"
+          ./scripts/ci/dockerfiles/build-injected.sh
+
+      - name: Login to Dockerhub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ inputs.owner }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Push Container image for ${{ matrix.binary }}
+        id: docker_push
+        env:
+          BINARY: ${{ matrix.binary }}
+        run: |
+          $ENGINE images | grep ${BINARY}
+          $ENGINE push --all-tags ${REGISTRY}/${DOCKER_OWNER}/${BINARY}
+
+      - name: Check version for the published image for ${{ matrix.binary }}
+        env:
+          BINARY: ${{ matrix.binary }}
+          RELEASE_TAG: ${{ steps.fetch_refs.outputs.release }}
+        run: |
+          echo "Checking tag ${RELEASE_TAG} for image ${REGISTRY}/${DOCKER_OWNER}/${BINARY}"
+          $ENGINE run -i ${REGISTRY}/${DOCKER_OWNER}/${BINARY}:${RELEASE_TAG} --version
diff --git a/.github/workflows/release-50_publish-docker-release.yml b/.github/workflows/release-50_publish-docker-release.yml
@@ -30,7 +30,7 @@ jobs:
         uses: docker/build-push-action@v4
         with:
           push: true
-          file: scripts/ci/dockerfiles/polkadot_injected_release.Dockerfile
+          file: scripts/ci/dockerfiles/polkadot/polkadot_injected_debian.Dockerfile
           tags: |
             parity/polkadot:latest
             parity/polkadot:${{ github.event.release.tag_name }}

diff --git a/.github/workflows/release-51_publish-docker-manual.yml b/.github/workflows/release-51_publish-docker-manual.yml
@@ -37,7 +37,7 @@ jobs:
         uses: docker/build-push-action@v4
         with:
           push: true
-          file: scripts/ci/dockerfiles/polkadot_injected_release.Dockerfile
+          file: scripts/ci/dockerfiles/polkadot/polkadot_injected_debian.Dockerfile
           tags: |
             parity/polkadot:latest
             parity/polkadot:${{ github.event.inputs.version }}

diff --git a/.gitignore b/.gitignore
@@ -10,3 +10,7 @@ polkadot.*
 !polkadot.service
 .DS_Store
 .env
+
+artifacts
+release-artifacts
+release.json
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -159,31 +159,39 @@ default:
     - if: $CI_COMMIT_REF_NAME =~ /^v[0-9]+\.[0-9]+.*$/ # i.e. v1.0, v2.1rc1
 
 .build-push-image:
+  variables:
+    CI_IMAGE: "${BUILDAH_IMAGE}"
+
+    REGISTRY: "docker.io"
+    DOCKER_OWNER: "paritypr"
+    DOCKER_USER: "${PARITYPR_USER}"
+    DOCKER_PASS: "${PARITYPR_PASS}"
+    IMAGE: "${REGISTRY}/${DOCKER_OWNER}/${IMAGE_NAME}"
+
+    ENGINE: "${BUILDAH_COMMAND}"
+    BUILDAH_FORMAT: "docker"
+    SKIP_IMAGE_VALIDATION: 1
+
+    PROJECT_ROOT: "."
+    BIN_FOLDER: "./artifacts"
+    VCS_REF: "${CI_COMMIT_SHA}"
+
   before_script:
     - !reference [.common-before-script, before_script]
     - test -s ./artifacts/VERSION || exit 1
     - test -s ./artifacts/EXTRATAG || exit 1
-    - VERSION="$(cat ./artifacts/VERSION)"
+    - export VERSION="$(cat ./artifacts/VERSION)"
     - EXTRATAG="$(cat ./artifacts/EXTRATAG)"
     - echo "Polkadot version = ${VERSION} (EXTRATAG = ${EXTRATAG})"
   script:
     - test "$DOCKER_USER" -a "$DOCKER_PASS" ||
       ( echo "no docker credentials provided"; exit 1 )
-    - cd ./artifacts
-    - $BUILDAH_COMMAND build
-      --format=docker
-      --build-arg VCS_REF="${CI_COMMIT_SHA}"
-      --build-arg BUILD_DATE="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
-      --build-arg IMAGE_NAME="${IMAGE_NAME}"
-      --tag "$IMAGE_NAME:$VERSION"
-      --tag "$IMAGE_NAME:$EXTRATAG"
-      --file ${DOCKERFILE} .
-    # The job will success only on the protected branch
+    - TAGS="${VERSION},${EXTRATAG}" scripts/ci/dockerfiles/build-injected.sh
     - echo "$DOCKER_PASS" |
-      buildah login --username "$DOCKER_USER" --password-stdin docker.io
+      buildah login --username "$DOCKER_USER" --password-stdin "${REGISTRY}"
     - $BUILDAH_COMMAND info
-    - $BUILDAH_COMMAND push --format=v2s2 "$IMAGE_NAME:$VERSION"
-    - $BUILDAH_COMMAND push --format=v2s2 "$IMAGE_NAME:$EXTRATAG"
+    - $BUILDAH_COMMAND push --format=v2s2 "$IMAGE:$VERSION"
+    - $BUILDAH_COMMAND push --format=v2s2 "$IMAGE:$EXTRATAG"
   after_script:
     - buildah logout --all
 

diff --git a/doc/docker.md b/doc/docker.md
@@ -1,43 +1,58 @@
-# Using Docker
+# Using Containers
+
+The following commands should work no matter if you use Docker or Podman. In general, Podman is recommended. All commands are "engine neutral" so you can use the container engine of your choice while still being able to copy/paste the commands below.
+
+Let's start defining Podman as our engine:
+```
+ENGINE=podman
+```
+
+If you prefer to stick with Docker, use:
+```
+ENGINE=docker
+```
 
 ## The easiest way
 
-The easiest/faster option to run Polkadot in Docker is to use the latest release images. These are small images that use the latest official release of the Polkadot binary, pulled from our package repository.
+The easiest/faster option to run Polkadot in Docker is to use the latest release images. These are small images that use the latest official release of the Polkadot binary, pulled from our Debian package.
 
-**_Following examples are running on westend chain and without SSL. They can be used to quick start and learn how Polkadot needs to be configured. Please find out how to secure your node, if you want to operate it on the internet. Do not expose RPC and WS ports, if they are not correctly configured._**
+**_The following examples are running on westend chain and without SSL. They can be used to quick start and learn how Polkadot needs to be configured. Please find out how to secure your node, if you want to operate it on the internet. Do not expose RPC and WS ports, if they are not correctly configured._**
 
 Let's first check the version we have. The first time you run this command, the Polkadot docker image will be downloaded. This takes a bit of time and bandwidth, be patient:
 
 ```bash
-docker run --rm -it parity/polkadot:latest --version
+$ENGINE run --rm -it parity/polkadot:latest --version
 ```
 
 You can also pass any argument/flag that Polkadot supports:
 
 ```bash
-docker run --rm -it parity/polkadot:latest --chain westend --name "PolkaDocker"
+$ENGINE run --rm -it parity/polkadot:latest --chain westend --name "PolkaDocker"
 ```
 
 ## Examples
 
-Once you are done experimenting and picking the best node name :) you can start Polkadot as daemon, exposes the Polkadot ports and mount a volume that will keep your blockchain data locally. Make sure that you set the ownership of your local directory to the Polkadot user that is used by the container. Set user id 1000 and group id 1000, by running `chown 1000.1000 /my/local/folder -R` if you use a bind mount.
-
-To start a Polkadot node on default rpc port 9933 and default p2p port 30333 use the following command. If you want to connect to rpc port 9933, then must add Polkadot startup parameter: `--rpc-external`.
+Once you are done experimenting and picking the best node name :) you can start Polkadot as daemon, exposes the Polkadot ports and mount a volume that will keep your blockchain data locally. Make sure that you set the ownership of your local directory to the Polkadot user that is used by the container.
 
-```bash
-docker run -d -p 30333:30333 -p 9933:9933 -v /my/local/folder:/polkadot parity/polkadot:latest --chain westend --rpc-external --rpc-cors all
-```
+Set user id 1000 and group id 1000, by running `chown 1000.1000 /my/local/folder -R` if you use a bind mount.
 
-Additionally if you want to have custom node name you can add the `--name "YourName"` at the end
+To start a Polkadot node on default rpc port 9933 and default p2p port 30333 use the following command. If you want to connect to rpc port 9933, then must add Polkadot startup parameter: `--rpc-external`.
 
 ```bash
-docker run -d -p 30333:30333 -p 9933:9933 -v /my/local/folder:/polkadot parity/polkadot:latest --chain westend --rpc-external --rpc-cors all --name "PolkaDocker"
+$ENGINE run -d -p 30333:30333 -p 9933:9933 \
+    -v /my/local/folder:/polkadot \
+    parity/polkadot:latest \
+    --chain westend --rpc-external --rpc-cors all \
+    --name "PolkaDocker
 ```
 
 If you also want to expose the webservice port 9944 use the following command:
 
 ```bash
-docker run -d -p 30333:30333 -p 9933:9933 -p 9944:9944 -v /my/local/folder:/polkadot parity/polkadot:latest --chain westend --ws-external --rpc-external --rpc-cors all --name "PolkaDocker"
+$ENGINE run -d -p 30333:30333 -p 9933:9933 -p 9944:9944 \
+    -v /my/local/folder:/polkadot \
+    parity/polkadot:latest \
+    --chain westend --ws-external --rpc-external --rpc-cors all --name "PolkaDocker"
 ```
 
 ## Using Docker compose
@@ -55,17 +70,19 @@ services:
       - 30333:30333 # p2p port
       - 9933:9933 # rpc port
       - 9944:9944 # ws port
+      - 9615:9615 # Prometheus port
     volumes:
       - /my/local/folder:/polkadot
     command: [
       "--name", "PolkaDocker",
       "--ws-external",
       "--rpc-external",
+      "--prometheus-external",
       "--rpc-cors", "all"
     ]
 ```
 
-With following docker-compose.yml you can set up a node and use polkadot-js-apps as the front end on port 80. After starting the node use a browser and enter your Docker host IP in the URL field: _<http://[YOUR_DOCKER_HOST_IP>_
+With following `docker-compose.yml` you can set up a node and use polkadot-js-apps as the front end on port 80. After starting the node use a browser and enter your Docker host IP in the URL field: _<http://[YOUR_DOCKER_HOST_IP]>_
 
 ```bash
 version: '2'
@@ -78,10 +95,12 @@ services:
       - 30333:30333 # p2p port
       - 9933:9933 # rpc port
       - 9944:9944 # ws port
+      - 9615:9615 # Prometheus port
     command: [
       "--name", "PolkaDocker",
       "--ws-external",
       "--rpc-external",
+      "--prometheus-external",
       "--rpc-cors", "all"
     ]
 
@@ -100,36 +119,39 @@ Chain syncing will utilize all available memory and CPU power your server has to
 
 If running on a low resource VPS, use `--memory` and `--cpus` to limit the resources used. E.g. To allow a maximum of 512MB memory and 50% of 1 CPU, use `--cpus=".5" --memory="512m"`. Read more about limiting a container's resources [here](https://docs.docker.com/config/containers/resource_constraints).
 
-Start a shell session with the daemon:
 
-```bash
-docker exec -it $(docker ps -q) bash;
-```
+## Build your own image
 
-Check the current version:
+There are 3 options to build a polkadot container image:
+- using the builder image
+- using the injected "Debian" image
+- using the generic injected image
 
-```bash
-polkadot --version
-```
+### Builder image
 
-## Build your own image
+To get up and running with the smallest footprint on your system, you may use an existing Polkadot Container image.
 
-To get up and running with the smallest footprint on your system, you may use the Polkadot Docker image.
-You can build it yourself (it takes a while...) in the shell session of the daemon:
+You may also build a polkadot container image yourself (it takes a while...) using the container specs `scripts/ci/dockerfiles/polkadot/polkadot_builder.Dockerfile`.
 
-```bash
-cd scripts/ci/dockerfiles/polkadot
-./build.sh
-```
+### Debian injected
+
+The Debian injected image is how the official polkadot container image is produced. It relies on the Debian package that is published upon each release. The Debian injected image is usually available a few minutes after a new release is published.
+It has the benefit of relying on the GPG signatures embedded in the Debian package.
+
+### Generic injected
+
+For simple testing purposes, the easiest option for polkadot and also random binaries, is to use the `binary_injected.Dockerfile` container spec. This option is less secure since the injected binary is not checked at all but it has the benefit to be simple. This option requires to already have a valid `polkadot` binary, compiled for Linux.
+
+This binary is then simply copied inside the `parity/base-bin` image.
 
 ## Reporting issues
 
 If you run into issues with Polkadot when using docker, please run the following command
 (replace the tag with the appropriate one if you do not use latest):
 
 ```bash
-docker run --rm -it parity/polkadot:latest --version
+$ENGINE run --rm -it parity/polkadot:latest --version
 ```
 
 This will show you the Polkadot version as well as the git commit ref that was used to build your container.
-Just paste that in the issue you create.
+You can now paste the version information in a [new issue](https://github.com/paritytech/polkadot/issues/new/choose).