From f289151cb64cd65707923b4eda880933344f7431 Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Fri, 27 Apr 2018 09:34:49 -0500 Subject: [PATCH 1/7] Update release procedures (#1637) (#1832) (cherry picked from commit 20bd476d0dbbb8bd83164fd544a2accf55e9b53e) --- installer/docs/BUILD.md | 4 +- installer/docs/RELEASE.md | 84 ++++++++++++++++++++++++++++----------- 2 files changed, 62 insertions(+), 26 deletions(-) diff --git a/installer/docs/BUILD.md b/installer/docs/BUILD.md index 59d71594e8..bd70a38b2d 100644 --- a/installer/docs/BUILD.md +++ b/installer/docs/BUILD.md @@ -155,13 +155,13 @@ Please note that you cannot trigger new CI builds manually, but have to promote Make sure `DRONE_SERVER` and `DRONE_TOKEN` environment variables are set before executing these commands. -To promote existing successful CI build to staging... +To promote existing successful CI build to staging (`vic-product-ova-builds` bucket): `` $ drone deploy --param VICENGINE= --param VIC_MACHINE_SERVER= --param ADMIRAL= --param HARBOR= vmware/vic-product staging `` -To promote existing successful CI build to release... +To promote existing successful CI build to release (`vic-product-ova-releases` bucket): `` $ drone deploy --param VICENGINE= --param VIC_MACHINE_SERVER= --param ADMIRAL= --param HARBOR= vmware/vic-product release diff --git a/installer/docs/RELEASE.md b/installer/docs/RELEASE.md index 318cc7f88e..69b53e3427 100644 --- a/installer/docs/RELEASE.md +++ b/installer/docs/RELEASE.md @@ -3,12 +3,14 @@ All examples in this document assume the user's fork is at `origin` and `vmware/vic-product` is at `upstream`. + ## Branching When the team is ready to build a release candidate, create a branch off of master based on the -release version number. A tag for ongoing development should also be created at the commit after -where the release branch begins. If there have been commits to master since the intended branch -point, first create a tag for the release candidate, then create the branch from that tag. +release version number. + +A tag for ongoing development should also be created at the commit **after** +the start of the release branch. The tagging procedure is documented in [Tagging](#Tagging). ``` git remote update @@ -22,8 +24,11 @@ git push upstream Configure branch protection on Github to have the same protection as the master branch. `Protect this branch` and `Require pull request reviews before merging` should be set. -Development should continue on master. Commits that need to be pulled into the release should be -cherry picked into the release branch after they are merged into master. + +## Cherry picking + +Commits that need to be pulled into the release should be cherry picked into the release branch +after they are merged into master. ``` git remote update @@ -38,13 +43,10 @@ git push upstream ## Tagging On the master branch, tag the commit for the first release candidate. On the -following commit, tag `dev` for ongoing development. For example, if the +following commit, tag `dev` for ongoing development. For example, if the current release is `v1.2.0`, the first release candidate will be `v1.2.0-rc1` and the tag for ongoing development will be `v1.3.0-dev`. -When the team is ready to release, tag the commit in the release branch (`v1.2.0`) and push the tag -to Github. - ``` git remote update git checkout upstream/releases/1.2.0 @@ -52,25 +54,46 @@ git tag -a v1.2.0-rc1 aaaaaaa git push upstream v1.2.0-rc1 ``` -Tag `dev` on the release branch after a release. For example, if `v1.2.0` was tagged on -`/releases/1.2.0` and there is work for `v1.2.1`, on the following commit, tag `v1.2.1-dev`. +If there is not yet a commit after the start of the release branch, create an empty commit after +the commit for the release branch. This empty commit will be tagged for ongoing development on master. + +``` +# Create empty commit on master +git remote update +git checkout upstream/master +git commit --allow-empty -m "v1.3.0-dev" +git push upstream + +# Tag empty commit for ongoing development +git remote update +git checkout upstream/master +git tag -a v1.3.0-dev bbbbbbb +git pubsh upstream v1.3.0-dev +``` + +After the release candidate has passed QA and the team is ready to release, tag the commit in the +release branch (`v1.2.0`) and push the tag to Github. ``` git remote update git checkout upstream/releases/1.2.0 -git tag -a v1.2.1-dev bbbbbbb -git push upstream v1.2.1-dev +git tag -a v1.2.0 ccccccc +git push upstream v1.2.0 ``` +### Point releases -## Building Release +After a release, tag `dev` on the release branch for ongoing development. +For example, if `v1.2.0` was tagged on `/releases/1.2.0` and there is work for `v1.2.1`, on the +following commit, tag `v1.2.1-dev`. ``` -git fetch --all --tags --prune -git checkout tags/v1.2.0 +git remote update +git checkout upstream/releases/1.2.0 +git tag -a v1.2.1-dev ddddddd +git push upstream v1.2.1-dev ``` -Follow instructions in [How to build VIC Product OVA](BUILD.md) ## Github Releases @@ -78,14 +101,15 @@ After pushing the tag to Github, go to https://github.com/vmware/vic-product/rel Select the appropriate tag -Title for follows form `VIC Product ` (`VIC Product v1.2.0-rc1`) +Release title follows form `vSphere Integrated Containers Appliance ` (e.g. `vSphere Integrated Containers Appliance v1.2.0-rc1`) Description template for release candidates: ``` OVA will contain: -Admiral `v1.2.0-rc3` -Harbor `harbor-offline-installer-v1.2.0-rc4.tgz` -VIC Engine `vic_1.2.0-rc4.tar.gz` +Admiral v1.2.0-rc3 +Harbor harbor-offline-installer-v1.2.0-rc4.tgz +VIC Engine vic_1.2.0-rc4.tar.gz +VIC Machine Server digest aaaaaaa ``` If the release is a release candidate, mark `This is a pre-release` @@ -94,12 +118,24 @@ If the release is a release candidate, mark `This is a pre-release` Description template for release version of the OVA: ``` -Admiral `v1.2.0` -Harbor `harbor-offline-installer-v1.2.0.tgz` -VIC Engine `vic_1.2.0.tar.gz` +Admiral v1.2.0 +Harbor harbor-offline-installer-v1.2.0.tgz +VIC Engine vic_1.2.0.tar.gz +VIC Machine Server digest aaaaaaa ef6b71d98bb6650240008b5281e97bf8592d5fd726833883718f471ed665fc5b vic-v1.2.0-aaaaaaaa.ova 85eabdf7e58fed8c09e4f3c45b2caa974ae89a16 vic-v1.2.0-aaaaaaaa.ova ebc669f7b4cebf7501cf141e3b6fa2e3 vic-v1.2.0-aaaaaaaa.ova 4741.43 MB ``` + + +## Building Release + +``` +git fetch --all --tags --prune +git checkout tags/v1.2.0 +``` + +Follow instructions in [How to build VIC Product OVA](BUILD.md) + From 28e01f4b39c4456be5e26f899839a39fb9f49b2f Mon Sep 17 00:00:00 2001 From: Jason Morris <10388115+morris-jason@users.noreply.github.com> Date: Tue, 24 Apr 2018 11:51:42 -0500 Subject: [PATCH 2/7] Rename ovf keys for registry port (#1642) (#1832) Changes the ovf key names for management_portal.port and registry.port so they do not conflict on the HTML5 client in vSphere 6.7. Fixes #1624. (cherry picked from commit 7c07bfe01d77d93e643c5f10232addde3d27a86a) --- installer/build/bootable/config/builder.ovf | 4 ++-- .../scripts/systemd/scripts/vic-appliance-environment.sh | 4 ++-- installer/docs/BUILD.md | 4 ++-- installer/fileserver/server.go | 2 +- installer/fileserver/tasks.go | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/installer/build/bootable/config/builder.ovf b/installer/build/bootable/config/builder.ovf index f8bf696e45..26d5908c1f 100644 --- a/installer/build/bootable/config/builder.ovf +++ b/installer/build/bootable/config/builder.ovf @@ -246,7 +246,7 @@ EVALUATION LICENSE. If You are licensing the Software for evaluation purposes, Y Registry Properties 3. Registry Configuration - + Specifies the port on which registry will be published. @@ -262,7 +262,7 @@ EVALUATION LICENSE. If You are licensing the Software for evaluation purposes, Y Management Portal Properties 4. Management Portal Configuration - + Specifies the port on which Management Portal will be published. diff --git a/installer/build/scripts/systemd/scripts/vic-appliance-environment.sh b/installer/build/scripts/systemd/scripts/vic-appliance-environment.sh index 55198d4513..5e71413a45 100755 --- a/installer/build/scripts/systemd/scripts/vic-appliance-environment.sh +++ b/installer/build/scripts/systemd/scripts/vic-appliance-environment.sh @@ -23,8 +23,8 @@ APPLIANCE_TLS_CERT="$(ovfenv -k appliance.tls_cert | sed -E ':a;N;$!ba;s/\r{0,1} APPLIANCE_TLS_PRIVATE_KEY="$(ovfenv -k appliance.tls_cert_key | sed -E ':a;N;$!ba;s/\r{0,1}\n//g')" APPLIANCE_TLS_CA_CERT="$(ovfenv -k appliance.ca_cert | sed -E ':a;N;$!ba;s/\r{0,1}\n//g')" -ADMIRAL_PORT="$(ovfenv -k management_portal.port)" -REGISTRY_PORT="$(ovfenv -k registry.port)" +ADMIRAL_PORT="$(ovfenv -k management_portal.management_portal_port)" +REGISTRY_PORT="$(ovfenv -k registry.registry_port)" NOTARY_PORT="$(ovfenv -k registry.notary_port)" FILESERVER_PORT="$(ovfenv -k appliance.config_port)" HOSTNAME="" diff --git a/installer/docs/BUILD.md b/installer/docs/BUILD.md index bd70a38b2d..fa81cac5df 100644 --- a/installer/docs/BUILD.md +++ b/installer/docs/BUILD.md @@ -112,8 +112,8 @@ docker run -it --net=host -v $GOPATH/src/github.com/vmware/vic-product/installer --net:Network="VM Network" \ --prop:appliance.root_pwd="password" \ --prop:appliance.permit_root_login=True \ - --prop:management_portal.port=8282 \ - --prop:registry.port=443 \ + --prop:management_portal.management_portal_port=8282 \ + --prop:registry.registry_port=443 \ /test-bin/$(ls -1t bin | grep "\.ova") \ vi://$VC_USER:$VC_PASSWORD@$VC_IP/$VC_COMPUTE ``` diff --git a/installer/fileserver/server.go b/installer/fileserver/server.go index 34e50be31b..37631f6b2e 100644 --- a/installer/fileserver/server.go +++ b/installer/fileserver/server.go @@ -142,7 +142,7 @@ func Init(conf *config) { if ip, err := ip.FirstIPv4(ip.Eth0Interface); err == nil { conf.serverHostname = getHostname(ovf, ip) - if port, ok := ovf.Properties["management_portal.port"]; ok { + if port, ok := ovf.Properties["management_portal.management_portal_port"]; ok { conf.admiralPort = port } } diff --git a/installer/fileserver/tasks.go b/installer/fileserver/tasks.go index 2771811ae5..3a2a715a27 100644 --- a/installer/fileserver/tasks.go +++ b/installer/fileserver/tasks.go @@ -74,7 +74,7 @@ func registerWithPSC(ctx context.Context) error { if err != nil { return err } - admiralPort := ovf.Properties["management_portal.port"] + admiralPort := ovf.Properties["management_portal.management_portal_port"] // Out of the box users defCreateUsers, foundCreateUsers := ovf.Properties["default_users.create_def_users"] From cd2aac5a671f5f85154200d53f2586d680f63842 Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Wed, 2 May 2018 10:29:16 -0500 Subject: [PATCH 3/7] Update unsupported upgrade path message (#1685) (#1832) (cherry picked from commit c7c1598b69fec4548bb16765df3ba2f2360775c3) --- installer/build/scripts/upgrade/upgrade.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installer/build/scripts/upgrade/upgrade.sh b/installer/build/scripts/upgrade/upgrade.sh index 2ec03ee9ee..a291c58717 100755 --- a/installer/build/scripts/upgrade/upgrade.sh +++ b/installer/build/scripts/upgrade/upgrade.sh @@ -214,7 +214,7 @@ function proceedWithUpgrade { fi log "" - log "Detected old appliance's version $ver as 1.2.0 or older." + log "Detected old appliance's version as $ver." log "Upgrade from this version is not a supported upgrade path." log "If the old appliance's version is not detected correctly, please contact VMware support." exit 1 From b54ec287e651a6d8219fa22c9cb768d01cb147e0 Mon Sep 17 00:00:00 2001 From: Yan Date: Wed, 2 May 2018 09:12:38 -0700 Subject: [PATCH 4/7] Unify Harbor and Clair DB password on first boot (#1684) (#1832) (cherry picked from commit 49623c76552fadd7102166bc11165196f3358763) --- installer/build/scripts/harbor/configure_harbor.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/installer/build/scripts/harbor/configure_harbor.sh b/installer/build/scripts/harbor/configure_harbor.sh index c8aa9e36e9..1d087527d3 100755 --- a/installer/build/scripts/harbor/configure_harbor.sh +++ b/installer/build/scripts/harbor/configure_harbor.sh @@ -115,9 +115,10 @@ configureHarborCfg ssl_cert $appliance_tls_cert configureHarborCfg ssl_cert_key $appliance_tls_key configureHarborCfg secretkey_path $data_dir -# Set MySQL and Clair DB passwords on first boot -configureHarborCfgOnce db_password "$(genPass)" -configureHarborCfgOnce clair_db_password "$(genPass)" +# Set Harbor DB and Clair DB passwords on first boot +random_pwd=$(genPass) +configureHarborCfgOnce db_password "$random_pwd" +configureHarborCfgOnce clair_db_password "$random_pwd" setPortInYAML $harbor_compose_file "${REGISTRY_PORT}" "${NOTARY_PORT}" From b646de8486761ad2cd1589b8a0835ce555aec686 Mon Sep 17 00:00:00 2001 From: Andrew Chin Date: Fri, 4 May 2018 15:22:49 -0500 Subject: [PATCH 5/7] Update release docs (#1704) (#1832) (cherry picked from commit 6bbb3a7bdfe6a7f9cf9ada6211f25ac2ccf5e855) --- installer/docs/BUILD.md | 33 ++++++++++++++++++++------- installer/docs/RELEASE.md | 48 +++++++++++++++++---------------------- 2 files changed, 46 insertions(+), 35 deletions(-) diff --git a/installer/docs/BUILD.md b/installer/docs/BUILD.md index fa81cac5df..f6559d2122 100644 --- a/installer/docs/BUILD.md +++ b/installer/docs/BUILD.md @@ -157,20 +157,37 @@ Make sure `DRONE_SERVER` and `DRONE_TOKEN` environment variables are set before To promote existing successful CI build to staging (`vic-product-ova-builds` bucket): -`` -$ drone deploy --param VICENGINE= --param VIC_MACHINE_SERVER= --param ADMIRAL= --param HARBOR= vmware/vic-product staging -`` +``` +$ drone deploy --param VICENGINE= \ + --param VIC_MACHINE_SERVER= \ + --param ADMIRAL= \ + --param HARBOR= \ + vmware/vic-product staging +``` To promote existing successful CI build to release (`vic-product-ova-releases` bucket): -`` -$ drone deploy --param VICENGINE= --param VIC_MACHINE_SERVER= --param ADMIRAL= --param HARBOR= vmware/vic-product release -`` +``` +$ drone deploy --param VICENGINE= \ + --param VIC_MACHINE_SERVER= \ + --param ADMIRAL= \ + --param HARBOR= \ + vmware/vic-product release +``` + +Example: + +``` +$ drone deploy --param VICENGINE=https://storage.googleapis.com/vic-engine-releases/vic_v1.4.0.tar.gz \ + --param VIC_MACHINE_SERVER=latest \ + --param ADMIRAL=v1.4.0 \ + --param HARBOR=https://storage.googleapis.com/harbor-releases/harbor-offline-installer-v1.5.0.tgz \ + vmware/vic-product release + +``` `vic_engine_version` and `harbor_version` can be specified as a URL or a file in `cwd`, eg. 'https://storage.googleapis.com/vic-engine-releases/vic_1.2.1.tar.gz' `admiral_tag` and `vic_machine_server` should be specified as docker image revision tag, eg. 'latest' `ci_build_number_to_promote` is the drone build number which will be promoted - -## Troubleshooting diff --git a/installer/docs/RELEASE.md b/installer/docs/RELEASE.md index 69b53e3427..e436474963 100644 --- a/installer/docs/RELEASE.md +++ b/installer/docs/RELEASE.md @@ -101,41 +101,35 @@ After pushing the tag to Github, go to https://github.com/vmware/vic-product/rel Select the appropriate tag -Release title follows form `vSphere Integrated Containers Appliance ` (e.g. `vSphere Integrated Containers Appliance v1.2.0-rc1`) +Release title follows form `vSphere Integrated Containers Appliance ` (e.g. `vSphere Integrated Containers Appliance v1.4.0-rc3`) -Description template for release candidates: -``` -OVA will contain: -Admiral v1.2.0-rc3 -Harbor harbor-offline-installer-v1.2.0-rc4.tgz -VIC Engine vic_1.2.0-rc4.tar.gz -VIC Machine Server digest aaaaaaa -``` +Obtain artifact hashes from CI build output at the end of `unified-ova-build` step -If the release is a release candidate, mark `This is a pre-release` +Obtain version information from `/etc/vmware/version` + +Description template for release candidates and releases: -## OVA Releases +``````` +### [Download OVA](https://storage.googleapis.com/vic-product-ova-releases/vic-v1.4.0-rc3-4824-d99cbdb4.ova) +Filesize: +SHA256: +SHA1: +MD5: -Description template for release version of the OVA: +### OVA will contain: ``` -Admiral v1.2.0 -Harbor harbor-offline-installer-v1.2.0.tgz -VIC Engine vic_1.2.0.tar.gz -VIC Machine Server digest aaaaaaa - -ef6b71d98bb6650240008b5281e97bf8592d5fd726833883718f471ed665fc5b vic-v1.2.0-aaaaaaaa.ova -85eabdf7e58fed8c09e4f3c45b2caa974ae89a16 vic-v1.2.0-aaaaaaaa.ova -ebc669f7b4cebf7501cf141e3b6fa2e3 vic-v1.2.0-aaaaaaaa.ova -4741.43 MB +appliance=v1.4.0-rc3-4824-d99cbdb4 +harbor=harbor-offline-installer-v1.5.0-rc4.tgz +engine=vic_v1.4.0-rc2.tar.gz +admiral=vmware/admiral:vic_v1.4.0-rc4 45a773ffae33 +vic-machine-server=gcr.io/eminent-nation-87317/vic-machine-server:latest b3412e003674 ``` +### [Changes from v1.3.1](https://github.com/vmware/vic-product/compare/v1.3.1...v1.4.0-rc3) +``````` -## Building Release +If the release is a release candidate, mark `This is a pre-release` -``` -git fetch --all --tags --prune -git checkout tags/v1.2.0 -``` +## Building Releases Follow instructions in [How to build VIC Product OVA](BUILD.md) - From 4bae4e48e439a0f11cd6857ee6fb3bf358193e56 Mon Sep 17 00:00:00 2001 From: Jason Morris <10388115+morris-jason@users.noreply.github.com> Date: Tue, 8 May 2018 11:29:39 -0500 Subject: [PATCH 6/7] Pull release components on tag (#1693) (#1832) Pulls release versions of ova components during release tags or a push to a releases branch. Uses gcloud credentials to pull in latest tagged version of vic-machine-server. Fixes #397. (cherry picked from commit 914d606613edbab828da31b91e0160023784746f) --- .drone.yml | 3 +- installer/build/build-ova.sh | 2 +- installer/build/container/Dockerfile | 6 ++-- installer/scripts/ci-build.sh | 52 +++++++++++++++++++--------- 4 files changed, 42 insertions(+), 21 deletions(-) diff --git a/.drone.yml b/.drone.yml index 8ae3f918cf..f3f4a56f8b 100644 --- a/.drone.yml +++ b/.drone.yml @@ -79,7 +79,7 @@ pipeline: unified-ova-build: group: build - image: 'gcr.io/eminent-nation-87317/vic-product-build:2ea9bdfd' + image: 'gcr.io/eminent-nation-87317/vic-product-build:33e3b968' pull: true privileged: true environment: @@ -91,6 +91,7 @@ pipeline: - harbor - vic_machine_server - vicengine + - gs_token_key volumes: - '/dev:/dev' - '/var/run/docker.sock:/var/run/docker.sock' diff --git a/installer/build/build-ova.sh b/installer/build/build-ova.sh index c9d38670e2..caf73ac59d 100755 --- a/installer/build/build-ova.sh +++ b/installer/build/build-ova.sh @@ -82,7 +82,7 @@ url=$(gsutil ls -l "gs://vic-engine-builds" | grep -v TOTAL | grep vic_ | sort - setenv VICENGINE "$url" #set Harbor -url=$(gsutil ls -l "gs://harbor-builds" | grep -v TOTAL | grep offline-installer | sort -k2 -r | (trap '' PIPE; head -n1) | xargs | cut -d ' ' -f 3 | sed 's/gs:\/\//https:\/\/storage.googleapis.com\//') +url=$(gsutil ls -l "gs://harbor-builds" | grep -v TOTAL | grep offline-installer | grep -v offline-installer-latest | sort -k2 -r | (trap '' PIPE; head -n1) | xargs | cut -d ' ' -f 3 | sed 's/gs:\/\//https:\/\/storage.googleapis.com\//') setenv HARBOR "$url" export BUILD_DCHPHOTON_VERSION="1.13" diff --git a/installer/build/container/Dockerfile b/installer/build/container/Dockerfile index 625957b8c9..c20ac0b1c4 100644 --- a/installer/build/container/Dockerfile +++ b/installer/build/container/Dockerfile @@ -1,12 +1,14 @@ FROM vmware/photon ENV GOVERSION=1.9.2 -ENV PATH=$PATH:/root/gsutil:/usr/local/go/bin +ENV PATH=$PATH:/root/gsutil:/usr/local/go/bin:/usr/local/google-cloud-sdk/bin/ RUN set -eux; \ tdnf install -y make tar gzip python2 python-pip sed git \ gawk docker gptfdisk e2fsprogs grub2 parted xz docker; \ - curl -L'#' -k https://storage.googleapis.com/pub/gsutil.tar.gz | tar xzf - -C $HOME; \ + curl -L'#' -k https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-200.0.0-linux-x86_64.tar.gz | tar xzf - -C /usr/local; \ + mkdir -p /root/.gsutil/; \ + /usr/local/google-cloud-sdk/install.sh --quiet; \ curl -L'#' -k https://storage.googleapis.com/golang/go$GOVERSION.linux-amd64.tar.gz | tar xzf - -C /usr/local; \ curl -o /usr/bin/jq -L'#' -k https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 && chmod +x /usr/bin/jq; diff --git a/installer/scripts/ci-build.sh b/installer/scripts/ci-build.sh index dbc517f0fc..2b5292fb9d 100755 --- a/installer/scripts/ci-build.sh +++ b/installer/scripts/ci-build.sh @@ -20,35 +20,53 @@ cd installer INSTALLER_DIR=$(pwd) OPTIONS="" -echo "BRANCH = $DRONE_BRANCH" +echo -e "BRANCH = $DRONE_BRANCH\nEVENT = $DRONE_BUILD_EVENT\nTAG = $DRONE_TAG\n" -# set options +# set options based on drone args, if present if [ -n "${ADMIRAL}" ]; then OPTIONS="--admiral $ADMIRAL" -elif [[ "$DRONE_BRANCH" == *"releases/"* ]]; then - admiral_release=$(curl -s https://hub.docker.com/v2/repositories/vmware/admiral/tags/\?page\=1\&page_size\=250 | jq '.results[] | .name'| cut -d "\"" -f2 | grep '^vic_v' | head -n 1 | cut -d'_' -f2) - OPTIONS="--admiral $admiral_release" fi - if [ -n "${HARBOR}" ]; then OPTIONS="$OPTIONS --harbor $HARBOR" -elif [[ "$DRONE_BRANCH" == *"releases/"* ]]; then - harbor_release=$(gsutil ls -l "gs://harbor-releases" | grep -v TOTAL | grep offline-installer | sort -k2 -r | (trap '' PIPE; head -n1) | xargs | cut -d ' ' -f 3 | sed 's/gs:\/\//https:\/\/storage.googleapis.com\//') - OPTIONS="$OPTIONS --harbor $harbor_release" fi - if [ -n "${VICENGINE}" ]; then OPTIONS="$OPTIONS --vicengine $VICENGINE" -elif [[ "$DRONE_BRANCH" == *"releases/"* ]]; then - vicengine_release=$(gsutil ls -l "gs://vic-engine-releases" | grep -v TOTAL | grep vic_ | sort -k2 -r | (trap '' PIPE; head -1) | xargs | cut -d ' ' -f 3 | sed 's/gs:\/\//https:\/\/storage.googleapis.com\//') - OPTIONS="$OPTIONS --vicengine $vicengine_release" fi - if [ -n "${VIC_MACHINE_SERVER}" ]; then OPTIONS="$OPTIONS --vicmachineserver $VIC_MACHINE_SERVER" -elif [[ "$DRONE_BRANCH" == *"releases/"* ]]; then - vicmachineserver_release="latest" - OPTIONS="$OPTIONS --vicmachineserver $vicmachineserver_release" +fi + +# set release options if drone args not present +if [[ ( "$DRONE_BUILD_EVENT" == "tag" && "$DRONE_TAG" != *"dev"* ) || "$DRONE_BRANCH" == *"releases/"* ]]; then + if [ -z "${ADMIRAL}" ]; then + admiral_release=$(curl -s https://hub.docker.com/v2/repositories/vmware/admiral/tags/\?page\=1\&page_size\=250 | jq '.results[] | .name'| cut -d "\"" -f2 | grep '^vic_v' | head -n 1 | cut -d'_' -f2) + OPTIONS="--admiral $admiral_release" + fi + if [ -z "${HARBOR}" ]; then + harbor_release=$(gsutil ls -l "gs://harbor-releases" | grep -v TOTAL | grep offline-installer | sort -k2 -r | (trap '' PIPE; head -n1) | xargs | cut -d ' ' -f 3 | sed 's/gs:\/\//https:\/\/storage.googleapis.com\//') + OPTIONS="$OPTIONS --harbor $harbor_release" + fi + if [ -z "${VICENGINE}" ]; then + vicengine_release=$(gsutil ls -l "gs://vic-engine-releases" | grep -v TOTAL | grep vic_ | sort -k2 -r | (trap '' PIPE; head -1) | xargs | cut -d ' ' -f 3 | sed 's/gs:\/\//https:\/\/storage.googleapis.com\//') + OPTIONS="$OPTIONS --vicengine $vicengine_release" + fi + if [ -z "${VIC_MACHINE_SERVER}" ]; then + # Listing container tags requires permissions + if [ -z "$(gcloud auth list --filter=status:ACTIVE --format='value(account)')" ]; then + if [ -z "${GS_TOKEN_KEY}" ]; then + echo "No google service account key found..." + exit 1 + fi + echo "Attempting to login with google account service key" + KEY_FILE=".tmp.token" + echo "${GS_TOKEN_KEY}" > ${KEY_FILE} + gcloud auth activate-service-account --key-file ${KEY_FILE} || (echo "Login with service account key failed..." && exit 1) + rm -f ${KEY_FILE} + fi + + vicmachineserver_release="$(gcloud container images list-tags gcr.io/eminent-nation-87317/vic-machine-server --filter='tags~.' | grep -v DIGEST | awk '{print $2}' | sed -rn 's/^(.*,)?(v([0-9]+\.){2}[0-9]+(-rc[0-9]+)?)(,.*)?$/\2/p' | head -n 1)" + OPTIONS="$OPTIONS --vicmachineserver $vicmachineserver_release" + fi fi echo "OPTIONS = $OPTIONS" From 257dfbe23122700da46a3b809b0c565053f1f307 Mon Sep 17 00:00:00 2001 From: Jason Morris <10388115+morris-jason@users.noreply.github.com> Date: Wed, 9 May 2018 13:12:38 -0500 Subject: [PATCH 7/7] Fix installer problems in ci (#1726) (#1832) Slightly increases the disk size of the ova during the installer build process. Fixes a syntax error during the manifest generation and removes target disk loops during disk export. Fixes #1725. (cherry picked from commit 7a3d84712ee2b0196c8cccc7669594a3cdf2399f) --- installer/build/bootable/build-disks.sh | 7 ++++--- installer/build/bootable/build-main.sh | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/installer/build/bootable/build-disks.sh b/installer/build/bootable/build-disks.sh index c5085b1d67..9448d1bfa4 100755 --- a/installer/build/bootable/build-disks.sh +++ b/installer/build/bootable/build-disks.sh @@ -177,8 +177,8 @@ fi # These sizes are minimal for install, since partitions are resized to full disk space after firstboot. IMAGESIZES=( - "4GiB" - "1GiB" + "6GiB" + "2GiB" ) IMAGES=( "vic-disk1" @@ -207,7 +207,8 @@ elif [ "${ACTION}" == "export" ]; then log1 "export images to VMDKs" for i in "${!IMAGES[@]}"; do log2 "exporting ${IMAGES[$i]}.img to ${IMAGES[$i]}.vmdk" - DEV=$(losetup -l -O NAME,BACK-FILE -a | tail -n +2 | grep "${IMAGES[$i]}" | awk '{print $1}') + echo "export ${PACKAGE}/${IMAGES[$i]}" + DEV=$(losetup -l -O NAME,BACK-FILE -a | tail -n +2 | grep "${PACKAGE}/${IMAGES[$i]}" | awk '{print $1}') convert "${DEV}" "${IMAGEROOTS[$i]}" "${IMAGES[$i]}.img" "${IMAGES[$i]}.vmdk" done diff --git a/installer/build/bootable/build-main.sh b/installer/build/bootable/build-main.sh index 06b6e2ecf8..6a4f29b0ed 100755 --- a/installer/build/bootable/build-main.sh +++ b/installer/build/bootable/build-main.sh @@ -137,7 +137,7 @@ function main { cd "${PACKAGE}" sed -i -e s~--version--~${BUILD_OVA_REVISION}~ vic-${BUILD_OVA_REVISION}.ovf log2 "rebuilding OVF manifest" - sha256sum --tag "vic-${BUILD_OVA_REVISION}.ovf" "vic-${BUILD_OVA_REVISION}.mf" *.vmdk | sed s/SHA256\ \(/SHA256\(/ > "vic-${BUILD_OVA_REVISION}.mf" + sha256sum --tag "vic-${BUILD_OVA_REVISION}.ovf" *.vmdk | sed s/SHA256\ \(/SHA256\(/ > "vic-${BUILD_OVA_REVISION}.mf" tar -cvf "${RESOURCE}/vic-${BUILD_OVA_REVISION}.ova" "vic-${BUILD_OVA_REVISION}.ovf" "vic-${BUILD_OVA_REVISION}.mf" *.vmdk OUTFILE=${RESOURCE}/vic-${BUILD_OVA_REVISION}.ova