Significant updates to the release process

Makes lots of updates related to us now being a top-level project, and
no longer in the Incubator.

website/developers/committers/release-process/environment-variables.md

@@ -11,11 +11,17 @@ releases. To allow these example commands to run unmodified, set these environme
 # The version currently set on the master branch (BROOKLYN_VERSION_BELOW)
 OLD_MASTER_VERSION=0.10.0-SNAPSHOT
 # The next version to be set on the master branch
-NEW_MASTER_VERSION=0.NNN+1.0-SNAPSHOT
+NEW_MASTER_VERSION=0.10.0-SNAPSHOT
 
 # The version we are releasing now.
-VERSION_NAME=0.NNN.0
+VERSION_NAME=0.9.0
 
 # The release candidate number we are making now.
 RC_NUMBER=1
+
+# Modules and submodules - these will come in handy later
+SUBMODULES="$( perl -n -e 'if ($_ =~ /path += +(.*)$/) { print $1."\n" }' < .gitmodules )"
+MODULES=". ${SUBMODULES}"
 {% endhighlight %}
+
+Alternatively, use the command `eval $( ./brooklyn-dist/release/environment.sh )` to set these values automatically.

website/developers/committers/release-process/index.md

@@ -11,11 +11,9 @@ children:
 - { path: publish-temp.md }
 - { path: vote.md }
 - { path: fix-release.md }
-- { path: vote-ipmc.md }
 - { path: publish.md }
 - { path: announce.md }
 ---
-0. Review and update these instructions, post-graduation from incubation!
 1. [Prerequisites](prerequisites.html) - steps that a new release manager must do (but which only need to be done once)
 2. [Set environment variables](environment-variables.html) - many example snippets here use environment variables to
    avoid repetition - this page describes what they are
@@ -25,7 +23,5 @@ children:
 5. [Publish the release artifacts to the staging area](publish-temp.html)
 6. [Vote on the dev@brooklyn list](vote.html)
    1. If the vote fails - [fix the release branch](fix-release.html) and resume from step 3
-7. [Vote on the general@incubator list](vote-ipmc.html)
-   1. If the vote fails - [fix the release branch](fix-release.html) and resume from step 3
-8. [Publish the release artifacts to the public location](publish.html)
-9. [Announce the release](announce.html)
+7. [Publish the release artifacts to the public location](publish.html)
+8. [Announce the release](announce.html)

website/developers/committers/release-process/make-release-artifacts.md

@@ -15,26 +15,26 @@ The release script will:
 The script has a single required parameter `-r` which is given the release candidate number - so `-r1` will create
 release candidate 1 and will name the artifacts appropriately.
 
-Before running this however, it is a good idea to check the `apache-release` profile build is healthy.
-This will catch glitches such as PGP or javadoc problems without doing time-consuming uploads:
+The script takes a `-n` parameter to work in *dry run* mode; in this mode, the script will NOT upload Maven artifacts
+or commit the release to the Subversion repository. This speeds up the process (the Maven deploy in particular slows
+down the build) and will catch any problems such as PGP or javadoc problems much sooner.
 
 {% highlight bash %}
-mvn clean install -Papache-release
-{% endhighlight %}
-
-To run the script:
+# A dry run to test everything is OK
+./brooklyn-dist/release/make-release-artifacts.sh -r$RC_NUMBER -n
 
-{% highlight bash %}
+# The real build, which will publish artifacts
 ./brooklyn-dist/release/make-release-artifacts.sh -r$RC_NUMBER
 {% endhighlight %}
 
-It will show you the release information it has deduced, and ask yes-or-no if it can proceed. Please note that the
-script will thoroughly clean the Git workspace of all uncommited and unadded files.
-
-**You really probably want to run this against a secondary checkout.** It will wipe `.project` files and other IDE metadata, and bad things can happen if an IDE tries to write while the script is running. Also as it takes a long time, this means your workspace is not tied up. One quick and easy way to do this is to `git clone` the local directory of your primary checkout to a secondary location.
+It will show you the release information it has deduced, and ask yes-or-no if it can proceed. Then you will be prompted
+for the passphrase to your GnuPG private key. You should only be asked this question once; the GnuPG agent will cache
+the password for the remainder of the build.
 
-A few minutes into the script you will be prompted for the passphrase to your GnuPG private key. You should only be
-asked this question once; the GnuPG agent will cache the password for the remainder of the build.
+Please note that the script will thoroughly clean the Git workspace of all uncommitted and unadded files **even in dry
+run mode**. Therefore **you really want to run this against a secondary checkout.** It will wipe `.project` files and
+other IDE metadata, and bad things can happen if an IDE tries to write while the script is running. Consider using the
+Vagrant configuration provided.
 
 Please note that uploading to the Nexus staging repository is a slow process. Expect this stage of the build to take
 2 hours.
@@ -49,10 +49,3 @@ The release script will:
 5. For each of the produced files, produce MD5, SHA1, SHA256 and GnuPG signatures
 
 At the end of the script, it will show you the files it has produced and their location.
-
-Make a signed tag for this release, and then push the tag:
-
-{% highlight bash %}
-git tag -s -m "Tag release ${VERSION_NAME} release candidate ${RC_NUMBER}" apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER}
-git push apache apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER}
-{% endhighlight %}

website/developers/committers/release-process/prerequisites.md

@@ -36,6 +36,8 @@ The following software packages are required during the build. Make sure you hav
 
 - A Java Development Kit, version 1.7
 - `maven` and `git`
+- Go Language 1.6 - usually provided by the `golang` package on popular distributions
+- The `rpmbuild` command - usually provided by the `rpm` package on popular distributions
 - `xmlstarlet` is required by the release script to process version numbers in `pom.xml` files;
   on mac, `port install xmlstarlet` should do the trick.
 - `zip` and `unzip`
@@ -45,6 +47,7 @@ The following software packages are required during the build. Make sure you hav
 - `md5sum` and `sha1sum` - these are often present by default on Linux, but not on Mac;
   `port install md5sha1sum` should remedy that.
 - if `gpg` does not resolve (it is needed for maven), create an alias or script pointing at `gpg2 "$@"`
+- the `mmv` command (usually in a package named `mmv`) will help with the final steps of the release process
 
 
 GPG keys
@@ -79,16 +82,26 @@ cd apache-dist-release-brooklyn
 svn --username $SVN_USERNAME commit -m 'Update brooklyn/KEYS for $GPG_KEY'
 {% endhighlight %}
 
-References: 
+References:
+
 * [Post on the general@incubator list](https://mail-archives.apache.org/mod_mbox/incubator-general/201410.mbox/%3CCAOGo0VawupMYRWJKm%2Bi%2ByMBqDQQtbv-nQkfRud5%2BV9PusZ2wnQ%40mail.gmail.com%3E)
 * [GPG cheatsheet](http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/gpg-cs.html)
 
+We recommend the use of the `gpg-agent`, as the release process invokes gpg to sign a large number of artifacts, one at
+a time. The agent stores its configuration in `~/.gnupg/gpg-agent.conf`. A sample configuration is shown below; it uses
+the Mac OSX `pinentry-mac` program which can be obtained through MacPorts or other sources. For other platforms you will
+need to change this; sometimes you can omit it completely and your OS will pick a suitable alternative. The following
+two lines cause your passphrase to be cached in memory for a limited period; it will expire from the cache 30 minutes
+after it was most recently accessed, or 4 hours after it was first cached.  
+
+```
+pinentry-program /Applications/MacPorts/pinentry-mac.app/Contents/MacOS/pinentry-mac
+default-cache-ttl 1800
+max-cache-ttl 14400
+```
+
 If you experience trouble with PGP subsequently (when running maven):
-* On Mac make sure that `~/.gnupg/gpg-agent.conf` refers to the right file for `pinentry-program`;
-  it must point at a *binary* and preferably a pop-up which is keychain-aware,
-  such as with macports `/Applications/MacPorts/pinentry-mac.app/Contents/MacOS/pinentry-mac`
-* When the `pinentry` dialog pops up, tick to "Save to keychain", otherwise it will keep popping up
-  and may time out and fail the build
+
 * See [GnuPG/Pinentry Enigmail debugging](https://www.enigmail.net/support/gnupg2_issues.php) for tips on diagnosing gpg-agent communication (from the process to this agent and from this agent to the pinentry program)
 * See [GnuPG Agent Options](https://www.gnupg.org/documentation/manuals/gnupg/Agent-Options.html) for extended gpg-agent debug
 

website/developers/committers/release-process/publish-temp.md

@@ -4,6 +4,22 @@ title: Publish to the staging area
 navgroup: developers
 ---
 
+Update the canonical Git repository
+-----------------------------------
+
+Make a signed tag for this release candidate:
+
+{% highlight bash %}
+for m in ${MODULES}; do ( cd $m && git tag -s -m "Tag release ${VERSION_NAME} release candidate ${RC_NUMBER}" rel/apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER} ); done
+{% endhighlight %}
+
+Now push the release branch and release candidate tag:
+
+{% highlight bash %}
+for m in ${MODULES}; do ( cd $m && git push apache-git $VERSION_NAME && git push apache-git rel/apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER} ); done
+{% endhighlight %}
+
+
 Publish the source and binary distributions to the pre-release area
 -------------------------------------------------------------------
 
@@ -17,8 +33,8 @@ In your workspace for the `dist.apache.org` repo, create a directory with the ar
 mkdir apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER}
 {% endhighlight %}
 
-Copy into this directory all of the artifacts from the previous step - `-src` and `-bin`, `.tar.gz` and `.zip`, and all
-associated `.md5`, `.sha1`, `.sha256` and `.asc` signatures. Then commit:
+Copy into this directory all of the artifacts from the previous step - `-src` and `-bin`, `.tar.gz`, `.zip` and `.rpm`,
+and all associated `.md5`, `.sha1`, `.sha256` and `.asc` signatures. Then commit:
 
 {% highlight bash %}
 svn add apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER}

website/developers/committers/release-process/publish.md

@@ -4,6 +4,21 @@ title: Publish to the public
 navgroup: developers
 ---
 
+Update the canonical Git repository
+-----------------------------------
+
+Make a signed tag for this release:
+
+{% highlight bash %}
+for m in ${MODULES}; do ( cd $m && git tag -s -m "Tag release ${VERSION_NAME}" rel/apache-brooklyn-${VERSION_NAME} rel/apache-brooklyn-${VERSION_NAME}-rc${RC_NUMBER} ); done
+{% endhighlight %}
+
+Now push the release tag:
+
+{% highlight bash %}
+for m in ${MODULES}; do ( cd $m && git push apache-git rel/apache-brooklyn-${VERSION_NAME} ); done
+{% endhighlight %}
+
 Publish the source and binary distributions to the pre-release area
 -------------------------------------------------------------------
 
@@ -40,8 +55,7 @@ Note that the PGP signatures do not embed the filename so they do not need to be
 As a final check, re-test the hashes and signatures:
 
 {% highlight bash %}
-for ext in -src.tar.gz -src.zip -bin.tar.gz -bin.zip; do
-    artifact=apache-brooklyn-${VERSION_NAME}${ext}
+for artifact in *.tar.gz *.zip *.rpm; do
     md5sum -c ${artifact}.md5
     shasum -a1 -c ${artifact}.sha1
     shasum -a256 -c ${artifact}.sha256

website/developers/committers/release-process/release-version.md

@@ -7,15 +7,13 @@ navgroup: developers
 This will allow development to continue on master without affecting the release; it also allows quick-fixes to the
 release branch to address last-minute problems (which must of course be merged/cherry-picked back into master later).
 
-Determine the correct name for the version. Note that while in incubation, we must include “incubating” in our release
-name - common practice for this is to append “-incubating” to the release version.
-
 Do not use -rc1, -rc2 etc. in version strings. Use the version that will be the actual published version. (The artifacts
 that get uploaded to the dist.apache.org/dev will include “-rc1” etc. in the folder name, but the contents will be *as
 final*. Therefore, turning the rc into the final is simply a case of taking the rc file and publishing it to the release
 folder with the correct name.)
 
 References:
+
 - [Post on general@incubator](https://mail-archives.apache.org/mod_mbox/incubator-general/201409.mbox/%3CCAK2iWdS1H9dkJcSdohky6hFqJdP0XyuhAG%2B%3D1Aspxcjt5RmnJw%40mail.gmail.com%3E)
 - [Post on general@incubator](https://mail-archives.apache.org/mod_mbox/incubator-general/201409.mbox/%3CCAOGo0VaEz4cEUbgMgqhh3hiiiubnspiGkQ%3DQv08bOwPqRtzAvQ%40mail.gmail.com%3E)
 
@@ -28,20 +26,26 @@ Create a branch with the same name as the version, based off master:
 {% highlight bash %}
 git checkout master
 git pull --rebase # assumes that the Apache canonical repository is the default upstream for your master - amend if necessary
-git checkout -b $VERSION_NAME
-git push -u apache $VERSION_NAME
+git submodule update --remote --merge --recursive
+for m in $MODULES; do ( cd $m && git checkout master && git checkout -b $VERSION_NAME ); done
 {% endhighlight %}
 
 Now change the version numbers in this branch throughout the project using the script `brooklyn-dist/release/change-version.sh` and commit it:
 
 {% highlight bash %}
 ./brooklyn-dist/release/change-version.sh BROOKLYN $OLD_MASTER_VERSION $VERSION_NAME
-git add .
-# Now inspect the staged changes and ensure there are no surprises
-git commit -m "Change version to $VERSION_NAME"
-git push
+# Now inspect the changes and ensure there are no surprises
+find . -name "*.bak" -delete
+for m in $SUBMODULES; do ( cd $m && git add . && git commit -m "Change version to $VERSION_NAME" ); done
+git add $SUBMODULES && git commit -m "Update submodules to $VERSION_NAME"
+git add . && git commit -m "Change version to $VERSION_NAME"
 {% endhighlight %}
 
+If you are happy with the changes, push them:
+
+{% highlight bash %}
+for m in $MODULES; do ( cd $m && git push apache-git $VERSION ); done
+{% endhighlight %}
 
 
 Update the version on master
@@ -55,10 +59,9 @@ The release notes should be cleared out and the version history augmented with t
 Example:
 
 {% highlight bash %}
-git checkout master
+for m in $MODULES; do ( cd $m && git checkout master ); done
 ./brooklyn-dist/release/change-version.sh BROOKLYN $OLD_MASTER_VERSION $NEW_MASTER_VERSION
-git add .
-# Now inspect the staged changes and ensure there are no surprises
+# Now inspect the changes and ensure there are no surprises
 {% endhighlight %}
 
 Open `brooklyn-docs/guide/misc/release-notes.md` and `brooklyn-docs/website/meta/versions.md` in your favourite editor and amend.
@@ -68,8 +71,16 @@ and putting some placeholder text elsewhere.
 Then:
 
 {% highlight bash %}
-git commit -m "Change version to $NEW_MASTER_VERSION"
-git push
+find . -name "*.bak" -delete
+for m in $SUBMODULES; do ( cd $m && git add . && git commit -m "Change version to $NEW_MASTER_VERSION" ); done
+git add $SUBMODULES && git commit -m "Update submodules to $NEW_MASTER_VERSION"
+git add . && git commit -m "Change version to $NEW_MASTER_VERSION"
+{% endhighlight %}
+
+If you are happy with the changes, push them:
+
+{% highlight bash %}
+for m in $MODULES; do ( cd $m && git push apache-git master ); done
 {% endhighlight %}
 
 
@@ -79,5 +90,5 @@ Switch back to the release branch
 Move back to the release branch:
 
 {% highlight bash %}
-git checkout $VERSION_NAME
+for m in $MODULES; do ( cd $m && git checkout $VERSION_NAME ); done
 {% endhighlight %}