Overhauling release scripts, adding documentation

release_scripts/README.txt

@@ -10,33 +10,32 @@ for signing, take online again to broadcast (upload to Amazon S3).
 The following is assumed to have been done already before starting this 
 process:
 
--- Local & remote machines and VMs have compiled & bundled installers
--- fetchlist.txt contains location data for each installer, specified by
-   cp and scp commands
--- Each remote system has our public key in its authorized_keys file
--- Offline computer has GPG key & Armory wallet spec'd at top of Step2 script
--- All announce files are updated (except for dllinks which will be updated
-   by the script itself once files are signed and hashes are known)
--- The Step2 script contains an accurate list of everything file/installer
--- The computer running Step3 has write-access to the git repo, and a 
-   configuration file with API key for uploading results to Amazon S3
--- Directories on the offline computer containing dependencies for each 
-   OS-specific offline-bundle
--- Already have an installed version of Armory offline in the /usr/lib/armory
-   directory, to be used for creating signature blocks
-
-
-The result of this process will be:
-
-- Signed git tag that can be pushed to the repo
-- All .deb installers will be signed using dpkg-sig
-- Offline bundles using the signed deb files
-- GPG-signed hashes file including all regular installers and offline bundles
-- Append URLs and hashes to dllinks.txt
-- New announce.txt file that contains URLs and hashes of all notify files
-  signed by offline BITCOIN private key
-- Full list of URLs of uploaded installers & bundles in HTML and forum
-  markdown, for easy updating of website and forum posts
+   - Local & remote machines and VMs have compiled & bundled installers
+   - master_list.py file that returns nested dictionary of all release/installer
+      information (see example at end of this doc)
+   - Each remote system has our public key in its authorized_keys file
+   - Offline computer has GPG key & Armory wallet spec'd at top of Step2 script
+   - All announce files are updated (except for dllinks which will be updated
+      by the script itself once files are signed and hashes are known)
+   - The Step2 script contains an accurate list of everything file/installer
+   - The computer running Step3 has write-access to the git repo, and a 
+      configuration file with API key for uploading results to Amazon S3
+   - Directories on the offline computer containing dependencies for each 
+      OS-specific offline-bundle
+   - Already have an installed version of Armory offline in the /usr/lib/armory
+      directory, to be used for creating signature blocks
+
+The output of this process will be:
+
+   - Signed git tag that can be pushed to the repo
+   - All .deb installers will be signed using dpkg-sig
+   - Offline bundles using the signed deb files
+   - GPG-signed hashes file including all regular installers and offline bundles
+   - Append URLs and hashes to dllinks.txt
+   - New announce.txt file that contains URLs and hashes of all notify files
+     signed by offline BITCOIN private key
+   - Full list of URLs of uploaded installers & bundles in HTML and forum
+     markdown, for easy updating of website and forum posts
 
 
 -----
@@ -47,35 +46,55 @@ repo. It should also include updates to the announcement files
 After that, put everything into a single directory that 
 can be copied to a USB key to be taken to the offline computer.
 
-   Directory tree to be transferred to offlie computer:
-
-      unsigned/BitcoinArmory       (clone of repo)
-      unsigned/release_scripts     (copy of release_scripts dir from repo)
-      unsigned/installers          (all non-offline-bundle packages)
-      unsigned/announceFiles       (all unsigned announcement files)
+   Script Arguments (* is optional)
+         argv[0]   <>
+         argv[1]   version string,  "0.91.1"
+         argv[2]   version type,    "-testing", "-beta", ""
+         argv[3]*  output directory      (default ~ ./exportToOffline)
+         argv[4]*  unsigned announce dir (default ~ ./unsignedannounce)
+         argv[5]*  Bitcoin Core SHA256SUMS.asc (default ~ "None")
+
+   Script Output:
+
+      <outputDir>/BitcoinArmory       (clone of repo)
+      <outputDir>/release_scripts     (copy of release_scripts dir from repo)
+      <outputDir>/installers          (all non-offline-bundle packages)
+      <outputDir>/announceFiles       (all unsigned announcement files)
+      <outputDir>/SHA256SUMS.asc      (if present)
+
+Note the release_scripts dir is itself copied because it has master_list.py
+which is needed by all three steps.  Plus, we most likely made tweaks to 
+these Step* scripts to support the current release, and it wouldn't be in 
+the cloned repo yet.  After the release is successful, we commit the updated 
+scripts as the basis for the next release.  
+
 
-Note the release_scripts dir is copied because we likely made modifications
-to it to support the current release, and it wouldn't be in the cloned repo
-yet.  After the release is successful, we commit the updated scripts as the
-basis for the next release.  
 
 
 -----
 Step2 Script:
 
 This script will be executed from the release_scripts directory above --
-we will copy the directory to the offline computer, then cd into
-the unsigned/release_scripts dir, then [modify if necessary and] run the 
+we will copy the directory to the offline computer, then cd into it then run 
 Step2 script from there.  It does not depend on the cloned repo -- it adds
-/usr/lib/armory to its python path, to use the currently installed version
+/usr/lib/armory to sys.path, to use the currently installed version
 of Armory for any non-generic-python operations.
 
-When it's done, it should create a similar directory tree to take back
-to the online computer:
+   Script Arguments (* is optional)
+         argv[0]   <>
+         argv[1]   inputDir  (from Step1)
+         argv[2]   outputDir (for Step3)
+         argv[3]   gpgKeyID
+         argv[4]   btcWltID
+         argv[5]*  git branch to tag (default ~ "master")
+
+   Script Output:
+
+      <outputDir>/BitcoinArmory       (same repo but with signed tag v0.91.1)
+      <outputDir>/release_scripts     (same copy, unmodified)
+      <outputDir>/installers          (now includes offline bundles)
+      <outputDir>/announceFiles       (all txt signed, added announce.txt)
 
-      signed/BitcoinArmory      (now with signed git tag v0.XX-beta)
-      signed/installers         (debs signed, bundles added, signed hash file)
-      signed/announceFiles      (dllinks.txt updated, announce.txt created)
 
 
 
@@ -97,6 +116,17 @@ It will do the following:
          version, you may not want to push the tag)
 
 
+   Script Arguments (* is optional)
+         argv[0]   <>
+         argv[1]   inputDir  (from Step2)
+         argv[5]*  git branch to tag (default ~ "master")
+
+   Script Output:
+
+      <outputDir>/BitcoinArmory       (same repo but with signed tag v0.91.1)
+      <outputDir>/release_scripts     (same copy, unmodified)
+      <outputDir>/installers          (now includes offline bundles)
+      <outputDir>/announceFiles       (all txt signed, added announce.txt)
 
 
 
@@ -105,4 +135,75 @@ It will do the following:
 
 
 
+################################################################################
+#                                                                              #
+# Example master_list.py file                                                  #
+#                                                                              #
+################################################################################
+
+#! /usr/bin/python
+import os
+
+def getMasterPackageList():
+   masterPkgList = {}
+   m = masterPkgList
+
+   pkg = 'Windows (All)'
+   m[pkg] = {}
+   m[pkg]['FetchFrom']  = ['cp', '~/windows_share/armory_%s_win32.exe']
+   m[pkg]['FileSuffix'] = 'winAll.exe'
+   m[pkg]['OSName']     = 'Windows'
+   m[pkg]['OSVar']      = 'XP, Vista, 7, 8+'
+   m[pkg]['OSArch']     = '32- and 64-bit'
+   m[pkg]['HasBundle']  = False
+
+   pkg = 'MacOSX (All)'
+   m[pkg] = {}
+   m[pkg]['FetchFrom']  = ['scp', 'joeschmode', '192.168.1.22', 22, '~/BitcoinArmory/osxbuild/armory_%s_osx.tar.gz']
+   m[pkg]['FileSuffix'] = 'osx.tar.gz'
+   m[pkg]['OSName']     = 'MacOSX'
+   m[pkg]['OSVar']      = '10.7+'
+   m[pkg]['OSArch']     = '64bit'
+   m[pkg]['HasBundle']  = False
+
+
+   pkg = 'Ubuntu 12.04-32bit'
+   m[pkg] = {}
+   m[pkg]['FetchFrom']  = ['scp', 'guest', '192.168.1.23', 5111, '~/buildenv/armory_%s-1_i386.deb']
+   m[pkg]['FileSuffix'] = 'ubuntu-32bit.deb'
+   m[pkg]['OSName']     = 'Ubuntu'
+   m[pkg]['OSVar']      = '12.04+'
+   m[pkg]['OSArch']     = '32bit'
+   m[pkg]['HasBundle']  = True
+   m[pkg]['BundleDeps'] = 'offline_deps_ubuntu32'
+   m[pkg]['BndlSuffix'] = 'offline_ubuntu_12.0432.tar.gz'
+
+
+   pkg = 'Ubuntu 12.04-64bit'
+   m[pkg] = {}
+   m[pkg]['FetchFrom']  = ['scp', 'joe', '192.168.1.80', 5111, '~/buildenv/armory_%s-1_amd64_osx.deb']
+   m[pkg]['FileSuffix'] = 'ubuntu-64bit.deb'
+   m[pkg]['OSName']     = 'Ubuntu'
+   m[pkg]['OSVar']      = '12.04+'
+   m[pkg]['OSArch']     = '64bit'
+   m[pkg]['HasBundle']  = True
+   m[pkg]['BundleDeps'] = 'offline_deps_ubuntu64'
+   m[pkg]['BndlSuffix'] = 'offline_ubuntu64.tar.gz'
+
+   pkg = 'RaspberryPi'
+   m[pkg] = {}
+   m[pkg]['FetchFrom']  = ['cp', '~/buildenv/rpibuild/armory_%s-1.tar.gz']
+   m[pkg]['FileSuffix'] = 'raspbian-armhf.tar.gz'
+   m[pkg]['OSName']     = 'Raspbian'
+   m[pkg]['OSVar']      = None
+   m[pkg]['OSArch']     = 'armhf'
+   m[pkg]['HasBundle']  = True
+   m[pkg]['BundleDeps'] = 'offline_deps_raspbian'
+   m[pkg]['BndlSuffix'] = 'rpi_bundle.tar.gz'
+
+
+   return masterPkgList
+
+
+
 

release_scripts/Step1_Online_PrepareForSigning.py

@@ -2,24 +2,68 @@
 import subprocess
 import os
 import time
-import sys
 import shutil
 import ast
-import fetchInstallers
+import textwrap
+from sys import argv
 from release_utils import execAndWait
 
+#####
+from master_list import masterPkgList
+#####
+
 CLONE_URL = 'https://github.com/etotheipi/BitcoinArmory.git'
 
-verStr   = sys.argv[1]
-typeStr  = sys.argv[2]
-localDir = sys.argv[3]
+if len(argv)<3:
+   print textwrap.dedent("""
+      Script Arguments (* is optional)
+            argv[0]   "python %s"
+            argv[1]   version string,  "0.91.1"
+            argv[2]   version type,    "-testing", "-beta", ""
+            argv[3]*  output directory      (default ~ ./exportToOffline)
+            argv[4]*  path to fetchlist.txt (default ~ ./fetchlist.txt)
+            argv[5]*  unsigned announce dir (default ~ ./unsignedannounce)
+            argv[6]*  Bitcoin Core SHA256SUMS.asc (default ~ "None")
+            """) % argv[0]
+   exit(1)
+
+verStr  = argv[1]
+verType = argv[2]
+outDir  = argv[3] if len(argv)>3 else './exportToOffline'
+annSrc  = argv[4] if len(argv)>5 else './unsignedannounce'
+shaCore = argv[5] if len(argv)>6 else None
+
+if os.path.exists(outDir):
+   shutil.rmtree(outDir)
+os.mkdir(outDir)
+
+check_exists(annSrc)
+if shaCore is not None:
+   check_exists(shaCore)
+   shutil.copy(shaCore, annDst)
 
+# Grab the list of scp and cp commands from fetchlist
+for pkgName,pkgDetails in masterPkgList.iteritems():
+   cmd,cmdArgs = pkgDetails[0],pkgDetails[1:]
+   if cmd=='cp':
+      copyFrom = src % verStr
+      copyTo = os.path.join(outDir,  % (verStr, verType))
+      print 'Copying: %s --> %s' % (copyFrom, copyTo)
+      shutil.copy(copyFrom, copyTo)
+   if cmd=='scp':
+      for usr,ip,port,path,rllist in cplist:
+         for remoteSrc,localDst in rllist:
+            remoteSrc = os.path.join(path, remoteSrc % verStr)
+            hostPath = '%s@%s:%s' % (usr, ip, remoteSrc)
+            localDst  = os.path.join(outDir, localDst % (verStr,verType))
+            execAndWait(['scp', '-P', str(port), hostPath, localPath])
 
-fetchInstallers.doFetch()
-cloneDir = os.path.join(localDir, 'BitcoinArmory')
-rscrDir  = os.path.join(localDir, 'release_scripts')
+cloneDir = os.path.join(outDir, 'BitcoinArmory')
+rscrDir  = os.path.join(outDir, 'release_scripts')
+annDst   = os.path.join(outDir, 'unsignedannounce')
 
 execAndWait(['git', 'clone', CLONE_URL, cloneDir])
 shutil.copytree('../release_scripts', rscrDir)
+shutil.copytree(annSrc, annDst)
 
 

release_scripts/release_utils.py

@@ -172,3 +172,19 @@ def getAllHashes(fnlist):
       hashes.append([fn, out.strip().split()[0]])
    return hashes
 
+
+################################################################################
+def check_exists(fullPath, onDNE='exit'):
+   fullPath = os.path.expanduser(fullPath)
+   if os.path.exists(fullPath):
+      print 'Found file: %s' % fullPath 
+   else:
+      print 'Path does not exist: %s' % fullPath
+      if onDNE=='skip':
+         return None
+      elif onDNE=='exit':
+         exit(1)
+
+   return fullPath
+
+