Rename RELEASE_ENGINEERING.txt to RELEASE_ENGINEERING.md

Add make rule for symlinking and removing /bin files

Makefile

@@ -45,6 +45,19 @@ lock-rpm:
 unlock-rpm:
 	sudo yum versionlock clear
 
+#
+# Manual install
+#
+manual-install:
+	@echo 'Symlinking files from ./bin to /usr/local/bin'
+	@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do ln -fs `pwd`/$$f /usr/local/bin/$$f; done
+	@echo 'Done.'
+
+manual-uninstall:
+	@echo 'Removing blackbox files from /usr/local/bin'
+	@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do rm /usr/local/bin/$$f; done
+	@echo 'Done.'
+
 #
 # DEB builds
 #
@@ -84,7 +97,7 @@ packages-macports: tools/mk_macports.vcs_blackbox.txt
 	mkdir -p $(DESTDIR)/bin
 	cd tools && ./mk_macports mk_macports.vcs_blackbox.txt
 
-# stow is a pretty easy way to manage simple local installs on GNU systems 
+# stow is a pretty easy way to manage simple local installs on GNU systems
 install-stow:
 	mkdir -p /usr/local/stow/blackbox/bin
 	cp bin/* /usr/local/stow/blackbox/bin

README.md

@@ -11,6 +11,31 @@ for Puppet, BlackBox now works with any Git or Mercurial repository.
 
 A slide presentation about an older release [is on SlideShare](http://www.slideshare.net/TomLimoncelli/the-blackbox-project-sfae).
 
+Table of Contents
+===============
+
+* [Overview](#overview)
+* [Why is this important?](#why-is-this-important)
+* [Installation Instructions](#installation-instructions)
+* [Commands](#commands)
+* [Compatibility](#compatibility)
+* [How is the encryption done?](#how-is-the-encryption-done)
+* [What does this look like to the typical user?](#what-does-this-look-like-to-the-typical-user)
+* [How to use the secrets with Puppet?](#how-to-use-the-secrets-with-puppet)
+* [How to enroll a new file into the system?](#how-to-enroll-a-new-file-into-the-system)
+* [How to remove a file from the system?](#how-to-remove-a-file-from-the-system)
+* [How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system)
+* [How to remove a user from the system?](#how-to-remove-a-user-from-the-system)
+* [Enabling Blackbox For a Repo](#enabling-blackbox-for-a-repo)
+* [Set up automated users or "role accounts"](#set-up-automated-users-or-role-accounts)
+* [Some common errors](#some-common-errors)
+* [Using Blackbox without a repo](#using-blackbox-without-a-repo)
+* [How to submit bugs or ask questions?](#how-to-submit-bugs-or-ask-questions)
+* [Developer Info](#developer-info)
+* [Alternatives](#alternatives)
+* [License](#license)
+
+
 Overview
 ========
 
@@ -78,9 +103,10 @@ Installation Instructions:
 ==========================
 
   * *The MacPorts Way*: `sudo port install vcs_blackbox`
-  * *The RPM way*: `make packages-rpm` and now you have an RPM you can install.
-  * *The Debian/Ubuntu way*: Install [fpm](https://github.com/jordansissel/fpm), then `make packages-deb` and now you have a DEB you can install.
+  * *The RPM way*: Check out the repo and make an RPM via `make packages-rpm`; now you can distribute the RPM via local methods.
+  * *The Debian/Ubuntu way*: Check out the repo and install [fpm](https://github.com/jordansissel/fpm). Now you can make a DEB `make packages-deb` that can be distributed via local methods.
   * *The hard way*: Copy all the files in "bin" to your "bin".
+  * *The manual way*: `make manual-install` to install. `make manual-uninstall` to uninstall.
   * *The Antigen Way*: Add `antigen bundle StackExchange/blackbox` to your .zshrc
   * *The Zgen Way*: Add `zgen load StackExchange/blackbox` to your .zshrc where you're loading your other plugins.
 
@@ -194,7 +220,7 @@ their network.
 *If you use Puppet, why didn't you just use hiera-eyaml?*
 There are 4 reasons:
 
-  1. This works works with any Git or Mercurial repo, even if you aren't using Puppet.
+  1. This works with any Git or Mercurial repo, even if you aren't using Puppet.
   2. hiera-eyaml decrypts "on demand" which means your Puppet Master now uses a lot of CPU to decrypt keys every time it is contacted. It slows down your master, which, in my case, is already slow enough.
   3. This works with binary files, without having to ASCIIify them and paste them into a YAML file.  Have you tried to do this with a cert that is 10K long and changes every few weeks?  Ick.
   4. hiera-eyaml didn't exist when I wrote this.
@@ -318,11 +344,15 @@ To join the list of people that can edit the file requires three steps; You crea
 
 ### Step 1: YOU create a GPG key pair on a secure machine and add to public keychain.
 
+If you don't already have a GPG key, here's how to generate one:
+
 ```
 gpg --gen-key
 ```
 
-Pick defaults for encryption settings, 0 expiration.  Pick a VERY GOOD passphrase.
+Pick defaults for encryption settings, 0 expiration.  Pick a VERY GOOD passphrase.  Store the private key securely.  Tip: Store it on a secure machine, or one with little or no internet access, with full-disk-encryption, etc.  Your employer problably has rules about how to store such things.
+
+Now that you have a GPG key, add yourself as an admin:
 
 ```
 blackbox_addadmin KEYNAME
@@ -334,23 +364,41 @@ blackbox_addadmin KEYNAME
 blackbox_addadmin tal@example.com
 ```
 
-When the command completes successfully, instructions on how to
-commit these changes will be output.  Run the command as give.
+When the command completes successfully, instructions on how to commit these changes will be output.  Run the command as given to commit the changes.  It will look like this:
 
 ```
-NEXT STEP: Check these into the repo.  Probably with a command like...
 git commit -m'NEW ADMIN: tal@example.com' keyrings/live/pubring.gpg keyrings/live/trustdb.gpg keyrings/live/blackbox-admins.txt
 ```
 
-Role accounts: If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter:
+Then push it to the repo:
 
 ```
-blackbox_addadmin puppetmaster@puppet-master-1.example.com /path/to/the/dir
+git push
+
+or
+
+ht push
+
+(or whatever is appropriate)
 ```
 
+NOTE: Creating a Role Account?  If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter: `blackbox_addadmin puppetmaster@puppet-master-1.example.com /path/to/the/dir`
+
+
 ### Step 2: SOMEONE ELSE adds you to the system.
 
-Ask someone that already has access to re-encrypt the data files. This gives you access.  They simply decrypt and re-encrypt the data without making any changes:
+Ask someone that already has access to re-encrypt the data files. This gives you access.  They simply decrypt and re-encrypt the data without making any changes.
+
+Pre-check: Verify the new keys look good.
+
+```
+$ gpg --homedir=keyrings/live --list-keys 
+```
+
+For example, examine the key name (email address) to make sure
+it conforms to corporate standards.
+
+Import the keychain into your personal keychain and reencrypt:
 
 ```
 gpg --import keyrings/live/pubring.gpg
@@ -406,7 +454,7 @@ so on just like when anyone that had privileged access leaves an
 organization.
 
 
-First Time Setup (enabling Blackbox for a repo)
+Enabling Blackbox For a Repo
 ===========================
 
 Overview:
@@ -631,8 +679,23 @@ rm -rf /tmp/NEWMASTER
 Also shred any other temporary files you may have made.
 
 
+Some common errors:
+=========================================
+
+`gpg: filename: skipped: No public key` -- Usually this means there
+is an item in `keyrings/live/blackbox-admins.txt` that is not the
+name of the key.  Either something invalid was inserted (like a
+filename instead of a username) or a user has left the organization
+and their key was removed from the keychain, but their name wasn't
+removed from the blackbox-admins.txt file.
+
+`gpg: decryption failed: No secret key` -- Usually means you forgot
+to re-encrypt the file with the new key.
+
+
 Using Blackbox without a repo
 ===========================
+
 If the files are copied out of a repo they can still be decrypted
 and edited.  Obviously edits, changes to keys, and such will be lost
 if they are made outside the repo. Also note that commands are most
@@ -646,27 +709,51 @@ The following commands have been tested outside a repo:
   * `blackbox_edit_end`
 
 
-Help out: Submit bugs, pull requests and ideas:
+How to submit bugs or ask questions?
 ============
 
-I welcome code changes, questions, bug reports and feedback!
+We welcome questions, bug reports and feedback!
 
-  * Submit code: https://github.com/StackExchange/blackbox
-  * Report bugs/questions: https://github.com/StackExchange/blackbox/issues
+  * https://github.com/StackExchange/blackbox/issues
 
-Tip for submitting code:
+Developer Info
+============
 
-After you make a change, please re-run the confidence tests.  This
-runs through various procedures and checks the results.
+Code submissions are gladly welcomed!  The code is
+fairly easy to read.
 
-To run the tests:
+
+Get the code:
+
+```
+git clone git@github.com:StackExchange/blackbox.git
+```
+
+Test your changes:
 
 ```
 make confidence
 ```
 
+This runs through a number of system tests. It
+creates a repo, encrypts files, decrypts files, and so on.
+You can run these tests to verify that the changes you made
+didn't break anything.  You can also use these tests to
+verify that the system works with a new operating system.
+
+Please submit tests with code changes:
+
+The best way to change Blackbox is via Test Driven Development.
+First add a test to `tools/confidence.sh`. This test should
+fail, and demonstrate the need for the change you are about to
+make. Then fix the bug or add the feature you want.  When
+you are done, `make confidence` should pass all tests.
+The PR you submit should include your code as well as the new
+test.  This way the confidence tests accumulate as the system
+grows as we know future changes don't break old features.
+
 Note: The tests currently assume "git" and have been tested
-on CentOS and Cygwin.
+only on CentOS, Mac OS X, and Cygwin.  Patches welcome!
 
 
 Alternatives
@@ -674,9 +761,12 @@ Alternatives
 
 Here are other open source packages that do something similar to Blackbox. If you like them better than Blackbox, please use them.
 
+  * git-crypt:  https://www.agwa.name/projects/git-crypt/
   * Pass: http://www.zx2c4.com/projects/password-store/
   * Transcrypt: https://github.com/elasticdog/transcrypt
-  * git-crypt:  https://www.agwa.name/projects/git-crypt/
+
+git-crypt has the best git integration.  Once set up it is nearly transparent to the users.  However it only works with git.
+
 
 License
 =======

RELEASE_ENGINEERING.md

@@ -66,25 +66,25 @@ github.setup and checksums updated.```
 * Type: `update`
 * Component: `ports`
 * Port: `vcs_blackbox`
-* Keywords: `maintainer`
+* Keywords: `maintainer haspatch`
 * Attach this file: `Portfile-vcs_blackbox.diff`
 
 Step 3: Watch for the update to happen.
 
 # Updating MacPorts (manual)
 
-This is the old, manual, procedure.  If the automated procedure work, these notes may or may not be helpful.
+This is the old, manual, procedure.  If the automated procedure fails to work, these notes may or may not be helpful.
 
-The ultimate result of the script should be a `diff -u Portfile.orig Portfile`.  The new `Portfile` should have these changes:
+The ultimate result of the script should be the output of `diff -u Portfile.orig Portfile` which is sent as an attachment to MacPorts.  The new `Portfile` should have these changes:
 
 1. The `github.setup` line should have a new version number.
 2. The `checksums` line(s) should have updated checksums.
 
 How to generate the checksums?
 
-The easiest way is to to make a Portfile with incorrect checksums, then run `sudo port -v checksum vcs_blackbox` to see what they should have been.  Fix the file, and try again.
+The easiest way is to to make a Portfile with incorrect checksums, then run `sudo port -v checksum vcs_blackbox` to see what they should have been.  Fix the file, and try again until the checksum command works.
 
-When the the checksum command works, run `port lint vcs_blackbox` and make sure it has no errors.
+Next run `port lint vcs_blackbox` and make sure it has no errors.
 
 Some useful commands:
 
@@ -95,12 +95,12 @@ sudo vi /opt/local/etc/macports/sources.conf
   file:///var/tmp/ports
 ```
 
-Add a local repo in an automated manner:
+Add a local repo:
 ```
 fgrep >/dev/null -x 'file:///var/tmp/ports' /opt/local/etc/macports/sources.conf || sudo sed -i -e '1s@^@file:///var/tmp/ports\'$'\n@' /opt/local/etc/macports/sources.conf
 ```
 
-Remove the local repo in an automated manner:
+Remove the local repo:
 ```
 sudo sed -i -e '\@^file:///var/tmp/ports@d' /opt/local/etc/macports/sources.conf
 ```
@@ -116,4 +116,4 @@ cp Portfile /var/tmp/ports/security/vcs_blackbox
 cd /var/tmp/ports && portindex
 sudo port -v checksum vcs_blackbox
 sudo port install vcs_blackbox
-```
+```

bin/_blackbox_common.sh

@@ -28,7 +28,7 @@ source "${0%/*}"/_stack_lib.sh
 function physical_directory_of() {
   local d=$(dirname "$1")
   local f=$(basename "$1")
-  (cd "$d" && echo $(pwd -P)"/$f" )
+  (cd "$d" && echo "$(pwd -P)/$f" )
 }
 
 # Set REPOBASE to the top of the repository
@@ -329,7 +329,11 @@ function enumerate_blackbox_repos() {
 function vcs_relative_path() {
   # Usage: vcs_relative_path file
   local name="$1"
-  python -c 'import os ; print(os.path.relpath("'"$(pwd -P)"'/'"$name"'", "'"$REPOBASE"'"))'
+  #python -c 'import os ; print(os.path.relpath("'"$(pwd -P)"'/'"$name"'", "'"$REPOBASE"'"))'
+  local p=$( printf "%s" "$( pwd -P )/${1}" | sed 's#//*#/#g' )
+  local name="${p#$REPOBASE}"
+  name=$( printf "%s" "$name" | sed 's#^/##g' | sed 's#/$##g' )
+  printf "%s" "$name"
 }
 
 # Removes a line from a text file
@@ -367,14 +371,11 @@ function md5sum_file() {
     Darwin )
       md5 -r "$1" | awk '{ print $1 }'
       ;;
-    Linux )
-      md5sum "$1" | awk '{ print $1 }'
-      ;;
-    CYGWIN* )
+    Linux | CYGWIN* | MINGW* )
       md5sum "$1" | awk '{ print $1 }'
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (md5sum_file)'
       exit 1
       ;;
   esac
@@ -386,11 +387,11 @@ function cp_permissions() {
     Darwin )
       chmod $( stat -f '%p' "$1" ) "${@:2}"
       ;;
-    Linux | CYGWIN* )
+    Linux | CYGWIN* | MINGW* )
       chmod --reference "$1" "${@:2}"
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (cp_permissions)'
       exit 1
       ;;
   esac

bin/_stack_lib.sh

@@ -54,14 +54,14 @@ function create_self_deleting_tempfile() {
 
   case $(uname -s) in
     Darwin )
-      : ${TMPDIR:=/tmp} ;
+      : "${TMPDIR:=/tmp}"
       filename=$(mktemp -t _stacklib_.XXXXXXXX )
       ;;
-    Linux )
+    Linux | CYGWIN* | MINGW* )
       filename=$(mktemp)
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (create_self_deleting_tempfile)'
       exit 1
       ;;
   esac
@@ -75,14 +75,14 @@ function create_self_deleting_tempdir() {
 
   case $(uname -s) in
     Darwin )
-      : ${TMPDIR:=/tmp} ;
-      filename=$(mktemp -d -t _stacklib_ )
+      : "${TMPDIR:=/tmp}"
+      filename=$(mktemp -d -t _stacklib_.XXXXXXXX )
       ;;
-    Linux )
+    Linux | CYGWIN* | MINGW* )
       filename=$(mktemp -d)
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (create_self_deleting_tempdir)'
       exit 1
       ;;
   esac
@@ -99,14 +99,14 @@ function make_self_deleting_tempfile() {
 
   case $(uname -s) in
     Darwin )
-      : ${TMPDIR:=/tmp} ;
-      name=$(mktemp -t _stacklib_ )
+      : "${TMPDIR:=/tmp}"
+      name=$(mktemp -t _stacklib_.XXXXXXXX )
       ;;
-    Linux )
+    Linux | CYGWIN* | MINGW* )
       name=$(mktemp)
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (make_self_deleting_tempfile)'
       exit 1
       ;;
   esac
@@ -121,14 +121,14 @@ function make_tempdir() {
 
   case $(uname -s) in
     Darwin )
-      : ${TMPDIR:=/tmp} ;
-      name=$(mktemp -d -t _stacklib_ )
+      : "${TMPDIR:=/tmp}"
+      name=$(mktemp -d -t _stacklib_.XXXXXXXX )
       ;;
-    Linux )
+    Linux | CYGWIN* | MINGW* )
       name=$(mktemp -d)
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (make_tempdir)'
       exit 1
       ;;
   esac
@@ -164,14 +164,7 @@ function fail_if_in_root_directory() {
         exit 1
       fi
       ;;
-    Linux )
-      if [[ $(stat -c'%i' / ) == $(stat -c'%i' . ) ]] ; then
-        echo 'SECURITY ALERT: The current directory is the root directory.'
-        echo 'Exiting...'
-        exit 1
-      fi
-      ;;
-    CYGWIN* )
+    Linux | CYGWIN* | MINGW* )
       if [[ $(stat -c'%i' / ) == $(stat -c'%i' . ) ]] ; then
         echo 'SECURITY ALERT: The current directory is the root directory.'
         echo 'Exiting...'
@@ -179,7 +172,7 @@ function fail_if_in_root_directory() {
       fi
       ;;
     * )
-      echo 'ERROR: Unknown OS. Exiting.'
+      echo 'ERROR: Unknown OS. Exiting. (fail_if_in_root_directory)'
       exit 1
       ;;
   esac

bin/blackbox_edit

@@ -27,7 +27,7 @@ for param in "$@" ; do
     esac
   fi
   "${BLACKBOX_HOME}/blackbox_edit_start" "$unencrypted_file"
-  "$EDITOR" "$(get_unencrypted_filename "$unencrypted_file")"
+  $EDITOR "$(get_unencrypted_filename "$unencrypted_file")"
   "${BLACKBOX_HOME}/blackbox_edit_end" "$unencrypted_file"
 
 done

tools/macports_report_upgrade.sh

@@ -43,7 +43,7 @@ fi
 
 # Generate the diff
 cp /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/security/vcs_blackbox/Portfile /var/tmp/ports/security/vcs_blackbox/Portfile.orig
-( cd /var/tmp/ports/security/vcs_blackbox && diff -u Portfile.orig Portfile ) > Portfile-vcs_blackbox.diff
+( cd /var/tmp/ports/security/vcs_blackbox && diff --ignore-matching-lines='Id:' -u Portfile.orig Portfile ) > Portfile-vcs_blackbox.diff
 open -R Portfile-vcs_blackbox.diff
 
 echo
@@ -62,6 +62,6 @@ github.setup and checksums updated.
     Type: update
     Component: ports
     Port: vcs_blackbox
-    Keywords: maintainer
+    Keywords: maintainer haspatch
 "
 echo 'Attach: Portfile-vcs_blackbox.diff'