Update RELEASE_ENGINEERING.md

Add support for NetBSD and SunOS (SmartOS)

CHANGELOG.md

@@ -1,3 +1,19 @@
+Release v1.20181219
+
+* New OS support: Add support for NetBSD and SunOS (SmartOS)
+* Testing: Improve confidence test.
+* .blackbox is now the default config directory for new repos. (#272)
+* Add blackbox_decrypt_file (#270)
+* Improved compatibility: change"/bin/[x]" to "/usr/bin/env [x]" (#265)
+* Add blackbox_less. (#263)
+* add nix method of install (#261)
+* Linked setting up of GPG key (#260)
+
+
+Release v1.20180618
+
+* Restore `make manual-install` with warning. (#258)
+
 Release v1.20180615
 
 * Standardize on .blackbox for config. Use keyrings/live for backwards compatibility.

Makefile

@@ -59,6 +59,13 @@ symlinks-install:
 	@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do ln -fs `pwd`/$$f /usr/local/bin/$$f; done
 	@echo 'Done.'
 
+manual-install:
+	@echo '***************************************************************'
+	@echo '*                      DEPRECATED                             *'
+	@echo '* `make manual-install` is now called `make symlinks-install` *'
+	@echo '***************************************************************'
+	$(MAKE) symlinks-install
+
 copy-install:
 	@echo 'Copying files from ./bin to /usr/local/bin'
 	@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do cp `pwd`/$$f /usr/local/bin/$$f; done

README.md

@@ -22,11 +22,14 @@ Table of Contents
 - [How to use the secrets with Puppet?](#how-to-use-the-secrets-with-puppet)
   - [Entire files](#entire-files)
   - [Small strings](#small-strings)
-- [How to enroll a new file into the system?](#how-to-enroll-a-new-file-into-the-system)
-- [How to remove a file from the system?](#how-to-remove-a-file-from-the-system)
-- [How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system)
-- [How to remove a user from the system?](#how-to-remove-a-user-from-the-system)
-- [Enabling BlackBox For a Repo](#enabling-blackbox-for-a-repo)
+- File Management
+  - [How to enroll a new file into the system?](#how-to-enroll-a-new-file-into-the-system)
+  - [How to remove a file from the system?](#how-to-remove-a-file-from-the-system)
+- User Management
+  - [How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system)
+  - [How to remove a user from the system?](#how-to-remove-a-user-from-the-system)
+- Repo Management
+  - [Enabling BlackBox For a Repo](#enabling-blackbox-for-a-repo)
 - [Set up automated users or “role accounts”](#set-up-automated-users-or-role-accounts)
 - [Replacing expired keys](#replacing-expired-keys)
 - [Some common errors](#some-common-errors)
@@ -50,7 +53,22 @@ Rather than one GPG passphrase for all the files, each person with access has th
 
 Automated processes often need access to all the decrypted files. This is easy too. For example, suppose Git is being used for Puppet files. The master needs access to the decrypted version of all the files. Simply set up a GPG key for the Puppet master (or the role account that pushes new files to the Puppet master) and have that user run `blackbox_postdeploy` after any files are updated.
 
-Getting started is easy. Just `cd` into a Git, Mercurial, Subversion or Perforce repository and run `blackbox_initialize`. After that, if a file is to be encrypted, run `blackbox_register_new_file` and you are done. Add and remove keys with `blackbox_addadmin` and `blackbox_removeadmin`. To view and/or edit a file, run `blackbox_edit`; this will decrypt the file and open with whatever is specified by your $EDITOR environment variable. When you close the editor the file will automatically be encrypted again and the temporary plaintext file will be shredded. If you need to leave the file decrypted while you update you can use the`blackbox_edit_start` to decrypt the file and `blackbox_edit_end` when you want to "put it back in the box."
+Getting started is looks like this.
+First, if you don't have a GPG key, set it up using instructions
+such as:
+[Set up GPG key](https://help.github.com/articles/generating-a-new-gpg-key/).
+Now you are ready to go.
+`cd` into a Git, Mercurial, Subversion
+or Perforce repository and run `blackbox_initialize`. After that,
+if a file is to be encrypted, run `blackbox_register_new_file` and
+you are done. Add and remove keys with `blackbox_addadmin` and
+`blackbox_removeadmin`. To view and/or edit a file, run `blackbox_edit`;
+this will decrypt the file and open with whatever is specified by
+your $EDITOR environment variable. When you close the editor the
+file will automatically be encrypted again and the temporary plaintext
+file will be shredded. If you need to leave the file decrypted while
+you update you can use the`blackbox_edit_start` to decrypt the file
+and `blackbox_edit_end` when you want to "put it back in the box."
 
 Why is this important?
 ======================
@@ -73,6 +91,7 @@ Installation Instructions
 - *The Debian/Ubuntu way*: Check out the repo and make a DEB via `make packages-deb`; now you can distribute the DEB via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).)
 - *The Antigen Way*: Add `antigen bundle StackExchange/blackbox` to your .zshrc
 - *The Zgen Way*: Add `zgen load StackExchange/blackbox` to your .zshrc where you're loading your other plugins.
+- *The Nix Way*: `nix-env -i blackbox`
 
 Commands
 ========
@@ -83,12 +102,14 @@ Commands
 | `blackbox_edit_start <file>`        | Decrypt a file so it can be updated                                     |
 | `blackbox_edit_end <file>`          | Encrypt a file after blackbox_edit_start was used                       |
 | `blackbox_cat <file>`               | Decrypt and view the contents of a file                                 |
+| `blackbox_view <file>`              | Like blackbox_cat but pipes to `less` or $PAGER                         |
 | `blackbox_diff`                     | Diff decrypted files against their original crypted version             |
 | `blackbox_initialize`               | Enable blackbox for a GIT or HG repo                                    |
 | `blackbox_register_new_file <file>` | Encrypt a file for the first time                                       |
 | `blackbox_deregister_file <file>`   | Remove a file from blackbox                                             |
 | `blackbox_list_files`               | List the files maintained by blackbox                                   |
 | `blackbox_list_admins`              | List admins currently authorized for blackbox                           |
+| `blackbox_decrypt_file <file>`      | Decrypt a file                                                          |
 | `blackbox_decrypt_all_files`        | Decrypt all managed files (INTERACTIVE)                                 |
 | `blackbox_postdeploy`               | Decrypt all managed files (batch)                                       |
 | `blackbox_addadmin <gpg-key>`       | Add someone to the list of people that can encrypt/decrypt secrets      |
@@ -113,6 +134,8 @@ BlackBox automatically determines which VCS you are using and does the right thi
   - MacOS X
   - Cygwin (Thanks, Ben Drasin!) **See Note Below**
   - MinGW (git bash on windows) **See Note Below**
+  - NetBSD
+  - SmartOS
 
 To add or fix support for a VCS system, look for code at the end of `bin/_blackbox_common.sh`
 
@@ -348,6 +371,7 @@ Ask someone that already has access to re-encrypt the data files. This gives you
 Pre-check: Verify the new keys look good.
 
 ```
+git pull    # Or whatever is required for your system
 gpg --homedir=.blackbox --list-keys
 ```
 

RELEASE_ENGINEERING.md

@@ -25,6 +25,17 @@ Build Tasks
 Stable Releases
 ===============
 
+Step 0. Test the software
+
+Run this command to run the unit and system tests:
+
+```
+make test
+```
+
+NOTE: The tests require pinentry-tty. On macOS with NIX this
+can be installed via: `nix-env -i pinentry`
+
 Marking the software to be "stable":
 
 Step 1.  Update CHANGELOG.md

bin/_blackbox_common.sh

@@ -16,10 +16,13 @@ source "${0%/*}"/_stack_lib.sh
 : "${BLACKBOX_HOME:="$(cd "${0%/*}" ; pwd)"}" ;
 
 # What are the candidates for the blackbox data directory?
+#
+# The order of candidates matter. The first entry of the array
+# sets the default Blackbox directory for all new repositories.
 declare -a BLACKBOXDATA_CANDIDATES
 BLACKBOXDATA_CANDIDATES=(
-  'keyrings/live'
   '.blackbox'
+  'keyrings/live'
 )
 
 # If $EDITOR is not set, set it to "vi":
@@ -140,7 +143,7 @@ function fail_if_not_on_cryptlist() {
 
   if ! is_on_cryptlist "$name" ; then
     echo "ERROR: $name not found in $BB_FILES" >&2
-    echo "PWD=$(/bin/pwd)" >&2
+    echo "PWD=$(/usr/bin/env pwd)" >&2
     echo 'Exiting...' >&2
     exit 1
   fi
@@ -408,6 +411,12 @@ function md5sum_file() {
     Darwin | FreeBSD )
       md5 -r "$1" | awk '{ print $1 }'
       ;;
+    NetBSD )
+      md5 -q "$1"
+      ;;
+    SunOS )
+      digest -a md5 "$1"
+      ;;
     Linux | CYGWIN* | MINGW* )
       md5sum "$1" | awk '{ print $1 }'
       ;;
@@ -424,10 +433,13 @@ function cp_permissions() {
     Darwin )
       chmod $( stat -f '%p' "$1" ) "${@:2}"
       ;;
-    FreeBSD )
+    FreeBSD | NetBSD )
       chmod $( stat -f '%p' "$1" | sed -e "s/^100//" ) "${@:2}"
       ;;
-    Linux | CYGWIN* | MINGW* )
+    SunOS )
+      chmod $( stat -c '%a' "$1" ) "${@:2}"
+      ;;
+    Linux | CYGWIN* | MINGW* | SunOS )
       if [[ -e /etc/alpine-release ]]; then
         chmod $( stat -c '%a' "$1" ) "${@:2}"
       else

bin/_stack_lib.sh

@@ -57,7 +57,7 @@ function create_self_deleting_tempfile() {
       : "${TMPDIR:=/tmp}" ;
       filename=$(mktemp -t _stacklib_.XXXXXXXX )
       ;;
-    Linux | CYGWIN* | MINGW* )
+    Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
       filename=$(mktemp)
       ;;
     * )
@@ -78,7 +78,7 @@ function create_self_deleting_tempdir() {
       : "${TMPDIR:=/tmp}" ;
       filename=$(mktemp -d -t _stacklib_.XXXXXXXX )
       ;;
-    Linux | CYGWIN* | MINGW* )
+    Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
       filename=$(mktemp -d)
       ;;
     * )
@@ -102,7 +102,7 @@ function make_self_deleting_tempfile() {
       : "${TMPDIR:=/tmp}" ;
       name=$(mktemp -t _stacklib_.XXXXXXXX )
       ;;
-    Linux | CYGWIN* | MINGW* )
+    Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
       name=$(mktemp)
       ;;
     * )
@@ -127,7 +127,7 @@ function make_tempdir() {
       # which needs to fit within sockaddr_un.sun_path (see unix(7)).
       name=$(mktemp -d -t SO )
       ;;
-    Linux | CYGWIN* | MINGW* )
+    Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
       name=$(mktemp -d)
       ;;
     * )
@@ -160,14 +160,14 @@ function fail_if_not_running_as_root() {
 function fail_if_in_root_directory() {
   # Verify nobody has tricked us into being in "/".
   case $(uname -s) in
-    Darwin | FreeBSD )
+    Darwin | FreeBSD | NetBSD )
       if [[ $(stat -f'%i' / ) == $(stat -f'%i' . ) ]] ; then
         echo 'SECURITY ALERT: The current directory is the root directory.'
         echo 'Exiting...'
         exit 1
       fi
       ;;
-    Linux | CYGWIN* | MINGW* )
+    Linux | CYGWIN* | MINGW* | SunOS )
       if [[ $(stat -c'%i' / ) == $(stat -c'%i' . ) ]] ; then
         echo 'SECURITY ALERT: The current directory is the root directory.'
         echo 'Exiting...'

bin/blackbox_decrypt_file

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+#
+# blackbox_decrypt_file -- Decrypt one or more blackbox files.
+#
+
+set -e
+source "${0%/*}/_blackbox_common.sh"
+
+if [ $# -eq 0 ]; then
+  echo >&2 "Please provide at least one file to decrypt"
+  exit 1
+fi
+
+"${BLACKBOX_HOME}/blackbox_edit_start" "$@"

bin/blackbox_shred_all_files

@@ -39,7 +39,12 @@ export -f exported_internal_shred_file
 DEREFERENCED_BIN_DIR="${0%/*}"
 MAX_PARALLEL_SHRED=10
 
+bash_args=
+if bash --help | grep import-functions >/dev/null 2>/dev/null; then
+  bash_args=--import-functions
+fi
+
 export IFS=
-tr '\n' '\0' <"$BB_FILES" | xargs -0 -I{} -n 1 -P $MAX_PARALLEL_SHRED bash -c "exported_internal_shred_file $DEREFERENCED_BIN_DIR \"{}\"" $DEREFERENCED_BIN_DIR/fake
+tr '\n' '\0' <"$BB_FILES" | xargs -0 -I{} -P $MAX_PARALLEL_SHRED bash $bash_args -c "exported_internal_shred_file $DEREFERENCED_BIN_DIR \"{}\"" $DEREFERENCED_BIN_DIR/fake
 
 echo '========== DONE.'

bin/blackbox_view

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# blackbox_view -- Decrypt a file, view it, shred it
+#
+set -e
+source "${0%/*}/_blackbox_common.sh"
+
+for param in "$@" ; do
+  shreddable=0
+  unencrypted_file=$(get_unencrypted_filename "$param")
+  if [[ ! -e "$unencrypted_file" ]]; then
+    "${BLACKBOX_HOME}/blackbox_edit_start" "$param"
+    shreddable=1
+  fi
+  ${PAGER:-less} "$unencrypted_file"
+  if [[ $shreddable = 1 ]]; then
+    shred_file "$unencrypted_file"
+  fi
+done

blackbox.plugin.zsh

@@ -1,4 +1,4 @@
-#!/bin/zsh
+#!/usr/bin/env zsh
 # The MIT License (MIT)
 
 # Copyright (c) 2014 Stack Exchange, Inc.

tools/confidence_test.sh

@@ -1,10 +1,13 @@
 #!/usr/bin/env bash
 
 blackbox_home=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../bin
-export PATH="${blackbox_home}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/local/bin:${blackbox_home}"
+export PATH="${blackbox_home}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/local/bin:/usr/pkg/bin:/usr/pkg/gnu/bin:${blackbox_home}"
 
 export LANG=C.UTF-8  # Required ro "gpg --export" to work properly.
 
+# TODO(tlim): The tests are hardcoded to this directory. This should be fixed.
+export BLACKBOXDATA=keyrings/live
+
 # This script requires many utilities, some are not
 # required by the usual blackbox scripts.  Test to make
 # sure we have them all.

tools/profile.d-usrblackbox-test.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env sh
 
 # Test profile.d-usrblackbox.sh
 

tools/test_functions.sh

@@ -24,6 +24,12 @@ function md5sum_file() {
     Darwin | FreeBSD )
       md5 -r "$1" | awk '{ print $1 }'
       ;;
+    NetBSD )
+      md5 -q "$1"
+      ;;
+    SunOS )
+      digest -a md5 "$1"
+      ;;
     Linux )
       md5sum "$1" | awk '{ print $1 }'
       ;;
@@ -47,7 +53,7 @@ function assert_file_missing() {
 function assert_file_exists() {
   if [[ ! -e "$1" ]]; then
     echo "ASSERT FAILED: ${1} should exist."
-    echo "PWD=$(/bin/pwd -P)"
+    echo "PWD=$(/usr/bin/env pwd -P)"
     #echo "LS START"
     #ls -la
     #echo "LS END"
@@ -72,10 +78,10 @@ function assert_file_group() {
   assert_file_exists "$file"
 
   case $(uname -s) in
-    Darwin|FreeBSD )
+    Darwin | FreeBSD | NetBSD )
       found=$(stat -f '%Dg' "$file")
       ;;
-    Linux )
+    Linux | SunOS )
       found=$(stat -c '%g' "$file")
       ;;
     CYGWIN* )
@@ -102,11 +108,11 @@ function assert_file_perm() {
   assert_file_exists "$file"
 
   case $(uname -s) in
-    Darwin|FreeBSD )
+    Darwin | FreeBSD | NetBSD )
       found=$(stat -f '%Sp' "$file")
       ;;
     # NB(tlim): CYGWIN hasn't been tested. It might be more like Darwin.
-    Linux | CYGWIN* )
+    Linux | CYGWIN* | SunOS )
       found=$(stat -c '%A' "$file")
       ;;
     * )