Safely store secrets in a VCS repo (i.e. Git, Mercurial, Subversion or Perforce). These commands make it easy for you to Gnu Privacy Guard (GPG) encrypt specific files in a repo so they are "encrypted at rest" in your repository. However, the scripts make it easy to decrypt them when you need to view or edit them, and decrypt them for use in production. Originally written for Puppet, BlackBox now works with any Git or Mercurial repository.
WARNING: The goal of this project is to be a simple wrapper around `gpg` so you and your coworkers don't have to remember its all those inscrutable and confusing flags. It is *not* intented to be a sophisticated encryption system that solves all problems or supports a large numbers of files. The ideal use-case is to keep secrets in a secure service such as Conjur, AWS KMS, Azure Key Vault or GCP KMS; then use Blackbox for safely storing the API keys needed to access that system. That way you are encrypting one tiny file.
Suppose you have a VCS repository (i.e. a Git or Mercurial repo) and certain files contain secrets such as passwords or SSL private keys. Often people just store such files "and hope that nobody finds them in the repo". That's not safe.
With BlackBox, those files are stored encrypted using GPG. Access to the VCS repo without also having the right GPG keys makes it worthless to have the files. As long as you keep your GPG keys safe, you don't have to worry about storing your VCS repo on an untrusted server. Heck, even if you trust your server, now you don't have to trust the people that do backups of that server, or the people that handle the backup tapes!
Rather than one GPG passphrase for all the files, each person with access has their own GPG keys in the system. Any file can be decrypted by anyone with their GPG key. This way, if one person leaves the company, you don't have to communicate a new password to everyone with access. Simply disable the one key that should no longer have access. The process for doing this is as easy as running 2 commands (1 to disable their key, 1 to re-encrypt all files.)
Automated processes often need access to all the decrypted files. This is easy too. For example, suppose Git is being used for Puppet files. The master needs access to the decrypted version of all the files. Simply set up a GPG key for the Puppet master (or the role account that pushes new files to the Puppet master) and have that user run `blackbox_postdeploy` after any files are updated.
NOT SO OBVIOUSLY when we store "secrets" in a VCS repo like Git or Mercurial, suddenly we are less able to share our code with other people. Communication between subteams of an organization is hurt. You can't collaborate as well. Either you find yourself emailing individual files around (yuck!), making a special repo with just the files needed by your collaborators (yuck!!), or just deciding that collaboration isn't worth all that effort (yuck!!!).
The ability to be open and transparent about our code, with the exception of a few specific files, is key to the kind of collaboration that DevOps and modern IT practitioners need to do.
- *The hard way (automatic)*: `make copy-install` will copy the bin files into $PREFIX/bin, default is /usr/local (uninstall with `make copy-uninstall`).
- *The symlinks way*: `make symlinks-install` will make symlinks of the bin files into $PREFIX/bin, default is /usr/local (uninstall with `make copy-uninstall`) (useful when doing development)
- *The MacPorts Way*: `sudo port install vcs_blackbox`
- *The Homebrew Way*: `brew install blackbox`
- *The RPM way*: Check out the repo and make an RPM via `make packages-rpm`; now you can distribute the RPM via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).)
- *The Debian/Ubuntu way*: Check out the repo and make a DEB via `make packages-deb`; now you can distribute the DEB via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).)
- *The Antigen Way*: Add `antigen bundle StackExchange/blackbox` to your .zshrc
BlackBox automatically determines which VCS you are using and does the right thing. It has a plug-in architecture to make it easy to extend to work with other systems. It has been tested to work with many operating systems.
To add or fix support for a new operating system, look for the case statements in `bin/_blackbox_common.sh` and `bin/_stack_lib.sh` and maybe `tools/confidence_test.sh`
- [Git for Windows](https://git-scm.com/) (not tested with Mercurial)
- Git Bash MINTTY returns a MinGW console. So when you install make sure you pick `MINTTY` instead of windows console. You'll be executing blackbox from the Git Bash prompt.
- You need at least version 2.8.1 of Git for Windows.
- [GnuWin32](https://sourceforge.net/projects/getgnuwin32/files/) - needed for various tools not least of which is mktemp which is used by blackbox
- after downloading the install just provides you with some batch files. Because of prior issues at sourceforge and to make sure you get the latest version of each package the batch files handle the brunt of the work of getting the correct packages and installing them for you.
- from a **windows command prompt** run `download.bat` once it has completed run `install.bat` then add the path for those tools to your PATH (ex: `PATH=%PATH%;c:\GnuWin32\bin`)
If you have 5 people ("admins") that should be able to access the secrets, each creates a GPG key and adds their public key to the keychain. The GPG command used to encrypt the file lists all 5 key names, and therefore any 1 key can decrypt the file.
To remove someone's access, remove that admin's key name (i.e. email address) from the list of admins and re-encrypt all the files. They can still read the .gpg file (assuming they have access to the repository) but they can't decrypt it any more.
*What if they kept a copy of the old repo before you removed access?* Yes, they can decrypt old versions of the file. This is why when an admin leaves the team, you should change all your passwords, SSL certs, and so on. You should have been doing that before BlackBox, right?
*Why don't you use symmetric keys?* In other words, why mess with all this GPG key stuff and instead why don't we just encrypt all the files with a single passphrase. Yes, GPG supports that, but then we are managing a shared password, which is fraught with problems. If someone "leaves the team" we would have to communicate to everyone a new password. Now we just have to remove their key. This scales better.
*How do automated processes decrypt without asking for a password?* GPG requires a passphrase on a private key. However, it permits the creation of subkeys that have no passphrase. For automated processes, create a subkey that is only stored on the machine that needs to decrypt the files. For example, at Stack Exchange, when our Continuous Integration (CI) system pushes a code change to our Puppet masters, they run `blackbox_postdeploy` to decrypt all the files. The user that runs this code has a subkey that doesn't require a passphrase. Since we have many masters, each has its own key. And, yes, this means our Puppet Masters have to be very secure. However, they were already secure because, like, dude... if you can break into someone's puppet master you own their network.
*If you use Puppet, why didn't you just use hiera-eyaml?* There are 4 reasons:
1. This works with any Git or Mercurial repo, even if you aren't using Puppet.
2. hiera-eyaml decrypts "on demand" which means your Puppet Master now uses a lot of CPU to decrypt keys every time it is contacted. It slows down your master, which, in my case, is already slow enough.
3. This works with binary files, without having to ASCIIify them and paste them into a YAML file. Have you tried to do this with a cert that is 10K long and changes every few weeks? Ick.
Wait... it can be even easier than that! Run `blackbox_edit FILENAME`, and it'll decrypt the file in a temp file and call `$EDITOR` on it, re-encrypting again after the editor is closed.
Entire files, such as SSL certs and private keys, are treated just like regular files. You decrypt them any time you push a new release to the puppet master.
Small strings, such as passwords and API keys, are stored in a hiera yaml file, which you encrypt with `blackbox_register_new_file`. For example, we use a file called `blackbox.yaml`. You can access them using the hiera() function.
`.blackbox/blackbox-admins.txt` is a file that lists which users are able to decrypt files. (More pedantically, it is a list of the GnuPG key names that the file is encrypted for.)
To join the list of people that can edit the file requires three steps; You create a GPG key and add it to the key ring. Then, someone that already has access adds you to the system. Lastly, you should test your access.
Pick defaults for encryption settings, 0 expiration. Pick a VERY GOOD passphrase. Store a backup of the private key someplace secure. For example, keep the backup copy on a USB drive that is locked in safe. Or, at least put it on a secure machine with little or no internet access, full-disk-encryption, etc. Your employer probably has rules about how to store such things.
When the command completes successfully, instructions on how to commit these changes will be output. Run the command as given to commit the changes. It will look like this:
NOTE: Creating a Role Account? If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter: `blackbox_addadmin puppetmaster@puppet-master-1.example.com /path/to/the/dir`
Ask someone that already has access to re-encrypt the data files. This gives you access. They simply decrypt and re-encrypt the data without making any changes.
Note that their keys will still be in the key ring, but they will go unused. If you'd like to clean up the keyring, use the normal GPG commands and check in the file.
Remember that this person did have access to all the secrets at one time. They could have made a copy. Therefore, to be completely secure, you should change all passwords, generate new SSL keys, and so on just like when anyone that had privileged access leaves an organization.
Blackbox stores its configuration data in the `.blackbox` subdirectory. Older
repos use `keyrings/live`. For backwards compatibility either will work.
All documentation refers to `.blackbox`.
You can convert an old repo by simply renaming the directory:
```
mv keyrings/live .blackbox
rmdir keyrings
```
There is no technical reason to convert old repos except that it is less
confusing to users.
This change was made in commit 60e782a0, release v1.20180615.
The details:
- First Blackbox checks `$BLACKBOXDATA`. If this environment variable is set, this is the directory that will be used. If it lists a directory that does not exist, Blackbox will print an error and exit.
- If `$BLACKBOXDATA` is not set: (which is the typical use case)
- Blackbox will first try `keyrings/live` and use it if it exists.
- Otherwise the default `.blackbox` will be used. If `.blackbox` does not exist, Blackbox will print an error and exit.
Once that is done, is a good idea to test the system by making sure a file can be added to the system (see "How to enroll a new file into the system?"), and a different user can decrypt the file.
The next step is to commit `foo.txt.gpg` and make sure another user can check out, view, and change the contents of the file. That is left as an exercise for the reader. If you are feel like taking a risk, don't commit `foo.txt.gpg` and delete it instead.
An automated user (a "role account") is one that that must be able to decrypt without a passphrase. In general you'll want to do this for the user that pulls the files from the repo to the master. This may be automated with Jenkins CI or other CI system.
GPG keys have to have a passphrase. However, passphrases are optional on subkeys. Therefore, we will create a key with a passphrase then create a subkey without a passphrase. Since the subkey is very powerful, it should be created on a very secure machine.
There's another catch. The role account probably can't check files into Git/Mercurial. It probably only has read-only access to the repo. That's a good security policy. This means that the role account can't be used to upload the subkey public bits into the repo.
Therefore, we will create the key/subkey on a secure machine as yourself. From there we can commit the public portions into the repo. Also from this account we will export the parts that the role account needs, copy them to where the role account can access them, and import them as the role account.
NOTE: Rather than a real email address, use the username@FQDN of the host the key will be used on. If you use this role account on many machines, each should have its own key. By using the FQDN of the host, you will be able to know which key is which. In this doc, we'll refer to username@FQDN as $KEYNAME
Any files that were temporarily copied in the first step so as to not be overwritten can now be copied back and re-encrypted with the `blackbox_edit_end` command.
(Thanks to @chishaku for finding a solution to this problem!)
`gpg: filename: skipped: No public key` -- Usually this means there is an item in `.blackbox/blackbox-admins.txt` that is not the name of the key. Either something invalid was inserted (like a filename instead of a username) or a user has left the organization and their key was removed from the keychain, but their name wasn't removed from the blackbox-admins.txt file.
`Error: can't re-encrypt because a key has expired.` -- A user's key has expired and can't be used to encrypt any more. Follow the [Replace expired keys](#replace-expired-keys) tip.
If the files are copied out of a repo they can still be decrypted and edited. Obviously edits, changes to keys, and such will be lost if they are made outside the repo. Also note that commands are most likely to only work if run from the base directory (i.e. the parent to the .blackbox directory).
The current implementation will store the blackbox in `/keyrings` at the root of the entire repo. This will create an issue between environments that have different roots (i.e. checking out `/` on development vs `/releases/foo` in production). To get around this, you can `export BLACKBOX_REPOBASE=/path/to/repo` and set a specific base for your repo.
This was originally written for git and supports a two-phase commit, in which `commit` is a local commit and "push" sends the change upstream to the version control server when something is registered or deregistered with the system. The current implementation will immediately `commit` a file (to the upstream subversion server) when you execute a `blackbox_*` command.
This runs through a number of system tests. It creates a repo, encrypts files, decrypts files, and so on. You can run these tests to verify that the changes you made didn't break anything. You can also use these tests to verify that the system works with a new operating system.
The best way to change BlackBox is via Test Driven Development. First add a test to `tools/confidence.sh`. This test should fail, and demonstrate the need for the change you are about to make. Then fix the bug or add the feature you want. When you are done, `make confidence` should pass all tests. The PR you submit should include your code as well as the new test. This way the confidence tests accumulate as the system grows as we know future changes don't break old features.
