george/blackbox
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement blackbox in Golang (#250)

Browse Source 
* Initial release
This commit is contained in:
Tom Limoncelli
2020-07-24 14:21:33 -04:00
committed by GitHub
parent e049c02655
commit 1c77c87555
86 changed files with 6074 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
docs/README.md Normal file
View File

@@ -0,0 +1,61 @@
BlackBox
========
Blackbox is an open source tool that enables you to safe store sensitive information in
Git (or other) repos by encrypting them with GPG. Only the encrypted
version of the file is available. You can be free to provide access
to the repo, as but only people with the right GPG keys can access the
encrypted data.
Things you should **never** store in a repo without encryption:
* TLS (SSL) certificates
* Passwords
* API keys
* And more!
Project Info:
* [Overview](user-overview.md)
* [Why is this important?](why-is-this-important.md)
* [Support/Community](support.md)
* [How BB encrypts](encryption.md)
* [OS Compatibility](compatibility.md)
* [Installation Instructions](installation.md)
* [Alternatives](alternatives.md)
User Info:
* [Enabling Blackbox on a Repo](enable-repo.md)
* [Enroll a file](enable-repo.md)
* [Full Command List](full-command-list.md)
* [Add/Remove users](admin-ops.md)
* [Add/Remove files](file-ops.md)
* [Advanced techiques](advanced.md)
* [Use with Role Accounts](role-accounts.md)
* [Backwards Compatibility](backwards-compatibility.md)
* [Replacing expired keys](expired-keys.md)
* [Git Tips](git-tips.md)
* [SubVersion Tips](subversion-tips.md)
* [GnuPG tips](gnupg-tips.md)
* [Use with Ansible](with-ansible.md)
* [Use with Puppet](with-puppet.md)
For contributors:
* [Developer Info](dev.md)
* [Code overview](dev-code-overview.md)
* [HOWTO: Add new OS support](dev-add-os-support.md)
* [HOWTO: Add new VCS support](dev-add-vcs-support.md)
A slide presentation about an older release [is on SlideShare](http://www.slideshare.net/TomLimoncelli/the-blackbox-project-sfae).
Join our mailing list: [https://groups.google.com/d/forum/blackbox-project](https://groups.google.com/d/forum/blackbox-project)
License
=======
This content is released under the MIT License.
See the [LICENSE.txt](LICENSE.txt) file.

148
docs/admin-ops.md Normal file
View File

@@ -0,0 +1,148 @@
User Management
===============
# Who are the current admins?
```
blackbox admin list
```
# Add a new user (admin)
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
`.blackbox/blackbox-admins.txt` is a file that lists which users are able to decrypt files. (More pedantically, it is a list of the GnuPG key names that the file is encrypted for.)
To join the list of people that can edit the file requires three steps; You create a GPG key and add it to the key ring. Then, someone that already has access adds you to the system. Lastly, you should test your access.
## Step 1: NEWPERSON creates a GPG key pair on a secure machine and add to public keychain.
If you don't already have a GPG key, here's how to generate one:
```
gpg --gen-key
```
WARNING: New versions of GPG generate keys which are not understood by
old versions of GPG. If you generate a key with a new version of GPG,
this will cause problems for users of older versions of GPG.
Therefore it is recommended that you either assure that everyone using
Blackbox have the exact same version of GPG, or generate GPG keys
using a version of GPG as old as the oldest version of GPG used by
everyone using Blackbox.
Pick defaults for encryption settings, 0 expiration. Pick a VERY GOOD
passphrase. Store a backup of the private key someplace secure. For
example, keep the backup copy on a USB drive that is locked in safe.
Or, at least put it on a secure machine with little or no internet
access, full-disk-encryption, etc. Your employer probably has rules
about how to store such things.
FYI: If generating the key is slow, this is usually because the system
isn't generating enough entropy. Tip: Open another window on that
machine and run this command: `ls -R /`
Now that you have a GPG key, add yourself as an admin:
```
blackbox admin add KEYNAME
```
...where "KEYNAME" is the email address listed in the gpg key you created previously. For example:
```
blackbox admin add tal@example.com
```
When the command completes successfully, instructions on how to commit these changes will be output. Run the command as given to commit the changes. It will look like this:
```
git commit -m'NEW ADMIN: tal@example.com' .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
```
Then push it to the repo:
```
git push
or
ht push
(or whatever is appropriate)
```
NOTE: Creating a Role Account? If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter: `blackbox admin add puppetmaster@puppet-master-1.example.com /path/to/the/dir`
## Step 2: AN EXISTING ADMIN accepts you into the system.
Ask someone that already has access to re-encrypt the data files. This
gives you access. They simply decrypt and re-encrypt the data without
making any changes.
Pre-check: Verify the new keys look good.
```
git pull # Or whatever is required for your system
gpg --homedir=.blackbox --list-keys
```
For example, examine the key name (email address) to make sure it conforms to corporate standards.
Import the keychain into your personal keychain and reencrypt:
```
gpg --import .blackbox/pubring.gpg
blackbox reencrypt --all shred
```
Push the re-encrypted files:
```
git commit -a
git push
or
hg commit
hg push
```
### Step 3: NEWPERSON tests.
Make sure you can decrypt a file. (Suggestion: Keep a dummy file in
VCS just for new people to practice on.)
# Remove a user
Simply run `blackbox admin remove` with their keyname then re-encrypt:
Example:
```
blackbox admin remove olduser@example.com
blackbox reencrypt --all shred
```
When the command completes, you will be given a reminder to check in the change and push it.
Note that their keys will still be in the key ring, but they will go unused. If you'd like to clean up the keyring, use the normal GPG commands and check in the file.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
```
gpg --homedir=.blackbox --list-keys
gpg --homedir=.blackbox --delete-key olduser@example.com
git commit -m'Cleaned olduser@example.com from keyring' .blackbox/*
```
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
The key ring only has public keys. There are no secret keys to delete.
Remember that this person did have access to all the secrets at one time. They could have made a copy. Therefore, to be completely secure, you should change all passwords, generate new SSL keys, and so on just like when anyone that had privileged access leaves an organization.

46
docs/advanced.md Normal file
View File

@@ -0,0 +1,46 @@
Advanced Techniques
===================
# Using Blackbox without a repo
If the files are copied out of a repo they can still be decrypted and
edited. Obviously edits, changes to keys, and such will be lost if
they are made outside the repo. Also note that commands are most
likely to only work if run from the base directory (i.e. the parent to
the .blackbox directory).
Without a repo, all commands must be run from the same directory
as the ".blackbox" directory. It might work otherwise but no
promises.
# Mixing gpg 1.x/2.0 and 2.2
WARNING: Each version of GnuPG uses a different, and incompatible,
binary format to store the keychain. When Blackbox was originally
created, I didn't know this. Things are mostly upwards compatible.
That said, if you have some admins with GnuPG 1.x and others with GnuPG 2.2,
you may corrupt the keychain.
A future version will store the keychain in an GnuPG-approved
version-neutral format.
# Having gpg and gpg2 on the same machine
NOTE: This is not implemented at this time. TODO(tlim) Use GPG to find
the binary.
In some situations, team members or automated roles need to install gpg
2.x alongside the system gpg version 1.x to catch up with the team's gpg
version. On Ubuntu 16, you can ```apt-get install gnupg2``` which
installs the binary gpg2. If you want to use this gpg2 binary, run every
blackbox command with GPG=gpg2.
For example:
```
GPG=gpg2 blackbox_postdeploy
```

14
docs/alternatives.md Normal file
View File

@@ -0,0 +1,14 @@
Alternatives
============
Here are other open source packages that do something similar to
BlackBox. If you like them better than BlackBox, please use them.
- [git-crypt](https://www.agwa.name/projects/git-crypt/)
- [Pass](http://www.zx2c4.com/projects/password-store/)
- [Transcrypt](https://github.com/elasticdog/transcrypt)
- [Keyringer](https://keyringer.pw/)
- [git-secret](https://github.com/sobolevn/git-secret)
git-crypt has the best git integration. Once set up it is nearly
transparent to the users. However it only works with git.

55
docs/backwards-compatibility.md Normal file
View File

@@ -0,0 +1,55 @@
Backwards Compatibility
=======================
# Where is the configuration stored? .blackbox vs. keyrings/live
Blackbox stores its configuration data in the `.blackbox` subdirectory. Older
repos use `keyrings/live`. For backwards compatibility either will work.
All documentation refers to `.blackbox`.
You can convert an old repo by simply renaming the directory:
```
mv keyrings/live .blackbox
rmdir keyrings
```
There is no technical reason to convert old repos except that it is less
confusing to users.
This change was made in commit 60e782a0, release v1.20180615.
# How blackbox fines the config directory:
## Creating the repo:
`blackbox init` creates the config directory in the root
of the repo. Here's how it picks the name:
- If `$BLACKBOX_TEAM` is set, `.blackbox-$BLACKBOX_TEAM` is used.
- If the flag `--team <teamname>` is set, it uses `.blackbox-<teamname>`
- Otherwise, it uses `.blackbox`
When searching for the configuration directory, the following
locations are checked. First match wins.
- `.blackbox-$BLACKBOX_TEAM` (only if `$BLACKBOX_TEAM` is set)
- The value of `--config value` (if the flag is set)
- `$BLACKBOX_CONFIGDIR` (the preferred env. variable to use)
- `$BLACKBOXDATA` (for backwards compatibility with v1)
- `.blackbox`
- `keyrings/live` (for backwards compatibility)
NOTE: The env variables and `--config` should be set to the full path
to the config directory (i.e.: `/Users/tom/gitstuff/myrepo/.blackbox`).
If it is set to a relative directory (i.e. `.blackbox` or
`../myrepo/.blackbox`) most commands will break.
NOTE: Why the change from `$BLACKBOXDATA` to `$BLACKBOX_CONFIGDIR`? We want
all the env. variables to begin with the prefix `BLACKBOX_`. If v1
supported another name, that is still supported. If you are starting
with v2 and have no other users using v1, please use the `BLACKBOX_`
prefix.

78
docs/compatibility.md Normal file
View File

@@ -0,0 +1,78 @@
Compatibility
=============
# Compatibility with Blackbox v1
The command names all changed from v1 to v2. The `binv2` directory
includes shell scripts that provide full backwards compatibility.
# Supported Architectures
Blackbox supports a plug-in archtecture to easily support multiple VCS
system. Current support is for:
## Supported VCS/DVCS systems
* git
* "none" (repo-less use is supported)
* WOULD LOVE VOLUNTEERS TO HELP ADD SUPPORT FOR: hg, svn, p4
## Supported GPG versions
* Git 1.x and 2.0
* Git 2.2 and higher
* WOULD LOVE VOLUNTEERS TO HELP ADD SUPPORT FOR:
golang.org/x/crypto/openpgp (this would make the code have no
external dependencies)
## Supported Operating systems
Blackbox should work on any Linux system with GnuPG installed.
Blackbox simply looks for `gpg` in `$PATH`.
Windows: It should work (but has not been extensively tested) on
Windows WSL2.
# Automated testing
While many combinations work, we do automated tests
on these combinations. If any of these fail it blocks the release:
* macOS: GnuPG 2.2 executables from https://gpgtools.org/
* CentOS: GnuPG 2.0.x executables from the "base" or "updates" repo.
Windows native: VOLUNTEER NEEDED to make a native Windows version
(should be rather simple as Go does most of the work)
NOTE: Version 1 worked on CentOS/RedHat, macOS, Gygwin, WinGW, NetBSD,
and SmartOS. Hopefully we can achieve that broad level of support in
the future. Any system that is supported by the Go language and
has GuPG 2.0.x or higher binaries available should be easy to achieve.
We'd also like to have automated testing for the same.
# Windows Support
BlackBox assumes that `blackbox-admins.txt` and `blackbox-files.txt` will have
LF line endings. Windows users should be careful to configure Git or other systems
to not convert or "fix" those files.
If you use Git, add the following lines to your `.gitattributes` file:
**/blackbox-admins.txt text eol=lf
**/blackbox-files.txt text eol=lf
The `blackbox init` (and newer versions of `blackbox_initialize`)
will create an appropriate `.gitattributes` file for you.
# Cygwin
TODO: List what packages are required for building the software.
TODO: List what packages are required for running the software.
# MinGW
MinGW (comes with Git for Windows) support requires the following:
TODO: FILL IN any requirements

36
docs/dev-code-overview.md Normal file
View File

@@ -0,0 +1,36 @@
Code Overview
=============
Here is how the code is laid out.
TODO(tlim): Add a diagram of the layers
```
cmd/blackbox/ The command line tool.
blackbox.go main()
cli.go Definition of all subcommands and flags
drive.go Processes flags and calls functions in verbs.go
NOTE: These are the only files that are aware of the
flags. Everything else gets the flag data passed to it
as a parameter. This way the remaining system can be
used as a module.
pkg/box/ High-level functions related to "the black box".
verbs.go One function per subcommand.
box.go Functions for manipulating the files in .blackbox
boxutils.go Helper functions for the above.
pkg/bblog/ Module that provides logging facilities.
pkg/bbutil/ Functions that are useful to box, plug-ins, etc.
pkg/tainedname/ Module for printing filenames escaped for Bash.
models/vcs.go The interface that defines a VCS plug-in.
models/crypters.go The interface that defines a GPG plug-in.
pkg/crypters/ Plug-ins for GPG functionality.
pkg/crypters/gnupg Plug-in that runs an external gpg binary (found via $PATH)
pkg/vcs/ Plug-ins for VCS functionality.
pkg/vcs/none Repo-less mode.
pkg/vcs/git Git mode.
```

36
docs/dev.md Normal file
View File

@@ -0,0 +1,36 @@
Developer Info
==============
Code submissions are gladly welcomed! The code is fairly easy to read.
Get the code:
```
git clone git@github.com:StackExchange/blackbox.git
```
Test your changes:
```
go test ./...
```
This runs through a number of system tests. It creates a repo,
encrypts files, decrypts files, and so on. You can run these tests to
verify that the changes you made didn't break anything. You can also
use these tests to verify that the system works with a new operating
system.
Please submit tests with code changes:
The best way to change BlackBox is via Test Driven Development. First
add a test to `tools/confidence.sh`. This test should fail, and
demonstrate the need for the change you are about to make. Then fix
the bug or add the feature you want. When you are done, `make
confidence` should pass all tests. The PR you submit should include
your code as well as the new test. This way the confidence tests
accumulate as the system grows as we know future changes don't break
old features.
Note: More info about compatibility are on the [Compatibility Page](compatibility.md)

58
docs/enable-repo.md Normal file
View File

@@ -0,0 +1,58 @@
Enabling Blackbox on a Repo
===========================
Overview:
1. Run the initialization command
2. Add at least one admin.
3. Add files. (don't add files before the admins)
The long version:
1. If you don't have a GPG key, set it up using instructions such as:
[Set up GPG key](https://help.github.com/articles/generating-a-new-gpg-key/). \
Now you are ready to go.
1. `cd` into a Git, Mercurial, Subversion or Perforce repository and run `blackbox init`.
1. Add yourself with `blackbox admin add YOUR@EMAIL`
1. Commit the files as directed.
That's it!
At this point you should encrypt a file and make sure you can decrypt
it. This verifies that everything is working as expected.
1. Pick a file to be encrypted. Since this is a test, you might want
to create a test file. Call it `secret.txt` and edit the file
so that it includes your mother's maiden name. Just kidding!
Store this sentence: `This is my test file.`
2. Run `blackbox file add secret.txt`
3. Decode the encrypted version: `blackbox cat secret.txt`
The "cat" subcommand only accesses the encrypted (`.gpg`) file and is
a good way to see that the file was encrypted properly. You should
see `This is my test file.`
4 Verify that editing the file works.
To view and/or edit a file, run `blackbox edit --shred secret.txt`
Now encrypt it and shred the original:
```
blackbox encrypt --shred secret.txt
```
Now make sure you can decrypt the new file:
```
blackbox cat secret.txt
```
You should see the changed text.
Now commit and push `secret.txt.gpg` and you are done!

54
docs/encryption.md Normal file
View File

@@ -0,0 +1,54 @@
How is the encryption done?
===========================
GPG has many different ways to encrypt a file. BlackBox uses the mode
that lets you specify a list of keys that can decrypt the message.
If you have 5 people ("admins") that should be able to access the
secrets, each creates a GPG key and adds their public key to the
keychain. The GPG command used to encrypt the file lists all 5 key
names, and therefore any 1 key can decrypt the file.
Blackbox stores a copy of the public keys of all admins. It never
stores the private keys.
To remove someone's access, remove that admin's key name (i.e. email
address) from the list of admins and re-encrypt all the files. They
can still read the .gpg file (assuming they have access to the
repository) but they can't decrypt it any more.
*What if they kept a copy of the old repo before you removed access?*
Yes, they can decrypt old versions of the file. This is why when an
admin leaves the team, you should change all your passwords, SSL
certs, and so on. You should have been doing that before BlackBox,
right?
*Why don't you use symmetric keys?* In other words, why mess with all
this GPG key stuff and instead why don't we just encrypt all the files
with a single passphrase. Yes, GPG supports that, but then we are
managing a shared password, which is fraught with problems. If someone
"leaves the team" we would have to communicate to everyone a new
password. Now we just have to remove their key. This scales better.
*How do automated processes decrypt without asking for a password?*
GPG requires a passphrase on a private key. However, it permits the
creation of subkeys that have no passphrase. For automated processes,
create a subkey that is only stored on the machine that needs to
decrypt the files. For example, at Stack Exchange, when our Continuous
Integration (CI) system pushes a code change to our Puppet masters,
they run `blackbox decrypt --all --overwrite` to decrypt all the files.
The user that
runs this code has a subkey that doesn't require a passphrase. Since
we have many masters, each has its own key. And, yes, this means our
Puppet Masters have to be very secure. However, they were already
secure because, like, dude... if you can break into someone's puppet
master you own their network.
*If you use Puppet, why didn't you just use hiera-eyaml?* There are 4
reasons:
1. This works with any Git or Mercurial repo, even if you aren't using Puppet.
2. hiera-eyaml decrypts "on demand" which means your Puppet Master now uses a lot of CPU to decrypt keys every time it is contacted. It slows down your master, which, in my case, is already slow enough.
3. This works with binary files, without having to ASCIIify them and paste them into a YAML file. Have you tried to do this with a cert that is 10K long and changes every few weeks? Ick.
4. hiera-eyaml didn't exist when I wrote this. (That's the real reason.)

62
docs/expired-keys.md Normal file
View File

@@ -0,0 +1,62 @@
Replacing expired keys
======================
If someone's key has already expired, blackbox will stop
encrypting. You see this error:
```
$ blackbox_edit_end modified_file.txt
--> Error: can't re-encrypt because a key has expired.
```
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
You can also detect keys that are about to expire by issuing this command and manually reviewing the "expired:" dates:
gpg --homedir=.blackbox --list-keys
or... list UIDs that will expire within 1 month from today: (Warning: this also lists keys without an expiration date)
gpg --homedir=.blackbox --list-keys --with-colons --fixed-list-mode | grep ^uid | awk -F: '$6 < '$(( $(date +%s) + 2592000))
Here's how to replace the key:
- Step 1. Administrator removes expired user:
Warning: This process will erase any unencrypted files that you were in the process of editing. Copy them elsewhere and restore the changes when done.
```
blackbox_removeadmin expired_user@example.com
# This next command overwrites any changed unencrypted files. See warning above.
blackbox_update_all_files
git commit -m "Re-encrypt all files"
gpg --homedir=.blackbox --delete-key expired_user@example.com
git commit -m 'Cleaned expired_user@example.com from keyring' .blackbox/*
git push
```
- Step 2. Expired user adds an updated key:
```
git pull
blackbox_addadmin updated_user@example.com
git commit -m'NEW ADMIN: updated_user@example.com .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
git push
```
- Step 3. Administrator re-encrypts all files with the updated key of the expired user:
```
git pull
gpg --import .blackbox/pubring.gpg
blackbox_update_all_files
git commit -m "Re-encrypt all files"
git push
```
- Step 4: Clean up:
Any files that were temporarily copied in the first step so as to not be overwritten can now be copied back and re-encrypted with the `blackbox_edit_end` command.
(Thanks to @chishaku for finding a solution to this problem!)

55
docs/file-ops.md Normal file
View File

@@ -0,0 +1,55 @@
How to add/remove a file into the system?
=========================================
# Adding files:
- If you need to, start the GPG Agent: `eval $(gpg-agent --daemon)`
- Add the file to the system:
```
blackbox file add path/to/file.name.key
# If you want to delete the old plaintext:
blackbox file add --shred path/to/file.name.key
```
Multiple file names can be specified on the command line:
Example 1: Register 2 files:
```
blackbox file add --shred file1.txt file2.txt
```
Example 2: Register all the files in `$DIR`:
```
find $DIR -type f -not -name '*.gpg' -print0 | xargs -0 blackbox file add
```
# Removing files
This command
```
blackbox file remove path/to/file.name.key
```
TODO(tlim): Add examples.
# List files
To see what files are currently enrolled in the system:
```
blackbox file list
```
You can also see their status:
```
blackbox status
blackbox status just_one_file.txt
blackbox status --type ENCRYPTED
```

34
docs/full-command-list.md Normal file
View File

@@ -0,0 +1,34 @@
Blackbox Command List
=====================
## Global Flags
### `--vcs`
### `--crypto`
### `--config`
### `--team`
### `--editor`
### `--umask`
### `--debug`
### `--help`
### `--help`
### `--version`
## User Commands
### `blackbox decrypt`
### `blackbox encrypt`
### `blackbox edit`
### `blackbox cat`
### `blackbox diff`
### `blackbox shred`
### `blackbox help`
## User Commands
### `blackbox init`
### `blackbox admin`
### `blackbox file`
### `blackbox status`
### `blackbox reencrypt`
## Debug
### `blackbox info`
## Integration Test (secret menu)
### `blackbox testing_init`
TODO(tlim): Can we automatically generate this? The data is all in cli.go

22
docs/git-tips.md Normal file
View File

@@ -0,0 +1,22 @@
GIT tips
========
# Configure git to show diffs in encrypted files
It's possible to tell Git to decrypt versions of the file before running them through `git diff` or `git log`. To achieve this do:
- Add the following to `.gitattributes` at the top of the git repository:
```
*.gpg diff=blackbox
```
- Add the following to `.git/config`:
```
[diff "blackbox"]
textconv = gpg --use-agent -q --batch --decrypt
````
Commands like `git log -p file.gpg` and `git diff master --` will display as expected.

31
docs/gnupg-tips.md Normal file
View File

@@ -0,0 +1,31 @@
GnuPG tips
==========
# Common error messages
* Message: `gpg: filename: skipped: No public key`
* Solution: Usually this means there is an item in
`.blackbox/blackbox-admins.txt` that is not the name of the key.
Either something invalid was inserted (like a filename instead of a
username) or a user has left the organization and their key was
removed from the keychain, but their name wasn't removed from the
blackbox-admins.txt file.
* Message: `gpg: decryption failed: No secret key`
* Solution: Usually means you forgot to re-encrypt the file with the new key.
* Message: `Error: can't re-encrypt because a key has expired.`
* Solution: A user's key has expired and can't be used to encrypt any more. Follow the [Replace expired keys](expired-keys.md) page.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
# GnuPG problems
Blackbox is just a front-end to GPG. If you get into a problem with a
key or file, you'll usually have better luck asking for advice on
the gnupg users mailing list TODO: Get link to this list
The author of Blackbox is not a GnuPG expert. He wrote Blackbox
because it was better than trying to remember GPG's horrible flag
names.

17
docs/installation.md Normal file
View File

@@ -0,0 +1,17 @@
Installation Instructions
=========================
Currently blackbox v2 is installed by compiling the code and
copying the binary someplace:
TODO:
```
git clone FILL IN
```
Future: We will have RPM, DEB, Chocolately packages.
Next step: [Enable on a repo](enable-repo.md)

158
docs/role-accounts.md Normal file
View File

@@ -0,0 +1,158 @@
Set up automated users or "role accounts"
=========================================
TODO(tlim): I think this is overly complex. With GnuPG 2.2 and later,
you can use `--password '' --quick-generate-key userid` and you are
done. No need for subkeys. Maybe rework this?
With role accounts, you have an automated system that needs to be able
to decrypt secrets without a password. This means the security of your
repo is based on how locked down the automation system is. This
is risky, so be careful.
i.e. This is how a Puppet Master can have access to the unencrypted data.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
An automated user (a "role account") is one that that must be able to decrypt without a passphrase. In general you'll want to do this for the user that pulls the files from the repo to the master. This may be automated with Jenkins CI or other CI system.
GPG keys have to have a passphrase. However, passphrases are optional on subkeys. Therefore, we will create a key with a passphrase then create a subkey without a passphrase. Since the subkey is very powerful, it should be created on a very secure machine.
There's another catch. The role account probably can't check files into Git/Mercurial. It probably only has read-only access to the repo. That's a good security policy. This means that the role account can't be used to upload the subkey public bits into the repo.
Therefore, we will create the key/subkey on a secure machine as yourself. From there we can commit the public portions into the repo. Also from this account we will export the parts that the role account needs, copy them to where the role account can access them, and import them as the role account.
ProTip: If asked to generate entropy, consider running this on the same machine in another window: `sudo dd if=/dev/sda of=/dev/null`
For the rest of this doc, you'll need to make the following substitutions:
- ROLEUSER: svc_deployacct or whatever your role account's name is.
- NEWMASTER: the machine this role account exists on.
- SECUREHOST: The machine you use to create the keys.
NOTE: This should be more automated/scripted. Patches welcome.
On SECUREHOST, create the puppet master's keys:
```
$ mkdir /tmp/NEWMASTER
$ cd /tmp/NEWMASTER
$ gpg --homedir . --gen-key
Your selection?
(1) RSA and RSA (default)
What keysize do you want? (2048) DEFAULT
Key is valid for? (0) DEFAULT
# Real name: Puppet CI Deploy Account
# Email address: svc_deployacct@hostname.domain.name
```
NOTE: Rather than a real email address, use the username@FQDN of the host the key will be used on. If you use this role account on many machines, each should have its own key. By using the FQDN of the host, you will be able to know which key is which. In this doc, we'll refer to username@FQDN as $KEYNAME
Save the passphrase somewhere safe!
Create a sub-key that has no password:
```
$ gpg --homedir . --edit-key svc_deployacct
gpg> addkey
(enter passphrase)
Please select what kind of key you want:
(3) DSA (sign only)
(4) RSA (sign only)
(5) Elgamal (encrypt only)
(6) RSA (encrypt only)
Your selection? 6
What keysize do you want? (2048)
Key is valid for? (0)
Command> key 2
(the new subkey has a "*" next to it)
Command> passwd
(enter the main key's passphrase)
(enter an empty passphrase for the subkey... confirm you want to do this)
Command> save
```
Now securely export this directory to NEWMASTER:
```
gpg --homedir . --export -a svc_sadeploy >/tmp/NEWMASTER/pubkey.txt
tar cvf /tmp/keys.tar .
rsync -avP /tmp/keys.tar NEWMASTER:/tmp/.
```
On NEWMASTER, receive the new GnuPG config:
```
sudo -u svc_deployacct bash
mkdir -m 0700 -p ~/.gnupg
cd ~/.gnupg && tar xpvf /tmp/keys.tar
```
<!---
Back on SECUREHOST, import the pubkey into the repository.
```
$ cd .blackbox
$ gpg --homedir . --import /tmp/NEWMASTER/pubkey.txt
```
-->
Back on SECUREHOST, add the new email address to .blackbox/blackbox-admins.txt:
```
cd /path/to/the/repo
blackbox_addadmin $KEYNAME /tmp/NEWMASTER
```
Verify that secring.gpg is a zero-length file. If it isn't, you have somehow added a private key to the keyring. Start over.
```
cd .blackbox
ls -l secring.gpg
```
Commit the recent changes:
```
cd .blackbox
git commit -m"Adding key for KEYNAME" pubring.gpg trustdb.gpg blackbox-admins.txt
```
Regenerate all encrypted files with the new key:
```
blackbox_update_all_files
git status
git commit -m"updated encryption" -a
git push
```
On NEWMASTER, import the keys and decrypt the files:
```
sudo -u svc_sadeploy bash # Become the role account.
gpg --import /etc/puppet/.blackbox/pubring.gpg
export PATH=$PATH:/path/to/blackbox/bin
blackbox_postdeploy
sudo -u puppet cat /etc/puppet/hieradata/blackbox.yaml # or any encrypted file.
```
ProTip: If you get "gpg: decryption failed: No secret key" then you forgot to re-encrypt blackbox.yaml with the new key.
On SECUREHOST, securely delete your files:
```
cd /tmp/NEWMASTER
# On machines with the "shred" command:
shred -u /tmp/keys.tar
find . -type f -print0 | xargs -0 shred -u
# All else:
rm -rf /tmp/NEWMASTER
```
Also shred any other temporary files you may have made.

21
docs/subversion-tips.md Normal file
View File

@@ -0,0 +1,21 @@
Subversion Tips
===============
NOTE: This is from v1. Can someone that uses Subversion check
this and update it?
The current implementation will store the blackbox in `/keyrings` at
the root of the entire repo. This will create an issue between
environments that have different roots (i.e. checking out `/` on
development vs `/releases/foo` in production). To get around this, you
can `export BLACKBOX_REPOBASE=/path/to/repo` and set a specific base
for your repo.
This was originally written for git and supports a two-phase commit,
in which `commit` is a local commit and "push" sends the change
upstream to the version control server when something is registered or
deregistered with the system. The current implementation will
immediately `commit` a file (to the upstream subversion server) when
you execute a `blackbox_*` command.

16
docs/support.md Normal file
View File

@@ -0,0 +1,16 @@
Support
=======
# Join our community!
Join the [blackbox-project mailing list](https://groups.google.com/d/forum/blackbox-project)!
# How to submit bugs or ask questions?
We welcome questions, bug reports and feedback!
The best place to start is to join the [blackbox-project mailing list](https://groups.google.com/d/forum/blackbox-project) and ask there.
Bugs are tracked here in Github. Please feel free to [report bugs](https://github.com/StackExchange/blackbox/issues) yourself.

124
docs/user-overview.md Normal file
View File

@@ -0,0 +1,124 @@
User Guide
==========
# Overview
Suppose you have a VCS repository (i.e. a Git or Mercurial repo) and
certain files contain secrets such as passwords or SSL private keys.
Often people just store such files "and hope that nobody finds them in
the repo". That's not safe. Hope is not a strategy.
With BlackBox, those files are stored encrypted using GPG. Access to
the repo without also having the right GPG keys makes those files as worthless
as random bits. As long as you keep your GPG keys safe, you don't
have to worry about storing your VCS repo on an untrusted server or
letting anyone clone the repo.
Heck, even if you trust your server, now you don't have to trust the
people that do backups of that server!
Each person ("admin") of the system can decrypt all the files using
their GPG key, which has its own passphrase. The authorized GPG keys
can decrypt any file. This is better than systems that use one
GPG key (and passphrase) that must be shared among a group of people.
It is much better than having one passphrase for each file (I don't
think anyone actually does that).
Since any admin's GPG key can decrypt the files, if one person leaves
the company, you don't have to communicate a new passphrase to everyone.
Simply disable the one key that should no longer have access.
The process for doing this is as easy as running 2 commands (1 to
disable their key, 1 to re-encrypt all files.) Obviously if they kept
a copy of the repo (and their own passphrase) before leaving the
company, they have access to the secrets. However, you should rotate
those secrets anyway. ("rotate secrets" means changing the passwords,
regenerating TLS certs, and so on).
# Sample session:
First we are going to list the files currently in the blackbox. In
this case, it is an SSH private key.
```
$ blackbox file list
modules/log_management/files/id_rsa
```
Excellent! Our coworkers have already registered a file with the
system. Let's decrypt it, edit it, and re-encrypt it.
```
$ blackbox decrypt modules/log_management/files/id_rsa
========== DECRYPTING "modules/log_management/files/id_rsa"
$ vi modules/log_management/files/id_rsa
```
That was easy so far!
When we encrypt it, Blackbox will not commit the changes, but it
will give a hint that you should. It spells out the exact command you
need to type and even proposes a commit message.
```
$ blackbox encrypt modules/log_management/files/id_rsa
========== ENCRYPTING "modules/log_management/files/id_rsa"
NEXT STEP: You need to manually check these in:
git commit -m"ENCRYPTED modules/log_management/files/id_rsa" modules/log_management/files/id_rsa.gpg
```
You can also use `blackbox edit <filename>` to decrypt a file, edit it
(it will call `$EDITOR`) and re-encrypt it.
Now let's register a new file with the blackbox system.
`data/pass.yaml` is a small file that stores a very important
password. In this example, we had just stored the unecrypted
password in our repo. That's bad. Let's encrypt it.
```
$ blackbox file add data/pass.yaml
========== SHREDDING ("/bin/rm", "-f"): "data/pass.yaml"
NEXT STEP: You need to manually check these in:
git commit -m"NEW FILES: data/pass.yaml" .gitignore keyrings/live/blackbox-files.txt modules/stacklb/pass.yaml modules/stacklb/pass.yaml.gpg
```
Before we commit the change, let's do a `git status` to see what else
has changed.
```
$ git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: .gitignore
modified: keyrings/live/blackbox-files.txt
deleted: modules/stacklb/pass.yaml
new file: modules/stacklb/pass.yaml.gpg
```
Notice that a number of files were modified:
* `.gitignore`: This file is updated to include the plaintext
filename, so that you don't accidentally add it to the repo in the
future.
* `.blackbox/blackbox-files.txt`: The list of files that are registered with the system.
* `data/pass.yaml`: The file we encrypted is deleted from the repo.
* `data/pass.yaml.gpg`: The encrypted file is added to the repo.
Even though pass.yaml was deleted from the repo, it is still in the
repo's history. Anyone with an old copy of the repo, or a new copy
that knows how to view the repo's history, can see the secret
password. For that reason, you should change the password and
re-encrypt the file. This is an important point. Blackbox is not
magic and it doesn't have a "Men In Black"-style neuralizer that
can make people forget the past. If someone leaves a project, you
have to change the old passwords, etc.
Those are the basics. Your next step might be:
* TODO: How to enable Blackbox for a repo.
* TODO: How to add yourself as an admin to a repo.
* TODO: Complete list of [all blackbox commands](all-commands)

17
docs/why-is-this-important.md Normal file
View File

@@ -0,0 +1,17 @@
Why encrypt your secrets?
=========================
OBVIOUSLY we don't want secret things like SSL private keys and
passwords to be leaked.
NOT SO OBVIOUSLY when we store "secrets" in a VCS repo like Git or
Mercurial, suddenly we are less able to share our code with other
people. Communication between subteams of an organization is hurt. You
can't collaborate as well. Either you find yourself emailing
individual files around (yuck!), making a special repo with just the
files needed by your collaborators (yuck!!), or just deciding that
collaboration isn't worth all that effort (yuck!!!).
The ability to be open and transparent about our code, with the
exception of a few specific files, is key to the kind of collaboration
that DevOps and modern IT practitioners need to do.

18
docs/with-ansible.md Normal file
View File

@@ -0,0 +1,18 @@
How to use the secrets with Ansible?
===================================
Ansible Vault provides functionality for encrypting both entire files
and strings stored within files; however, keeping track of the
password(s) required for decryption is not handled by this module.
Instead one must specify a password file when running the playbook.
Ansible example for password file: `my_secret_password.txt.gpg`
```
ansible-playbook --vault-password-file my_secret_password.txt site.yml
```
Alternatively, one can specify this in the
`ANSIBLE_VAULT_PASSWORD_FILE` environment variable.

68
docs/with-puppet.md Normal file
View File

@@ -0,0 +1,68 @@
How to use the secrets with Puppet?
===================================
# Entire files:
Entire files, such as SSL certs and private keys, are treated just
like regular files. You decrypt them any time you push a new release
to the puppet master.
Example of an encrypted file named `secret_file.key.gpg`
* Plaintext file is: `modules/${module_name}/files/secret_file.key`
* Encrypted file is: `modules/${module_name}/files/secret_file.key.gpg`
* Puppet sees it as: `puppet:///modules/${module_name}/secret_file.key`
Puppet code that stores `secret_file.key` in `/etc/my_little_secret.key`:
```
file { '/etc/my_little_secret.key':
ensure => 'file',
owner => 'root',
group => 'puppet',
mode => '0760',
source => "puppet:///modules/${module_name}/secret_file.key", # No ".gpg"
}
```
# Small strings:
For small strings such as passwords and API keys, it makes sense
to store them in an (encrypted) YAML file which is then made
available via hiera.
For example, we use a file called `blackbox.yaml`. You can access the
data in it using the hiera() function.
*Setup:*
Edit `hiera.yaml` to include "blackbox" to the search hierarchy:
```
:hierarchy:
- ...
- blackbox
- ...
```
In blackbox.yaml specify:
```
---
module::test_password: "my secret password"
```
In your Puppet Code, access the password as you would any hiera data:
```
$the_password = hiera('module::test_password', 'fail')
file {'/tmp/debug-blackbox.txt':
content => $the_password,
owner => 'root',
group => 'root',
mode => '0600',
}
```
The variable `$the_password` will contain "my secret password" and can be used anywhere strings are used.