Locked Vault

During a recent project, I needed to test against an implementation of HashiCorp's Vault [1]. If you haven't come across Vault before, it's the industry leader for cloud-native secrets and certificate management that, by design, also plays very nicely with software in the cloud, such as Kubernetes. According to the Vault website, you can safely keep and fine-tune access for users and applications for "tokens, passwords, certificates, encryption keys … and other sensitive data using a UI, CLI, or HTTP API."

Working on security in DevOps teams, I had seen and used Vault, built by Terraform, on two or three different Amazon Web Services (AWS) estates but had never set up a lab myself on a local machine for testing. I wanted a simple solution that was slick and quickly reproducible, and, because it was for a lab environment, it didn't have to support high availability (HA). Thankfully, the clever Vault provides what is called dev server mode for just that requirement.

In this article, I deploy the venerable Vault locally to offer a centralized, flexible, and highly secure place to keep secrets and certificates; then, I'll run through a few of its core features to get to grips with the basics.

Is It Safe?

I'll dive straight into getting Vault up and running. I'm using Linux Mint 19, which sits atop Ubuntu 18.04. The set up should be mostly the same configuration, however, if you use another Linux distribution.

First, I'll navigate to the binary download page [2] and choose Linux 64-bit. I'm going to become the root user in this terminal and in a second terminal a little later on. After clicking the Download button, a ZIP file starts downloading. In the following command, you can see the version I'm using as I decompress the file:

$ unzip vault_1.3.2_linux_amd64.zip

The next thing to do is move the resulting vault binary file to the user path:

$ mv vault /usr/local/bin

To see if it works, I enter:

$ vault --version
Vault v1.3.2

One nice touch is that the vault command can use autocompletion, just like the Bash shell. To switch that feature on, enter,

$ vault -autocomplete-install

which will drop the line complete -C /usr/local/bin/vault vault into your .bashrc file. Close and reopen your terminal to load that up, or in your home directory run

$ . .bashrc

to load that file into your shell again. Try hitting the Tab key with the vault prompt on your command line now, and you'll see a number of options, which is a great way of checking syntax when you're learning Vault.

Under Lock and Key

As mentioned, I'm going to create a dev server for the purposes of running a lab environment. The Vault docs warn you that it shouldn't be used in production, isn't very secure, stores loads of things in memory, and automatically initializes (or "unseals") the strong locking or "sealing" mechanism at install time. You will also lose your secrets when you kill the terminal in which the dev server launches. Consider yourself suitably warned! For more information, read the "Concepts" box.

All commands that you send from the vault binary (think of it as a client) are encrypted by TLS as they travel over to the Vault server. To fire the dev server up, use the command:

$ vault server -dev

The uppermost part of the abbreviated output in Listing 1 shows what you need to know. Note that you will need to open another terminal to continue with other commands from this point on, because the server will keep that window open as it runs in the foreground.

==> Vault server configuration:
        Api Address: http://127.0.0.1:8200
                Cgo: disabled
    Cluster Address: https://127.0.0.1:8201
         Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
          Log Level: info
              Mlock: supported: true, enabled: false
      Recovery Mode: false
            Storage: inmem
            Version: Vault v1.3.2
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variable:
  $ export VAULT_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: PrAbjZWj2NYYnV6Lh/FRpf5Nu5f2E5fwZZf95JQiLdo=
Root Token: s.pKGs7KObPOp0Cx2ybiUWuEIW
Development mode should NOT be used in production installations!

After opening up a second terminal, you can now interact with the Vault server. For good measure, I'll offer the new terminal the VAULT_ADDR environment variable with the command

$ export VAULT_ADDR="http://127.0.0.1:8200"

so that the shell can access the correct IP address and TCP port for the server.

The next step is using an environment variable for the root token, copied from the dev server start-up output in the first terminal, within the second terminal:

$ export VAULT_DEV_ROOT_TOKEN_ID="s.pKGs7KObPOp0Cx2ybiUWuEIW"

From the second terminal, you can check that the client – the vault binary that provides its command-line interface (CLI)) – can connect to the server (Listing 2). The output shows a successful response, which means that your client to server comms are working as hoped.

$ vault status
Key            Value
---            -----
Seal Type      shamir
Initialized    true
Sealed         false
Total Shares   1
Threshold      1
Version        1.3.2
Cluster Name   vault-cluster-e5878f26
Cluster ID     54ef0ecc-5f31-0926-e904-1bd94bc152ae
HA Enabled     false

Lockdown, Lockdown

With a working Vault instance, I will now look at a few ways to make use of it. The first, most obvious thing to try is saving a secret to the secured vault. This secret could be any bit of data – not just a password:

$ vault kv put secret/chrisbinnie liverpool=best
Key              Value
---              -----
created_time     2020-03-13T12:56:24.857347664Z
deletion_time    n/a
destroyed        false
version          1

That output looks promising; Vault has responded as expected. If that doesn't work correctly for you for some reason, then you've probably not exported the two environment variables correctly, so check them. Take note that the secret/chrisbinnie part in the command above relates to the path where the secret is stored. The "Secrets Engines" box has more information to help explain how the path is configured. As you might expect, the liverpool=best element is the key-value pair.

Now that you know how to write a secret, I'll try to retrieve one with a get command (Listing 3). You can verify that you are seeing the correct secret by the timestamp in the command's output; also, you can see the key-value pair data under the Data section.

$ vault kv get secret/chrisbinnie
====== Metadata ======
Key            Value
---            -----
created_time   2020-03-13T12:56:24.857347664Z
deletion_time  n/a
destroyed      false
version        1
====== Data ======
Key         Value
---         -----
liverpool   best

Simply replace the get in the command in Listing 3 with delete to remove a secret. Note that it's possible to write multiple secrets to the same path. At an uber-basic level, that's how secrets are written, retrieved, and deleted in Vault.

However, another very powerful feature concerns dynamic secrets [6]. According to the docs: "Dynamic secrets do not exist until they are read, so there is no risk of someone stealing them … dynamic secrets can be revoked immediately after use, minimizing the amount of time the secret existed."

This sophisticated functionality is perfect for an environment such as AWS, wherein an IAM role is given access via a role with a policy attached to it. Vault then temporarily offers access to permitted resources with the use of dynamic credentials that can be revoked or set to expire within customized time periods. Additionally, ephemeral, single-use tokens are powerful.

Locked Shut

Of course, another important feature of Vault is authenticating so that you can interact with the stored data. A dev server automatically allows you straight into the locked vault, but usually you would need to authenticate, for obvious reasons.

The need to authenticate translates into Vault creating a token for access. In the dev server's start-up output, you saw the root token being created so you could interact. The root token is how you'd set up Vault initially under normal circumstances. The eagle-eyed among you will see that the command

$ vault token create
Key                  Value
---                  -----
token                s.1xnAreJvUEKnKwKs1KVtcNyQ
token_accessor       DulIsR7BVxv6k4YF6JoMagb0
token_duration       ?
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

creates a child token. The parent-child design means that, because tokens always have a parent token, if that parent is revoked, then all child tokens under it will also be revoked. Note that the child token above inherits the root token permissions, as shown by the token_policies line.

Once you've created a token like that, it's possible to log in as in Listing 4. Study the output carefully. As you can see, "Future Vault requests will automatically use this token," so there's no need to log in repeatedly to Vault. The time to live (TTL) or token_duration symbol is shown as a lemniscate (i.e., infinity symbol), meaning it won't expire.

$ vault login s.1xnAreJvUEKnKwKs1KVtcNyQ
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key                  Value
---                  -----
token                s.1xnAreJvUEKnKwKs1KVtcNyQ
token_accessor       DulIsR7BVxv6k4YF6JoMagb0
token_duration       ?
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Running the command

$ vault revoke s.1xnAreJvUEKnKwKs1KVtcNyQ

revokes the same child token just created.

Turn the Key

Obviously, you also need to make sure you can change a child token's policies to refine the access that token is allowed.

Policies in Vault use HashiCorp's intuitive HashiCorp Configuration Language (HCL) [7] – or JSON otherwise. The policy example in the Vault docs looks something like:

path "secret/data/*" {
  capabilities = ["create", "update"]
}

Here, the path secret/data and everything within it (denoted by the asterisk) can be changed (update) by that policy, and new information can be written (create). However, if you added this to the following policy, what would happen?

path "secret/data/chrisbinnie" {
  capabilities = ["read"]
}

Although you'd still be able to write and change data under the path secret/data, you would not be able to do so in secret/data/chrisbinnie, which only allows you to read from it.

A top tip is to use the fmt formatting command when writing policies (HashiCorp's Terraform has this ability, too) to get feedback on typos and formatting issues. If I add the two path snippets above to a file called lab-policy.hcl and check it with the fmt command,

$ vault policy fmt lab-policy.hcl

a happy policy will output:

Success! Formatted policy: lab-policy.hcl

The next step is to upload the policy to Vault (just a write to storage, really) with a name:

$ vault policy write lab-policy lab-policy.hcl
Success! Uploaded policy: lab-policy

Here, I've called the policy lab-policy, which I'll reference again in a moment.

If I want to list the policies currently stored in Vault, I'd use the command:

$ vault policy list
lab-policy
default
root

It looks like the previous policy upload command worked as hoped. To check inside a policy, you can use the view command:

$ vault policy read lab-policy
path "secret/data/*" {
  capabilities = ["create", "update"]
}
path "secret/data/chrisbinnie" {
  capabilities = ["read"]
}

To continue, you create an auth method. According to the docs [8], these methods are "components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user."

The auth prefix is used for authentication methods just as secrets has its own prefix, and authentication follows the same philosophy when it comes to referring to it by path. Look at this command which uses the userpass plugin:

$ vault auth enable userpass
Success! Enabled userpass auth method at: userpass/

Here, I'm enabling userpass as the name of the auth method. Now, I'll create a user with a password and attach the policy lab-policy to the user:

$ vault write auth/userpass/users/chris password="nothingtoseehere" policies="lab-policy"
Success! Data written to: auth/userpass/users/chris

This command is working under the userpass auth method and under the path users/chris while adding a password and a policy to the user.

Now I log back in as chris:

$ vault login -method=userpass username=chris password=nothingtoseehere

Listing 5 shows the results. Armed with the correct commands, it's now possible to limit the access rights of a user very effectively. You can see that both default and lab-policy are attached to the username under token_policies. Now the token generated by that vault login command can be used to access Vault as required.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key                    Value
---                    -----
token                  s.PWHPg5hvBYnxkFHOyC1ZgoRu
token_accessor         DRBFdwUjwQd8UgfZbE1XxmBt
token_duration         768h
token_renewable        true
token_policies         ["default" "lab-policy"]
identity_policies      []
policies               ["default" "lab-policy"]
token_meta_username    chris

Once you have the concepts and basics under your belt, you can read further about internal groups and external groups [9] to assist with multiple users. If you visit the web page in a fully fledged browser and click Show Tutorial at the bottom, a nice browser-friendly shell window pops up so you can run shell commands yourself (Figure 1). That functionality is powered by the truly excellent Katacoda [10], which is definitely worth a visit if you've never used their Live environments in a browser before. For example, I use it for Kubernetes testing sometimes.

The End Is Nigh

A highly secure, trusted place to keep your secrets and certificates is an invaluable addition to any estate. A common way to interact with and manipulate the Vault API is with the curl command, which I hope should now make sense.

If you get stuck, you can try the autocompletion feature mentioned earlier and run the command

$ vault kv help

for assistance. I'm sure you'll find the docs and the built-in CLI help very clear and readable.

Once you're comfortable with how the innards of Vault work, the next thing to do is configure HA in a cloud environment (e.g., AWS) so that it will play nicely with software like Kubernetes.

For HA, you need a suitable back end to avoid locking headaches from multiple users or applications trying to write to the same bit of stored data at the same time. You have quite a few solutions from which to choose, such as Consul [11] (included with Vault Enterprise and also made by HashiCorp) or DynamoDB from AWS.

A nice article on the AWS site [12] can get you going. Because Vault is so popular, you'll find lots of useful online literature to help out if you get stuck.

One thing that I didn't covered in this article is the useful browser-based user interface (UI), which is a nice, clean, and simple way of interacting with Vault. When you have a more mature installation of Vault up and running, be certain to have a rummage around the UI.