Secure Containers

Any self-respecting DevOps engineer knows of the hidden dangers that can be found, without much hunting at all, inside container images. Handling security problems with packages stored inside images in a timely fashion is imperative to prevent a successful attack on containers that, as a result, would put your servers at risk.

The sheer volume of CVEs (Common Vulnerabilities and Exploits) found in today's popular images and listed on the CVE website [1] might surprise you.

According to a new report from the venerable Snyk [2], who knows about all things security focused, the top 10 most popular container images each boast at least 30 vulnerabilities, with node leading the way. The report, which I would recommend reading carefully and in detail, reveals some frightening information. For example, you might be surprised to learn that different strategies are required for Alpine images over other operating systems (OSs) in use within your container images. Alpine is super-popular among containers as the base OS because of its tiny footprint.

Now that you're suitably alarmed, the good news is that you can automate how you are alerted to CVEs that apply to your container images. Although you have a choice of many tools in the open source space, in this article I'll look at the open source version of a well-respected, enterprise tool called Anchore [3] that describes itself as the "only end-to-end container security and compliance platform built on open source." To keep things lightweight and portable, I will run Anchore in two containers: one for the main engine and one for the database holding useful generated information.

Fear, Uncertainty, and Doubt

To get Anchore up and running, you first need to install Docker Compose. If I'm honest, I've not always been a massive fan, but Docker Compose certainly has its place and can speed up scenarios in which you need to bring up multiple containers.

On my laptop, I'm using a Debian derivative (courtesy of Ubuntu 18.04 and Linux Mint "Tara"). When installing Docker Compose, numerous packages are promised (Listing 1; note that the application relies heavily on Python).

$ apt install docker-compose
The following NEW packages will be installed...
docker-compose golang-docker-credential-helpers python-backports.ssl-match-hostname python-cached-property python-certifi python-chardet python-docker python-dockerpty python-dockerpycreds python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-pbr python-pkg-resources python-requests python-texttable python-urllib3 python-websocket python-yaml

Having installed Docker Compose and its dependencies, the next step is to create a local directory at which to point the Docker container. In terms of file paths, I'm going to go a little off-piste here from the documentation. I'm assuming, of course, that you're working in the root user's home directory. Note that you should create another subdirectory and work from that if you don't want the db/ directory created in the top level of the root home directory:

$ mkdir -p /root/config

Now that you have a home for the main configuration file and those files created when your Docker containers come up, you can download a configuration file [4] from GitHub and save it as config.yaml within that local config/ directory.

I'm not going to fiddle with the default login details. (See the "Troubleshooting" box for some tips if you get an Unauthorized error after modifying your credentials). The snippet below shows the two fields (password and email) I would edit in the config/config.yaml configuration file:

credentials:
  users:
    admin:
      password: 'franticfrenetic'
      email: 'chris@binnie.tld'

Here, I've personalized the credentials with my own administrative login details.

Always Something

Before Docker Compose would behave properly, I had to get past an odd error I hadn't see before; the error (abbreviated here) looked something like:

/usr/lib/python2.7/dist-packages/  requests/__init__.py:80:
RequestsDependencyWarning: urllib3   (1.24.1) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning)

To get it working I had to downgrade urllib3:

$ pip install --upgrade "urllib3==1.22"

Next, I downloaded the main docker-compose.yaml file from Anchore's GitHub pages [6] to the top-level of /root.

Pick It, Pack It, Fire It Up

Now I'm just about set to fire my freshly installed Docker Compose up, and once it's running I'll get a chance to pick which images Anchore will analyze:

$ docker-compose pull

As you can see in Figure 2, the images anchore-engine:latest and postgres:9 are pulled down when Docker Compose runs the pull command. The command seems completely successful, so the next command I need to run will fire up Anchore:

$ docker-compose up -d

As you can see from the output in Figure 3, in true spinning-many-plates style, Docker Compose will also fire up the Postgres database at the same time.

Note that both anchore-db_1 and anchore-engine_1 are needed to start Anchore's service. If you want to bring the services down later, use the simple command:

$ docker-compose down

After starting Anchore, you'll need to wait a minute or two depending on the specification of your server or laptop before the Anchore service is fully accessible.

You can, of course, use standard Docker commands to check the status of Anchore as you wait for the service and database to finish coming up. With the docker ps command, I can see that my anchore-engine container has a hash that begins with 0f8a0, so running the command in Listing 2 as I wait for the service to finish starting reveals the internal logging within the container. (I had to abbreviate the output for clarity.)

$ docker logs 0f8a0
[INFO] Syncing group took 5.293130159378052 sec
[INFO] Processing group: alpine:3.4
"172.20.0.3" "POST /v1/queues/watcher_tasks/is_inqueue HTTP/1.1" 200 3 "-" "python-requests/2.20.0"

My Needs Are Not Being Met

The Anchore Engine is accessible from a Python-based command-line interface (CLI) program, and I'll use the pip Python package manager to install it. More usage details are on the GitHub page [7] if you get stuck, or check the "Troubleshooting" box in this article. First, you should make sure pip is accessible so you can install the anchorecli package:

$ apt install python-pip
$ pip install anchorecli

Running the second command resulted in some missing pip dependencies, which I fixed by installing the named package before entering

$ pip install wheel
$ pip install anchorecli

which indicated (Figure 4) that all was indeed well again.

Password123

To prevent saving your credentials to a file, you have to set the environment variables manually each time you log in to a terminal to use Anchore:

$ export ANCHORE_CLI_USER=admin
$ export ANCHORE_CLI_PASS=foobar

Alter the variables as you see fit; you'll use these credentials (the default values are shown) to connect to the Anchore API. Additionally, you can explicitly set the URL of the Anchore service,

$ export ANCHORE_CLI_URL=http://myserver.example.com:8228/v1

although I didn't have to take this step.

Ready Player One

The next step (if you can't get the next commands to work, check out the "Troubleshooting" box) adds an image to Anchore:

$ anchore-cli image add docker.io/library/debian:latest

Figure 5 shows that the stock Debian OS image has been accepted by Anchore and is currently showing not_analyzed in the Analysis Status line; that is, Anchore has retrieved that image and is processing it now, checking each package and version for known CVEs and compiling a list of each file that is present.

If you want to check which images were pulled into Anchore, you run the image list command,

$ anchore-cli image list

which also reports the status of the images (Figure 6), so you can see whether it's been analyzed after recently being subsumed by Anchore.

Are You Sitting Down?

Once Anchore has happily finished analyzing your image, you can get a full readout of what's going on inside with the following command, which uses Nginx:

$ anchore-cli image vuln docker.io/library/nginx:latest os

I'm assuming larger images take a little longer to complete, because this command returned empty output for a few minutes after running it against some images.

I'm sorry to say that Anchore came back with some unwelcome news (Figure 7), reporting a vast number of issues found in the latest nginx image. Each CVE is marked as High, Medium, and so on for clarity. I'm sure you can see why tools as powerful as Anchore are so critical to improving your security posture.

For comparison, when Anchore was run over Debian's latest image, the total was 43 CVEs, and my own image (a handful of tools for security auditing using an old 2017 Debian base OS) [8] contained a whopping 260 CVEs!

You can incorporate Anchore into your continuous integration and continuous delivery pipelines nicely with webhooks and receive notifications when new CVEs appear in an image. To activate this functionality, use the command:

$ anchore-cli subscription activate vuln_update docker.io/library/debian:latest
Success

As you might expect, the result Success is a welcome message, meaning you've subscribed to notices about that image.

So Many Files

You can also use Anchore to show details of what a container image holds with the command:

$ anchore-cli image content chrisbinnie/super:latest files

You might want to redirect the output to an empty text file so you can look at it more closely later. Figure 8 shows the content of the image, with the size of each file listed to the far right for reference.

Starship Enterprise

At this stage, I would be remiss not to mention the Enterprise version of Anchore, which describes itself as offering the ability to "start utilizing the most comprehensive container image inspection and policy management platform available today" [9]. With the Enterprise version, you can view Anchore in a dashboard and drill down into items of interest with ease.

It Ends Here

Container images have so many CVEs today that you simply can't risk blindly avoiding their potential pitfalls. Anchore is quick to set up and comprehensive, and the enterprise version looks sophisticated with an easy-to-use dashboard.

To my mind, rather than just listing CVE reference numbers Anchore cuts down on your workload most effectively by offering a link to the pertinent CVEs. For instance, the libc-bin-2.24-11+deb9u1 package reported a "high" CVE rating; by clicking on the reported URL [10], I was presented with lots of useful information relating to which culprits might make a container vulnerable.

As Figure 9 shows, a CVE can have complexities of its own, which means that offering as much information as possible is critical, so you can make informed decisions as to whether an image is safe to use inside a container.

I hope you enjoy trying Anchore; I recommend it whenever possible.