Safe Harbor

Docker containers are a great way to produce lean and portable services, not only for the enterprise, but also at home. Docker takes the complexity out of setting up software services that are inherently complicated, like an email or a Nextcloud server. However, it's difficult to know when and how to update the software inside the Docker container to keep it as secure as possible.

In this article, I present a scripted approach to alerting you about when to update Docker containers. The first script checks Docker Hub for image tags or versions available. If you want to update, you need to know your software version and adjust the image tag accordingly, so updates stay within a safe range of versions. The second set of scripts checks your local version against possible updates on Docker Hub. One mini-script produces an ID file containing the two Docker image IDs and another checks for updates available on the Hub.

With these scripts, you can automate your checks for available updates. The updating process itself is left in your hands.

Updating a Docker Container

Many official software projects like Nextcloud, MariaDB, Nginx, and so on maintain official Docker images. Often, though, their prescription for Docker is a Docker Compose file in which you install some latest version of the image; however, it is not advisable to keep blindly updating to a latest version. For instance, the Nginx container defined in the Docker Compose file shown in Listing 1, when run, pulls an nginx:alpine image from Docker Hub, where nginx is the image and alpine is the tag (referred to hereafter as <image><tag>).

version: '2'
services:
  web:
    image: nginx:alpine
    restart: always
    container_name: nginx
    ports:
      - 8080:80
    environment:
      - NGINX_HOST=www.somewebsite.com
      - NGINX_PORT=80
    command: "nginx -g 'daemon off;'"

If you pull nginx:alpine, you pull a so-called "latest" version. Several commonly used tags also pull a latest version, such as <image>:latest, <image>:stable, nginx:alpine, nextcloud:fpm-alpine, and so on. By "latest," I mean the tag has no version (e.g., alpine-10.4).

If you leave the latest tag in your configuration files while updating the container, you are in for trouble: The software version might go from nginx-8.3 to nginx-11.2, which, bottom line, could break things. On the other hand, if you do not update at all, the software after some time will contain known vulnerabilities (which is why you update in the first place, of course).

Depending on the software in the container, you should decide what version changes are probably safe or, in other words, the version range in which you want to update. By version range, I mean it's probably fine to upgrade nextcloud-10.4.1 to nextcloud-10.4.8, but it's not advisable to update blindly from version 10.4 to version 12.3. A safe version range, then, would be to stay within version 10.4. Of course, it's entirely up to you what risks to take and what you consider a safe version range.

To update a Docker container within a safe version range, you need two pieces of information: the image version you are using right now and what <image><tag> you will use when updating, so you don't run into trouble.

Image Version

To know whether your Docker container needs an update within a safe version range, you need to know what version you are running. The answer lies in a set of identifiers and tags to be found in the manifest that is also downloaded when you pull an image. You can inspect the manifest in two steps: Find the image ID on your local machine, and inspect its manifest for a version tag.

You can inspect container or image manifests on a Docker host (a host running Docker containers) as shown in Listing 2. The docker image command produces the image ID of running containers. In this case, my running container uses the nginx image with the id bf85f2b6bf52.

docker images
REPOSITORY   TAG      IMAGE ID       CREATED         SIZE
nginx_web    latest   c100b674c0b5   13 months ago   19MB
nginx        alpine   bf85f2b6bf52   13 months ago   15.5MB

With the image ID in hand, you can inspect the image manifest:

docker inspect bf85f2b6bf52 | egrep -i version

In this case, it yields: NGINX_VERSION=1.13.7.

An easy way to get the versions of all running Docker containers is shown in Listing 3. When I run the script on my Nextcloud machine, it yields (together with some lines of no use):

"NEXTCLOUD_VERSION=12.0.4"
MARIADB_MAJOR=10.2   MARIADB_VERSION=10.2.12+maria~jessie

What Tag?

## get versions for running containers
for i in `docker ps | egrep -v NAMES | awk '{print $1}'`; do
  echo " image-ID is now $i"
  docker inspect --format='{{.Config.Env}}' $i
  docker inspect $i | egrep VERSION
done

Now that you know your software versions, the next question to ask is what tag to use in docker-compose.yml. You can't use nginx:alpine:1.13, so what tag can you use?

The answer depends, of course, on what tags a software project has issued for its images on Docker Hub. I like to use the script in Listing 4, which asks the Docker Hub Registry API to list all available tags. To download the script, make it executable, and run it, enter:

curl https://raw.githubusercontent.com/  hanscees/dockerscripts/master/scripts/get-docker-hub--image-tags.sh > get-public-image-tags.sh
chmod +x get-public-image-tags.sh
./get-public-image-tags.sh library/nginx

01 #!/bin/bash
02
03 # Retrieves image tags from public
04 # images in DockerHub
05 # if someone improves this script, please let me know, preferably in Python
06 # hanscees@AT@hanscees.com
07 # modified from https://gist.github.com/cirocosta/17ea17be7ac11594cb0f290b0a3ac0d1x
08
09 set -o errexit
10
11 main() {
12   check_args "$@"
13
14   local image=$1
15   local token=$(get_token $image)
16   local tags=$(get_tags $image $token)
17   echo "tags reported are:"
18   echo $tags
19 }
20
21 get_token() {
22   local image=$1
23
24   echo "Retrieving Docker Hub token.
25     IMAGE: $image
26   " >&2
27
28   curl --silent "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" | jq -r '.token'
29 }
30
31 # Retrieve the digest, now specifying in the header
32 # that we have a token (so we can ...
33 get_tags() {
34   local image=$1
35   local token=$2
36
37   echo "Retrieving image tags.
38     IMAGE:  $image
39   " >&2
40
41   curl --silent --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $token" "https://registry-1.docker.io/v2/$image/tags/list" | jq -r '.tags'
42 }
43 check_args() {
44   if (($# != 1)); then
45     echo "Error:
46     One argument must be provided - $# provided.
47
48     Usage:
49       ./get-docker-hub-image-tags.sh <image>
50       for instance ./get-docker-hub-image-tags.sh library/mariadb
51 Aborting."
52     exit 1
53   fi
54 }
55
56 main "$@"

In the last line, I prepended library/ to nginx: All official Docker Hub repositories require this prefix.

The output comprises many tags, including various versions of alpine, perl, and alpine-perl, along with their mainline and stable versions (Table 1).

The get-public-image-tags.sh script uses jq (a command-line JSON processor) in lines 28 and 41 to parse the JSON. On Ubuntu, the command to install the utility is:

apt-get install jq

As you can see, Nginx has quite a few tags. It also becomes clear how to reference a version of nginx:alpine: Simply use nginx:<version>-alpine. Because Nginx seems unlikely to change in any important way, I would suggest considering the nginx:1.15-alpine tag in the YAML file.

Automating Update Checks

When I first pulled down an Nginx image as nginx:alpine in Listing 1, it turned out to be version 1.13. Using safe version logic, I should change the nginx:alpine tag in the YAML file to nginx:1.15-alpine.

Now I can update manually by running:

docker-compose pull
docker-compose up -d

Alternatively, you can let an automated solution update your containers. Watchtower [1] (a Docker container), can do this for you, although I have not tested it.

However, I want to stick with manual updating, and I simply want to know when updates are available for <image><tag> (compared with my current image, of course). If an update is available, I can make a snapshot of the virtual machine on which Docker runs, update, and roll back if things go wrong. This time, I use a script and learn something about image IDs, as well.

First, you need two more pieces of information: what version and image ID are you running and how it compares with the latest container ID of the version to which you want to update. The first item was already solved above, but not the second.

Repo ID vs. Image ID

The image ID was discovered by analyzing the manifest file of an image and extracting the version of the software. However, for an automated solution, this process won't do because every software project uses versions in a slightly different way. Therefore, you need a more general ID.

A Docker manifest uses several IDs, two of which are the image ID, which is a hash of the uncompressed image, and the repo ID (repo digest), which is a hash of the compressed image when it sits in the registry (from which you pull your images). Both are digests, by the way, but with different data payload forms.

Confusingly, different Docker command-line options show both IDs or parts of both, as you can see in the confusing output of the docker images command in Listing 5, which shows information about images you pulled to your computer.

docker images --digests
REPOSITORY   TAG           DIGEST                                                                    IMAGE ID       CREATED         SIZE
nginx        1.15-alpine   sha256:385fbcf0f04621981df6c6f1abd896101eb61a439746ee2921b26abc78f45571   315798907716   5 days ago      17.8MB
nginx        alpine        sha256:f1ca87d9adb678b180c31bf21eb9798b043c22571f419ed844bca1d103f2a2f7   bf85f2b6bf52   13 months ago   15.5MB

The output shows information on both the original nginx:alpine image and the updated nginx:1.15-alpine. The Image ID column refers to a short version of the (local unpacked) image ID and the Digest column refers to the repo ID.

If you want to see both the full image hash and the repo digest of an image on a Docker host, the command in Listing 6 shows both.

01 docker image inspect 315798907716 | head
02     {
03         "Id": "sha256:315798907716a51610bb3c270c191e0e61112b19aae9a3bb0c2a60c53d074750",
04         "RepoTags": [
05             "nginx:1.15-alpine"
06         ],
07         "RepoDigests": [
08             "nginx@sha256:385fbcf0f04621981df6c6f1abd896101eb61a439746ee2921b26abc78f45571"

The "Id" (line 3) shows the image ID: The first 12 characters are the same as the output from docker images; the repo ID is the repo digest. By the way, docker image inspect reads a downloaded image manifest file that you get when you pull an image.

For the automated update check, you will collect both the image and repo IDs, but only the image ID can be used to automate a check without pulling an image because of the Docker Registry API (as far as I know).

A Scripted Update Check

After searching and finding some Bash scripts on GitHub that work on the Docker Registry API, I have come up with this solution:

1. A local Docker script first produces a local file on a Docker host with per-line <image><tag><current image ID> and a second line with <image><tag><current repo ID>. You can run this script on many Docker hosts, combine lines, and use one central host to check for updates on many Docker images you might have in production.

2. A second script reads the local file and, for each line, checks Docker Hub for the <image ID> of the latest version of <image><tag>; compares the local image tag to the image ID of latest on the Hub; and lets you know what local Docker services thus can be updated (if local and remote IDs don't match).

Although you can probably find many ways to solve this problem, the script lets you check a number of different images at the same time. Also, a centralized check is possible by combining with a centralized scanning solution.

The scripts to do this can be found on GitHub [2], which you can either clone or download one by one (Listing 7), because Git is quite a large package [3].

mkdir dockerscripts
cd dockerscripts
curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/check-docker-image-updates.sh > check-docker-image-updates.sh
 curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/get-docker-hub--image-tags.sh > get-docker-hub--image-tags.sh
 curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/get-docker-hub-image-tag-digest.sh > get-docker-hub-image-tag-digest.sh
 curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/emailer.py > emailer.py
#-chmod them
   chmod +x *.sh
   chmod +x  *.py
#- install qs
  apt-get install jq

Local Docker Script

The script to get Docker file IDs on a Docker host is not much of a script, really; just copy and paste the code in Listing 8 in a terminal to produce the ImageId-file output file, which has two lines per image, as shown in Listing 9. The first line holds the repo ID, the second the image ID. Each line contains the image name, the tag name, and either the image ID or repo ID. If the last word of the line is RepoId, the line holds the repo ID, making it easy to grep lines with the right kind of ID.

nginx 1.15-alpine nginx@sha256:385fbcf0f04621981df6c6f1abd896101eb61a439746ee2921b26abc78f45571 RepoId
nginx 1.15-alpine 315798907716a51610bb3c270c191e0e61112b19aae9a3bb0c2a60c53d074750
nextcloud latest nextcloud@sha256:78515af937fe6c6d0213103197e09d88bbf9ded117b9877db59e8d70dbdae6b2 RepoId
nextcloud latest 8757ce9de782c2dd746a1dd702178b8309ca6d2feb5e84bad9184441170d4898
mariadb latest mariadb@sha256:12e32f8d1e8958cd076660bc22d19aa74f2da63f286e100fb58d41b740c57006 RepoId
mariadb latest b468922dbbd73bdc874c751778f1ec0ec10817691624976865cb3ec5c70cd4e0
mvance/unbound 1.8.3 mvance/unbound@sha256:d67469fad9cc965f032e4ea736c509df6d009245dac81339e2c6e1caef9b65ac RepoId
mvance/unbound 1.8.3 mvance/unbound latest a88e44773675294dcd10e08a9a2a5ee2a39796e5a832c99606b3c8a54901ea75

> ImageId-file
for i in `docker images -f dangling=false| egrep -v TAG | awk '{print $3}'` ; do
  echo copying name, tag and image ID digest to file
  ImageId=`docker image inspect $i | jq -r '.[0] | {Id: .Id}' | egrep Id | awk -F":" '{print $3'} | awk -F"\"" '{print $1}'`
  RepoId=`docker image inspect $i | jq -r '.[0] | {RepId: .RepoDigests}' | jq --raw-output  '.RepId | .[]'`
  ImageData=`docker images | egrep $i | awk '{print $1," " , $2}'`
  echo $ImageData $ImageId
  echo $ImageData $RepoId RepoId >> ImageId-file
  echo $ImageData $ImageId >> ImageId-file
done

You should check ImageId-file, because it might contain lines about images you do not use anymore: The script will produce lines on all images used in containers, even if you no longer run those containers or have not yet run them. Make sure to discard any "old" containers.

A Central Docker Host Check

If you run the script in Listing 8 on a Docker host, it will collect ID lines on all containers on that host. If you have multiple Docker hosts, you can run the script on multiple hosts and combine the lines into one big file on a central host; of course, you could automate this process and run checks on a central Docker host for all the containers you are using, run checks for updates for your <image><tag> combinations, and download the images and check them against known vulnerabilities.

With the repo ID lines collected, all images can be downloaded on the central host, because the repo ID can be used to pull images with docker pull <repo ID>; for instance,

docker pull mariadb@sha256:12e32f8d1e8958cd076660bc22d19aa74f2da63f286e100fb58d41b740c57006

will download that specific MariaDB image from Docker Hub. Therefore, you can download many images to a central host without actually running them in a container with,

for image in `cat ImageId-file | egrep RepoId | awk '{print $3}'` ; do
   echo $image ;
   docker pull $image
done

and show them by entering docker images.

Once you have your images on one host, you will find multiple solutions to check your images against known vulnerabilities. Examples are Anchore, Clair, or Dagda, which all run on Docker, so they are relatively easy to set up [4].

In the next section, however, I show you how to check for container updates – without the need to download the images, by the way.

Docker Updates Script

At this point you have the image ID file, which knows what Docker <image><tag> pairs you have, and the IDs of these images as used in production. Now, you need to see whether these images can be updated.

The first part of the update check script downloads a manifest file from Docker Hub for a specified Docker <image><tag> using the Registry API. The second part of the script does two things: Loops over all image lines, downloading a manifest file for each, and checks the local image ID against the ID from the downloaded manifest. If they differ, an update is available.

Listing 10 shows the first part of the script, which fetches a digest from Docker Hub. As it states (if you run it without arguments), you should feed it two arguments:

./get-docker-hub-digest library/nginx 1.15-alpine

01 #!/bin/bash
02
03 # Retrieves image digest from public
04 # images in DockerHub
05 # modified from https://gist.github.com/cirocosta/17ea17be7ac11594cb0f290b0a3ac0d1x
06
07 set -o errexit
08
09 main() {
10   check_args "$@"
11
12   local image=$1
13   local tag=$2
14   local token=$(get_token $image)
15   local digest=$(get_digest $image $tag $token)
16
17 echo " $digest"
18
19 }
20
21 get_token() {
22   local image=$1
23
24   echo "Retrieving Docker Hub token.
25     IMAGE: $image
26   " >&2
27
28   curl --silent "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" | jq -r '.token'
29 }
30
31 # Retrieve the digest, now specifying in the header
32 # that we have a token (so we can ...
33 get_digest() {
34   local image=$1
35   local tag=$2
36   local token=$3
37
38   echo "Retrieving image digest.
39     IMAGE:  $image
40     TAG:    $tag
41   " >&2
42 #    TOKEN:  $token # add to echo for debug
43
44   curl --silent --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $token" "https://registry-1.docker.io/v2/$image/manifests/$tag"
45 #    | jq -r '.config.digest'
46 }
47
48 check_args() {
49   if (($# != 2)); then
50     echo "Error:
51     Two arguments must be provided - $# provided.
52
53     Usage:
54       ./get-docker-hub-digest.sh <image> <tag>
55      for instance ./get-docker-hub-digest library/nginx 1.15-alpine
56
57 Aborting."
58     exit 1
59   fi
60 }
61 main "$@"

The script spits out a JSON file with a series of digests, of which the first is the image ID (not the repo ID, for some reason).

The second part of the check script (Listing 11) you will actually use. It checks for updates on Docker images and uses the script in Listing 10 to get digests. You can feed it the image ID file produced in Listing 10 (without the RepoId lines) with:

cat ImageId-file | egrep -v "RepoId|none|^$" | ./check-docker-image-updates.sh

001 #!/bin/bash
002
003 # Checks Docker Hub for updates on <image><tag><ImageId>
004 # ImageId without sha256: part
005 # script accepts <image> <tag> <ImageId> or sdtin
006
007 # reads file from standard in: lines in the form of
008 # nginx 1.14-alpine #315798907716a51610bb3c270c191e0e61112b19aae9a3bb0c2a60c53d074750
009 # mvance/unbound latest 4568745687569875689745689756
010 #
011 # Script calls other script ./getdigest. Can be found at #https://github.com/hanscees/dockerscripts
012
013 set -o errexit
014 > UpdateTheseImages
015 main() {
016
017 REPOS=""
018 TAG=""
019 ID=""
020
021 if [ -t 0 ] ; then
022   echo terminal input; #and no stdin
023   check_args "$@"  #if input not from stdin
024   REPOS=$1
025   TAG=$2
026   ID=$3
027   report_result $REPOS $TAG $ID
028 else
029   echo "not a terminal, so reading stdin";
030   while read line;  do
031   declare -a ImageData=($line) #bash array
032   REPOS=${ImageData[0]}
033   TAG=${ImageData[1]}
034   ID=${ImageData[2]}
035   report_result $REPOS $TAG $ID
036   done
037 fi
038 }  #end main
039
040 report_result () {
041   REPOS=$1
042   TAG=$2
043   ID=$3
044   echo "repo/image and tag are "
045   echo $REPOS $TAG
046
047   check_result=$(check_for_updates $REPOS $TAG $ID)
048   echo check_result is $check_result
049
050   if [ $check_result ]
051   then
052    echo $REPOS $TAG can be updated: a new version is available
053    echo also check file UpdateTheseImages
054   else
055    echo NO update found for $REPOS $TAG
056   fi
057   }
058
059 check_for_updates () {
060   REPOS=$1
061   TAG=$2
062   ID=$3
063   local myresult=0 #default return value
064   timestamp=`date --rfc-3339=seconds`
065   echo $timestamp >> debug
066   echo $REPOS $TAG >> debug
067   blub=`echo $REPOS | egrep "\/"`
068   if [ ! "$blub" ] ; then REPOS=library/$REPOS ;fi  #add library/ before repo if needed
069   DockerIdNew=`./get-docker-hub-image-tag-digest.sh $REPOS $TAG | jq -r '.config.digest' | awk -F':' '{print $2}' `
070   echo DockeridNew is $DockerIdNew >> debug
071   echo DockerIdLocal is $ID >>debug
072   if [ ! $DockerIdNew ] ; then myresult="" ;echo $myresult ; exit ;fi    #if empty image was probably built locally
073
074   if [ "$DockerIdNew" == "$ID" ]
075   then
076     #echo "you r good, no updates for $REPOS:$TAG"
077     myresult=""
078     echo $myresult
079   else
080     #echo "update available for $REPOS:$TAG"
081     echo "update available for $REPOS:$TAG" >> UpdateTheseImages
082     echo "update available for $REPOS:$TAG" >> debug
083     myresult=1
084     echo $myresult
085   fi
086   }
087
088 check_args() {
089   if (($# != 3)); then
090     echo "Error:
091     Three argument must be provided - $# provided.
092
093     Usage:
094       ./check-docker-image-updates.sh <image> <tag> <imageID>
095       for instance ./check-docker-image-updates.sh library/mariadb latest 2345623745234753647
096       imageID is local digest image without sha256: part
097 Aborting."
098     exit 1
099   fi
100 }
101
102 main "$@"

The output will be:

nginx alpine can be updated: a new version is available
also check file UpdateTheseImages

Two files are produced: UpdateTheseImages, which holds lines on images that need updating, and debug, which holds logging on found IDs. Use cat to see the content of UpdateTheseImages:

cat UpdateTheseImages
update available for library/nextcloud:latest
update available for library/mariadb:latest
update available for mvance/unbound:1.8.3

The GitHub repository also has a Python email script, should you want to email update messages to a local SMTP server.

Conclusion

Playing around with Docker is fun, and before you know it, you have multiple Docker hosts running services. However, all software needs updating to keep it safe, and software in containers is no exception. In this article, I showed a practical way to start automating update notices of Dockerized services, so you know when to update.