
Update your Docker containers safely
Safe Harbor
Docker containers are a great way to produce lean and portable services, not only for the enterprise, but also at home. Docker takes the complexity out of setting up software services that are inherently complicated, like an email or a Nextcloud server. However, it's difficult to know when and how to update the software inside the Docker container to keep it as secure as possible.
In this article, I present a scripted approach to alerting you about when to update Docker containers. The first script checks Docker Hub for image tags or versions available. If you want to update, you need to know your software version and adjust the image tag accordingly, so updates stay within a safe range of versions. The second set of scripts checks your local version against possible updates on Docker Hub. One mini-script produces an ID file containing the two Docker image IDs and another checks for updates available on the Hub.
With these scripts, you can automate your checks for available updates. The updating process itself is left in your hands.
Updating a Docker Container
Many official software projects like Nextcloud, MariaDB, Nginx, and so on maintain official Docker images. Often, though, their prescription for Docker is a Docker Compose file in which you install some latest version of the image; however, it is not advisable to keep blindly updating to a latest version. For instance, the Nginx container defined in the Docker Compose file shown in Listing 1, when run, pulls an nginx:alpine
image from Docker Hub, where nginx
is the image and alpine
is the tag (referred to hereafter as <image><tag>
).
Listing 1: docker-compose.yml
version: '2' services: web: image: nginx:alpine restart: always container_name: nginx ports: - 8080:80 environment: - NGINX_HOST=www.somewebsite.com - NGINX_PORT=80 command: "nginx -g 'daemon off;'"
If you pull nginx:alpine
, you pull a so-called "latest" version. Several commonly used tags also pull a latest version, such as <image>:latest
, <image>:stable
, nginx:alpine
, nextcloud:fpm-alpine
, and so on. By "latest," I mean the tag has no version (e.g., alpine-10.4
).
If you leave the latest
tag in your configuration files while updating the container, you are in for trouble: The software version might go from nginx-8.3
to nginx-11.2
, which, bottom line, could break things. On the other hand, if you do not update at all, the software after some time will contain known vulnerabilities (which is why you update in the first place, of course).
Depending on the software in the container, you should decide what version changes are probably safe or, in other words, the version range in which you want to update. By version range, I mean it's probably fine to upgrade nextcloud-10.4.1
to nextcloud-10.4.8
, but it's not advisable to update blindly from version 10.4 to version 12.3. A safe version range, then, would be to stay within version 10.4. Of course, it's entirely up to you what risks to take and what you consider a safe version range.
To update a Docker container within a safe version range, you need two pieces of information: the image version you are using right now and what <image><tag>
you will use when updating, so you don't run into trouble.
Image Version
To know whether your Docker container needs an update within a safe version range, you need to know what version you are running. The answer lies in a set of identifiers and tags to be found in the manifest that is also downloaded when you pull an image. You can inspect the manifest in two steps: Find the image ID on your local machine, and inspect its manifest for a version tag.
You can inspect container or image manifests on a Docker host (a host running Docker containers) as shown in Listing 2. The docker image
command produces the image ID of running containers. In this case, my running container uses the nginx
image with the id bf85f2b6bf52
.
Listing 2: Inspecting Image Manifests
docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx_web latest c100b674c0b5 13 months ago 19MB nginx alpine bf85f2b6bf52 13 months ago 15.5MB
With the image ID in hand, you can inspect the image manifest:
docker inspect bf85f2b6bf52 | egrep -i version
In this case, it yields: NGINX_VERSION=1.13.7.
An easy way to get the versions of all running Docker containers is shown in Listing 3. When I run the script on my Nextcloud machine, it yields (together with some lines of no use):
"NEXTCLOUD_VERSION=12.0.4" MARIADB_MAJOR=10.2 MARIADB_VERSION=10.2.12+maria~jessie
What Tag?
Listing 3: Docker Container Versions
## get versions for running containers for i in `docker ps | egrep -v NAMES | awk '{print $1}'`; do echo " image-ID is now $i" docker inspect --format='{{.Config.Env}}' $i docker inspect $i | egrep VERSION done
Now that you know your software versions, the next question to ask is what tag to use in docker-compose.yml
. You can't use nginx:alpine:1.13
, so what tag can you use?
The answer depends, of course, on what tags a software project has issued for its images on Docker Hub. I like to use the script in Listing 4, which asks the Docker Hub Registry API to list all available tags. To download the script, make it executable, and run it, enter:
curl https://raw.githubusercontent.com/ hanscees/dockerscripts/master/scripts/get-docker-hub--image-tags.sh > get-public-image-tags.sh chmod +x get-public-image-tags.sh ./get-public-image-tags.sh library/nginx
Listing 4: get-public-image-tags.sh
01 #!/bin/bash 02 03 # Retrieves image tags from public 04 # images in DockerHub 05 # if someone improves this script, please let me know, preferably in Python 06 # hanscees@AT@hanscees.com 07 # modified from https://gist.github.com/cirocosta/17ea17be7ac11594cb0f290b0a3ac0d1x 08 09 set -o errexit 10 11 main() { 12 check_args "$@" 13 14 local image=$1 15 local token=$(get_token $image) 16 local tags=$(get_tags $image $token) 17 echo "tags reported are:" 18 echo $tags 19 } 20 21 get_token() { 22 local image=$1 23 24 echo "Retrieving Docker Hub token. 25 IMAGE: $image 26 " >&2 27 28 curl --silent "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" | jq -r '.token' 29 } 30 31 # Retrieve the digest, now specifying in the header 32 # that we have a token (so we can ... 33 get_tags() { 34 local image=$1 35 local token=$2 36 37 echo "Retrieving image tags. 38 IMAGE: $image 39 " >&2 40 41 curl --silent --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $token" "https://registry-1.docker.io/v2/$image/tags/list" | jq -r '.tags' 42 } 43 check_args() { 44 if (($# != 1)); then 45 echo "Error: 46 One argument must be provided - $# provided. 47 48 Usage: 49 ./get-docker-hub-image-tags.sh <image> 50 for instance ./get-docker-hub-image-tags.sh library/mariadb 51 Aborting." 52 exit 1 53 fi 54 } 55 56 main "$@"
In the last line, I prepended library/
to nginx
: All official Docker Hub repositories require this prefix.
The output comprises many tags, including various versions of alpine
, perl
, and alpine-perl
, along with their mainline
and stable
versions (Table 1).
Tabelle 1: get-public-image-tags.sh Output
Tags |
||
---|---|---|
1.13.3-alpine |
1.13.8-alpine |
1.8-alpine |
1.13.3-perl |
1.13.8-perl |
1.8.1-alpine |
1.13.3 |
1.13.8 |
1.8.1 |
1.13.5-alpine-perl |
1.13.9-alpine-perl |
1.8 |
1.13.5-alpine |
1.13.9-alpine |
alpine |
1.13.5-perl |
1.13.9-perl |
latest |
1.13.5 |
1.13.9 |
mainline-alpine-perl |
1.13.6-alpine-perl |
1.13 |
mainline-alpine |
1.13.6-alpine |
1.14-alpine-perl |
mainline-perl |
1.13.6-perl |
1.14-alpine |
mainline |
1.13.6 |
1.14-perl |
perl |
1.13.7-alpine-perl |
1.15-alpine |
stable-alpine-perl |
1.13.7-alpine |
1.15.3-alpine |
stable-alpine |
1.13.7-perl |
1.15.3-perl |
stable-perl |
1.13.7 |
1.15.3 |
stable |
1.13.8-alpine-perl |
1.15.4-alpine-perl |
The get-public-image-tags.sh
script uses jq
(a command-line JSON processor) in lines 28 and 41 to parse the JSON. On Ubuntu, the command to install the utility is:
apt-get install jq
As you can see, Nginx has quite a few tags. It also becomes clear how to reference a version of nginx:alpine
: Simply use nginx:<version>-alpine
. Because Nginx seems unlikely to change in any important way, I would suggest considering the nginx:1.15-alpine
tag in the YAML file.
Automating Update Checks
When I first pulled down an Nginx image as nginx:alpine
in Listing 1, it turned out to be version 1.13. Using safe version logic, I should change the nginx:alpine
tag in the YAML file to nginx:1.15-alpine
.
Now I can update manually by running:
docker-compose pull docker-compose up -d
Alternatively, you can let an automated solution update your containers. Watchtower [1] (a Docker container), can do this for you, although I have not tested it.
However, I want to stick with manual updating, and I simply want to know when updates are available for <image><tag>
(compared with my current image, of course). If an update is available, I can make a snapshot of the virtual machine on which Docker runs, update, and roll back if things go wrong. This time, I use a script and learn something about image IDs, as well.
First, you need two more pieces of information: what version and image ID are you running and how it compares with the latest container ID of the version to which you want to update. The first item was already solved above, but not the second.
Repo ID vs. Image ID
The image ID was discovered by analyzing the manifest file of an image and extracting the version of the software. However, for an automated solution, this process won't do because every software project uses versions in a slightly different way. Therefore, you need a more general ID.
A Docker manifest uses several IDs, two of which are the image ID, which is a hash of the uncompressed image, and the repo ID (repo digest), which is a hash of the compressed image when it sits in the registry (from which you pull your images). Both are digests, by the way, but with different data payload forms.
Confusingly, different Docker command-line options show both IDs or parts of both, as you can see in the confusing output of the docker images
command in Listing 5, which shows information about images you pulled to your computer.
Listing 5: docker images Output
docker images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE nginx 1.15-alpine sha256:385fbcf0f04621981df6c6f1abd896101eb61a439746ee2921b26abc78f45571 315798907716 5 days ago 17.8MB nginx alpine sha256:f1ca87d9adb678b180c31bf21eb9798b043c22571f419ed844bca1d103f2a2f7 bf85f2b6bf52 13 months ago 15.5MB
The output shows information on both the original nginx:alpine
image and the updated nginx:1.15-alpine
. The Image ID column refers to a short version of the (local unpacked) image ID and the Digest column refers to the repo ID.
If you want to see both the full image hash and the repo digest of an image on a Docker host, the command in Listing 6 shows both.
Listing 6: docker image inspect
01 docker image inspect 315798907716 | head 02 { 03 "Id": "sha256:315798907716a51610bb3c270c191e0e61112b19aae9a3bb0c2a60c53d074750", 04 "RepoTags": [ 05 "nginx:1.15-alpine" 06 ], 07 "RepoDigests": [ 08 "nginx@sha256:385fbcf0f04621981df6c6f1abd896101eb61a439746ee2921b26abc78f45571"
The "Id"
(line 3) shows the image ID: The first 12 characters are the same as the output from docker images
; the repo ID is the repo digest. By the way, docker image inspect
reads a downloaded image manifest file that you get when you pull an image.
For the automated update check, you will collect both the image and repo IDs, but only the image ID can be used to automate a check without pulling an image because of the Docker Registry API (as far as I know).
A Scripted Update Check
After searching and finding some Bash scripts on GitHub that work on the Docker Registry API, I have come up with this solution:
1. A local Docker script first produces a local file on a Docker host with per-line <image><tag><current image ID>
and a second line with <image><tag><current repo ID>
. You can run this script on many Docker hosts, combine lines, and use one central host to check for updates on many Docker images you might have in production.
2. A second script reads the local file and, for each line, checks Docker Hub for the <image ID>
of the latest version of <image><tag>
; compares the local image tag to the image ID of latest
on the Hub; and lets you know what local Docker services thus can be updated (if local and remote IDs don't match).
Although you can probably find many ways to solve this problem, the script lets you check a number of different images at the same time. Also, a centralized check is possible by combining with a centralized scanning solution.
The scripts to do this can be found on GitHub [2], which you can either clone or download one by one (Listing 7), because Git is quite a large package [3].
Listing 7: Downloading Update Scripts
mkdir dockerscripts cd dockerscripts curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/check-docker-image-updates.sh > check-docker-image-updates.sh curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/get-docker-hub--image-tags.sh > get-docker-hub--image-tags.sh curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/get-docker-hub-image-tag-digest.sh > get-docker-hub-image-tag-digest.sh curl https://raw.githubusercontent.com/hanscees/dockerscripts/master/scripts/emailer.py > emailer.py #-chmod them chmod +x *.sh chmod +x *.py #- install qs apt-get install jq
Local Docker Script
The script to get Docker file IDs on a Docker host is not much of a script, really; just copy and paste the code in Listing 8 in a terminal to produce the ImageId-file
output file, which has two lines per image, as shown in Listing 9. The first line holds the repo ID, the second the image ID. Each line contains the image name, the tag name, and either the image ID or repo ID. If the last word of the line is RepoId, the line holds the repo ID, making it easy to grep lines with the right kind of ID.
Listing 9: ImageId-file List
nginx 1.15-alpine nginx@sha256:385fbcf0f04621981df6c6f1abd896101eb61a439746ee2921b26abc78f45571 RepoId nginx 1.15-alpine 315798907716a51610bb3c270c191e0e61112b19aae9a3bb0c2a60c53d074750 nextcloud latest nextcloud@sha256:78515af937fe6c6d0213103197e09d88bbf9ded117b9877db59e8d70dbdae6b2 RepoId nextcloud latest 8757ce9de782c2dd746a1dd702178b8309ca6d2feb5e84bad9184441170d4898 mariadb latest mariadb@sha256:12e32f8d1e8958cd076660bc22d19aa74f2da63f286e100fb58d41b740c57006 RepoId mariadb latest b468922dbbd73bdc874c751778f1ec0ec10817691624976865cb3ec5c70cd4e0 mvance/unbound 1.8.3 mvance/unbound@sha256:d67469fad9cc965f032e4ea736c509df6d009245dac81339e2c6e1caef9b65ac RepoId mvance/unbound 1.8.3 mvance/unbound latest a88e44773675294dcd10e08a9a2a5ee2a39796e5a832c99606b3c8a54901ea75
Listing 8: Get Docker File IDs
> ImageId-file for i in `docker images -f dangling=false| egrep -v TAG | awk '{print $3}'` ; do echo copying name, tag and image ID digest to file ImageId=`docker image inspect $i | jq -r '.[0] | {Id: .Id}' | egrep Id | awk -F":" '{print $3'} | awk -F"\"" '{print $1}'` RepoId=`docker image inspect $i | jq -r '.[0] | {RepId: .RepoDigests}' | jq --raw-output '.RepId | .[]'` ImageData=`docker images | egrep $i | awk '{print $1," " , $2}'` echo $ImageData $ImageId echo $ImageData $RepoId RepoId >> ImageId-file echo $ImageData $ImageId >> ImageId-file done
You should check ImageId-file
, because it might contain lines about images you do not use anymore: The script will produce lines on all images used in containers, even if you no longer run those containers or have not yet run them. Make sure to discard any "old" containers.
A Central Docker Host Check
If you run the script in Listing 8 on a Docker host, it will collect ID lines on all containers on that host. If you have multiple Docker hosts, you can run the script on multiple hosts and combine the lines into one big file on a central host; of course, you could automate this process and run checks on a central Docker host for all the containers you are using, run checks for updates for your <image><tag>
combinations, and download the images and check them against known vulnerabilities.
With the repo ID lines collected, all images can be downloaded on the central host, because the repo ID can be used to pull images with docker pull <repo ID>
; for instance,
docker pull mariadb@sha256:12e32f8d1e8958cd076660bc22d19aa74f2da63f286e100fb58d41b740c57006
will download that specific MariaDB image from Docker Hub. Therefore, you can download many images to a central host without actually running them in a container with,
for image in `cat ImageId-file | egrep RepoId | awk '{print $3}'` ; do echo $image ; docker pull $image done
and show them by entering docker images
.
Once you have your images on one host, you will find multiple solutions to check your images against known vulnerabilities. Examples are Anchore, Clair, or Dagda, which all run on Docker, so they are relatively easy to set up [4].
In the next section, however, I show you how to check for container updates – without the need to download the images, by the way.
Docker Updates Script
At this point you have the image ID file, which knows what Docker <image><tag>
pairs you have, and the IDs of these images as used in production. Now, you need to see whether these images can be updated.
The first part of the update check script downloads a manifest file from Docker Hub for a specified Docker <image><tag>
using the Registry API. The second part of the script does two things: Loops over all image lines, downloading a manifest file for each, and checks the local image ID against the ID from the downloaded manifest. If they differ, an update is available.
Listing 10 shows the first part of the script, which fetches a digest from Docker Hub. As it states (if you run it without arguments), you should feed it two arguments:
./get-docker-hub-digest library/nginx 1.15-alpine
Listing 10: get-docker-hub-digest
01 #!/bin/bash 02 03 # Retrieves image digest from public 04 # images in DockerHub 05 # modified from https://gist.github.com/cirocosta/17ea17be7ac11594cb0f290b0a3ac0d1x 06 07 set -o errexit 08 09 main() { 10 check_args "$@" 11 12 local image=$1 13 local tag=$2 14 local token=$(get_token $image) 15 local digest=$(get_digest $image $tag $token) 16 17 echo " $digest" 18 19 } 20 21 get_token() { 22 local image=$1 23 24 echo "Retrieving Docker Hub token. 25 IMAGE: $image 26 " >&2 27 28 curl --silent "https://auth.docker.io/token?scope=repository:$image:pull&service=registry.docker.io" | jq -r '.token' 29 } 30 31 # Retrieve the digest, now specifying in the header 32 # that we have a token (so we can ... 33 get_digest() { 34 local image=$1 35 local tag=$2 36 local token=$3 37 38 echo "Retrieving image digest. 39 IMAGE: $image 40 TAG: $tag 41 " >&2 42 # TOKEN: $token # add to echo for debug 43 44 curl --silent --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $token" "https://registry-1.docker.io/v2/$image/manifests/$tag" 45 # | jq -r '.config.digest' 46 } 47 48 check_args() { 49 if (($# != 2)); then 50 echo "Error: 51 Two arguments must be provided - $# provided. 52 53 Usage: 54 ./get-docker-hub-digest.sh <image> <tag> 55 for instance ./get-docker-hub-digest library/nginx 1.15-alpine 56 57 Aborting." 58 exit 1 59 fi 60 } 61 main "$@"
The script spits out a JSON file with a series of digests, of which the first is the image ID (not the repo ID, for some reason).
The second part of the check script (Listing 11) you will actually use. It checks for updates on Docker images and uses the script in Listing 10 to get digests. You can feed it the image ID file produced in Listing 10 (without the RepoId lines) with:
cat ImageId-file | egrep -v "RepoId|none|^$" | ./check-docker-image-updates.sh
Listing 11: check-docker-image-updates.sh
001 #!/bin/bash 002 003 # Checks Docker Hub for updates on <image><tag><ImageId> 004 # ImageId without sha256: part 005 # script accepts <image> <tag> <ImageId> or sdtin 006 007 # reads file from standard in: lines in the form of 008 # nginx 1.14-alpine #315798907716a51610bb3c270c191e0e61112b19aae9a3bb0c2a60c53d074750 009 # mvance/unbound latest 4568745687569875689745689756 010 # 011 # Script calls other script ./getdigest. Can be found at #https://github.com/hanscees/dockerscripts 012 013 set -o errexit 014 > UpdateTheseImages 015 main() { 016 017 REPOS="" 018 TAG="" 019 ID="" 020 021 if [ -t 0 ] ; then 022 echo terminal input; #and no stdin 023 check_args "$@" #if input not from stdin 024 REPOS=$1 025 TAG=$2 026 ID=$3 027 report_result $REPOS $TAG $ID 028 else 029 echo "not a terminal, so reading stdin"; 030 while read line; do 031 declare -a ImageData=($line) #bash array 032 REPOS=${ImageData[0]} 033 TAG=${ImageData[1]} 034 ID=${ImageData[2]} 035 report_result $REPOS $TAG $ID 036 done 037 fi 038 } #end main 039 040 report_result () { 041 REPOS=$1 042 TAG=$2 043 ID=$3 044 echo "repo/image and tag are " 045 echo $REPOS $TAG 046 047 check_result=$(check_for_updates $REPOS $TAG $ID) 048 echo check_result is $check_result 049 050 if [ $check_result ] 051 then 052 echo $REPOS $TAG can be updated: a new version is available 053 echo also check file UpdateTheseImages 054 else 055 echo NO update found for $REPOS $TAG 056 fi 057 } 058 059 check_for_updates () { 060 REPOS=$1 061 TAG=$2 062 ID=$3 063 local myresult=0 #default return value 064 timestamp=`date --rfc-3339=seconds` 065 echo $timestamp >> debug 066 echo $REPOS $TAG >> debug 067 blub=`echo $REPOS | egrep "\/"` 068 if [ ! "$blub" ] ; then REPOS=library/$REPOS ;fi #add library/ before repo if needed 069 DockerIdNew=`./get-docker-hub-image-tag-digest.sh $REPOS $TAG | jq -r '.config.digest' | awk -F':' '{print $2}' ` 070 echo DockeridNew is $DockerIdNew >> debug 071 echo DockerIdLocal is $ID >>debug 072 if [ ! $DockerIdNew ] ; then myresult="" ;echo $myresult ; exit ;fi #if empty image was probably built locally 073 074 if [ "$DockerIdNew" == "$ID" ] 075 then 076 #echo "you r good, no updates for $REPOS:$TAG" 077 myresult="" 078 echo $myresult 079 else 080 #echo "update available for $REPOS:$TAG" 081 echo "update available for $REPOS:$TAG" >> UpdateTheseImages 082 echo "update available for $REPOS:$TAG" >> debug 083 myresult=1 084 echo $myresult 085 fi 086 } 087 088 check_args() { 089 if (($# != 3)); then 090 echo "Error: 091 Three argument must be provided - $# provided. 092 093 Usage: 094 ./check-docker-image-updates.sh <image> <tag> <imageID> 095 for instance ./check-docker-image-updates.sh library/mariadb latest 2345623745234753647 096 imageID is local digest image without sha256: part 097 Aborting." 098 exit 1 099 fi 100 } 101 102 main "$@"
The output will be:
nginx alpine can be updated: a new version is available also check file UpdateTheseImages
Two files are produced: UpdateTheseImages
, which holds lines on images that need updating, and debug
, which holds logging on found IDs. Use cat
to see the content of UpdateTheseImages
:
cat UpdateTheseImages update available for library/nextcloud:latest update available for library/mariadb:latest update available for mvance/unbound:1.8.3
The GitHub repository also has a Python email script, should you want to email update messages to a local SMTP server.
Conclusion
Playing around with Docker is fun, and before you know it, you have multiple Docker hosts running services. However, all software needs updating to keep it safe, and software in containers is no exception. In this article, I showed a practical way to start automating update notices of Dockerized services, so you know when to update.