Container Inspection

If a customs officer wants to know whether a freighter from, say, Brazil really has bananas on board, they simply look inside the containers. Google's test tool with the somewhat unwieldy name Container Structure Test (CST) [1] basically follows the same principle.

The tool walks into a predefined Docker container, performs various user-defined tests, and logs the results of the individual checks in detail at the command line. Automatically, you can ensure, among other things, that a container actually contains MySQL and not bananas.

Compact Friend

CST currently supports four different test types. First, developers can use it to inspect the filesystem to find out whether the configuration file for the database is in the right place and has the appropriate access privileges. Additionally, the tool takes a look at text files and searches for given strings.

For example, the user-generated tests check whether the database configuration file reserves enough cache space. If required, CST can call any command in the Docker container and examine its output. In this way, you could ask the package manager whether a specific package is installed.

Finally, the tool also checks the configuration and metadata of the container on request, which makes it possible, for example, to ensure that the operator has set all the required environment variables correctly in the container.

With the appropriate tests, the test suite ensures that a new Docker container complies with the required or prescribed specification, even before it is put into operation. For example, if a prepackaged MySQL container from Docker Hub suddenly comes with a new version of the database, the CST tool immediately notices this. In this way, you don't have to wait for the web application to deliver mysterious error messages during live operation to know the API has changed.

CST also ensures more robust operation of the complete infrastructure. The tool can also be integrated easily into an existing workflow and launched automatically (e.g., whenever a container is updated).

However, the tool does not take a close look at running Docker containers. Moreover, the quality of the test results depends to a large extent on the stored tests. For example, if a developer forgets to check the MySQL version, the test tool will not complain about an outdated database.

Command Set

Although Google refers to CST as a test framework, it actually only consists of a command-line tool written in Go. CST carries out all the intended tests and returns a test report when done. The source code is licensed under the Apache 2.0 license and is available from GitHub [1].

For 64-bit systems, the prebuilt tool is available on the Google servers [2]. You only have to download the program and make it executable, thus removing the need to install. If you want to use the tool globally on a system, the Google developers recommend installing with the one-liner:

curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

All following examples assume that the test tool file name is container-structure-test and is accessible in the search path.

Delivery Notes

Developers add the tests to be performed to a text file with either YAML or JSON notation, which means that the tests can even be generated or modified by other tools or scripts. A simple test example in YAML notation is shown in Listing 1, which first checks whether the MySQL server is installed in the container. It then inspects the associated configuration file, which must be found at the specified location and must contain a special setting. CST ignores all lines starting with a hash (#).

01 schemaVersion: "2.0.0"
02
03 globalEnvVars:
04 - key: "MYSQL_USER"
05 value: "dev"
06 - key: "MYSQL_PASSWORD"
07 value: "123456"
08
09 commandTests:
10
11 # Is the MySQL package installed?
12 - name: "MySQL package installed"
13 command: "rpm"
14 args: ["-q", "mysql-community-server-minimal"]
15 expectedOutput: "["mysql-community-server-minimal*"]
16
17 fileExistenceTests:
18
19 # Does the MySQL client exist in the container?
20 - name: "mysql client exists"
21 path: '/usr/bin/mysql'
22 shouldExist: true
23
24 # Does the MySQL configuration file exist at the right place?
25 - name: "my.cnf exists and has appropriate permissions"
26 path: '/etc/my.cnf'
27 shouldExist: true
28 permissions: '-rw-r--r--'
29
30 fileContentTests:
31
32 # Does the configuration file my.conf have the correct content?
33 - name: 'Content of my.cnf'
34 path: '/etc/my.cnf'
35 expectedContents: "["datadir=/var/lib/mysql\n"]
36
37 metadataTest:
38 env:
39 - key: "PATH"
40 value: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
41 exposedPorts: ["3306", "33060"]
42 volumes: "["/test"]
43 entrypoint: [/entrypoint.sh]
44 cmd: ["mysqld"]

How developers need to define each test is specified in a matching schema. At the beginning of Listing 1, schemaVersion: "2.0.0" reveals that the test descriptions that follow use the 2.0.0 schema. Without this information, the tool immediately denies service.

Under globalEnvVars (lines 3-7; note that the colon is left out in the text explanation throughout to avoid punctuation confusion) you set environment variables in the container before starting the tests; key is followed by the environment variable itself, and the variable's content follows the value keyword. Listing 1 uses this to pass the logon credentials for the database into the container. The container then has a MYSQL_USER environment variable containing dev, and MYSQL_PASSWORD has the password 123456. These environment variables can now be used for the tests.

The commandTests section (lines 9-15) lets you list the checks for which the tool must execute a command in the Docker container. The single test in Listing 1 asks the RPM package manager whether the mysql-community-server-minimal package is installed in the container.

Like all other tests, it has a short description after name. In Listing 1 this is MySQL package installed. The test framework will use this in the log later, so you should choose intuitive and unambiguous names.

Chain of Commands

The command in line 13 states the program to be executed, to which the test tool immediately passes all the parameters that follow args. In YAML format, a list comprises comma-separated elements inside square brackets. Listing 1 calls the rpm -q mysql-community-server-minimal command.

The output returned by this command must match the text entered after expectedOutput (line 15), which is the condition for passing the test. As with all other text comparisons, the test tool interprets the term between the quotation marks as a regular expression. For a successful test in Listing 1, the name of the identified package must start with mysql-community-server.

In addition to command and expectedOutput are setup and teardown (not used in this example), which specify commands that precede and follow the test, respectively. They can be used, for example, to create a virtual environment in a Python container and then fetch the Django framework with the pip package manager:

setup: [["virtualenv", "/env"], ["pip", "install", "django"]]

You need to encapsulate each command in a separate YAML list and save separately in the list each parameter to be passed in. In the example, the test tool would first run virtualenv /env and then pip install django.

Face Check

The checks on the filesystem are grouped below fileExistenceTests (lines 17-28). The example in Listing 1 tests whether the MySQL client and the database configuration file exist, where path is the file the tool is to examine. Both tests are considered to have passed if the file exists (shouldExist: true). The configuration file must also have the access rights listed after permissions.

The section below fileContentTests (lines 30-35) groups all the tests for which the tool needs to inspect a text file. The file to be inspected again follows path. The test is passed if the file contains the text listed after expectedContents. CST again interprets this as a regular expression. In Listing 1, the tool searches the database configuration file for a line containing datadir=/var/lib/mysql\n and ensures that the MySQL server stores its data below /var/lib/mysql. The \n stands for a line break, as usual.

The final checks, grouped below metadataTest, take a close look at the configuration of the container itself. Although developers can use the other sections shown earlier to store any number of tests in line with the schema, each test is only allowed once under metadataTest. The env parameter first checks whether the specified environment variables are set in the container; key is followed by the environment variable again, and its required value follows value. Listing 1 tests for a PATH environment variable with the specified paths in the container.

The exposedPorts entry then ensures that the listed ports are open. Similarly, volumes tests whether the specified volume exists, entrypoint checks the entry point, and cmd checks the command called automatically when the container is started (CMD in the Docker configuration file).

Container Inspection

Once all the tests are set up, it's time to pull the Docker image to be examined onto your hard drive. For example, to check out the official MySQL Docker image, mysql/mysql-server, from Docker Hub, you must run the command:

docker pull mysql/mysql-server

You then feed the name of the container image to be examined and the file containing the tests to be executed to the executable container-structure-test tool (Figure 1). In the following example, CST would run the tests from the mysql-tests.yaml configuration file against the mysql/mysql-server Docker image:

container-structure-test test --image mysql/mysql-server --config mysql-tests.yaml

To run the tests, container-structure-test starts the container. However, this is not always desirable (e.g., as part of a continuous integration process). In such cases, you can run container-structure-test directly against the image:

container-structure-test test --driver tar --image mysql/mysql-server --config mysql-tests.yaml

The --driver tar parameter ensures that container-structure-test packages the container image in a TAR archive and then executes the tests directly in the archive. Of course, this approach means that the CST tool can only perform the file-based tests and examine the container itself; it does not execute any commands (Figure 2).

Historic Research

Several more test options exist in addition to those presented in Listing 1. For example, you can check the access rights of a file and the user and group IDs with Uid and Gid. The official documentation on GitHub [1] explains the currently available test set but only provided a brief reference when this issue went to press.

For your first steps, Google offers the gcr.io/google-appengine/python Docker image, which essentially contains a Python interpreter. Matching tests can be found in a separate GitHub repository [3].

Parameters of the container-structure-test command-line tool have changed slightly in the past. Many examples online still refer to an older version. Even the official documentation on GitHub contains some broken sample commands.

Compared with older versions, you need to introduce the parameters with two hyphens and replace -test.v with the shorter test. Furthermore, the YAML file must be loaded explicitly with the --config parameter. In many obsolete examples, container-structure-test is also referred to as structure-test. The sample call that you frequently come across,

structure-test -test.v -image gcr.io/google-appengine/python python_command_tests.yaml

thus needs to be converted to the following for the current 1.4.0 version:

container-structure-test test --image gcr.io/google-appengine/python --config python_command_tests.yaml

In case of doubt, you can call

container-structure-test help

to reveal the valid parameters.

Conclusions

CST lets developers and admins validate containers extremely quickly. In most cases, you do not even need to start the containers. Thanks to the short YAML (or JSON) notation, the tests are quick and easy to enter, without a long learning curve.

However, the simple structure of the test descriptions does tend to limit your options. To find out whether version 5.7 or newer of MySQL is installed involves some contortions. Furthermore, CST is designed to test containers and images before they are used and not during operation. If you need to carry out extensive tests on live containers, you might want take a look at another product, such as Testinfra [4].