Container Check

Thanks to the unremitting, ever-present threat of a multitude of attacks to which a Linux system can be subjected, it's critical to capture important changes and events made by users and processes on your running systems.

Highlighting such changes could potentially point toward something as innocuous as a simple misconfiguration but, equally, might proactively help stop an impending attack dead in its tracks. Additionally, having trustworthy, detailed logging data is exceptionally useful for post-event forensic analysis, especially when you are trying to discern how an attacker originally managed to compromise your system and get a foothold.

One such package I have been using recently on a large AWS server estate is called auditd. Its man page states: "auditd is the userspace component to the Linux Auditing System."

One of the pages on the Linux Audit Documentation project GitHub site describes its (very old) original design as being based around the following aims:

The main goals were to provide system call auditing 1) with as low overhead as possible and 2) without duplicating functionality that is already provided by SELinux (and/or other security infrastructures).

For the uninitiated, "system calls," which are more commonly referred to as "syscalls," occur when processes ask the kernel for a hand with something. The syscalls man page reports, "The system call is the fundamental interface between an application and the Linux kernel."

Simply think of every task, such as opening a socket for network communications, mounting a disk volume, or even creating a directory, as needing some form of assistance and therefore validation from a host's kernel. In many cases, a handling program like the all-pervasive glibc (GNU C Library) will invoke the syscall directly and not the underlying application that is asking for help.

If you're keen to encourage narcolepsy, you can find a list of system calls with some more glibc wrapper detail thrown in for good measure on the syscalls man page mentioned above.

In this article, I'll help you set up auditd and give you some food for thought in relation to what syscall information you might want to capture when it comes to containers.

Bring out the DevOps

The advantages of carefully monitoring such key system changes are clear, but you might rightly ask: "Why is this even more important in a containerized environment?"

The reason is surprisingly simple. On a host running many containers (I'll use the popular Docker as an example) all share the same Linux kernel. Without some jiggery-pokery your superuser root inside one container or more is effectively the root user on your host, which, from a security perspective, is unwelcome for a myriad of reasons.

As a result, if a vulnerability is used to exploit a container to a high enough degree, then the other containers and potentially the host itself are at risk of being compromised. Such an event, wherein a bad actor (an attacker) breaks into the larger host system by way of a relatively minuscule container, is known as a "container escape." The implications are varied, but a favorite is gaining access to the host's filesystem, reading secrets (passwords), and being able to access anything visible to the host itself using its credentials. That might be far beyond the host's local cluster or VLAN and into other trusting, geographically disparate environments.

Vendors such as Docker have made a number of welcome strides to mitigate such risks, but do not imagine for one moment that the issues around container security are their responsibility. The Docker Security page opens candidly with the following warning:

There are four major areas to consider when reviewing Docker security: the intrinsic security of the kernel and its support for namespaces and cgroups; the attack surface of the Docker daemon itself; loopholes in the container configuration profile, either by default, or when customized by users; the "hardening" security features of the kernel and how they interact with containers.

For these reasons, among others, tracking issues to a high degree of detail is critical to the uptime of your containerized environments, especially in an enterprise environment. You can lessen the possibility of a compromise in one environment being successful in another environment if the attack vector that was exploited is determined early enough using such information.

In simple terms, a container suddenly opening up a network port that it's never used before or catching a misbehaving container trying to set the host's clock to an arbitrary time would almost definitely not be welcome surprises.

Needless to say, if you like squinting at kernel logs or want to monitor users and applications on your systems to a really high level of detail, then the auditd package is for you. It is noisy in its output for good reason: detail.

The installation command on Debian derivatives such as Ubuntu (as per Debian 8 Jessie) is shown in Listing 1.

$ apt install auditd
chrisbinnie ~ # apt install auditd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libauparse0
Suggested packages:
  audispd-plugins
The following NEW packages will be installed:
  auditd libauparse0
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 253 kB of archives.
After this operation, 833 kB of additional disk space will be used.

Notice that a dependency is listed when installing auditd on Debian derivatives. Somewhat surprisingly, on Red Hat derivatives (CentOS 7, anyway), I need to install the audit package and not the auditd package, as on Debian:

$ yum install audit

Don't let this trip you up if it's not already installed for some reason.

Audit This

The all-seeing, omnipotent auditd and supporting packages allow you to report on fine-grained system actions at varying levels of detail. With too many configuration options to cover in this article, bear in mind the following aspects if you want to try it out for yourself.

First, auditing takes place directly from the kernel and is saved, usually at least to /var/log/audit/audit.log. That file is readable by the root superuser only and for very good reason: You don't want anyone or anything tampering with this file if you are relying on it to examine your system forensically after either an attempted or successful compromise. For this reason, the auditd package also insists that permissions are set to root:root after rotating logs, which leads me to the next aspect of auditd to be considered.

Professionally, I deal with Docker and Kubernetes security. If you've used it before, you might be able to guess how often the Docker run time (the main Docker binary file running as a daemon) makes syscalls to the kernel. If you can't imagine, then let me tell you: It's a lot!

With a few containers running on a host, your auditd logs will soon be creaking at the seams with pull, push, run, and kill commands if a human is interacting with the Docker API; however, if an orchestrator like Kubernetes or OpenShift is accessing that API, then the logs will be filling up even quicker. As a result, it's important to forward the logs off-host to a syslog server or something similarly functional (e.g., Splunk in an enterprise environment) for security, in case an attacker manages to elevate to root and somehow affect your logging. Meanwhile, you are preserving resources because you only need to keep a few days of logs on a host.

Inside the main auditd configuration file (/etc/audit/auditd.conf), you can change the maximum size of each logfile and how many logs you want to keep; it is unquestionably wise to set these to something sane as soon as you possibly can. Following each change, be sure to restart the daemon as root in systemd:

$ systemctl restart auditd

Also, you should make sure that the daemon is set to load after each reboot:

$ systemctl enable auditd

Note that to keep you on your toes, the daemon/service is named auditd and the package is called audit. I am always using the command

$ systemctl list-unit-files

to find systemd service names; if you forget and think you're losing your marbles because your service isn't starting or stopping, simply run this command and search for audit.

Where Did They Go?

The filesystem file and directory structure under the directory /etc/audit is shown in Listing 2. Remember, you make changes in the rules.d/ directory, which is then written to the audit.rules file after regenerating rules with augenrules.

chrisbinnie ~ # cd /etc/audit/
chrisbinnie audit # ls
auditd.conf  audit.rules  rules.d
chrisbinnie audit # ls -al
total 20
drwxr-x---  3 root root 4096 Dec 16 12:34 .
drwxr-xr-x 95 root root 4096 Dec 16 12:34 ..
-rw-r-----  1 root root  701 Dec  9  2014 auditd.conf
-rw-r-----  1 root root  373 Dec  9  2014 audit.rules
drwxr-x---  2 root root 4096 Dec 16 12:34 rules.d
chrisbinnie audit # ls -al rules.d/
total 12
drwxr-x--- 2 root root 4096 Dec 16 12:34 .
drwxr-x--- 3 root root 4096 Dec 16 12:34 ..
-rw-r----- 1 root root  373 Dec  9  2014 audit.rules

Apologies that I've been a little "now hit the Enter key" with the directory listings displayed in Listing 2. It's purely to emphasize that the changes you make inside the file rules/auditd.rules then appear magically inside the file audit.rules in the directory /etc/auditd after the execution of the augenrules command, which I will look at now.

As mentioned, after you make a change (assuming "immutability" isn't enforced, which I'll cover a little later), you follow a procedure each time you update your rules. The file /sbin/augenrules is used to merge (concatenate, really, in a "sorted" order) all the rules files that are present in the rules.d directory. The rationale is that you might have several files in that directory grouped into rules by application. In this way, it's much easier to administer an organized set of rules when you potentially can have a heap of configured rules. The mighty auditd offers some examples to put inside this directory:

$ ls /usr/share/doc/auditd/examples
auditd.cron capp.rules.gz lspp.rules.gz nispom.rules.gz stig.rules.gz

I will leave you to read the comments in these files if you want to pick up some tips and sample rules to add to your own monitoring configuration. Containerized environments or not, you can definitely make good use of monitoring (e.g., user creation and network event rules, among a zillion other things).

Back to the matter in hand, however: The command to run after each rule change in rules.d/audit.rules is:

$ augenrules --load

Running --check instead of --load will tell you whether the trimmed-down, concatenated /etc/audit/audit.rules file needs updating because it differs from the rules.d/audit.rules file.

Your Point Is?

To whet your appetite further, I'll now get to the good stuff and cover a useful Docker-orientated auditing rule. Inside your rules config file (this is the last time I'll remind you that it's rules/auditd.rules and not /etc/audit/audit.rules), you can add a variety of rules along with what they monitor.

With auditd, you have three distinct types of rules, all of which are configured in rules.d/audit.rules. They comprise "control rules," which do things like set buffer sizes and flush currently running rulesets to keep things ticking over sensibly, avoid duplication, and so on; "watch rules," which are focused on a specific file's activity; and "syscall rules," which catch and log kernel system activity around any matching task, exit, user, and exclude lists (essentially filters).

The syntax for watch rules is broken down as follows,

-w <path-to-file> -p <permissions> -k <keyname>

where <permissions> are:

• r – read access
• w – write access
• x – execute access
• a – change in a file or directory attribute

The syntax for syscall rules is:

-a <action>,<list> -S <syscall> -F <field>=<value> -k <keyname>

The <action> is either always or never, and <list> is a kernel rule-matching filter (i.e., task, exit, user, exclude); <sys_call> is the name or number of a system call. You can see a list of <field> types and <value>s on the auditctl man page.

Watching the Watcher

Using the layout above for watch rules, I'll set up a command that keeps a close eye on a Docker daemon binary. Thankfully it's not as tricky as you might think when you know how.

If you prefer that a rule does not persist after a reboot, you use the auditctl command, which I might do if I were testing in a sandbox (or if my /var/log/audit/audit.log logfile is really noisy and I need a quieter logfile to grep). Generally, I find building up the rules.d/audit.rules file section by section is fine, as long as I remember to run augenrules afterward to check for errors (e.g., watch rules pointing at non-existent files).

The auditctl approach is as simple as entering:

$ auditctl -w /usr/bin/docker -p rwxa -k docker-daemon

Briefly, this simple rule takes the path of the Docker binary; adds a watch for events relating to read, write, execute, and attribute changes; and writes to the trusty logfile, appending a docker-daemon label (the <keyname> in the command template) at the end of each line to help with searches.

To check which rules are currently applied on your system enter:

$ auditctl -l

The output will look just like the command minus the auditctl at the start:

chrisbinnie ~ # auditctl -l
-w /usr/bin/docker -p rwxa -k docker-daemon

In the /var/log/audit/audit.log logfile, you can now see that your faithful Docker daemon is being monitored closely. To give yourself something to look for (more on searching effectively in a moment), run the command:

$ docker pull chrisbinnie/super

(If you're container crazy, check out the article "Troubleshooting Kubernetes and Docker with a SuperContainer" [1] to see why I pulled the super image.)

Now when you run the Docker pull command to grep for the latest command, you see

$ cat /var/log/audit/audit.log | grep -i christype=EXECVE msg=audit(1513507468.995:124): argc=4 a0="/bin/sh" a1="/usr/bin/docker" a2="pull" a3="chrisbinnie/super"

which I've abbreviated for ease of reading.

That log entry hopefully makes some sense. Although interpreting the above is logical enough, the correct way to search is with the ausearch command, which the considerate auditd makes available as part of its bundled tools.

Cast your mind back to the watch rule's syntax with a -k <keyname> (label) added to the end of the rule (in my example, it was docker-daemon). Using the ausearch tool, you can search for that label or keyname to retrieve the logs for ALL of the activity that the Docker binary has generated. As warned, these logs can be noisy, so the output in Listing 3 has been heavily abbreviated. For those familiar with SELinux, you might be able to spot some recognizable information.

$ ausearch -k docker-daemon
type=SYSCALL msg=audit(1513507481.041:143): arch=c000003e syscall=59 success=yes exit=0 a0=e5c9f0 a1=e5a110 a2=e5c660 a3=7ffee7cf14b0 items=3 ppid=1140 pid=1541 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="docker" exe="/usr/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="docker-daemon"

This apparently simple search tool is actually very sophisticated. You can put it through its paces for a specific process ID (PID) (Listing 4).

$ ausearch -p 1431
----
time->Nov 11 11:11:11
type=VIRT_CONTROL msg=audit(1513507481.075:145): pid=1431 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='user=root auid=0 exe=? reason=api op=create vm=? vm-pid=? hostname=? exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'

The ausearch command has a heap of options, such as searching on event IDs, user groups, syscall names, SELinux context, and words and strings that exactly match, to name a few. The man page

$ man ausearch

helps out if you want to explore further.

I would be remiss not to mention an accompanying tool that sits alongside ausearch very nicely called aureport. The man page describes it as having the ability to summarize auditd system logfiles into reports as follows:

The reports have a column label at the top to help with interpretation of the various fields. Except for the main summary report, all reports have the audit event number. You can subsequently lookup the full event with ausearch -a event number. You may need to specify start & stop times if you get multiple hits. The reports produced by aureport can be used as building blocks for more complicated analysis.

If you think this will be useful, this example checks for login activity:

$ aureport -au --summary
Authentication Summary Report
=============================
total  acct
=============================
366  somedude
21  chrisbinnie
16  root

For a count of executable events with ausearch, the command output (abbreviated) is:

$ ausearch --start today --raw | aureport -x --summary
Executable Summary Report
=================================
total  file
=================================
1223  /usr/bin/dpkg
591  /bin/ls
118  /bin/cp

As the man page stated about using start and stop times, be warned that this is very CPU-heavy and dependent on the time range you select.

Boot Up the Behind

Another important component to think carefully about is what happens when the newer init system, systemd, starts up a modern day Linux after a reboot. The command

$ systemd-analyze
Startup finished in 1.026s (kernel) + 5.925s (userspace) = 6.952s

shows you the slowest services to load when you boot up your system, so you can name and shame those services (and then fine tune them afterward). Just under six seconds of userland applications are slowing down the system. The next command shows the top five worst offenders:

$ systemd-analyze blame | head -5
          2.084s fail2ban.service
          1.958s cloud-init.service
          1.851s cloud-init-local.service
          1.597s apache2.service
          1.134s postfix.service

As you can see, during the last reboot, the fail2ban rules took a whopping two seconds. Be aware these values change with each boot depending on a number of events, so don't focus entirely on these metrics, just the slower packages. Remember that systemd can also load its services using parallelism.

What's this got to do with auditing, you might ask? Quite a lot, actually.

It's very important to log services during the boot process with auditd and not just let those services do whatever they like (e.g., think about trojans, keyloggers, and boot-level malware) without generating any logging that can then be referenced later.

Out of the box, auditd needs a little help. I suspect this might not be the default (on Red Hat machines, anyway) because of performance-related concerns. If you have a multitude of badly organized rules, you can incur a slight kernel performance hit as each bit of activity is checked then logged before being executed. You should do more research if you're using a shipload of rules in a production environment.

Assuming you're using the GRUB bootloader, you can append a simple audit=1 to the kernel or linux line within your GRUB config, depending on your Linux flavor.

In Debian, I add the line

GRUB_CMDLINE_LINUX="audit=1"

to the /etc/default/grub file, which then is added to /boot/grub/grub.cfg after the update command:

$ grub-update

This is slightly different on Red Hat derivatives (which, strictly speaking, create a new config file; potentially, there might also be an update equivalent that you can use instead).

$ grub2-mkconfig > /boot/grub2/grub.cfg

As you can probably guess, it's now just a case of rebooting the system and looking at the start of the logfile to see which services are now also to be logged to the auditd logfile /var/log/audit/audit.log, as well.

Talkin' Loud and Sayin' Nothing

All this log output isn't very useful if you've no idea what it means. Interpreting the audit.log file is clearly very important, and I won't claim it's always easy, but with some exposure you'll get the hang of it eventually.

Bear in mind that many of the logs are event based and the timestamps, which are written something like

msg=audit(1513507481.075:145)

are how the system clock records each event: a timestamp (the longer number) followed by a colon and then the ID applied to it (the shorter, trailing number). If a number of events happen at once (which you'd expect on modern systems with high-power threaded CPUs), many entries could have the same time and ID. Using this knowledge, you can check when and what happened, in what order.

If you've used Linux for a while, then fields such as pid (process ID) and ppid (parent process ID) should make some sense. A handy entry to look out for is auid, which is the "actual user ID" of the user that generated this activity (even if they became root as UID 0, the original user is logged).

With time, this becomes easier to fathom, so don't despair if it initially induces eye strain; just refine searches with queries per event across short time periods.

Immutability

One important thing to note is the "immutability" feature in auditd. As mentioned, auditd makes its configuration and logfiles accessible only to the root user for very good reasons. Imagine if an attacker got into a system and wanted to affect an application (e.g., sshd) to hide a second SSH port for back door logins. If they knew auditd was dutifully logging all their actions, they would clearly want to disable some, if not all, of the SSH ruleset to hide their actions.

Fear not. Simply append this tiny additional config parameter to the very end of your rules.d/audit.rules file:

-e 2

The result of adding this line to your config is that the only way to alter the auditd rules on a running system after it has been booted up with that setting in place is to rewrite the rules in the config file (i.e., edit, then rerun augenrules, and perform another reboot). As a result, of course, you need to pay much more attention to rebooting systems and treat random reboots as suspicious activity, which you should be doing anyway.

One gotcha is being mindful of the fact that the system is using the immutability flag in its configuration.

You can tell if immutability is enabled by running the --load option with augenrules:

$ augenrules --load
/sbin/augenrules: No change

In this case, the audit system is in immutable mode, so no rule changes are allowed.

If You Strike Me Down Now

A plethora of auditd options are available, and I've only looked at catching one binary so far; however, the sheer number of other rules (watch rules, control rules, and syscall rules) are a little mind-blowing. Ultimately, this is what makes auditd so powerful: its ability to capture anything and everything going on within your systems. The following are samples from the bottom of the auditctl man page (note the -a for the syscall rules)

• To watch a file for changes (two ways):

auditctl -w /etc/shadow -p wa
auditctl -a always,exit -F path=/etc/shadow -F perm=wa

• To watch a directory recursively for changes (two ways):

auditctl -w /etc/ -p wa
auditctl -a always,exit -F dir=/etc/ -F perm=wa

• To see if an admin is accessing other users' files:

auditctl -a always,exit -F dir=/home/ -F uid=0 -C auid!=obj_uid

I'd suggest using your favorite online hunter-gatherer engine for more information and example rules.

If you're interested in threat modeling, then the powerful auditd also provides a tool called autrace, which you can point at specific binaries and glean a whole host (pun intended) of useful logging data. A simple example command is:

$ autrace /bin/ls

Again, the manual offers much more detail, so look there if you're interested.

This Is the End

As you can tell, I have barely scratched the surface of the venerable auditd package. You can switch on user and group changes (e.g., the creation of new users or their group membership), and you can catch filesystem access from a particular application, yet ignore other events entirely.

With some forethought, a pinch of trial and error, and a teaspoon of patience, you can help mitigate the immediate confusion of how an attacker has breached a system if such an incident ever occurs. If you have set up the package correctly and monitored the affected system events, then auditd will be a true lifesaver in such a scenario: I expect my containers to benefit dramatically as a result.