Fingerprints

When you think of IT forensics, you usually have the analysis of non-volatile data carriers, such as hard disks or SSDs, in mind. But volatile RAM is also worth a look. It usually contains important traces (e.g., of processes or network connections) and thus provides indications of a successful attack.

The detective work is preceded by the task of creating a RAM image. This memory dump must then be analyzed. With Linux onboard tools, users already can find out and learn a lot, but it's also quite a time-consuming process. Luckily, you can find support in Volatility [1], a framework written in Python that identifies the most important memory structures of an operating system and presents the content in a human-readable form. Its big advantage is the many plugins that support a wide variety of analysis activities (Table 1).

Imaged

The first step is to create a memory image. If the potentially compromised system is running on a virtual machine, the Virtual Machine Manager can create a snapshot, which appropriate tools then take apart [2] [3]. A bare metal Linux presents forensic experts with greater challenges: Every change to the system also changes the memory content, which could destroy valuable clues.

The approach of cooling the memory and reading it out in a special device [4] is not practicable outside a laboratory environment. If the computer has a FireWire connection, a security hole could be exploited, and the memory could be copied over the bus [5]. This has also been described for Thunderbolt and PCIe, but doesn't really work under Linux as of yet [6]-[8].

The only path that remains, then, is to accept that a part of the memory changes while copying. With older Linux versions, this was quite simple, because dd could copy /dev/mem locally to USB sticks, as could netcat over the network. Newer distributions, however, limit read permissions considerably, which is a good idea from a security point of view. Linux Memory Grabber [9] can help you in these cases.

Tapped

The lmg script assumes that a Linux machine is available that can compile a kernel module suitable for the target system; this requires detailed knowledge of the target and a root account. In many cases, forensic experts have to give up here, unless their own system is affected. The instructions on GitHub [9] will help you create a memory image. Although the installation of the kernel module changes the memory content of the target system, and lmg also uses binaries on the target system, which is not entirely without risk on a cracked system, it is probably the only viable solution in most cases.

If you want to be particularly careful about modified binaries, you could bring them along on a USB stick, accept another memory change before installation, and change the path so that the programs from the stick are used. First indications as to whether such a measure is necessary are provided by a host-based intrusion detection system [10]. Linux currently has no particularly secure solutions, such as those proposed in a paper presented at The Digital Forensic Research Workshop in 2013 [11]. Although annoying in forensic investigations, because attackers could also exploit such backdoors, it is definitely a security gain in everyday life.

Forensics

The admin, who now owns a memory image of the possibly compromised computer, can now turn to the analysis. If you want to install Volatility from source, you need numerous Python modules. Alternatively, you can find binaries for popular operating systems on the project page [1]. Simply unzip the ZIP file, and the tool is ready to use. If you find the long name unwieldy, you can create a symlink:

ln -s volatility_2.6_lin64_standalone vol

Also, you should create a profiles subdirectory to give you space for the profiles of the systems to be analyzed. To evaluate a memory image, Volatility requires information about the memory layout. A profile must therefore always precisely match the kernel version. The Volatility wiki [12] links to ready-made Linux profiles and shows how you can use dwarfdump to generate the necessary information about kernel data structures and debug symbols to create your own profiles.

All Set Up

A prebuilt memory image from a book by the Volatility authors [13] was used for all the examples in this article. After downloading and unpacking the ZIP file, you will find a subdirectory named linux with the book.zip archive that needs to be moved to the previously created profiles folder. Whether you use mv, cp, or a hard link is a matter of taste; a symlink did not work on my lab computer. The command

./vol --plugin=profiles/ --info | grep -i book

should return the output:

Volatility Foundation Volatility Framework 2.6
Linuxbookx64 - A Profile for Linux book x64

If Volatility does not find the profile, the plugin directory may be incorrect. To search several folders, separate their names with colons.

Lazy users can also create a Volatility configuration file, which saves having to type long parameters and lets you reconstruct the settings later. The tool expects its setup either in ~/.volatilityrc or in the current directory in the volatilityrc file. Listing 1 shows an example that defines the plugin directory, a profile, and an image. Alternatively, you can specify the image with the -f switch:

[DEFAULT]
PLUGIN=profiles/
LOCATION=file://AMF_MemorySamples/linux/linux-sample-1.bin
PROFILE=Linuxbookx64

./vol --plugin=profiles/ --profile=Linuxbookx64 -f AMF_MemorySamples/linux/linux-sample-1.bin

The order of the parameters is important: If --profile precedes --plugin, the tool will not find the profile. If you specify a path to the profiles in the configuration file, you also need to assign a name for the profile, because command-line options take priority over the setup file.

A Question of Trust

A first look into the memory image with the ./vol linux_pslist command returns a ps-style list. The tool also supports linux_psaux, linux_psenv, and linux_pstree (Figure 1). The lists come from a data structure (defined in sched.h) in which the kernel manages processes.

Kernel processes are easily recognized because they have no address translation entry in the Directory Table Base (DTB); however, is this output trustworthy? Can an attacker manipulate it? One simple approach to spoofing can be found in the base.c file in the sources of older kernel versions:

res = access_process_vm(task, mm->arg_start, buffer, len, 0);

For each entry in the process list, the kernel reads the shell environment variables. A process that manipulates its environment could therefore appear under a different name.

Because forensic experts must always assume that attackers know the targets at least as well as they do, it is essential to verify the retrieved process list. Volatility offers the linux_psscan command, which does not search the kernel data structures for processes, but the whole memory, where an attacker cannot conceal their actions as easily. Manual adjustment of linux_pslist and linux_psscan is not necessary, because linux_psxview provides a plain text table showing where processes exist.

If you run the commands against the sample image, you will notice that linux_psscan displays numerous processes that linux_pslist does not find. These could be threads that the volatility plugin linux_threads can identify, but an immediately plausible explanation is not always obvious. The swapper process, for example, is missing in both lists, which is certainly allowed as part of the scheduler – it manages idle.

Tracking Rootkits

Sometimes it makes sense to create an overview first. Admins who are wondering if a machine is infected by a rootkit [14] can also turn to Volatility. The tool has several plugins that can provide information, including the command

./vol --plugin=profiles/ linux_check_idt

which returns the interrupt descriptor table (Figure 2). If a redirected (hooked) entry is present, the evidence of a manipulated system is strong. The same applies to the exception handlers and the system calls. Just like lsof, linux_lsof looks for the open files of a process; linux_chk_tty and linux_keyboard_notifier check whether an attacker has installed a typical keymapper. Volatility also determines the ARP table, open network connections, mounted filesystems, or kernel modules in use – you can find plugins for almost everything (a complete overview is available in the wiki on GitHub [12]).

Why should an admin resort to Volatility when looking for rootkits and not a special malware scanner that analyzes the disk? A rootkit in memory will not be affected by the check, which increases the chances of discovering the attacker. If a suspicious process is identified, linux_proc_maps returns the memory content for better analysis.

Interpretation Needed

Analyzing the Bash history (Figure 3) is useful for detecting user misbehavior. Volatility also detects commands if the length of the history has been changed to zero and its location to /dev/null to hide the last entries.

The real challenge in using Volatility, as with all analysis tools, is not so much using the correct parameters, but interpreting the program's output correctly. Only practice and a good knowledge of the system with all its data structures will be useful.