Security Volatility Lead image: Lead Image © Bruce Rolff, 123RF.com
Lead Image © Bruce Rolff, 123RF.com
 

Forensic main memory analysis with Volatility

Fingerprints

When you examine the memory of a computer after a break-in, take advantage of active support from the Volatility framework to analyze important memory structures and read the volatile traces of an attack. By Prof. Dr. Tobias Eggendorfer

When you think of IT forensics, you usually have the analysis of non-volatile data carriers, such as hard disks or SSDs, in mind. But volatile RAM is also worth a look. It usually contains important traces (e.g., of processes or network connections) and thus provides indications of a successful attack.

The detective work is preceded by the task of creating a RAM image. This memory dump must then be analyzed. With Linux onboard tools, users already can find out and learn a lot, but it's also quite a time-consuming process. Luckily, you can find support in Volatility [1], a framework written in Python that identifies the most important memory structures of an operating system and presents the content in a human-readable form. Its big advantage is the many plugins that support a wide variety of analysis activities (Table 1).

Tabelle 1: Volatility Enhancements

Plugin

Function

Processes

linux_apihooks

Checks for userland API hooks

linux_bash

Extracts the Bash history from the process memory

linux_check_creds

Checks whether processes share credential structures

linux_dump_map

Writes selected memory mappings to a disk

linux_dynamic_env

Fetches dynamic environment variables of a process

linux_getcwd

Shows the current directory for each process

linux_kernel_opened_files

Files opened by kernel

linux_library_list

Libraries loaded by a process

linux_librarydump

Copies the shared libraries of a process to disk

linux_list_raw

Lists processes with suspicious sockets

linux_malfind

Looks for suspicious process mappings

linux_memmap

Shows the memory map of Linux tasks

linux_plthook

Scans Procedure Linkage Table (PLT) of ELF binaries for unnecessary hooks

linux_proc_maps

Prints memory maps of a process

linux_procdump

Writes the executable of a process to disk

linux_process_hollow

Checks for signs of process hollowing

linux_psaux

Shows processes with start time and command-line arguments

linux_psenv

Shows processes with their static environment variables

linux_psscan

Scans the RAM of processes

Plugin

Function

Kernel and Scheduling

linux_check_idt

Checks whether IDT has changed

linux_check_inline_kernel

Searches for inline kernelhooks

linux_check_modules

Compares the module list with the sysfs specifications

linux_check_syscall

Checks whether the system call table has changed

linux_check_tty

Searches for hooks on TTY devices

linux_hidden_modules

Browses RAM for hidden kernel modules

linux_info_regs

As with GDB info registers, prints out contents of process registers

linux_lsmod

Prints loaded kernel modules

linux_moddump

Extracts loaded kernel modules

linux_pslist

Prints active tasks from the task_struct->tasks list

linux_pslist_cache

Shows processes from kmem_cache

linux_pstree

Shows parent-child relationships between processes

linux_psxview

Finds hidden processes

linux_threads

Shows the threads of a process

imagecopy

Copies a physical address area into an image

Files and Filesystems

linux_check_fop

Finds file_operations structures modified by rootkits

linux_dentry_cache

Collects files from the dentry cache

linux_enumerate_files

Lists file references from the filesystem cache

linux_elfs

Finds ELF binaries in process mappings

linux_find_file

Lists files and restores them from RAM

linux_lsof

Lists file descriptors

linux_mount

Prints info on mounted filesystems or devices

linux_mount_cache

Lists info on mounted filesystems from kmem_cache

linux_recover_filesystem

Recovers RAM-cached filesystem

linux_tmpfs

Restores tmpfs filesystems

linux_truecrypt_passphrase

Recovers cached TrueCrypt passphrases

mbrparser

Scans for master boot records

Network

linux_arp

Shows the Address Resolution Protocol (ARP) table

linux_check_afinfo

Checks operation function pointers of network protocols

linux_ifconfig

Prints active interface info

linux_netfilter

Lists Netfilter hooks

linux_netscan

Examines network structures

linux_netstat

Lists open sockets

linux_route_cache

Recovers routing cache

linux_sk_buff_cache

Recovers packets from the sk_buff of the kmem_cache structure

Imaged

The first step is to create a memory image. If the potentially compromised system is running on a virtual machine, the Virtual Machine Manager can create a snapshot, which appropriate tools then take apart [2] [3]. A bare metal Linux presents forensic experts with greater challenges: Every change to the system also changes the memory content, which could destroy valuable clues.

The approach of cooling the memory and reading it out in a special device [4] is not practicable outside a laboratory environment. If the computer has a FireWire connection, a security hole could be exploited, and the memory could be copied over the bus [5]. This has also been described for Thunderbolt and PCIe, but doesn't really work under Linux as of yet [6]-[8].

The only path that remains, then, is to accept that a part of the memory changes while copying. With older Linux versions, this was quite simple, because dd could copy /dev/mem locally to USB sticks, as could netcat over the network. Newer distributions, however, limit read permissions considerably, which is a good idea from a security point of view. Linux Memory Grabber [9] can help you in these cases.

Tapped

The lmg script assumes that a Linux machine is available that can compile a kernel module suitable for the target system; this requires detailed knowledge of the target and a root account. In many cases, forensic experts have to give up here, unless their own system is affected. The instructions on GitHub [9] will help you create a memory image. Although the installation of the kernel module changes the memory content of the target system, and lmg also uses binaries on the target system, which is not entirely without risk on a cracked system, it is probably the only viable solution in most cases.

If you want to be particularly careful about modified binaries, you could bring them along on a USB stick, accept another memory change before installation, and change the path so that the programs from the stick are used. First indications as to whether such a measure is necessary are provided by a host-based intrusion detection system [10]. Linux currently has no particularly secure solutions, such as those proposed in a paper presented at The Digital Forensic Research Workshop in 2013 [11]. Although annoying in forensic investigations, because attackers could also exploit such backdoors, it is definitely a security gain in everyday life.

Forensics

The admin, who now owns a memory image of the possibly compromised computer, can now turn to the analysis. If you want to install Volatility from source, you need numerous Python modules. Alternatively, you can find binaries for popular operating systems on the project page [1]. Simply unzip the ZIP file, and the tool is ready to use. If you find the long name unwieldy, you can create a symlink:

ln -s volatility_2.6_lin64_standalone vol

Also, you should create a profiles subdirectory to give you space for the profiles of the systems to be analyzed. To evaluate a memory image, Volatility requires information about the memory layout. A profile must therefore always precisely match the kernel version. The Volatility wiki [12] links to ready-made Linux profiles and shows how you can use dwarfdump to generate the necessary information about kernel data structures and debug symbols to create your own profiles.

All Set Up

A prebuilt memory image from a book by the Volatility authors [13] was used for all the examples in this article. After downloading and unpacking the ZIP file, you will find a subdirectory named linux with the book.zip archive that needs to be moved to the previously created profiles folder. Whether you use mv, cp, or a hard link is a matter of taste; a symlink did not work on my lab computer. The command

./vol --plugin=profiles/ --info | grep -i book

should return the output:

Volatility Foundation Volatility Framework 2.6
Linuxbookx64 - A Profile for Linux book x64

If Volatility does not find the profile, the plugin directory may be incorrect. To search several folders, separate their names with colons.

Lazy users can also create a Volatility configuration file, which saves having to type long parameters and lets you reconstruct the settings later. The tool expects its setup either in ~/.volatilityrc or in the current directory in the volatilityrc file. Listing 1 shows an example that defines the plugin directory, a profile, and an image. Alternatively, you can specify the image with the -f switch:

Listing 1: volatilityrc

[DEFAULT]
PLUGIN=profiles/
LOCATION=file://AMF_MemorySamples/linux/linux-sample-1.bin
PROFILE=Linuxbookx64
./vol --plugin=profiles/ --profile=Linuxbookx64 -f AMF_MemorySamples/linux/linux-sample-1.bin

The order of the parameters is important: If --profile precedes --plugin, the tool will not find the profile. If you specify a path to the profiles in the configuration file, you also need to assign a name for the profile, because command-line options take priority over the setup file.

A Question of Trust

A first look into the memory image with the ./vol linux_pslist command returns a ps-style list. The tool also supports linux_psaux, linux_psenv, and linux_pstree (Figure 1). The lists come from a data structure (defined in sched.h) in which the kernel manages processes.

The output generated by the Volatility linux_pstree command is similar to the output generated by the Linux pstree command.
Figure 1: The output generated by the Volatility linux_pstree command is similar to the output generated by the Linux pstree command.

Kernel processes are easily recognized because they have no address translation entry in the Directory Table Base (DTB); however, is this output trustworthy? Can an attacker manipulate it? One simple approach to spoofing can be found in the base.c file in the sources of older kernel versions:

res = access_process_vm(task, mm->arg_start, buffer, len, 0);

For each entry in the process list, the kernel reads the shell environment variables. A process that manipulates its environment could therefore appear under a different name.

Because forensic experts must always assume that attackers know the targets at least as well as they do, it is essential to verify the retrieved process list. Volatility offers the linux_psscan command, which does not search the kernel data structures for processes, but the whole memory, where an attacker cannot conceal their actions as easily. Manual adjustment of linux_pslist and linux_psscan is not necessary, because linux_psxview provides a plain text table showing where processes exist.

If you run the commands against the sample image, you will notice that linux_psscan displays numerous processes that linux_pslist does not find. These could be threads that the volatility plugin linux_threads can identify, but an immediately plausible explanation is not always obvious. The swapper process, for example, is missing in both lists, which is certainly allowed as part of the scheduler – it manages idle.

Tracking Rootkits

Sometimes it makes sense to create an overview first. Admins who are wondering if a machine is infected by a rootkit [14] can also turn to Volatility. The tool has several plugins that can provide information, including the command

./vol --plugin=profiles/ linux_check_idt

which returns the interrupt descriptor table (Figure 2). If a redirected (hooked) entry is present, the evidence of a manipulated system is strong. The same applies to the exception handlers and the system calls. Just like lsof, linux_lsof looks for the open files of a process; linux_chk_tty and linux_keyboard_notifier check whether an attacker has installed a typical keymapper. Volatility also determines the ARP table, open network connections, mounted filesystems, or kernel modules in use – you can find plugins for almost everything (a complete overview is available in the wiki on GitHub [12]).

Everything seems to be fine: The interrupt service routines are not redirected.
Figure 2: Everything seems to be fine: The interrupt service routines are not redirected.

Why should an admin resort to Volatility when looking for rootkits and not a special malware scanner that analyzes the disk? A rootkit in memory will not be affected by the check, which increases the chances of discovering the attacker. If a suspicious process is identified, linux_proc_maps returns the memory content for better analysis.

Interpretation Needed

Analyzing the Bash history (Figure 3) is useful for detecting user misbehavior. Volatility also detects commands if the length of the history has been changed to zero and its location to /dev/null to hide the last entries.

Clearly visible: Here the forensic scientist created the memory dump with lime.
Figure 3: Clearly visible: Here the forensic scientist created the memory dump with lime.

The real challenge in using Volatility, as with all analysis tools, is not so much using the correct parameters, but interpreting the program's output correctly. Only practice and a good knowledge of the system with all its data structures will be useful.