Reconnaissance

Quite a lot has been written about pen testing and hacker lifecycles. Over the past few months, for example, I've written a couple of articles for ADMIN about penetration testing: one about automated tools for pen testing [1] and the other about improving defense through pen testing [2]. However, comparatively little has been written about the knowledge, techniques, and tools necessary to analyze an attack or pen test as it occurs (i.e., the "other side," as it were, of an attack).

Indicators of Attack and Compromise

Before I stampede into the tools an analyst uses, it's important to identify an essential principle of the security analyst: As an attack occurs, certain things are left behind. This concept was first articulated by Edmond Locard [3] almost 100 years ago, well before the first modern computers were created. In fact, the concept that attackers leave behind signatures and traces is named "Locard's Exchange Principle."

The defender – in this case, the security analyst – needs to figure out what indicators of attack (IoAs) and indicators of compromise (IoCs) were left behind. An IoA is evidence left behind even if a particular attack doesn't lead to a break-in or data breach. An IoC is evidence left behind if an attack has successfully tricked or breached a security control. For example, an IoA could be a system scan or an unsuccessful attempt to create or exploit a buffer overflow condition. An IoC could be a case in which an attacker was able to exploit a buffer overflow successfully or otherwise gain unauthorized access to a system – same activity, two perspectives, and two sets of tools.

According to the Exchange Principle, Table 1 gives a quick overview of the tools that pen testers and security analysts use. This "quick and dirty" table is nowhere near perfect or thorough, but it's sometimes useful to think about the same activity – hacking – from two different perspectives. This model isn't perfect, because, after all, a pen tester can use Wireshark just as effectively as a security analyst. The pen tester, though, simply would use it in a different way.

Red Team/Blue Team Exercises

In my experience, pen testers are often called "red team" workers and security analysts "blue team" workers, often because pen testers and security analysts work together in teams during security exercises or war games.

Sometimes, these exercises are conducted to upskill a team of security workers. However, I've found that the most proactive, forward-thinking companies use these exercises to help preprogram and reprogram automated security tools, because the best reason to have pen testers isn't really to play "whack a mole" to try and find hackers or weak systems: The real reason to have pen testers is to improve the quality of the security analysts.

Several companies I know regularly conduct exercises that generate various IoAs and IoCs so that the security analysts can identify them in real time. In this way, security analysts can help fine tune their intrusion detection system (IDS) and security information event management (SIEM) tools.

Isn't There an App for That?

Over the last five years of traveling around the world talking with IT techs and cybersecurity workers, I've asked them about "killer cyber analytics tools." Invariably, they talk about monitoring logfiles and system processes and reviewing packet captures. At first, this kind of surprised me, because I thought that they would tell me about nifty, cool SIEM tools and host- and network-based IDS tools, such as Snort, Suricata, and Bro. Pretty consistently, the security pros I've talked to tell me that, although these tools are, of course, very useful and essential, you can't just "plug and pray" these systems into your network or network hosts. You have to pre-program them with information (i.e., customize them).

The best way to get this customization info is to break out the tried and true tools, so I'll take a quick look at these tools and then look through an applied example of what a security analyst does with them. (See also the "Steps in the Hacker Lifecycle" box.)

Process Monitor

Following Locard's Exchange Principle that attackers always leave something behind, it's only logical that when investigating a Windows system, you look for system processes that have been spawned. For that, I've always found that Process Monitor (Figure 1) is a terrific tool for tracking unknown activity on any Windows system, no matter how new or old. Useful features include the ability to identify and trace sub-processes, as well as filter the many thousands of processes running on any particular Windows host. You can download Process Monitor directly from Microsoft [6].

The following general principle is true: The more powerful the machine, the more powerful the potential shell or process. That's why Process Monitor is so useful. It allows me to investigate processes that have been spawned and tracks those processes right up the execution tree to the kernel. It also allows me to see whether these processes lead to a socket, which is where an IP address has been mapped to a port.

Wireshark: The Analyst's Best Friend

Wireshark is in many ways the best friend of a security analyst. Although I might be overstating this, because system logs and process logs are also vital for any analyst, if you really want to get into what's happening on the network, Wireshark is terrific. Once again, you can't simply expect an IDS such as Snort, Bro, SolarWinds, or Suricata to do this work for you. Wireshark is freely available online [7] and is preinstalled on many Linux operating systems.

Analyzing an Attack

A month or so ago, I was working with a security analyst who monitors Fortune 500 banks, as well as large oil companies. He showed me how many of these organizations still use older Windows and Linux systems to provide essential banking services and control physical systems, such as oil pipelines. I've had other security analysts confirm that many healthcare, manufacturing, and retail point of sale systems use older Windows hosts, so I thought I'd demonstrate how a security analyst identifies IoCs on these systems.

The Security Analyst's Perspective

What does this attack look like from the perspective of the defender? A security analyst generally doesn't use Metasploit or custom code to conduct an attack. A defender uses different tools to "listen," as it were, for evidence of the attack.

First, I'll start with what the reconnaissance/discovery process looks like from the perspective of both the attacker and security analyst with the use of a simple two-system example:

• Attacker: 172.16.190.158 (system1)
• Victim: 172.16.190.159 (system2)

Figure 2 shows an attacker using Nmap to conduct a quick discovery with the command:

system1$ sudo nmap -sV -O -A 172.16.190.159 -p 445

This command tells Nmap to do a verbose discovery of a host (-sV), and this scan will (one hopes) reveal the type of operating system (-O) and the specific service it is running (-A) at port 445 (-p). The result is shown in Figure 2.

This scan has revealed quite a bit of information, including:

• The operating system version (Windows 7, Service Pack 1)
• The computer name/NetBIOS name (WIN-NJKFH7RKUAQ)
• The computer's workgroup
• That it is running Microsoft Networking/Server Message Blocks (SMB)

From a security analyst's perspective, Figure 3 shows what the scan looks like through Wireshark. I've marked the scanning packets black and red and filtered all network traffic so that I am only viewing TCP-based traffic between the attacker and victim systems mentioned earlier.

Although this traffic is encrypted in this particular example, you can trace it. To get to the TCP trace function in Wireshark, first highlight a TCP packet, and then right-click on the packet. A pop-up window will appear. When it does, select Follow | TCP Stream. An example of an encrypted TCP trace is shown in Figure 4. You can also trace UDP packets in a similar fashion. Even though the traffic shown here is mostly encrypted, notice that certain Microsoft Networking header information is still easily readable.

Security analysts often use information like this to preprogram IDS and SIEM systems, which, believe it or not, don't often discover these details automatically. Figure 5 shows the Windows NetBIOS Name Service (NBNS) traffic that was generated by the Nmap scan.

Now that you've seen what a quick scan looks like, it's possible to (1) program IDS and SIEM systems with this information and (2) inform security operations workers about this traffic and its appearance.

From the discovery/reconnaissance phase, I'll consider what an actual attack looks like from both perspectives. Figure 6 shows an attack from a publicly available EternalBlue exploit, which is one of the many tools that the US National Security Agency developed to break into systems. Tools such as these were released by The Shadow Brokers hacking group back in 2017; many of these tools are still in use and find successful victims to this day.

Notice that system1 (172.16.190.158) first configures some shellcode that contains a hard-coded IP address, which is the address of the system being attacked (system2, with IP address 172.16.190.159). In the last line, I execute the command to attack system2:

./eternalblue exploit7.py 172.16.190.159 ./shellcode/sc x

From the security analyst (blue team) worker's perspective, Figure 7 shows what the result of the attack looks like.

Yep, that's a Windows 7 "blue screen of death." The result of the attack with the shellcode I downloaded from the Internet resulted in nothing more than a particularly obvious denial of service attack.

I'm being a bit of a smart aleck, here, but I do have a very serious point: In many cases, an attacker uses various tools that result in a system crash, rather than a system compromise. The security analysts I have talked with over the years are always on the lookout for system reboots and blue screens, because they could be indicators of an attack. In fact, I've seen evidence before of hackers who use tools like this simply to conduct denial of service attacks.

In this case, however, the mission is to do more than just crash a system, so I've simply switched tactics and used Metasploit's Meterpreter to conduct an attack (Figure 8). In this attack, I've set the remote host to the victim system (system1) and used a really obvious port (4444).

A good hacker would usually specify a much more "stealthy" port, which would be one commonly used on the network (e.g., TCP port 445, port 80, or port 8080). A good IDS would likely get pretty curious about why port 4444 is being used when it has never been seen before in this way. Then again, a good IDS would also possibly alert about SMB-based traffic being used on a web port (e.g., 80, 8080). As you can see, a good hacker needs to use good judgment, too.

Figure 9 shows what the very beginning of this attack looks like in Wireshark. Notice the SMB-based communication and how the traffic starts at port 445 from system1 (IP address 172.16.190.158, marked with the red circle). Then, you can see how system2 (172.16.190.159) begins the SMB protocol version negotiation and how NT LAN Manager 1.0 and 1.2 are being negotiated.

This observation is important, because the EternalBlue attack takes advantage of Microsoft Windows systems that mishandle specially crafted packets. When the affected Microsoft systems mishandle requests, a shell is left behind. Attackers can then use this shell to execute remote code and administer the system remotely without proper authentication credentials. In other words, you are seeing what the beginning of an EternalBlue attack looks like from the perspective of a security analyst.

Figure 10 shows the point where the EternalBlue code from Meterpreter has been able to open the reverse shell on port 4444 and begin creating a login shell for me to execute arbitrary code. If I try to trace the traffic stemming from system1 to system2, it is fully encrypted, much like the Microsoft SMB traffic I profiled earlier in Figure 4.

This is where my trusty copy of Wireshark really can't do much more good for me as a security analyst. Still, it is often the case that other exploits don't have a fully encrypted data stream. If that's the case, you can use information gathered in a Wireshark TCP or UDP trace to better program an IDS application or SIEM implementation.

On the Windows System

So far, I've focused on network-based traffic. Figure 11 shows the types of processes that the Metasploit EternalBlue attack spawns on the Windows system. Although it's awfully difficult to differentiate this process from any other legitimate instance of PowerShell, if you right-click on this instance, you can further trace what it is doing, as shown in Figure 12. Notice that this particular instance has opened a suspicious socket – in this case, one that goes back to the attack on system1 using port 4444.

Successful attackers are really never happy with simply gaining access to a remote system. A compromised, admin-level shell on a remote system is a precious but fleeting thing. In other words, all it would take is for a user to reboot the Windows system or for an antivirus update to eliminate the hacker's hard work and shut down port 4444, or whatever port the attacker is using.

Therefore, a hacker will always try to establish persistence, which is a hacker trying to create a more legitimate way back into the system that is more difficult to discover. That way, the hacker can take their time and return to the system using a more legitimate method. That makes the hacker's activity all the more difficult to discover.

Credential Harvesting

One of the more typical attack methods is to glean usernames and passwords stored in the registry. This activity is an example of "credential harvesting." Two common ways to harvest usernames and passwords in Windows systems are with the use of:

• Windows Credential Editor (WCE), which allows you to "dump" a credential, as well as change a user password. Installed by default in Metasploit, you can upload it and activate it during a Meterpreter session.
• The Meterpreter lsa_dump_secrets module, also installed by default, which allows you to dump password hashes and, if you are lucky, cracks the passwords during the process.

You can use either or both of these tools. In the exploit shown in Figure 13, the attacker has been able to use Metasploit's WCE to download passwords from system2, the victim system. The highlighted code is a Windows LAN Manager hash that an exploiter can then decode using an application such as John the Ripper, or any of the dozens of online hash-cracking tools (e.g., CrackStation [8] or GPUHash.me [9]).

An attacker could even download the hashes of previously used passwords that are stored in the password history portion of the registry. Windows systems often store the hashes of the system's password history to enforce password reuse policies so that end users don't simply reuse old passwords when it comes time for them to change. You can do this with Meterpreter's lsa_dump_secrets module, as shown in Figure 14.

In this case, the lsa_dump_secrets command dumps various password hashes. I also got lucky – Metasploit's Local Security Authority (LSA) feature was able to decrypt the default admin password and even give the old password for that account.

All of these activities are traceable from a security administrator's perspective. Figure 15 shows how each of the above activities spawned a new command-line shell on the Windows system. Now you know how to do more than just create a credential-harvesting exploit; you know what one looks like, as well.

Hackers often activate legitimate services and ports in their quest for persistence. For example, many Windows systems administrators still use Microsoft Remote Desktop Protocol (RDP), so if an attacker still has access from the EternalBlue exploit, why not enable RDP by making a simple little change in the registry (Figure 16)? In the lengthy command, the attacker knew exactly which registry entry to enter: It is easily found online.

Figure 17 shows what a security analyst will see if they are auditing this process manually with Process Explorer. You can see how the reg.exe process attempted to edit the registry. At first, access is denied, but then, the highlighted entry shows that the change is granted. With this change, I, as the attacker, now have the ability to use RDP. I can simply use any RDP client (e.g., rdesktop) to access the victim system with the use of the legitimate username and password that I had already harvested:

rdesktop -u user 172.16.190.159

Now, even if someone discovers my EternalBlue exploit and shuts down port 4444, I can still log back in at any time via RDP (Figure 18), or I can walk up to the system and log on locally. I have therefore just established persistence on my system.

Summary

Now you have a good idea of what an exploit looks from both the attacker's and the defender's perspective. I value the defender's perspective the most; the blue team worker is the one, at least in my mind, who accomplishes that essential step of customizing various security controls, such as IDS and SIEM systems.

Of course, you can automate the process of discovering and even responding to these types of changes. Well-known exploits are already preprogrammed in host- and network-based IDS solutions, but a good security analyst knows that hackers are always changing their tactics and techniques. Therefore, an IT administrator or security worker must know how to investigate and review these types of attacks with the use of non-automated tools.

Once you know how to view an attack from both perspectives, you can take this relatively simple example and trace more activities related to the hacker process, and you can discover more IoCs and use this information to make your IDS and SIEM tools work more efficiently.