Private Eye

The activity of penetration testing has been around for decades, but the past five years or so have seen radical changes in networking, beginning with the dissolved perimeter. Remember the good days of threats "inside the firewall" and "outside the firewall?" Those days are long gone, because of the advent of the cloud and the use of mobile devices.

A second problem is radically changed endpoints. From mobile phones and new peripherals, to the cloud, to the Internet of Things (IoT), the current environment is a long way from the typical end-user PC. Today's endpoint can be the cloud, a wearable device, or a traditional PC, notebook, mobile phone, or tablet.

A third threat is social engineering. The con man has been around for thousands of years in one form or another, but the majority of successful attacks start with clever pretexting. Good pen testers know this and use it to their advantage.

Finally, you must consider social media. Hackers have focused on end users. Why? One reason is because so many are revealing nearly everything about themselves to the world.

Considering all of the changes mentioned above, it is all the more important to consider automating certain steps of the penetration test, and it is especially important to obtain accurate information about the organization you are testing. Still, the step of gathering information for a penetration test can be quite time-consuming, so given all of the changes you need to address, what can you do to speed up that process? Also, what is the relationship between the first part of the penetration test, and all of the other parts?

To begin, I talk about the hacker lifecycle and its relevance to the penetration test.

Typical Hacker Lifecycle

In 2011, researchers from Lockheed Martin created their version of the hacker lifecycle called the Cyber Kill Chain. Figure 1 shows each of the steps.

Each "kill chain" step describes typical activities an attacker takes when infiltrating a network. The idea behind calling it a kill chain is that it helps defenders think about how to thwart each step. You can learn more about the Lockheed Martin model on their website [1].

Reconnaissance involves gathering information, including vulnerabilities. Weaponization is where the attacker obtains or creates malware. The Delivery and Exploitation steps include where the attacker is able to place and activate malware and use it to exploit network resources. The Command & Control stage involves the ability to usurp control over networked systems, including endpoints, servers, and connectivity devices. In the Actions on Objectives stage, the attacker begins stealing and modifying data and taking additional steps, such as covering up evidence of intrusion.

Since its introduction, the Lockheed Martin Cyber Kill Chain model has received quite a few critiques. Some focus on how the model is based on the old perimeter-focused model, which has largely dissolved. The model was clearly created before the days of the IoT, and even before the mobile/bring your own device (BYOD) revolution. As I noted above, the environment has changed greatly.

Customized Lifecycles and Frameworks

When it comes to penetration testing, I maintain it is important to create a lifecycle that makes sense for the environment you are investigating. You might want to emphasize certain steps or break out a certain step into multiple steps. In Figure 2, for example, I present a scheme I created for a few networks I have audited over the years.

In the above model, the Discovery stage involves reconnaissance. In Discovery, attackers do more than simply identify network hosts; they also identify and begin profiling individual users susceptible to social engineering and start identifying the likely steps for exploitation. As you will see later in this article, I spend some time discussing how to automate the process of gathering information during this stage.

The Penetration/Foothold step involves substeps, such as introducing malware, hijacking a session, or starting a reverse shell session. It is basically the initial compromise in which one gains unauthorized access. From here, the attacker generally escalates privileges, then ends the process, and gets out. This happens with the typical "smash and grab" hacker. They just grab something (e.g., an account and some information) and then leave.

Many hackers, however, tend to use this initial compromise as a way to learn more about the network and then lurk. These attackers are often part of an advanced persistent threat (APT), in which attackers lurk in systems for a long period of time – months or even years – before moving on to penetrate additional systems. They also engage in modifying networked systems (e.g., endpoints, routers, firewalls) to facilitate remaining in the network over a long period of time. This is the activity I call Persistence.

During either the Penetration or Persistence phase, attackers will conduct various activities, including:

• Data egress: stealing data for later use. Data can include user account information, databases, or sensitive intellectual property. It can also include personally identifiable information (PII).
• Data manipulation: modifying sensitive data. Attackers today often would rather alter data to their advantage over a long period of time than simply steal it. After all, why simply grab a quick data snapshot when you can manipulate information in your favor?
• Malware installation. Ransomware is quite common, but it can also include keylogging software, viruses, and other payloads.
• Sandbox escape. Lately, I have seen networks use mandatory access control (MAC) and "sandboxed" environments that do not allow software to be launched as in a traditional Windows, Linux, or Mac environment. You have to choose from menus or from choices on a web form; yet, in these cases, attackers have simply gone after code flaws in the web browsers and stored session cookies to escape these environments.

Usually, attackers will try to cover their tracks throughout the entire hacker lifecycle. Covering tracks involves modifying or removing logs, as well as behaving in ways that fall below usual intrusion detection thresholds. Pen testers, because they are working under terms of a contract and a Statement of Work (SOW), do more than cover tracks. They also clean up the systems and generate reports.

To begin, I will look at the Discovery/Reconnaissance stage and see what kinds of automated software the penetration tester can use to conduct a "black box" audit.

Discovery Stage: Nmap and Maltego

Open source intelligence tools (OSINTs) are extremely useful during the discovery stage. Pen testers were once relegated to using whois, nslookup, and good old nmap, along with plenty of other useful tools. However, you have many more automated choices available now.

One of the first additions is the Shodan API. Shodan [2] is a search engine service that provides granular information about Internet-aware devices. A few years ago, the maintainers of Shodan created an API that allows you and your security software to "plug in" to their database. Various tools now have interfaces, including Nmap.

Figure 3 shows a typical Nmap scan using the OSScan option (-0) that you've seen a thousand times. Notice with this particular scan that Nmap is having a hard time figuring out the exact nature of the operating system, although it outputs some details, including Oracle Virtualbox, but no more. Enter Shodan.

Automated Searches: Shodan and Nmap

The Nmap creators have allowed you to use Shodan for some time by specifying the --script option with the shodan-api argument, which allows you to specify your Shodan key:

$ sudo nmap -sn -Pn -n --script shodan-api --script-args shodan-api.apikey= UsYM89GnfM86IP5aodgbTf6voFQL7kae www.foocompany1.com

In a moment, I'll show you how to get your own Shodan key, but first I want to explain the above command. It simply tells Nmap to do a stealth scan using null sessions, not to use ping, and not to do Domain Name System (DNS) resolution. The --script option tells Nmap that you're going to use a script of some sort – in this case, it will be the Shodan API. The Shodan API script argument specifies your API key. The command then finishes by specifying the target URL, http://www.foocompany1.com.

The result is much more accurate information about your target. Specifying the Shodan API also results in having Shodan, not nmap, conduct the actual scan; in this way, you are doing an indirect scan, which is much more stealthy. You are also conducting one that is more accurate and informative about the hosts you're scanning. Note, however, that you can only use the Shodan API this way on publicly-facing IP addresses, and not private IP addresses. By specifying the Shodan API, I was able to gather more information during the Discovery phase (Figure 4), and the resulting scan gives more detailed information about the services on a particular target host. In this case, it reveals port 22 and the specific operating system of the server, Debian 5.

Obtaining the Shodan Key

You can obtain the Shodan key by going to the Shodan website [2] and creating an account. Figure 5 shows the Overview screen with an API key and Quick Response (QR) code. You can use either in Nmap and many other applications. (Also see the box titled "Getting a New API Key.")

Automated Discovery: Maltego

Maltego (Figure 6) is another tool often used during the pen testing reconnaissance step. Maltego automates the process of discovering network resources. By simply entering common information, such as a DNS domain name or a Twitter handle, you can automate discovery of:

• All DNS servers, including primary/master and secondary/slave.
• Email account names. Maltego can discover defined email accounts, thereby quickly giving you the structure of accounts on a server. From here, it is a quick two-step to effective social engineering.
• Twitter information. Specifying a simple Twitter handle lets you identify conversation threads quickly. By parsing these conversations, you can discover an enormous amount of information about a particular user or set of users. In Figure 6, Maltego has returned information about a specific network domain.

Accounts, Machines, and Transforms

On first starting Maltego, you need to create an account, which you will have to log in to for every use (Figure 7). In this case, I am using the Community account, which is free. Many professional pen testers purchase an upgraded account.

Once you've logged in, you can then begin a scan using any number of Machines (Figure 8). A machine is basically a type of scan. Two that I have used are the Company Stalker and Twitter Digger X. With the Company Stalker machine, you simply select it, enter the domain of a company that has authorized you to conduct a scan, and click Run.

Social Media and Maltego

The vast majority of security incidents begins with social engineering. Similarly, authorized pen testers use automated tools to cull information from social media sites, including Twitter. The Twitter Digger X machine, for example, allows you to trace and investigate any Twitter account.

Figure 9 shows the result of my own Twitter conversations. Using information gathered from these conversations, attackers could craft effective social engineering campaigns, because they now know my particular interests. An attacker could begin a profile that helps engage in spear phishing and whaling campaigns, for example.

Notice the NCSL icon at the bottom left of the figure. The castle denotes an authoritative DNS domain. Additional icons show different tweets. The yellow icons indicate key words that can be used to obtain more information about the subject being profiled.

Transforms

In Maltego, a transform is a method used to map a particular network protocol (e.g., a DNS MX record) to a particular user. As with any useful program, you can use predefined transforms or create your own. Figure 10 shows a picture of Maltego's Transform Manager.

If you click on the Transform Servers tab, you can then specify the Shodan API key you generated. Your subsequent scans will contain additional information about the domains and accounts you scan.

Integrating Shodan with Maltego

Automating the Reconnaissance stage is even easier in Maltego if you integrate Shodan into your searches. To do this, get a copy of your Shodan key, then go to the Home page in Maltego. Make sure that the Transform Hub page is open, and then click on the Shodan transform. Make sure it is installed. Then, click on Details | Settings, and then enter your key, as shown in Figure 11. You will then be able to use the Shodan API from within Maltego to conduct more specific searches. You can integrate Shodan with many other pen testing tools, including Metasploit.

Conclusion

Automating each element of the pen test lifecycle is possible, of course. In many cases, the Discovery phase is the most time-consuming part of the pen test, but it is very important to get it right. Once you have created the right foundation for the rest of the penetration test, you will be able to move on with confidence.

Of course, you are not limited to OSINT tools such as Nmap, the Shodan API, and Maltego, but they are very useful tools in many cases, and they do a nice job of showing how the pen test tools of the trade are automating many of the steps involved in the Discovery stage. Your responsibility is to use various techniques to gain a clear picture of the organization you are testing. No single tool – or even collection of tools – can take the place of an experienced pen tester, but if you use the right tools the right way, you can speed up the process and make your work more efficient.