Always On

Commonly, a mixture of open source and expensive proprietary tools are shoehorned into a pipeline to perform tests on nightly as well as ad hoc builds. However, anyone who has used such tests soon realizes that the maturity of a smaller number of time-honored tests is sometimes much more valuable than the extra detail you get by shoehorning too many tests into the pipe then waiting three hours for a nightly build to complete. The maturity of your battle-hardened tests is key.

The tests you require might involve interrogating the quality of code from developers or checking code for licensing issues. A continuous testing strategy can be onerous to set up but brings unparalleled value to your end product, including improvements in uptime, performance, compliance, and security.

To make any of the tests you run within your pipeline useful, you should be able to integrate them with existing tools and fire them following simple event-based hooks or triggers.

Once licensing test errors are safely classed as non-fatal, for example, your code may proceed by passing a "yes" to the next phase. Later, if Ansible or Puppet reports that all changes were executed properly from your playbooks or manifests without generating unwelcome errors, you are ready for the next step. After your code has moved successfully through all the phases of testing, your changes can then be accepted into your production environment.

The popular security tool Zed Attack Proxy (ZAP) [1] is a useful addition to your continuous integration security testing strategy. According to the project website, ZAP can "help you automatically find security vulnerabilities in your web applications while you are developing and testing your applications." My professional curiosity stems from a DevSecOps perspective and, with some tinkering, I have discovered that ZAP will help you avoid a great number of issues in your code.

Laser Beams

ZAP is brought to you by the not-for-profit organization called Open Web Application Security Project, or OWASP [2]. The ZAP security testing framework is regularly visible near the top of security tool review lists, thanks to its accessibility and its feature set.

What first piqued my interest in ZAP was its intuitive interface. Although dissimilar, it reminded me a little of WebGoat's tutorial-based approach [3]. ZAP is unquestionably a fantastic security learning tool, as well as an excellent automated testing tool. I've read before that it's ideal for beginners, but used by professionals. To my mind, that's a mantra many other software products should think of adopting.

Before you go any further, remember to try out anything and everything in a sandbox first before aiming it at any production systems. On that note, here's a genuinely serious point: ZAP is highly functional, so only use it on your own systems, or you might cause criminal damage.

It is safe to say that ZAP has been carefully considered, well devised, and expertly written. A handy Docker image comes with a GUI that you can run remotely over TightVNC [4], which I'll look at in a moment; however, first, I'll whet your appetite with some of ZAP's many features.

What's in the Box?

Worth mentioning is the superb quality of online documentation that ZAP offers, including videos and a teaser pamphlet titled "Zap" Your App's Vulnerabilities [5]. ZAP sits near the top of the stack in the Application Layer (OSI model Layer 7) [6]. Although many fantastic network security tools exist that will probe every nook and cranny of a network stack, ZAP specializes in the application vulnerabilities side, whether that's strictly Layer 7 or below.

Among ZAP's app-centric list of features is its ability to act as an intercepting proxy, which means you point your browser at it and click away merrily while ZAP tells you what's broken. You can then adjust code to remedy issues on the fly.

For automation, ZAP presents a sophisticated API that can be used in daemon mode (which I'll come to later when discussing continuous integration (CI)) and a plugin architecture with modular components that are simple to update. ZAP includes a traditional spider (a crawler that very efficiently checks websites without complex JavaScript) and an AJAX spider that launches a browser to click through interactive parts of a site, taking a little longer than its less intelligent counterpart.

If that list of features isn't enough, there's my favorite, "passive" scanning, which means that ZAP can isolate vulnerabilities without bringing your site to its knees. If you're feeling brave, the much less safe "active" scanning attack mode (which should only ever be used on your own sites) is equally clever, and, yes, it attacks the site with great fury and anger.

With ZAP, you can debug your code and initiate penetration testing (automated and manual). It has wide-ranging support for scripting languages, support for the API, and, if that's not enough, an auto-update option.

A marketplace for ZAP add-ons is easily accessible through a GUI interface [7]. ZAP add-ons come with a quality level, so you can choose whether to trust them implicitly or consider them "nice to have" for the future. Check the maturity of the add-ons by looking for the Alpha, Beta, or Release status.

Zapping from a Docker Container

The path of least resistance to getting started is by installing Docker on a host or virtual machine and running the ZAP Docker image as a container. Even inside complex environments, if you can open up TCP port 5900, you will be able to connect over VNC and get access to ZAP's GUI (firewalls permitting).

To get Docker installed successfully on a Debian derivative (I'm using Ubuntu on a server and Mint on a laptop) use the command

$ apt install docker.io

to install the package. You'll be glad to hear that even a cheap and cheerful AWS or Digital Ocean cloud instance will happily serve ZAP and its GUI on Docker. Some tests might take a bit longer, but certainly you won't be too put out by server specifications at first. However, you might want to up the RAM and CPU specs for CI integration. Sadly, sometimes 512MB of RAM even on Linux boxes doesn't get you very far these days if you want to run a feature-filled application.

Now it's time to drop the theory and move on to the juicy stuff. The lengthy Docker command shown below does a few things automatically and in a relatively sophisticated way:

$ docker run -u zap -p 5900:5900 -p 81:8080 -i owasp/zap2docker-stable x11vnc --forever --usepw --create

First, you can see that, from a security perspective, I'm running the Docker container application as user zap and not as root, which might help with some host security worries. Second, as mentioned before, TCP port 5900 is open to access ZAP's GUI over VNC, but you can choose an arbitrary port by adjusting the first 5900 to your preferred port. Third, because some applications (e.g., Kubernetes) need TCP port 8080, in this case, I've opened up TCP port 81 on my host, and not TCP port 8080. Finally, you might spot the mention of X11 and VNC, which, especially when you're dealing with containers, usually involves lots of heartache and effort to get working; however, ZAP is way ahead, and thankfully it just works.

On my host running Docker, it's a case of running this command and waiting for the image to get pulled down and run. When it's finished, you might get a slightly strange ioctl error (Figure 1), which is innocuous and your prompt (pun intended) to enter your TightVNC password.

Immediately afterward, you're asked to repeat the password and then just answer y to save the password inside the container. It's worth mentioning at this point that your container is going to be destroyed immediately after you stop it, unless you log in and run a script to get the GUI working with the -d (detach) option (Listing 1). If you're just testing the GUI to see if it is working correctly and your container will be short lived, then using a simple VNC password is fine to get you going.

root@chris:~# docker run -d -u zap -p 5900:5900 -p 81:8080 -i owasp/zap2docker-stable
03736c2a2088ef47dc4e2a82fbdf5f153e34f05834fecf23ede46c2061fda423
root@chris:~# docker exec -it 03736 bash
zap@03736c2a2088:/zap$ x11vnc --forever --usepw --create
Enter VNC password:
Verify password:
Write password to /home/zap//.vnc/passwd?  [y]/n y
Password written to: /home/zap//.vnc/passwd
[snip...]
The VNC desktop is:      03736c2a2088:0
PORT=5900

Assuming your firewalls and the planets are aligned correctly, it's a case of installing a package called xtightvncviewer and pointing it at the preset PORT in the output in Listing 1. I'm using Debian Mint, so I just run a command to install the simple viewer:

$ apt install xtightvncviewer

Now all you need to do is point xtightvncviewer at your server's IP address. You don't even have to add a colon and a port number in the terminal if you haven't changed the default port from 5900. (I recommend you change the default port for extended use to mitigate automated attacks on the well-known TCP 5900 port, which, believe me when I say, has some unwelcome history.) If you don't add an IP address, you'll see an old X Windows-style dialog box, but for ease, assuming you've stuck with TCP port 5900, simply type the following line in a new terminal, inserting your IP address instead of <1.2.3.4>:

$ xtightvncviewer <1.2.3.4>

If you'd like more Docker information, you'll find more options on the GitHub page [8].

The Gooey

Figure 2 shows the great reveal: ZAP's GUI served courtesy of a friendly neighborhood Docker over VNC. A generous level of detail, help, and options are available within the GUI. Notice the questions about persistence before you begin your session and see Listing 1 for Docker's part in making that possible.

I will leave you to explore the application (carefully and targeting only your own systems) and return to the API, which can be integrated in true DevOps fashion within a CI context.

However, first, I would be remiss not to mention the Check for Updates option under the Help menu. If you want to see a list of the innards of the beast (and by that I mean which add-ons give ZAP its functionality), check for updates. Figure 3 shows what I see when asked which add-ons I want to update.

Incidentally, if you mouse over the Description field of an add-on, you are offered a very helpful yellow pop-up box with details of the add-on, including a note related to its level of maturity.

Piping Hot

With the basics under your belt and fully appreciating that by harnessing the power of ZAP you can get some very useful scanning results (and enjoy the minimal setup time involved and not very steep learning curve), you now need to focus on how to access the ZAP API.

When dealing with automated execution from triggered, event-fired scans, you do not need to be overly concerned with persistence or the GUI. In some circumstances, you might just automate tasks using Docker commands and not by accessing the API directly. In the following simple example, I fire up a Docker container for a specific task, such as a baseline scan (a safe, passive scan) of a website:

$ docker run -t owasp/zap2docker-stable zap-baseline.py -t https://www.chrisbinnie-devsecops.com

ZAP is a potentially destructive tool, so choose your target URLs with great care, unless you want to find out how good you look in an orange jumpsuit. The above command will, by default, run for one minute against the target with passive scanning and warn on all issues. All parameters are configurable, as you'd expect from such a sophisticated tool, including adding regex to keep certain URLs from being tested.

If you want to check what options are available, access the zap-baseline.py script by running it without options through Docker or by entering the container, as in Listing 1, and running the script. Here's the simple no-options route:

$ docker run -t owasp/zap2docker-stable zap-baseline.py

Running the command without options brings up a useful help page (Figure 4).

My favorite choice is to create my own config file with the -g switch, which generates a file that can be edited to switch certain tests off in future scans. The docs suggest the following set of command-line options to generate a config file and a report:

$ docker run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap-baseline.py -t https://www.chrisbinnie-devsecops.com -g generated_file.conf -r shiny_report.html

In Listing 2, you can see some partial results from a potentially automated baseline scan.

Total of 18 URLs
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Password Autocomplete in Browser [10012]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Content-Type Header Missing [10019]
PASS: Information Disclosure - Debug Error Messages [10023]
PASS: Information Disclosure - Sensitive Informations in URL [10024]
PASS: Information Disclosure - Sensitive Information in HTTP Referrer Header [10025]
PASS: HTTP Parameter Override [10026]
PASS: Information Disclosure - Suspicious Comments [10027]
PASS: Viewstate Scanner [10032]
PASS: Secure Pages Include Mixed Content [10040]
PASS: Weak Authentication Method [10105]
PASS: Absence of Anti-CSRF Tokens [10202]
PASS: Private IP Disclosure [2]
PASS: Session ID in URL Rewrite [3]
PASS: Script Passive Scan Rules [50001]
PASS: Insecure JSF ViewState [90001]
PASS: Charset Mismatch [90011]
PASS: Application Error Disclosure [90022]
PASS: Loosely Scoped Cookie [90033]
FAIL-NEW: 0      FAIL-INPROG: 0 WARN-NEW: 4 WARN-INPROG: 0 INFO: 0 IGNORE: 0        PASS: 21

Massive Attack

The functionality of the baseline scripts are highly useful, but for mass deployment across many machines, the clever ZAP offers an alternative that lets you aim for multiple targets at once. These ZAP scripts are available in the Community Scripts repository on GitHub [9].

API Options

ZAP stores everything in a database, which doesn't suit automated integrations with a CI pipeline, so you should use the handy -config database.recoverylog=false option, which speeds things up significantly. If you want to update add-ons to make sure that ZAP tests against the very latest attacks, use the -addonupdate option.

Face Your Daemons

The impressive API can be presented to a number of clients for all your scripting needs, including PHP, Python, Java, Node.js, .NET, and Go. You can run the API in a Docker container in -daemon mode and then expose the host port as usual. The API even provides an HTTP version that you can check and make basic queries (GET requests) against. The developers called it RESTful-ish at one point. The basic command is along these lines:

$ docker run -p 9090:8090 -i owasp/zap2docker-stable zap.sh -daemon -port 8090 -host 0.0.0.0

The last line of the container output should be similar to:

73280 [ZAP-daemon] INFO org.zaproxy.zap.DaemonBootstrap - ZAP is now listening on 0.0.0.0:9090

I had greater success from the command line by using curl on the server itself, as opposed to using a remote web browser with the following command:

$ docker run -p 9090:8090 -i owasp/zap2docker-stable zap.sh -daemon -port 8090 -host 0.0.0.0 -config 'api.disablekey=true' -config 'api.addrs.addr.name=.*' -config 'api.addrs.addr.regex=true'

which I found on the ZAP Issues page [10].

If you get not permitted messages in your container output, then within the VNC GUI you can try using Options | Tools | API options to alter which IP addresses can connect. The preceding example allows all clients to connect, so use it with care. Figure 5 shows Curl output connecting on TCP port 8090 and the host's HTML output. The documentation mentions that you probably have the best chance of connecting to the API locally by using the 172 IP address for your container. Use the following command:

docker inspect CONTAINER-HASH | grep Address

to see which IP address your container has been assigned by Docker.

If you struggle with external browser access to your container because of firewalling and container port exposure, remember that command-line web browsers such as ELinks [11] are available, and although not fully functional, they are still useful in a closed, non-DMZ environment. In Debian derivatives, the command

$ apt install elinks

installs the ELinks browser. Figure 6 shows ELinks' view of the ZAP API user interface.

The Client Is Always Right

If you need to make any complex changes to your configuration files, you should consider making changes like adding credentials for a login in the fully blown GUI (over VNC or otherwise) and then exporting them to a file so you can use them in the API afterward.

To install the Python client, run the pip command:

$ pip install python-owasp-zap-v2.4

Don't be too put off if a search within your distribution's package manager (e.g., Apt or Yum) for the pip package offers unusual results. My results reported in the package description that v2.4 was version 2.6, which confused me about the possibility of API versus client version incompatibilities.

After you've started a set of tests, you can return to the browser user interface that's offered by the API and enter a scanID, which then returns JSON-style output giving a percentage of the scan under the key status. Take note that the AJAX spider is different and only reports a running status, as opposed to a percentage, until it's completed.

Markups

Once you've grasped the API, you can produce some shiny, useful reports to complement your testing strategy, and there's certainly no harm in compressing and archiving them for future reference. The marvel that is ZAP offers XML and HTML output with simple scripting examples to add to your arsenal.

Additionally, you can produce alert-only reports that boil down the results to display just the salient details. In addition to paging through alerts, the API lets you stop ZAP dead in its tracks if it's hit a fatal error, should it ever find something too nasty to continue.

Other reports include status responses from the API for logins and logouts, scan statistics, and active scan results. You can even tune your timeouts meticulously and make sure any problematic tests don't prevent other tests from completing.

The End

The deeper you delve into your application with ZAP tests, the richer the results you receive. ZAP encourages you to run unit tests proxied through ZAP for best results. As mentioned, over time, the eventual maturity of your tests will provide greater efficacy.

If you want to continue learning about ZAP, refer to the OWASP page with video help [1]. The User Guide link on that page jumps to the GitHub page [12], which is the official site and the most useful starter page, for beginners.

For up-to-date images, you can swap stable in the image name with weekly for a newer, slightly unstable version of ZAP for your automation needs; you would then run the owasp/zap2docker-weekly image instead. Not only do ZAP's developers recommend doing this for automated testing, they have a good historical failure rate on weekly releases, so they're probably quite safe.

Used internally and integrated with your pipeline, ZAP is a genuinely powerful addition to any CI setup. At the risk of sounding like a stuck record, you should pay close attention to how you're using ZAP and only aim it at systems you own. As you continue to use it, pay heed to the fact that the maturity of tests are what counts.

I'm looking forward to learning more about ZAP. It's a relatively safe bet that your organization will thank you for deploying ZAP today and then again at some point in the future when it keeps a potentially expensive mistake from making it into production.