Safe Clouds

The recent notable milestones in the evolution of software development and how applications are served over the Internet, not least of which are the popularity of portable containers and Docker's remarkable rate of adoption, had security professionals scratching their heads until the security challenges that they presented were fully understood. Few niche vendors fully embraced container security headaches as Docker's toolbox took the developer world by storm. The sophisticated security suite Twistlock, now Prisma Cloud Compute Edition [1] after their acquisition by Palo Alto Networks (see the "What's in a Name?" box), stood out from the crowd and, through natural selection, took center stage in the container security space.

Now, having increased its feature set significantly, the cloud-native security suite is exceptionally sophisticated. After the introduction of multiple new features (e.g., the ability to protect host machines that aren't running containers, serverless function protection, and improvements in continuous integration-continuous delivery), it's safe to say that the product has raised its game significantly. Moreover, Prisma Cloud Compute no longer focuses only on protecting Amazon Web Services (AWS); the security suite now supports the Google Cloud and Microsoft Azure platforms, as well.

Admittedly, it's difficult to do the product justice in a single article, so I will have to skip some details to fit as much content as possible. Although Prisma Cloud Compute is not an open source product, I will point you at some open source code that might be useful in the container security niche. Once you've had a look at what's included in the box, I hope you will be suitably motivated to request a demo to try it out yourself.

To get started, I'll begin with some basic terminology, and then briefly look at the three areas most important to supporting the DevSecOps life cycle: runtime defense, vulnerability management, and the all-important area of compliance, which is especially key in enterprises. In this article, I'll run through a basic installation, get your hands dirty with a look at Prisma Cloud Compute in action, and take a look at how Prisma Cloud Compute can be configured to prevent serverless functions from being compromised.

The Automation

To my mind one of the most powerful aspects of Prisma Cloud Compute is its inclusion of machine learning, which creates behavior models for discovered resources automatically. After an initial period of monitoring a resource, any deviations to a model triggers alerts of varying severities. Once the (sometimes thousands of) intricate profiles for all of your resources have been created, you can then fine-tune associated rules at a surprisingly granular level. This process not only saves vast amounts of time (not forgetting typing errors and misconfigurations commonly made by humans), but after discovering the "normal" behavior of your resources, you can then tweak the rulesets to account for anomalies with great ease.

The Mothership

The architecture includes a centralized server known as the Console, to which other components phone home with their audit logs and receive new rules in return. The Console currently needs Docker running in the background and is provided as a container that listens on TCP port 8084 for what are known as Defenders. In addition to chatting away merrily to Defenders, the Console presents a well-designed and easy-to-use dashboard over a web interface. Normally this would be over TCP port 8083, but it's not uncommon to add your own TLS/SSL certificate and move it to HTTPS to avoid site is not to be trusted errors from your browser and simplify internal firewall rules. Take note that if you write scripts around the API then some settings will default to TCP port 8083 and will need to be tweaked.

The Enforcers

Currently three types of Defenders are offered: Container, Host, and Serverless. Each is responsible for monitoring a specific type of resource. Once your Console is up and running, most of your time will likely be spent tweaking the rules to which your Defenders adhere and ensuring that all your resources have a Defender monitoring their behavior.

Prisma Cloud Compute not only offers the Defenders to protect your resources, but it also offers two types of firewalls (which make some use of the machine learning models too). One, as you might expect, deals with simple network access and allows whitelisting and blacklisting of IP address ranges. The other is a relatively simple but effective web application firewall named the Cloud Native Application Firewall (CNAF). Note that rolling CNAF rules across all resources might affect performance.

Also employed to keep common threat information current is a custom threat intelligence system that works by aggregating a number of commercial online feeds. The intel is updated frequently to catch the latest security issues and can be tweaked manually so that you can add, for example, your own malware signatures, whitelisted vulnerabilities, and banned IP address ranges.

Runtime Defense

The underlying engines, or runtimes, that power applications are subject to detailed investigation by Prisma Cloud Compute.

Container Defender. From the container perspective, the runtime might have more facets than initially thought. For example, sometimes very unwelcome (privileged) interactions occur between a host machine's kernel and any running containers that need to be closely monitored. This type of Defender is also responsible for scanning the images in your container image registry. Note that a Container Defender via the host machine's kernel monitors the host for you so that no Host Defender is required. As a result, if you're using a managed Kubernetes service like the Elastic Kubernetes Service (EKS) on AWS, for example, a Container Defender will additionally scan the host operating system (OS) on the nodes that are running the Console.

Host Defender. As you'd expect in the main filesystem access, RAM and all OS and hardware interactions are picked up at high levels of detail. This relatively new addition to the feature list means that you can dutifully check for any issues on each machine on your estate that isn't running Docker directly.

Serverless Defender. I'll look at this Defender in action a little later when using AWS Lambda functions. Take note that this type of Defender interacts with the Console noticeably differently from the other Defender types. Unlike Container Defenders and Host Defenders, the runtime protection for serverless functions is separate from the vulnerabilities and compliance configuration. You might even say that the runtime protection is deployed at arms length so it is distinctly different from the other types of protection. In this article, I look at how to set up all three serverless protection modes.

Vulnerability Management

Across all three Defender types, the ability to monitor which vulnerabilities are present is a key part of securing an estate. Common Vulnerabilities and Exploits (CVEs) [3] are the most popular way of listing, publishing, and categorizing security bugs. Within Prisma Cloud Compute, you can configure alert thresholds to filter out unwanted noise from logging data. For example, you might want to receive alerts only for High and Critical grades of vulnerability and ignore those listed as Low and Medium grades.

Compliance

The venerable Prisma Cloud Compute comes with a number of built-in compliance frameworks, including the National Institute of Standards and Technology (NIST), Payment Card Industry (PCI), and Center for Internet Security (CIS) benchmarks. As well as saving lots of time going through relatively tedious rule configuration, you can fine tune a template of compliance rules to suit your bespoke needs once it has been applied. The security suite's Compliance feature has been officially certified by my favorite benchmark, CIS [4], which are consensus-based, industry-respected guides detailing what you would be wise to attend to when it comes to security.

What's in the Box?

The graphical Console can be accessed by way of a sophisticated API on the same network port on which you install it. To get a Console up and running, I'll follow the instructions for a basic "Onebox" installation, because both a Container Defender and a Console are installed on a single machine. Remember, no Host Defender is needed on this occasion.

Once I have a working Console, run through some of the data the Container Defender is picking up, and feed back to the Console, I'll look at integrating an AWS Lambda function.

To get started, I'll fire up an EC2 instance in AWS to install the Console. I'll assume you have working AWS credentials. Of course, it's up to you whether you're using Terraform or some other programmatic method or doing this manually. I opted to use the latest Ubuntu LTS (18.04, Bionic) OS; you'll probably want to make the Console instance quite beefy, with a good bit of RAM, and provide it with decent disk I/O. Detailed system requirements are available if needed in the docs. I ran the commands apt update; apt upgrade -y to keep up with the latest patches and packages; and then, I performed a reboot.

The next task is to download the latest version of Prisma Cloud Compute to your instance by visiting the release links [2] within the docs. From there, you can find a PDF (boasting a staggering 754 pages of about 63MB) with details on where to find the latest release. In my case, this was version 19.11.480. The docs are also now public, which is more convenient (an access token attached to the license was always required before the acquisition and is still needed for the tarball download). In this case, however, you have a URL to use for downloading a tarball with wget.

The new Prisma Cloud Compute docs have a nicely compiled list of possible routes to take at this point. I can't help but be impressed with how broad a reach the product now has, especially having embraced non-containerized hosts and serverless functions alike. Installation options include Kubernetes (Elastic Kubernetes Service/Azure Kubernetes Service), OpenShift, Docker Swarm, Pivotal Container Service, Amazon Elastic Container Service, DC/OS (the Distributed Cloud Operating System), and Windows. I chose the simplest Onebox option, which the docs remind you will scale with additional Defenders (on other host machines), if you like.

You need to be root to perform the installation. As mentioned, the Onebox installation installs a Console container and a Defender container onto your machine. First makes sure you have Docker installed: docker-ce is the latest version [5] (the older versions are called docker, docker.io, or docker-engine). On Debian derivatives, you can install the older version with:

$ apt install docker.io

Now simply extract the files from the tarball, which still sits at about 687MB:

$ mkdir twistlock
$ tar -xzf prisma_cloud_compute_edition_19_11_480.tar.gz -C twistlock/

If you cd into that newly created directory and look for the twistlock.cfg file, the options you might tweak are critical to an enterprise installation and include, for example, which network ports your Console and Defenders communicate over and where backups are saved. You can also tweak read-only and root user security settings for the Console and set up high availability. For the Onebox installation, however, the defaults will suffice.

As part of the next command, you automatically agree to the end-user license agreement with the -s option and opt for the Onebox install route:

$ ./twistlock.sh -s onebox

If you didn't want a Defender on this same instance, then replacing onebox with console would achieve just that. If you see a disk space warning here, then it's possible to append a y to the switch (i.e., -sy) to ignore the warning and proceed.

After a little patience and familiar Dockeresque output, you should see some notices in green text denoting success.

Batteries Definitely Included

The next step is accessing the Console over TCP port 8083. In the AWS Management Console, open this port to your local IP address within the instance's security group to make it possible to navigate directly to the public URL over HTTPS: https://34.XXX.XXX.XXX:8083. After entering that URL, you're immediately asked about creating your initial admin account (Figure 1), which needs to be completed to log in.

The next screen asks for a license key, and once that's pasted in, you are presented with the main Console screen (Figure 2). The default view is a zoomed-in Radar view displaying your Console and Defender containers and their relationship in terms of network ports. Additionally, the Defender container is selected and shows any issues in the black box at the foot of the display. Usually this screen is extremely busy with resources, as opposed to just the two containers seen here. At its busiest, the ability to be able to zoom in and out of the topology on the display is invaluable. In addition to the container view, you have host and serverless views, too.

Before configuring serverless protection, I would be remiss not to cover some of the common interfaces and dashboards you will use to get the best out of the product.

The Console

On the left side of the Console you can see a number of options. The options under the Defend menu are, for all intents and purposes, the place to configure your rules. You will be most concerned with Runtime, Vulnerabilities, and Compliance. Other options in the list are Firewall and Access.

I covered firewalling a moment ago, so I'll take a quick look at Access. This area is one of the reasons I started working with what was Twistlock a few years ago. To my mind, this functionality is unquestionably exceptionally powerful. I was initially drawn to working in DevSecOps because of the minefield that was Docker security at the time (which has improved since then).

If you're interested, the open source Twistlock AuthZ Broker software is available on the Twistlock GitHub page [6]. It is described as a "basic extendable Docker authorization plugin that runs directly on the host or inside a container" and includes clear documentation.

Having discovered AuthZ – and, as a result, Twistlock as an organization – it soon became clear that container security required a great deal of consideration to get it right. Thankfully, Prisma Cloud Compute now means that all of the heavy lifting is done for you. In Figure 3 you can see the granularity of some of the container rules. Further down that list are image_delete rules and network and volume options, among many others. The blue boxes at the bottom of the figure show the Groups, Containers, Hosts, Images, and Labels to which the rules apply. This is just one of the options available from the Access menu, which has Secrets management and Kubernetes auditing, too.

Again, one of the most powerful features of Prisma Cloud Compute is its ability to use machine learning to monitor resources when they are first introduced to the security suite. After a period of an hour, a model is created. Should a resource (container, host machine, function, or otherwise) start misbehaving, the Console will flag it as an issue to be investigated. If in-house requirements mean that the models need to be tweaked, then you can alter the rules with the fine-tuned rulesets shown in Figure 3.

As you might expect, the Monitor section is where you can see the results of the rules created between you and machine learning under the Defend options. Briefly, Prisma Cloud Compute picks up the details of components (in this case, a Console and a Defender). With the Defender container selected under the Runtime option, you can see in Figure 4 that Prisma Cloud Compute's timesaving machine learning process is still checking how that container usually goes about its business. Figure 5 shows that the modeling has recorded a number of details from its day-to-day behavior. If Prisma Cloud Compute sees any deviation from these processes, it will trigger alerts, blocking actions, or both.

In that same model is common information relating to filesystem access, network ports, kernel capabilities, and syscalls. Tweaking all of these details manually would be nearly impossible and highly prone to errors; at the risk of repeating myself, this is where machine learning makes such a difference.

Under the Vulnerabilities option, the security suite checks include container images, hosts, serverless functions, container registries, and jobs triggered by the Jenkins plugin.

As an example, I'll just take a quick look at one particular CVE that's been flagged and how Prisma Cloud Compute deals with it. A redacted Figure 6 (because Prisma Cloud Compute is transparent about issues present within its own containers, too) shows how vulnerabilities are presented. Note that usually it's possible to see whether a vendor fix is available, in addition to being presented with clickable links to drill down into other ways to mitigate the flagged CVEs, again saving precious time in which you would normally be researching an issue.

As you can imagine, when dealing with thousands of running containers on a busy estate, this level of detailed information is genuinely invaluable for remediating issues promptly.

The Prisma Cloud Compute Compliance section presents vulnerabilities analysis. Among the information is a screen detailing the layers of containers and where issues might exist in your Dockerfile, which is a very useful addition to assist in the operational side of running infrastructure. Figure 7 shows a little taster of that which hosts, container images, and functions might all be subject when it comes to compliance. In this section, you will also find AWS compliance tests that can highlight cloud platform issues.

Less Is More

Now that the basics have been covered, it's time to set up a working example of how a serverless function can be protected by Prisma Cloud Compute. The runtime protection and the vulnerabilities and compliance protection are run separately within Prisma Cloud Compute, as mentioned earlier – partly, I suspect, because this functionality is fairly new and partly because it makes logical sense. Soon you'll see how they slot together from the Console perspective.

As an example, I have a simple Lambda function written in Python that will perform a DNS lookup of devsecops.cc (Listing 1). To begin, I set up the vulnerabilities and compliance protection, leave it to run a couple of scans, and then return to it in a little while. In reality, the Lambda functions in my AWS account aren't too complex and don't change much over time, so the initial scan results aren't going to change. However, it should convey that this functionality involves automated periodic scans once credentials are added to the Console and is separate from the runtime protection.

On AWS you'll need to add your own security credentials for this task by creating an Access Key and Secret Key in Identity and Access Management (IAM). In Prisma Cloud Compute, go to Manage | Authentication | Credentials Store (Figure 8). Although IAM roles are supported, an IAM user account is used for scanning all regions.

Listing 2 is a redacted set of IAM permissions that essentially just lists and describes resources in an AWS account and aren't too invasive. Check the policy carefully before creating such a policy, however, and attach them to a user who has Programmatic access to the account. These permissions allow Prisma Cloud Compute to discover any AWS Lambda functions present within an account and then scan them for vulnerabilities and compliance. A word of warning is to click Save twice after creating the rule, not once.

Figure 9 shows the Console Defend | Vulnerabilities | Functions screen for adding a scanning rule that will set up the AWS account for serverless functions. In this case, the asterisk will pull in all functions, of any name, up to a Cap (limit) of five functions.

After a quick test to make sure my test Lambda function (the simple Python test script in Listing 1) executes correctly, the next stage is to create a ZIP file of the refreshed function that includes the Prisma Cloud Compute files. (Support for other popular runtimes now includes Java, Node.js, and .NET.) To do so, first export the Lambda function and then save it as a ZIP file locally. The next task is to plug in the Prisma Cloud Compute library and certificates generated from the Console (these certs are unique to this Console).

01 import socket
02
03 def handler(event, context):
04   data = socket.gethostbyname_ex('devsecops.cc')
05   print (data)
06   return

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"apigateway:GET",
"cloudfront:ListDistributions",
"elasticloadbalancing:DescribeListeners",
[...snip]

From Manage | Defenders | Single Defender | Option 2-Serverless | Python, you can download the ZIP file from the Console (or via the API), extract the files from that compressed ZIP file, and add the code that performs DNS lookups after the Prisma Cloud Compute wrapping code (Listing 3). Save the file as function.py. Make sure the twistlock/ directory contains all the code from the Console's ZIP file and your function code is at the upper level; then, compress all the files into a new ZIP file. Next, import that new ZIP into AWS as the Lambda function's new payload, first making sure the handler is pointing at the correct place.

01 import socket
02 import twistlock.serverless
03 @twistlock.serverless.handler
04 def handler(event, context):
05     data = socket.gethostbyname_ex('devsecops.cc')
06     print (data)
07     return

Now that the freshly zipped Lambda function has been imported, you can run it to generate some logs. If the Console does not enforce any rules from CloudWatch logs, the DNS lookup will be successful.

As you might guess, the next test is applying a rule from the Console that matches the function's name (asterisks to cover all names are permitted here, too). To add rules to a Lambda function, you need to create an environment variable in the AWS config.

The environment variable for Prisma Cloud Compute uses a key called TW_POLICY and a Base64-encoded set of rules (coming in at around 24 lines) that deal with the network stack, process table, and filesystem of serverless functions (mostly with block/alert and whitelist/blacklist options). It's very clever in its simplicity. Other useful additions include detection of crypto mining and the ability to monitor and potentially block certain DNS lookups. By default, only the main process will be allowed to execute; other processes are blocked, which apparently helps to mitigate against command injection attacks.

In Defend | Runtime | Serverless Policy, you now create a rule (Figure 10) that alerts on devsecops.cc lookups. Figure 11 shows what CloudWatch says when the Lambda function is invoked again, now having pasted the environment variable details into the AWS configuration. The line to pay attention to is:

[Twistlock] (ALERT) /var/lang/bin/python3.7triggered a DNS resolution ofsuspicious name devsecops.cc

With the correct plumbing in place, this runtime protection data will feed back to the Console for analysis and easier digestion. A handy tip is that if you are using a Console in EC2 and want to forward data back from Lambda functions to the Console, you should attach the Lambda function in question to a virtual private cloud (VPC), a private subnet, and a security group with suitable permissions. From within the private subnet, add a network address translation (NAT) gateway, which is the AWS-recommended way of speaking securely and internally to non-public instances from AWS Lambda.

Feeling Vulnerable and Non-Compliant

Now that you know runtime protection works, go back to your Console and check that the serverless vulnerabilities analysis and compliance functionality is working with the user and policy added earlier (see Figure 8 and Listing 2).

If you look at Monitor | Vulnerabilities | Functions (Figure 12), you can see that all five Lambda functions received the all-clear for known vulnerabilities. It's not uncommon to see payloads with older libraries triggering alerts to remind you to patch and update your Lambda functions, as well as your other software.

When checking the compliance of the five discovered Lambda functions, the results for the prisma-dns-lookup function created for the runtime protection example registers a high alert (Figure 13). The certificate key used to communicate back to the Console, which was analyzed from inside the ZIP file payload uploaded in the runtime protection example, has been flagged as the issue.

Inside another Lambda function, Prisma Cloud Compute is flagging a number of other useful issues that would have almost certainly been missed otherwise (Figure 14). One problem is a potential denial-of-service risk because of permissions that allow EC2 instances to be stopped and started. Additionally, other permissions – such as being allowed access to AWS Systems Manager (SSM), which can wield all sorts of power within an AWS account – might cause headaches.

After seeing the three areas of serverless protection provided by Prisma Cloud Compute, I hope you can see the power serverless protection brings to an estate when used in tandem with Host Defenders that protect and monitor your host machines and Container Defenders that inspect the innards of your containers. Along with a range of other impressive features, to my mind it's an invaluable product when you consider how expensive a security compromise to both online and offline services can be.

The End

As pioneers of the container security market, Twistlock – now Prisma Cloud Compute Compute Edition – has matured into an exceptionally comprehensive security suite. This space moves quickly, so the occasional crinkle can appear when new features are released. However, the ability to improve the security posture of all cloud-native components and integrate them into one security suite is difficult at the best of times.

Notice that I didn't look at the continuous integration-continuous delivery options that are available to users who want to plug their builds into Prisma Cloud Compute. Needless to say, the security suite considers the whole DevSecOps life cycle. A (slightly outdated) document [7] can give you an idea of how the Jenkins plugin works, for example. A binary called twistcli also allows the scanning of data, and you can find more information about tracking "… vulnerability findings for each build, in each layer, and whether the function or image passed or failed …" [8].

You can explore the Prisma Cloud Compute SaaS version in more detail by requesting a demo from the main website [9].