A Code Bash

Amazon Web Services (AWS) Lambda functions [1] have grown immeasurably popular over the last year or two and will unquestionably have a place in the future when it comes to running repetitive tasks on cloud infrastructure while managing to avoid the costs and overhead of maintaining servers.

Something that tripped me up initially when I began adopting Lambda functions, however, was the growth in favoritism for the Python programming language. It seems that Node.js and Python have become the de facto sweetheart languages (or run times) of choice when it comes to Lambda functions, with Ruby appearing at points, too. Although I can pick away and tweak Python scripts, it's never been a priority for me to learn the language; as a result, creating a payload for Lambda functions was never easy.

I discovered something very clever the other day that allows Bash to be used as a custom run time in AWS Lambda functions. A GitHub project [2] clearly extols the virtues of using Bash as a serverless run time. In this article, I explore a relatively new addition to Lambda functions in AWS called "layers" and, with the use of a Bash layer provided on the GitHub page by author Graham Krizek [3], I run a Bash script that uses the latest and greatest serverless tech on AWS.

Bashes and Bumps

In simple terms, you might say that an AWS layer is an additional library you can inject into your Lambda function to run alongside your payload. Why would you want to do that, you might well ask? Well, you might have a common piece of code that you want to share around your infrastructure that gets updated periodically. It might contain anything, such as SSL/TLS certificates for your payload to connect securely into a service, or a shared library from a vendor that's useful to bolt onto your payloads and treat as a module.

According to AWS, a layer "… lets you keep your deployment package small, which makes development easier" and is a "… ZIP archive that contains libraries, a custom runtime, or other dependencies" [4].

Currently, you can add a maximum of five layers to a function, with a few size limits so that the "… total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB" [4]. If you're likely to embrace serverless tech to a massive degree, the AWS page on Lambda limits [5] will help explain the relatively sane limitations currently enforced.

When I've created Python Lambda functions in the past, I've always added layers – after getting the Python working. Of course, that won't be possible when it comes to running Bash scripts because they're not native to Lambda functions in AWS. Therefore, the symbiotic relationship between the function and the layer means that they need each other to exist and, as a result, must be present at the same time.

With that little bit of context in place and without further procrastination, I'll crack on and show you how to create a new function.

Injection

After a non-trivial amount of eye strain, I recently deployed layers in Lambda functions with Terraform on a large AWS estate. At the time of writing, it still needs a little maturation to meet some people's desired requirements. That said, Terraform development moves like lightning, and features are improved and added all the time. Whether you are interested in using Terraform or not, the makers of Terraform discuss the important aws_lambda_layer_version resource on their website [6]:

resource "aws_lambda_layer_version" "lambda_layer"{
  filename   = "lambda_layer_payload.zip"
  layer_name = "lambda_layer_name"
  compatible_runtimes = ["nodejs8.10"]
}

The long and short of using this resource is that you simply reference the Amazon Resource Name (ARN) layer from within your aws_lambda_function resource to inject it into a Lambda function and then configure the layer as per the example. One gotcha is that if the filename argument for the payload is changed and the payload is therefore pulled in from an AWS S3 bucket with the s3_key instead, the S3 bucket must exist in the same region where the Lambda function will run.

In the example in this article, I'm going to use a combination of the AWS command-line interface (CLI) and the AWS Management Console to create and populate a Lambda function, so I'll leave Terraform behind for now.

No Gooey

To begin, I presume you have set up your AWS credentials. If you're not sure how to do that, AWS has instructions [7] that walk you through the process step-by-step. You'll need to ensure that you have a relatively new version of the CLI installed on your machine (I always use the pip Python package installer when possible to avoid older packages) and have created an access key for your user along with a secret key. The AWS instructions also mention using the Identity and Access Management (IAM) section of the AWS Management Console to create your credentials.

Once you're ready, log in to the AWS Management Console in your browser and navigate to the Lambda section of the website. Look for the orange Create function button at top right to make sure you're in the right place. Here, you'll be able to view any newly created Lambda functions.

The GitHub repository offers instructions on how to make use of the Bash layer so you can run Bash scripts natively. The README file [8] contains all the information needed and starts off on how to create a brand new Lambda function from scratch via the AWS CLI, which also includes the Bash layer already injected. The command from the GitHub README is:

$ aws lambda create-function --function-name bashFunction --role bashFunctionRole --handler index.handler --runtime provided --layers arn:aws:lambda:<region>:744348701589:layer:bash:8 --zip-file fileb://function.zip

You'll note that <region> is left unpopulated, which as mentioned before, is one limitation to bear in mind when it comes to AWS Lambda layers. (I will be using the Dublin, Ireland, region (eu-west-1) for my command.) You'll also need to create a Bash script that can be loaded into your Lambda function as a payload.

Listing 1 is a Bash script, acting as the payload, inside a ZIP file, that runs in a Lambda function. It is a slight adaptation of the Krizek example [9]. Having saved the script to index.sh somewhere on your filesystem where you can find it, you'll need to compress that into a ZIP file to be compatible with AWS Lambda:

01 handler () {
02   set -e
03
04   # Event Data is sent as the first parameter
05   EVENT_DATA=$1
06
07   # This is the Event Data
08   echo $EVENT_DATA
09
10   # Example of command usage
11   EVENT_JSON=$(echo $EVENT_DATA | jq .)
12
13   # Example of AWS command that's output will show up in CloudWatch Logs
14   echo "I am a Bash command!"
15
16   # This is the return value because it's being sent to stderr (>&2)
17   echo "{\"success\": true}" >&2
18 }

$ zip list_buckets.zip index.sh
 adding: list_buckets.sh (deflated 39%)

As you can see, you should now have list_buckets.zip available on your filesystem.

Before continuing, however, you'll have to pay attention to some minor gotchas, which include explicitly stating which handler to use when invoking the Lambda function, because Bash behaves differently with error handling. (See the "Take Note" box for more details.)

All Got A Role To Play

Before creating a function and attaching the Bash layer, you need an AWS IAM role to provide permissions for its execution ("invocation" in AWS Lambda-speak) and to allow it to write to CloudWatch logs correctly. Note that these permissions are created by default if you create a function inside the AWS Management Console, so re-use those if you get stuck. Also note that Elastic Network Interface creation, deletion, and description of VPC permissions are also needed if you plan to use a VPC with your Lambda function so it can access internal AWS resources (e.g., Elastic Compute Cloud (EC2) instances and the like).

Listing 2 shows the IAM permission policy to use with your role. You need to replace the XXXXX entries with your own AWS account number to keep permissions tight. You can find the aforementioned account number by clicking the top-right, pull-down menu in the AWS Management Console and looking for Billing in the options. Save Listing 2 to your filesystem as policy.json. Note that bashFunction under Resource is the name of your function and needs to be changed if you change the function name.

01 {
02     "Version": "2012-10-17",
03     "Statement": [
04         {
05             "Effect": "Allow",
06             "Action": "logs:CreateLogGroup",
07             "Resource": "arn:aws:logs:eu-west-1:XXXXX:*"
08         },
09         {
10             "Effect": "Allow",
11             "Action": [
12                 "logs:CreateLogStream",
13                 "logs:PutLogEvents"
14             ],
15             "Resource": [
16                 "arn:aws:logs:eu-west-1:XXXXX:log-group:/aws/lambda/bashFunction:*"
17             ]
18         }
19     ]
20 }

For the IAM role, you will also need a trusted entity to tell AWS which services can use this role. Listing 3 names Lambda (!) as the service of choice. Call this file trust.json.

01 {
02   "Version": "2012-10-17",
03   "Statement": [
04     {
05       "Effect": "Allow",
06       "Principal": {
07         "Service": "lambda.amazonaws.com"
08       },
09       "Action": "sts:AssumeRole"
10     }
11   ]
12 }

You're almost there. You just need to create a role with that trust policy attached to it; then, you'll attach a permission policy. The command that creates the role with that trust policy is:

$ aws iam create-role --role-name bashFunctionRole --assume-role-policy-document file://trust.json

Any illegal character errors you receive might be caused by cut and paste errors [10], so check your input.

If that command is successful, you'll see lots of output, including a RoleId. If you want to be certain that role does exist, check the IAM section of the AWS Management Console.

Next, create a permissions policy in AWS and attach it to your role (Listing 2). Listing 4 creates a permissions policy called bashFunctionPolicy in AWS with the policy.json code.

$ aws iam create-policy --policy-name bashFunctionPolicy --policy-document file://policy.json
{
    "Policy": {
        "PolicyName": "bashFunctionPolicy",
        "PermissionsBoundaryUsageCount": 0,
        "CreateDate": "2001-11-11T11:11:11Z",
        "AttachmentCount": 0,
        "IsAttachable": true,
        "PolicyId": "ABCDEFGHIJKLMNOPQ",
        "DefaultVersionId": "v1",
        "Path": "/",
        "Arn": "arn:aws:iam::XXXXX:policy/bashFunctionPolicy",
        "UpdateDate": "2001-11-11T11:11:11Z"
    }
}

The next command attaches the policy to your IAM role. (Note that I've replaced my AWS account number with Xs again for security reasons.)

$ aws iam attach-role-policy --policy-arn arn:aws:iam::XXXXX:policy/bashFunctionPolicy --role-name bashFunctionRole

You might be a little surprised to see that the successful execution of that command is a simple, empty newline with no output whatsoever. To verify, check in the IAM section of the AWS Management Console to see that it worked. Figure 1 proves, having looked up the role name in IAM, that it works as hoped.

So Close …

Now you can create a shiny new Lambda function and attach your Bash layer by using the AWS CLI to execute the long aws lambda create-function command shown at the beginning of this article. When confronted with long commands, I can find them tricky to cut and paste, so I often just drop them into a script (adding #!/bin/bash as the first line), which also makes them easier to tweak and reference later. To use this script, save it as a file called run.sh (Listing 5) and make it executable with:

$ chmod +x run.sh

01 #!/bin/bash
02
03 aws lambda create-function \
04     --function-name bashFunction \
05     --role arn:aws:iam::XXXXX:role/bashFunctionRole \
06     --handler index.handler \
07     --runtime provided \
08     --layers arn:aws:lambda:eu-west-1:744348701589:layer:bash:8 \
09     --zip-file fileb://list_buckets.zip

In my case, the only things I replace are the <region> entry with eu-west-1 and function.zip to the newly named ZIP payload with the filename list_buckets.zip. To run that unwieldy command, enter:

$ ./run.sh

If you get stuck, you can copy and paste the code in Listing 5 (which is available on FTP [10]) and replace the Xs with your AWS account number, as usual.

The output from this relatively long command is shown in Listing 6, which shows that I've created a new Lambda function (in Dublin, Ireland), added a layer to it from another account, and given it permissions from a role with a policy attached to it. (Again, the output is redacted a bit.) The next step invokes the function and checks its output.

01 {
02     "Layers": [
03         {
04             "CodeSize": 26955425,
05             "Arn": "arn:aws:lambda:eu-west-1:744348701589:layer:bash:8"
06         }
07     ],
08     "FunctionName": "bashFunction",
09     "LastModified": "2001-11-11T11:11:11Z",
10     "RevisionId": "aa63-b5c0-4ec2-a5e99",
11     "MemorySize": 128,
12     "Version": "$LATEST",
13     "Role": "arn:aws:iam::XXXXX:role/bashFunctionRole",
14     "Timeout": 3,
15     "Runtime": "provided",
16     "TracingConfig": {
17         "Mode": "PassThrough"
18     },
19     "CodeSha256": "37n/rzJz4o2lyvh4s2aet2aBlY=adc",
20     "Description": "",
21     "CodeSize": 432,
22     "FunctionArn": "arn:aws:lambda:eu-west-1:XXXXX:function:bashFunction",
23     "Handler": "index.handler"
24 }

Alrighty Then

Of course, you can send variables to the Lambda function to execute it from the AWS CLI, but you should look at the AWS Management Console, as well, so you can get used to how a function looks there. In my case, I would navigate to https://eu-west-1.console.aws.amazon.com/lambda/home?region=eu-west-1#/functions in my Browser, which should list, among my other functions, bashFunctionRole. The eagle-eyed also will spot Custom runtime, on which I'll click to enter its configuration.

Once on that screen (Figure 2), double-click on the index.sh script. You can see that AWS Lambda has read the ZIP file payload as hoped and displays it on request.

Jump back up the page a moment, though, expand the little black menu arrow under the word Designer, and click on the attached layer. All in all, you should be able to see the config (Figure 3, blue highlight).

The next thing to check is whether the permissions are as intended. If you scroll down the very long page, you can see the attached role (Figure 4). Clicking on the link lets you see that your role has your policy attached.

The last stage requires invoking your new Lambda function's payload, written in the Bash scripting language and sitting atop your custom run time, to see what happens.

Get Set, Go

Navigate up to the top-right part of the function's page (Figure 5). Once there, click on Configure test events to create a test with variables to be sent to the function.

In my case, I'm going to leave an empty set of variables and not pass anything to the Lambda function; instead, I'll just send an empty run command to the Lambda function. I do, however, need to add the name EmptyTest to the test event under Event name. In this example, I'll just use the Hello World template. Click Save once you've added the event name and run the test by clicking the Test button. Figure 6 shows a successful invocation of the Lambda function.

Look back at the payload's content in Figure 2. Note that it runs the jq JSON formatting command and outputs I am a Bash command! (Listing 1). Lo and behold, that's what is present in the CloudWatch logs. If you get stuck, CloudWatch is your friend. The AWS Management Console main page has more than one link to click to read the most recent logs for the Lambda function, so follow a link.

The End Is Nigh

I hope you agree that the sophisticated functionality illustrated in this article brings a welcome addition to Lambda functions for those not familiar with other languages. Imagine the number of legacy scripts that are too much work to refactor but would suit a serverless invocation.

With the ever-growing features available in AWS, the use of Lambda functions will only grow and, I suspect, continue at the exponential rate it has shown so far. I hope you will enjoy using Bash for serverless tasks in the future as much as I intend to. Now, let me think, where's that uptime dashboard script I wrote a few years ago?