Terraforming

If you ask people within DevOps circles, they will tell you that Infrastructure as Code is the way forward. To create infrastructure automatically in a consistent and predictable manner, at least, the DevOps community has embraced a number of extensible tools.

Terraform [1] from HashiCorp is one of the market leaders. It is not a particularly new technological offering, in that it is more than two years old, but it is still absolutely key to running infrastructure in the cloud in an effective and efficient way.

Terraform can speak to multiple cloud providers making it highly flexible. This functionality offers management teams a somewhat conditional promise of having the ability to run cloud-agnostic infrastructure. Cloud providers also offer their own tools, of course, such as CloudFormation [2] from Amazon Web Services (AWS), but to my mind, Terraform is more popular for good reason.

Terraform's flexible features also mean you can run server-specific code with Terraform when you are creating servers, for example. One approach is to use a configuration management tool like Ansible or Puppet to alter the configuration on servers you have created with Terraform once the infrastructure is spun up.

In this article, I will look at some very simple uses of Terraform on AWS, along with the last bit of the jigsaw (i.e., testing the setup to make sure it is performing as designed), which I still need to get my head around and will at some point in the future turn into Terraform code.

I wrote this Terraform code because a thought occurred to me recently about the simplest (non-enterprise-oriented) way to provide failover for a personal website I was working on. The failover scenario was simple, and I just wanted a mechanism to check whether a service was available and then trip over to another server if a web server failed for some reason.

I was thinking along the lines of a geographically diverse failover, as opposed to a cloud server sitting next to the primary server (or in another Availability Zone in the same AWS region). Many a moon ago, this was not as easy as it sounds and involved cabling between data centers or relying on (unreliable) third-party providers. However, I remembered that the Amazon Route 53 [3] DNS service offered some useful DevOps-oriented tools and hacked out some basic code to get me started.

Duck Egg

The Amazon Route 53 service is sophisticated, but some elements are embraced less than others. If you are familiar with blue-green deployments, for example, you can "weight" your DNS traffic with Route 53 to shift incoming requests from your customers between your blue or green environment when testing a new release. That is a relatively simple example, but you might be surprised to find out that AWS provides some pretty clever DNS mechanisms (Figure 1).

Hashtag Fail

The content delivery network-style geoproximity provision is very slick. The failover mechanism that I was ruminating over recently became available shortly after Route 53 was introduced and was designed for a super-simple scenario (e.g., if A is down, then use B).

At this juncture, I will briefly remind you about the downsides of using DNS for failover: Internet access providers caching DNS records. In other words, the possibility is high that the changes reflected by Route 53 updates will not be picked up by broadband providers, mobile operators, and others in a consistent way, so you could lose some traffic until they are.

Against a Dark Background

Before looking more at the Route 53 mechanism, I will show you some of my very basic terraforming. One caveat is the backward compatibility issues when it comes to different versions of Terraform, so for the sake of completeness, the versions I am using are Terraform v0.11.7 and provider.aws v1.19.0.

A very brief "Getting Started" run-through for Terraform might be as follows:

1. Make sure your AWS key and secret key (from the AWS Identity and Access Management, IAM) work, and add them between the double-quotes as shown below on your command line to create environment variables (replace "KEY" and "SECRET-KEY"):

$ export AWS_ACCESS_KEY_ID="KEY" ; export AWS_SECRET_ACCESS_KEY="SECRET-KEY"

2. Download [5] and unzip the Terraform binary into your user path.

3. Once your code is in place, run the following commands in the directory in which your code resides, checking for errors (some are cryptic!) as you go:

$ terraform init # set up your env and grab the correct cloud provider module
$ terraform plan # dry-run your code for the sake ofbeing cautious
$ terraform apply # carefully create your infrastructure

Tidiness Is Next to Godliness

Usually you would split up your code in a sensible manner between multiple files, so I will demonstrate how to do so and then display all the code in one file for ease afterward.

The simple file in Listing 1 will go at the start of my single long file. The variables defined here should be in files of their own called something like variables.tf. The file defines two variables that I can change globally. I will use one value for the Time To Live (TTL) for each of the DNS records. The aws_region config in this file is not relevant to Route 53 because it is a global AWS service. In this case, I use Dublin, Ireland.

01 variable "aws_region" {
02   description = "Preferred AWS region (Route 53 doesn't need this because it's Global)."
03   default     = "eu-west-1"
04 }
05
06 variable "ttl" {
07   description = "TTL record"
08   default = 300
09 }

If I want to extract information from my code once it has been executed (for informational purposes or to respond programmatically to the results), I can use a separate file called outputs.tf that I will include at the end of my lengthy file:

output "ip" {
value = "${aws_route53_record.chris_A_record.records}"
}

As you can see, I substitute value with a variable name. Anybody who is familiar with Terraform has almost certainly seen the error that says the formatting of a variable should be TYPE.NAME.ATTR. If you translate aws_route53_record.chris_A_record.records correctly, it should make sense.

Now it is time to put these two minuscule snippets at the top and bottom of my slightly longer file, which I call main.tf. In Listing 2, you can see the creation of an A record (which could be, e.g., www or blog or indeed anything arbitrary) and the MX records, which are where email for the domain name's DNS queries are sent. The stanzas for the MX records and the A record should be self-explanatory.

01 variable "aws_region" {
02   description = "Preferred AWS region (Route 53 doesn't need this as it's Global)."
03   default     = "eu-west-1"
04 }
05
06 variable "ttl" {
07   description = "TTL record"
08   default = 300
09 }
10
11 provider "aws" {
12   region = "${var.aws_region}"
13 }
14
15 resource "aws_route53_zone" "chris_DNS_zone" {
16   name = "devsecops.cc"
17 }
18
19 resource "aws_route53_record" "chris_A_record" {
20   zone_id = "${aws_route53_zone.chris_DNS_zone.zone_id}"
21   name    = "www"
22   type    = "A"
23
24   records = ["1.2.3.4"]
25
26   ttl = "${var.ttl}"
27 }
28
29 resource "aws_route53_record" "chris_MX_record" {
30   zone_id = "${aws_route53_zone.chris_DNS_zone.zone_id}"
31   name    = ""
32   type    = "MX"
33
34   records = [
35     "5 mail.devsecops.cc",
36     "10 mail2.devsecops.cc",
37     "15 mail3.devsecops.cc",
38   ]
39
40   ttl = "${var.ttl}"
41 }
42
43 output "ip" {
44   value = "${aws_route53_record.chris_A_record.records}"
45 }

Fail2plan = Plan2fail

When I hit the Go button (or at least the dry-run Plan button) with

$ terraform plan

I am offered the output shown in Figure 2. Note the "+" signs on the left-hand side, which indicate that the mighty Terraform is creating and not deleting the resources shown. Clearly this is especially important when you are working in critical environments. You can see the creation of a single A record, three MX records, and a DNS zone called chris_DNS_zone.

Terraforming

Now I will hit Go for real. (i.e., run the terraform apply command.) Terraform sensibly asks me to type the word yes to confirm that the three plus signs shown are actually what I want. After typing yes, I then wait patiently for a little while before receiving the output shown in Figure 3.

If you look a little closer at Figure 3, you can see I was offered a countdown timer of sorts. At the bottom of the output, the Apply complete! message offers the good news that I added three resources but did not delete or change any other existing resources.

A quick look in the GUI (the AWS web console) shows that I did indeed affect the configuration of AWS (Figure 4). Set at five minutes (look for 300 seconds), the TTLs were created, and AWS or Terraform in its wisdom created the DNS zone using default values of 172800 seconds for the name server (NS) TTLs and 900 seconds for the SOA (Start of Authority record) within the zone.

Now I will make a quick but very practical change to the configuration. What if your web server IP address changes and you need to point your DNS somewhere else? If you refer to Listing 2 again (and Figure 4), you can see this exists in line 24 of the main.tf file:

records = ["1.2.3.4"]

I will quickly change the www record to a different IP address now. Figure 5 shows the result of editing main.tf and then running the

$ terraform apply

command again. Terraform uses a tilde character for changes and not a plus or minus for additions or deletions.

Once again, I am prompted for the word yes to continue and apply the change. The outcome is much quicker than before. As Figure 6 shows: Et voila! An updated A record will be propagating around the Internet's name servers.

Work in Progress

At the start of the article, I mentioned that I wanted to set up a geographically diverse failover between two web servers, which led me to tinker with Terraform and Route 53.

The part of the jigsaw that is still missing is not too complex to get working manually but might require some squinting to get slightly more advanced Terraform features working. The Route 53 health check, which Figure 7 demonstrates, will definitely be my first port of call.

This Is the End

If simple examples like the DNS config in AWS here do not pique your interest, then Infrastructure as Code might not be for you. The code is relatively easy to read, extensible, and delivers consistent results that you can depend on. I apply similar techniques to the Security as Code work that I do, which in turn provides a business with a level of comfort that resources are created in a predictable way.

I would encourage anyone to try Terraform on a cloud platform to see how it might help in the future. The learning curve, thankfully, is not too steep for simple tasks.