Mixing Allowed

According to reliable estimates, half of all corporations will be operating a hybrid cloud architecture by next year. Obviously, the advantages of mixing the historic IT landscape, which physically resides in-house, and public or private clouds from external service providers is something that appeals to admins. Lower running costs are often the deciding factor in favor of hybrid setups. In many cases, however, the combination also might have functional advantages. Companies with hybrid IT structures are said to be more successful than their competitors [1].

The reason the hybrid cloud is not yet widely used is probably because of the great effort needed to set it up. In this article, I seek to demonstrate that every admin can take a considerable step toward achieving a hybrid cloud with little effort, while gaining enough experience to outsource more parts of the local IT structure. With Amazon Web Services (AWS) as an example, I show how you can use two Linux virtual machines (VMs) to build a secure infrastructure that is even capable of multitenancy. The best part is that you do not need to do all this work yourself; instead, Ansible will carry out the setup steps. (For more information on Ansible, refer to earlier articles in this magazine [2]–[4].)

Many users still think that VMs in the cloud are always publicly available. However, administrators who have been involved with virtual network infrastructures, such as AWS or Azure, know that private networks, VPNs, and routers are also common in the cloud. AWS and Azure even offer VPNs to the private corporate network as a (commercial) service (see the "Amazon Network Glossary" box).

Target Architecture

Figure 1 shows the architecture I set up for the project here. In the local data center on the right, a VPN gateway establishes the connection to the gateway in the AWS universe. This can be implemented with a firewall or through a separate gateway. Ansible also configures a local OpenVPN instance; since I have already built VPN playbooks for Libreswan, Fortinet, and Juniper firewalls, I could easily replace the component.

On the AWS side, the structure is a bit more complicated, because I want to develop a multitenant solution – a VM facing the Internet that forwards to different closed environments. The VPN data in the closed environments will not be known to the front end. However, an incoming control based on IP addresses will still take place there. Therefore, the AWS VM uses socat to forward the incoming packets to the actual VPN service in the closed network so that an encrypted end-to-end connection is still established. If you do not want multitenant capability, you can omit this redirection from the structure.

The AWS subnet has a routing table that ensures the routing between the local networks of your data center and the AWS VM with OpenVPN occurs correctly.

Networks and Peering

As with other virtualization environments, you start by setting up the virtual network infrastructure so that the VMs ultimately end up on the correct networks. For AWS, you first create two VPCs: the inner one for the closed environment, the outer one for the gateway VM (Figure 2). Both have a private IP network.

The inner VPC should be selected so that it does not collide with the network in your data center, because on the local side, the routes into this network area must point to the local VPN gateway. In the external block, it is sufficient to select a small network area, because it only serves as a transition point. AWS now needs subnet objects whose IP networks each have to be a subset of the VPC's total range.

In the next step, you set up the VPC peering connection, which is like adding a virtual cable between your two VPCs (Figure 3). You need to accept this peering in a separate step. So that the VMs can find their way out of the outer VPC later on, you now need to create an Internet gateway.

VPN Routing

The configurations up to now simulate Layer 2 cabling, which would also be necessary for a physical structure. Now the IP routing follows. In the routing table for the inner VPC, you should define a route to the outer subnet. Because the VPN connections are forwarded at the application level (socat), there is no direct IP connection between the VPN gateway and the Internet. The goal of these routers is VPC peering – comparable to setting an interface route with:

ip route add 1.2.3.0/24 dev eth0

Conversely, the external VPC must know how to find the internal VPC, which requires routes to the internal subnet and also to VPC peering. Additionally, the external VPC's default route must point to the previously created Internet gateway.

The security configuration completes the setup of the virtual network: The VPN gateway in the internal VPC needs the VPN port (for OpenVPN, port 1194/UDP; for IPsec with NAT traversal, ports 500/UDP and 4500/UDP). To allow the admin to access the VM itself, SSH access is also required (i.e., incoming port 22/TCP). The same applies to the external gateway computer. Here, the VPN input can then be limited to the local VPN gateways' static IPs – if available. This does not offer very strong protection (especially for UDP), but it at least limits the Internet background noise a bit.

Two VMs

Now, you need just two VM instances to do the work. The intended software is installed and configured on each: socat as a service on the external VM and OpenVPN as VPN software on the internal VM.

To create a VM, you need an image ID (ami-id in AWS-speak) and an instance size. The small t2.micro instances are perfectly sufficient for the setup here, because the performance is sufficient for the Internet connections of most companies. If you have a 10Gb Internet connection, you should adjust the instance's size accordingly. Then, assign the correct VPCs and subnets, as well as the previously set up security groups, to both instances.

Both VMs are based on an Ubuntu 16.04 server image (Figure 4). socat is assigned to the front VM as a port forwarder; it launches systemd as a daemon. This would be enough for the VPN, but the internal VM has no Internet access, so it needs a proxy to install the packages. Therefore, the front VM is assigned a Squid web proxy, which is restricted so that only internal subnet clients can use it. To match this, you need to extend the security group to include port 3128/TCP from this network.

OpenVPN is installed on the inner VM in server mode. VMs on the inner network need a route through the OpenVPN gateway to the local data center network.

Ansible

If you want to configure all of this manually, it would require some work in the AWS web GUI. Additionally, the parameters must all match, such as the IP addresses of the networks and the corresponding entries in the routing tables. Later, the addresses are also included in the configuration of the services on the VMs. This leaves the admin with many opportunities for errors, especially since asking "What was the address again?" interrupts the workflow in the web GUI.

Ansible's cloud modules, on the other hand, cover everything necessary for the configuration. When creating one component in the playbook, always store the results, because the IDs of the individual components are required to create the next one. For example, you also need to enter the inner VM IP address assigned by Amazon in the socat configuration on the outer VM.

Thanks to Ansible, the configuration is a holistic process resulting in fewer errors. The whole ensemble has two roles for configuration on the inner and outer VM. Initially, the playbook builds the entire infrastructure.

Some data is parameterized, like the AWS region, the address space for the inner VPC and the subnet, and the subnet on the local LAN so that the VPN configuration can be generated. The playbook reads this data from a YAML file at the outset.

To allow the playbook to access AWS at all, you need either a file in ~.aws/credentials containing the aws_access_key and aws_secret_access_key entries, or you can set the values in the playbook as variables (or preferably via Ansible Vault for security reasons) or store them in environment variables. The cloud module documentation [4] explains the variants.

The only way to access a VM in AWS is with an SSH connection. Unlike other hosting service providers, there is no console. Therefore, the admin SSH key also needs to be stored in AWS. Listing 1 shows the list of tasks that upload the key (only do this if the key is not already in place), create the VPCs with one subnet each, and generate the VPC peering.

01 - name: Load Data
02   include_vars: "{{ datafile }}"
03   tags: getdata
04
05 - name: SSH Key
06   ec2_key:
07     name: ansible-admin-key
08     key_material: "{{ item }}"
09     state: present
10     region: "{{ region }}"
11   with_file: /home/user/.ssh/id_rsa.pub
12   register: sshkey
13   tags: sshkey
14
15 - name: Create VPC INT
16   ec2_vpc_net:
17     name: "{{ netname }}-int"
18     cidr_block: "{{ cidr_master }}"
19     region: "{{ region }}"
20   tags: create_vpc_int
21   register: myvpcint
22
23 - name: Create Subnet INT
24   ec2_vpc_subnet:
25     cidr: "{{ subnet }}"
26     vpc_id: "{{ myvpcint.vpc.id }}"
27     region: "{{ region }}"
28     state: present
29   tags: create_subnet_int
30   register: mysubnetint
31
32 - name: Create VPC Ext
33   ec2_vpc_net:
34     name: "{{ netname }}-ext"
35     cidr_block: 172.25.0.0/28
36     region: "{{ region }}"
37   tags: create_vpc_ext
38   register: myvpcext
39
40 - name: Create Subnet Ext
41   ec2_vpc_subnet:
42     cidr: 172.25.0.0/28
43     vpc_id: "{{ myvpcext.vpc.id }}"
44     region: "{{ region }}"
45     state: present
46   tags: create_subnet_ext
47   register: mysubnetext
48
49 - name: Create VPC Peering
50   ec2_vpc_peer:
51     region: "{{ region }}"
52     vpc_id: "{{ myvpcint.vpc.id }}"
53     peer_vpc_id: "{{ myvpcext.vpc.id }}"
54     state: present
55   register: myvpcpeering
56   tags: createvpcpeering
57
58 - name: Accept VPC Peering
59   ec2_vpc_peer:
60     region: "{{ region }}"
61     peering_id: "{{ myvpcpeering.peering_id }}"
62     state: accept
63   register: action_peer
64
65 - name: Create Internet Gateway
66   ec2_vpc_igw:
67     vpc_id: "{{ myvpcext.vpc.id }}"
68     region: "{{ region }}"
69     state: present
70   register: igw
71   tags: igw

At the end of each ec2 instruction is a register block that stores the operation's result in a variable. To create a subnet, you need the VPC's ID in which the subnet is to be located. The same thing happens starting on line 49, first to create and then accept VPC peering. The final task creates the Internet gateway.

Creating Routing and VMs

Listing 2 shows the second part of the playbook. The Gather Route tables task searches the routing table in the internal VPC to enter the route on the outer network. Then, the playbook sets the route from the inside out and in the opposite direction. The next two tasks create the security groups for both VMs.

01 - name: Gather Route tables
02   ec2_vpc_route_table_facts:
03     region: "{{ region }}"
04     filters:
05       vpc-id: "{{ myvpcint.vpc.id }}"
06   register: inttables
07   tags: gatherroutes
08
09 - name: Set Route out
10   ec2_vpc_route_table:
11     vpc_id: "{{ myvpcint.vpc.id }}"
12     region: "{{ region }}"
13     route_table_id: "{{ inttables.route_tables[0].id }}"
14     tags:
15       Name: "{{ netname }}-int"
16     subnets:
17       - "{{ mysubnetint.subnet.id }}"
18     routes:
19       - dest: 172.25.0.0/28
20         vpc_peering_connection_id: "{{ myvpcpeering.peering_id }}"
21   register: outboundroutetable
22   tags: routeout
23
24 - name: Set Route in
25   ec2_vpc_route_table:
26     vpc_id: "{{ myvpcext.vpc.id }}"
27     region: "{{ region }}"
28     tags:
29   name: "{{ netname }}-ext"
30     subnets:
31       - "{{ mysubnetext.subnet.id }}"
32     routes:
33       - dest: "{{ subnet }}"
34         vpc_peering_connection_id: "{{ myvpcpeering.peering_id }}"
35       - dest: 0.0.0.0/0
36         gateway_id: igw
37   register: outboundroutetable
38   tags: routein
39
40 - name: internal Secgroup
41   ec2_group:
42     name: "{{ netname }}-int-secgroup"
43     vpc_id: "{{ myvpcint.vpc.id }}"
44     region: "{{ region }}"
45     purge_rules: true
46     description: Ansible-Generated internal rule
47     rules:
48       - proto: udp
49         from_port: 12345
50         to_port: 12345
51         cidr_ip: 0.0.0.0/0
52       - proto: tcp
53         from_port: 22
54         to_port: 22
55         cidr_ip: 0.0.0.0/0
56       - proto: tcp
57         from_port: 443
58         to_port: 443
59         cidr_ip: 0.0.0.0/0
60   register: intsecg
61   tags: internalsec
62
63 - name: external Secgroup
64   ec2_group:
65     name: "{{ netname }}-ext-secgroup"
66     vpc_id: "{{ myvpcext.vpc.id }}"
67     region: "{{ region }}"
68     purge_rules: true
69     description: Ansible-Generated internal rule
70     rules:
71       - proto: udp
72         from_port: 12345
73         to_port: 12345
74         cidr_ip: 0.0.0.0/0
75       - proto: tcp
76         from_port: 22
77         to_port: 22
78         cidr_ip: 0.0.0.0/0
79       - proto: tcp
80         from_port: 443
81         to_port: 443
82         cidr_ip: 0.0.0.0/0
83       - proto: tcp
84         from_port: 3128
85         to_port: 3128
86         cidr_ip: "{{ subnet }}"
87   register: extsecg
88   tags: externalsec
89
90 - name: Update Auto
91   ec2_auto_assign_public_ip_for_subnet:
92     subnet: "{{ mysubnetext.subnet.id }}"
93     region: "{{ region }}"
94     state: present

The last task in Listing 2 (line 90) is necessary because of an Ansible peculiarity – according to the AWS API, this task is unnecessary without Ansible. The API can tell a subnet whether hosts on this subnet should always be assigned a public IP automatically and is only possible in Ansible with the added task shown in Listing 2.

Listing 3 creates the two VMs. The tasks create the VMs and add them to a group. These groups will immediately use the following plays to install and configure the software via SSH. The last task of the first play now waits until the external VM is accessible via SSH.

01 - name: Deploy Backend
02   ec2:
03     key_name: ansible-user-key
04     instance_type: t2.micro
05     image: ami-d15a75c7
06     region: "{{ region }}"
07     wait: yes
08     id: test-backend
09     assign_public_ip: no
10     vpc_subnet_id: "{{ mysubnetint.subnet.id }}"
11     group_id: "{{ intsecg.group_id }}"
12   register: backendvm
13   tags: createbackend
14
15 - name: add frontend to group
16   add_host:
17     hostname: "{{ item.private_ip }}"
18     groupname: backend
19   with_items: "{{ backendvm.instances }}"
20
21
22 - name: Deploy Frontend
23   ec2:
24     key_name: ansible-user-key
25     instance_type: t2.micro
26     image: ami-d15a75c7
27     region: "{{ region }}"
28     wait: yes
29     id: test-frontend
30     assign_public_ip: yes
31     vpc_subnet_id: "{{ mysubnetext.subnet.id }}"
32     group_id: "{{ extsecg.group_id }}"
33   register: frontendvm
34   tags: createfrontend
35
36 - name: add frontend to group
37   add_host:
38     hostname: "{{ item.public_ip }}"
39     groupname: frontend
40   with_items: "{{ frontendvm.instances }}"
41
42 - name: Wait for ssh of frontend
43   wait_for:
44     host: "{{ item.public_dns_name }}"
45     port: 22
46     state: started
47   with_items: "{{ frontendvm.instances }}"

Role-Play

The playbook's second and third plays use a role to configure the software on the two VMs. The last play in Listing 4 is especially important. Because the inner VM is only accessible from the outer VM, it needs to reroute the SSH command via ansible_ssh_common_args so that the inner VM uses the outer as a proxy. I only packed the configuration into roles here because I wanted the configuration to be reusable on the VMs.

01 - name: Configure Frontend
02   hosts: frontend
03   remote_user: ubuntu
04   gather_facts: no
05   roles:
06     - agouros.aws.frontend
07
08 - name: Configure Backend
09   hosts: backend
10   remote_user: ubuntu
11   gather_facts: no
12   vars:
13     ansible_ssh_common_args: "-o ProxyCommand='ssh -W %h:%p -q ubuntu@{{ hostvars['localhost']['frontendvm']['instances'][0]['public_ip'] }}'"
14   roles:
15     - agouros.aws.backend

Listing 5 shows the front-end play or tasks from the role. The first task installs Python in raw mode, because Python was not installed on the AWS Ubuntu image I used. Then Ansible installs and configures the socat and squid packages. What is striking here is socat, which is not really a service. Only a systemd unit file makes it a service on UDP and 443/TCP. The target address of the inner VM for connecting is taken from the previous playbook run. The play starting in line 43 (Listing 5) activates the services and starts them.

01 - name: Install Python
02   raw: test -e /usr/bin/python || (sudo -s apt-get -y install python)
03
04 - name: Install Software
05   become: true
06   become_method: sudo
07   apt:
08     name: "{{ item }}"
09     state: present
10     cache_valid_time: 86400
11   with_items:
12     - socat
13     - squid
14
15 - name: Configure Squid
16   become: true
17   become_method: sudo
18   template:
19     src: templates/squid.conf.j2
20     dest: /etc/squid/squid.conf
21   notify: restart squid
22
23 - name: Configure socat
24   become: true
25   become_method: sudo
26   template:
27     src: templates/socat.service.j2
28     dest: /etc/systemd/system/socat.service
29   notify:
30     - daemon-reload
31     - start socat
32
33 - name: Configure socat443
34   become: true
35   become_method: sudo
36   template:
37     src: templates/socat443.service.j2
38     dest: /etc/systemd/system/socat443.service
39   notify:
40     - daemon-reload
41     - start socat
42
43 - name: Start and enable Services
44   become: true
45   become_method: sudo
46   systemd:
47     enabled: yes
48     state: started
49     daemon-reload: yes
50     name: "{{ item }}"
51   with_items:
52     - squid
53     - socat
54     - socat443

Listing 6 handles the internal VM's configuration. This can't start with the installation of Python, because apt first has to get to know the upstream proxy; this again happens in raw mode here.

01 - name: Debug1
02   debug: msg="{{ inventory_hostname }}"
03
04 - name: Set Proxy
05   raw: test -e /etc/apt/apt.conf.d/80proxy || (sudo sh -c 'echo "Acquire::http::proxy \"http://{{ hostvars['localhost']['frontendvm']['instances'][0]['private_ip']  }}:3128\";" >  /etc/apt/apt.conf.d/80proxy')
06
07 - name: Install Python
08   raw: test -e /usr/bin/python || (sudo -s apt-get -y install python)
09
10 - name: Install Software
11   become: true
12   become_method: sudo
13   apt:
14     name: "{{ item }}"
15     state: present
16     cache_valid_time: 86400
17   with_items:
18     - openvpn
19
20 - name: Set OpenVPN Name
21   become: true
22   become_method: sudo
23   lineinfile:
24     path: /etc/default/openvpn
25     regexp: "^AUTOSTART"
26     line: "AUTOSTART=\"{{ hostvars['localhost']['netname'] }} {{ hostvars['localhost']['netname'] }}-443\""
27
28 - name: Send SSL Files
29   become: true
30   become_method: sudo
31   copy:
32     src: "{{ item }}"
33     dest: /etc/openvpn
34     owner: root
35     mode: 0600
36   with_fileglob:
37     - "files/*"
38
39 - name: ccd-dir
40   become: true
41   become_method: sudo
42   file:
43     dest: /etc/openvpn/ccd
44     state: directory
45     mode: 0755
46     owner: root
47     group: root
48
49 - name: clientfile
50   become: true
51   become_method: sudo
52   template:
53     src: templates/clientfile.j2
54     dest: /etc/openvpn/ccd/clientfile
55
56 - name: OpenVPN Group
57   become: true
58   become_method: sudo
59   group:
60     name: openvpn
61     state: present
62
63 - name: OpenVPN user
64   become: true
65   become_method: sudo
66   user:
67     name: openvpn
68     state: present
69     groups: openvpn
70     system: yes
71
72 - name: Configure Openvpn
73   become: true
74   become_method: sudo
75   template:
76     src: templates/openvpn.conf.j2
77     dest: "/etc/openvpn/{{ hostvars['localhost']['netname']}}.conf"
78
79 - name: Configure Openvpn
80   become: true
81   become_method: sudo
82   template:
83     src: templates/openvpn.conf-443.j2
84     dest: "/etc/openvpn/{{ hostvars['localhost']['netname']}}-443.conf"
85
86 - name: Enable and start service
87   become: true
88   become_method: sudo
89   systemd:
90     name: openvpn
91     enabled: true
92     state: restarted

In terms of software packages, the role only installs OpenVPN and configures two instances: one on the UDP port and one on TCP/443. These instances set an entry in the /etc/default/openvpn file to AUTOSTART. The last play in the listing enables and starts the OpenVPN service. The SSL keys in this example were already preconfigured. It is up to the admin to generate and upload certificates with Ansible.

Objective Achieved

Extending your data center with resources from the cloud can significantly reduce total costs. You'll mainly see this benefit if you need to purchase very expensive services or the services are only occasionally needed, such as additional instances of your own web store or web server for a foreseeable customer rush.

However, to ensure that the cloud services – unless currently desired – are not freely available on the Internet, the infrastructure allows AWS to be connected almost like your own data center. However, for temporary resources, a manual assembly and disassembly would be labor intensive. Automation with Ansible, as shown in this article, or with something like the HashiCorp Terraform tool, lets the admin quickly deploy and dismantle a data center infrastructure.