Operation Pushbutton

When investigating DevOps, you will come across various definitions that ultimately refer to an agile work approach. DevOps, it's said, is ultimately quite simple: Admins take over responsibility for workflows and various processes from developers to manage systems more efficiently.

Companies realize that DevOps and agility are more than buzzwords when it comes to operating large platforms, such as public clouds, for which several assumptions from classic setups no longer apply. A setup with 10 nodes might be manageable manually and without automation, but if several hundred servers or more are part of the installation, conventional approaches won't get you anywhere.

Agile Work Needed

Large environments such as OpenStack pose a real challenge in terms of both development and maintenance. They comprise a large number of components that do not all follow the same release cycle. They are so powerful that they cover many aspects, including storage, network, and user management. They continue to evolve, resulting in the need for regular updates on a production platform, and the need to roll out new components that were not previously part of the operation.

They also require so many computers that it is impossible to maintain the platform manually because of the sheer number of managed systems. In short: In a large environment, the principles of agile work are a prerequisite for the platform to work well at all.

If you operate such a platform, you need to coordinate its components. Design documents help developers, operations, and architects plan new functions. Classic issue tracking makes it possible to track who is currently working on which function or problem, which is especially important for distributed teams. Automation is a prerequisite.

Such teams usually combine their automation solution with classic revision management. For example, Ansible playbooks can be managed easily and efficiently with Git. Once a new feature has been developed and tested, it is sufficient to check out the appropriate repository on the central Ansible server and start an Ansible run to roll out the change.

This kind of development also makes continuous integration easy: Each commit to the Git directory causes Jenkins to launch in the background and fires various tests at the new code. Possible errors are already noticeable during the development stage – not months later, when the change is finally implemented in a major update, as would be the case with the classic waterfall principle.

A Question of Tools

If you want to set up an agile, DevOps-style workflow, start by asking yourself what tools you require. Atlassian has established itself as a supplier of such tools. For example, many users use Confluence as a wiki to collaborate on design documents or to manage their documentation.

Jira is often added as an issue tracker, because it not only offers Scrum and Kanban, but other possibilities that not only manage tasks, but also visualize them. If you don't want to store your playbooks for Ansible on GitHub, you can choose GitLab, which allows a local centralized Git repository that can also be integrated easily into the previously mentioned continuous integration and continuous development concepts.

However, if you start with a plain vanilla Ubuntu 16.04 and want to roll out Confluence, Jira [1], and GitLab [2], you still have some work to do. Quite a few companies take this step without any automation at all, thus violating the automation principle in the process.

A lack of automation is a problem because, on the one hand, the setup is difficult to maintain and, on the other hand, it is virtually impossible to restore working Jira, Confluence, and GitLab instances with imported backups after a total failure. Therefore, when a working agile tool chain is needed most, it might not be available.

Ansible to the Rescue

The good news is that Confluence, Jira, and GitLab can be automated easily with Ansible. Prebuilt playbooks for almost every combination of a certain distribution and the tools for agile work, often including the latest releases, can be found online.

I was confronted with the task of building an agile work center that had to contain instances of Confluence, Jira, and GitLab. The goal was to install the three services automatically on a fresh instance of Ubuntu 16.04 by invoking a single Ansible playbook. The resulting playbook is the basis for this article.

Preparatory Measures

Before beginning the Ansible playbook, a little preparation is necessary. For this article, I assume that a newly installed system with Ubuntu 16.04 LTS already exists (e.g., as a virtual machine in a public cloud). In the example, Ansible runs on the same host that later hosts Confluence, Jira, and GitLab, so it is an all-in-one (AiO) setup.

Because Ansible connects via SSH, but ideally not as the root user, a separate user account is required. The example assumes that this user's name is ubuntu. If you start a virtual machine (VM) for Confluence and its associates in an OpenStack cloud and use an official Ubuntu cloud image for this purpose, you will discover this account in the image and not have to create it yourself.

Allowing users to log in with an SSH key is also useful, because it is more secure than logging in with a password, and the admin does not have to type in the password every time the ubuntu user calls Ansible. As usual, you just need to store the public part of the respective SSH key in the .ssh/authorized_keys file in the ubuntu user's home directory.

As an AiO setup, Ansible and Ansible Vault must be present on the system. You can handle this task by installing the appropriate packages directly from the Ansible repositories for Ubuntu 16.04:

sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible

Once the three commands [3] have run successfully, you can use the ansible command to see whether everything worked: The Ansible help text should appear.

Finally, allowing the ubuntu user to execute all commands with sudo, without entering the user password, is helpful. If the playbook is used in a cloud VM, this is already the case out the box: In this case, ubuntu has no password at all, so password-free sudo is inevitably the default setting.

If you are not replicating the example in a cloud VM, you can become root and run visudo to allow the sudo group to run commands without a password, and if necessary, you can use

adduser ubduser ubuntu sudo

to make sure the user belongs to the sudo group.

One of the final preparatory steps is to create the required entries in the DNS zone of the target domain. These should be present and propagated in the DNS system before the rollout of Confluence, Jira, and GitLab is due. The example assumes that the three services each have their own DNS entry and subsequently have their own virtual hosts in Apache.

The Ansible Playbook

To download the AiO playbook for Ansible, you should install the git package, so you can then check out the playbook directly from GitHub:

git clone https://github.com/madkiss/ansible-playbook-agile-aio

If the checkout worked, the folder in which the command was executed will now contain the ansible-playbook-agile-aio directory. The folder follows the Ansible recommendations for directory structures [4].

You will thus find several playbooks whose file names end in .yml, along with the devops.hosts file, which Ansible uses as an inventory file. In the vars folder is a file that defines general variables for each playbook. You will later create another file named devops-secret.yml in this folder, which stores different passwords of the setup, after encryption by Ansible Vault. In the roles folder are the roles and, indirectly, the playbooks that roll out the respective services directly on the host.

The playbook.yml playbook contains links to all other playbooks, enabling you to use playbook.yml in Ansible to roll out all required services in a single action (Figure 1).

In the rest of this article, I go through the individual playbooks, explain their meaning, and point out places where you have to adjust your local configuration.

Configuring the Inventory

Before working with the different playbook roles, you need to create a suitable inventory file for your setup. A template is already included in the AiO playbook in the devops.hosts file. The devops group contains a single host. In the first field of the file, enter the full hostname of the server on which the AiO setup will run.

The IP address of the target system belongs in ansible_host. The ansible_port and ansible_python_interpreter variables ideally remain unchanged: They cause Python to connect to the host on standard SSH port 22 and to use Python 3 when calling commands on the target system.

Next, check whether the devops.hosts setup has worked, and then check the admin with the ping command:

ansible devops -i devops.hosts -m ping

If the command executed successfully, Ansible is ready for use.

Specifying Passwords

Before the Ansible roles start working, you must configure the required passwords. Naturally, the thought of storing passwords in the clear in text files, which could later be openly visible in an internal Git repository, should make your hair stand on end.

Luckily, Ansible offers an alternative: It can use Vault to store secret information on the disk in encrypted form. When running ansible-playbook, simply pass in two more parameters – the playbook and --ask-vault-pass: Ansible prompts for the Vault password and then decrypts the files automatically.

Vault works well, but with a minor limitation: If the content from different files has to be encrypted, all Vault files must be protected with exactly the same password, which is the case in this example: The role that rolls out the SSL certificate on the target system also saves the SSL key that belongs to the certificate as a Vault file. Remember the password used for the first Vault container, because you will need it later for the SSL key vault, too.

However, Vault is first used to create a file called devops-secret in the vars folder in the AiO playbook.yml folder:

ansible-vault create devops-secret.yml

After entering a password of your choice, the editor stored in the $EDITOR variable opens. The file should ultimately contain three lines:

apache2_vhosts_pass: "<secret>"
mysql_root_pass: "<secret>"
apache2_vhosts_hash_salt: "<secret>"

The <secret> placeholder has to be replaced by a sensible password. Once the file has been set up accordingly, just save it and close the editor. If you want to see or change the content of the file at a later time, use one of the following commands:

ansible-vault view devops-secret.yml
ansible-vault edit devops-secret.yml

(Figure 2).

Basic Configuration

The Ansible playbook used here is based on the work of three developers: The majority was penned by Wong Hoi Sing Edison (hswong3i), who mapped the entire installation of Jira and Confluence in Ansible [5]. It is the backbone of the AiO playbook with minor adjustments and not only covers Jira and Confluence, but also allows idempotent configuration of various basic system parameters.

The basic playbook uses many of Edison's roles: In addition to the basic Apt configuration, it also rolls out configuration files for the hostname and the locales to be used. It also switches the system's time zone to UTC and polls NTP to get a reliable source of time. Additionally, it installs a basic iptables configuration that, of the privileged ports below 1024, only opens port 22 for SSH. Other Ansible roles extend the firewall configuration later – if necessary.

Now check that you can connect to the target system via Ansible and test whether the execution of Ansible playbooks really works by calling the basics playbook:

ansible-playbook -i devops.hosts -e @vars/devops.yml -e @vars/devops-secret.yml --ask-vault-pass basics.yml

The command might take some time to execute because Ansible installs many packages on the future agile host when executing the roles in basics.yml. If the playbook runs without error, Ansible reports this at the end and outputs 0 as a return value. This completes the basic system configuration; the specific roles for Confluence, Jira, and GitLab are next.

Many Requirements

When installing Confluence, Jira, and GitLab manually, the various prerequisites expected by the tools are often overlooked. For example, Jira and Confluence require Java and MySQL, as well as the MySQL Java Connector. GitLab relies on PostgreSQL in the Omnibus edition, but at least it installs it itself.

SSL also plays a role: The three services usually run on the same host and can share the same IP but should, of course, support HTTPS on port 443. However, this setup can only be achieved by way of a web server with different virtual hosts that also serves as an SSL terminator. Manually installing Confluence, Jira, and GitLab with all their dependencies and the necessary configurations on top, often turns into a tedious task.

The present AiO playbook takes care of the dependencies and is also designed to set up Apache as a terminating SSL proxy. Jira, Confluence, and GitLab each configure their own virtual hosts via the Apache host role, which Edison also wrote.

After successfully rolling out the basics playbook, create the prerequisites for the three services by calling the java.yml, apache.yml, and mysql.yml playbooks with a command like the one used for the basics playbook. Afterward, Apache, MySQL, and Java should be running on the AiO host for Confluence, and Jira should be installed. After that, it's down to the nitty gritty – that is, the deployment of the desired services.

Launching SSL Certificates

Edison also contributes a ready-to-go Ansible role for Jira, but unlike the previous playbooks, you have to take an additional step: Several placeholders in the jira.yml file of the AiO playbook need to be replaced with actual values – in particular, the apache2_vhosts_server_name for the hswong3i.jira role and apache2_vhosts_server_name for the hswong3i.apache2_vhosts role (Figure 3).

Note that jira.yml calls a role named madkiss.certificate, which is a rudimentary role that I wrote myself; it drops the SSL certificate, the appropriate SSL key, and an SSL CA or SSL intermediate certificate into the correct place in the system. Before you can roll out jira.yml, you need to equip this role with the required files and configure them.

The madkiss.certificate role is based on three required files: key.pem.enc, an Ansible Vault file that contains the SSL key for the certificate; certificate.pem, the certificate itself; and intermediate.pem, which contains any certificates required to establish a direct trust path between the CA certificates in the browsers and a provider's certificate. Because most CAs now work with sub-signing keys, this step is essential.

The three files belong in the roles/madkiss.certificate/files folder in the AiO playbook. Entering

ansible-vault create key.pem.enc

creates the key file as described above; in doing so, be sure to use the same password as before when creating devops-secret.yml. The other two files are simply stored in plain text at the same location.

The rest works automatically: madkiss.certificate stores the three files in the standard configuration in /etc/ssl, and the other roles are configured by default to search for the certificate, intermediate certificate, and key in the same locations (Figure 4).

Postfix as a Prerequisite

Confluence, Jira, and GitLab share one property: They all communicate heavily with registered users by email. Without a working mail setup, you can't even add user accounts, because the initial user access data is delivered by mail for all services.

The playbook thus contains a postfix.yml file that configures Postfix on the target host and allows mail dispatching through a mail relay. You should create the appropriate configuration and roll out the playbook with the ansible-playbook command in postfix.yml. The Postfix role in the playbook doesn't come from Edison, but from the Dutch provider Oefenweb.

Incidentally, the Postfix playbook configures Postfix to work as a relay host for local services, which allows GitLab, Jira, and Confluence to be configured as mail servers on 127.0.0.0.1. Authentication is not a prerequisite for the service, but it does require that the mail server is not accessible through the usual SMTP ports – a firewall prevents access.

Jira and Confluence

Once the three files for madkiss.certificate are placed and you have defined the appropriate parameters in jira.yml, run the familiar ansible-playbook call as shown earlier, but this time with jira.yml as the target. This may take some time: The playbook loads the Jira software, rolls it out, stores an account for the service in MySQL, and configures Apache.

Once the Jira setup is working, the confluence rollout is handled in the same way: In confluence.yml, replace the values for apache2_vhosts_server_name and apache2_vhosts_server_name, as for jira.yml, and then call ansible-playbook again.

Setting Up Tools

After rolling out Confluence and Jira, it is important that you connect to the installed services via HTTPS immediately after the ansible-playbook call. Both initially greet you with the configuration wizard. Because this wizard only sets up the admin password, the page is open to anyone without a password. Using Confluence and Jira together, it makes sense to run the user database only in Jira and connect Confluence to Jira (Figure 5). During the initial setup, Confluence and Jira also ask for the license, which you should ideally have at hand.

The agile work center is almost finished: The only thing missing is GitLab. A separate playbook (gitlab.yml) rolls it out, but again you need to adjust different values up front – in particular, for the mail configuration that GitLab will use later on and in the usual entries for managing the Apache host configuration.

Finally, a last call to ansible-playbook rolls out the Omnibus edition of GitLab. GitLab also comes with its own configuration dialog, which you should close immediately after deployment.

Meaningful Approach

The presented setup with Confluence, Jira, and GitLab has several advantages. First, it's idempotent: You can thus call the individual playbooks or playbook.yml again at any time to restore the default setup. Any changes to the setup should first be implemented in Ansible and then rolled out.

The three services get along well: As mentioned before, Jira can serve as a user database for Confluence. If you want, you can connect GitLab and Jira so that GitLab commits lead to log entries in Jira issues; also, Jira tickets can be referenced directly in Confluence pages. All these features are very useful for distributed teams that want to stay agile.