Pre-Assembled

Special Thanks: This article was made possible by support from Linux Professional Institute [1].

Enterprise-grade automation solutions like Puppet and Chef are very powerful, but they require a huge amount of learning. Ansible has gained popularity as an alternative to Puppet and the other similar enterprise automation tools. The strict but simple Ansible syntax lets you define roles that are essentially self-documenting (Figure 1), which makes access far easier and ensures that you can integrate your existing knowledge faster.

Even though Ansible is simpler than Puppet or Chef, writing your own roles takes time. Wouldn't it be great if you had a library of prebuilt Ansible recipes for standard services that you could install in seconds by pressing a button?

DebOps [2] is a huge collection of prebuilt Ansible roles that you can use on fresh Debian systems.

What Is DebOps?

The collection of prebuilt Ansible roles currently totals more than 110. The authors also provide userland tools composed in Python for these roles. A tool called debops is part of the collection; you call debops with the appropriate parameters to launch an Ansible playbook for the desired effect.

The debops tool helps you install and maintain your DebOps Ansible roles, thanks to the debops-init and debops-update commands, whereas debops-defaults helps you find the correct configuration for your system, and debops-padlock makes it possible to encrypt and store sensitive passwords. The debops-task command is a wrapper for the ansible command. Note that after debops is run once and ansible.cfg is generated, you can use the ansible command directly.

The best way to approach DebOps is quite simply learning by doing, but be careful: The solution may make your life easier, but it comes with some pitfalls that you need to avoid at all costs. Having console access to a host in the beginning (either physical, through out-of-band VNC access to the VM console, or through the attach command of a container) can be an useful debugging tool. Also, check out the project's Getting Started guide online [3].

DebOps – Hands On

To get started, you'll need at least two Debian systems that are newly installed and are still plain vanilla. In theory, this example would also work with a single host, but that would not be very relevant to real-world IT practice: Tools like Ansible are typically used to automate the configuration of multiple systems. This example is based on the assumption that Ansible is running on a central automation host, which logs on to other systems via SSH.

DebOps is designed for Debian, but it is also supposed to work with the Debian-based Ubuntu. (Most of the packages on both systems are similar and many system tools have the same name.) Our tests revealed some issues with the Ubuntu support at various points, although other tasks completed successfully on both Debian and Ubuntu systems.

The first step is to install Ansible. It is included with the latest Debian and Ubuntu versions, but the default versions are naturally a bit outdated. It is better to pick up suitable packages [4] from Ansible upstream to make sure you have the latest version for your own system. You can normally install the packages using your own package management system.

Once you have set up Ansible, you can move on to installing DebOps. A prebuilt DebOps package exists for Debian, but it is only in the experimental branch and is fairly ancient. The DebOps developers recommend installing DebOps using the pip Python package tool [5]. The command is

pip install debops

which puts DebOps in /usr/local/bin/debops, where it is now ready for use.

The next step largely depends on whether you supervise your hosts in a team with other administrators or whether you work on your own. If you set up DebOps as a normal user, all the configuration files end up in your home directory. If you want co-workers to benefit from DebOps, as well, define the following parameters in a file named /etc/debops.cfg:

[paths]
data-home: /usr/local/debops

The /etc/debops.cfg file is useful only if you plan on having multiple users and you want all of them to use the same codebase; otherwise it's not needed.

Now you need to run the debops-update command from a regular user account. The command downloads all the debops-playbooks available on GitHub for Ansible and stores them in ~/.local/share/debops/debops/. This is followed by debops-init with the destination directory for the Ansible configuration.

The Ansible configuration is not in the data-home directory in DebOps. If you work as root, you can call the debops-init <local-setup> command below /root. The rest of the path is the typical folder structure that Ansible expects (Figure 2).

As the administrator, you will want to pass in many configuration parameters to DebOps along the way. The debops command is a wrapper for the ansible-playbook command, and it ultimately calls ansible with the correct parameters. Ansible expects ansible.cfg, which DebOps generated previously from the available configuration values. After debops is run once and ansible.cfg is generated, you can use the ansible command directly.

Make sure you do not change anything in the ansible.cfg file created with the ansible command, because DebOps will overwrite any changes later. Instead, /etc/debops.cfg or (preferably) .debops.cfg are available for configuration in the directory previously created by debops-init.

Connection Required

Once Ansible is set up on the configuration server, the next question is how to set up the connection with other servers (i.e., the ones you want DepOps to configure). One thing is clear: Ansible does not rely on its own client-server protocol but uses the well-established SSH to perform commands. The next step is to set up Ansible and the other servers so that the login works.

One option is to create an unprivileged user account, which is then given root privileges via sudo. However, this approach is a little time consuming, because you then need to enter the sudo password manually each time you run Ansible, unless you want to store it in the DebOps configuration. It is also possible to enable sudo without the user's password and instead manage access based on SSH keys.

On paper, this solution has significant advantages; the login is carried out via an SSH key that doesn't require a password. Because SSH keys are already standard in most setups, you immediately switch off the password login via SSH.

If you want to take the risk, you could theoretically generate an SSH key with an empty password and store its public part on all target servers for root with ssh-copy-id. In .debops.cfg in the configuration directory, the [ansible defaults] section then needs to be extended to include the remote_user = root entry. This ensures that Ansible actually uses the root account.

All hosts in the Ansible inventory layout used by DebOps should be included in the [debops_all_hosts] inventory group so that a common configuration can be applied. The current naming scheme for group names is [debops_service_<role_name>]. Therefore, to enable PostgreSQL server support on a host, it needs to be included in the [debops_service_postgresql_server] inventory group:

[debops_service_postgresql_server]
server1.example.net

From previous painful experience, the DebOps developers have learned that they need to check the condition of the target systems. The DebOps documentation recommends you enter the command

debops-task all -m setup

after you set up the connection and the inventory file.

Tricks for the Test

DebOps has not yet generated the necessary ansible.cfg file; after all, you have not even called debops thus far. A crude hack helps: Enter debops immediately followed by Ctrl+C to cancel the operation as soon as possible. Then, you can launch ansible.

The reason for canceling the operation is that the debops command would start running and use all the playbooks on the hosts for which they are configured, which is not what you want at this stage.

One thing that is very useful while working with DebOps is to ensure that the hosts have a properly configured DNS domain; you can check this by running the dnsdomainname command. If the domain is empty, most of the project should still work; however, more useful features like PKI infrastructure might not work correctly or at all.

An important feature in a multihost distributed environment is support for encrypted connections. The DebOps project provides this support using X.509 certificates and internal Root CA out of the box, including the free certificates of the Let's Encrypt project.

A Nasty Surprise

The aim of this example is to install a working instance of PostgreSQL, such as the basis for a classic LAPP setup, on at least one of the servers. Once the setup step for ansible has completed successfully, you have one more problem to deal with before you get to work.

During a normal run, Debops runs the common.yaml playbook on each host. This step configures certain basic settings and also ensures that certain basic packages are installed and that the apt configuration is identical; however, it also installs the ferm (For Easy Rulemaking) [6] and TCP Wrapper tools. These tools nail down the host with iptables so that it is no longer possible to log in remotely. Ansible thus locks itself out.

DebOps is designed from the ground up for a production environment, and roles assume that it's correctly configured to allow Ansible Controller to make connections over SSH. The project itself tries to ensure that possibility by checking the source IP address from which the connection is coming and adding it to the firewall and TCP Wrapper whitelist; however, the code is not perfect and lockouts can still happen.

Try to think of this as a "rite of passage" at this point. Because an unknown IP address (one not allowed by the firewall) has been blocked, it seems that the project is working as intended. Still, if you want to avoid the headache (e.g., in a development environment), one of the ways to deal with this might be disabling the firewall and TCP Wrapper support through the Ansible inventory with:

ferm__enabled: False
tcpwrappers__enabled: False

Turning off the firewall might affect some functionality, like setting up internal networks, but should be fine at first. After this step, TCP Wrapper still can't fully keep its hands off iptables, but at least it no longer installs the deadly catch-all rule.

Rolling Out PostgreSQL

After you work around the login problem, continue with the installation of the first real application: PostgreSQL. The packages for Debian and Ubuntu are meaningfully preconfigured so that you just have to lend a hand, and only in one place: You need to define the password for admin in the PostgreSQL database. To set a global password, you can add the parameter

postgresql_server__admin_password:'top-secret'

in ansible/inventory/group_vars/all/debops.postgresql_server.yml: All PostgreSQLs rolled out by DebOps are then equipped with this password.

If the hosts file is modified as described so that server1.example.net belongs to the debops_service_postgresql_server group, you just need to call debops from the DebOps configuration directory: DebOps then starts Ansible rolling out the desired service on the target host.

Administrators are – quite rightly – not usually happy about the passwords for critical services being stored in the clear on any host. Ansible even provides a vault for this purpose, which means that the passwords can be stored with GPG encryption. Only the owner of the GPG key can open the encrypted file and see the password when calling Ansible.

One of DebOp's greatest strengths is that it automatically generates random passwords to all important services. Using EncFS and the debops-padlock tool, you can store your password in encrypted form so that it can only be used with your GPG key. The root account passwords are randomized, and database passwords and others are all stored on the Ansible Controller host in the secret/ directory beside the Ansible inventory. Therefore, there's no need to set passwords for the services manually – unless you want to for some reason. You need to exercise caution: a password should be known to all stakeholding administrators. The developers reveal details about debops-padlock in the documentation [7].

The example described in this article looks at DebOps with PostgreSQL. Other Ansible roles work in the same way: By adding individual hosts to groups in the host file, you can determine which services should run on them. DebOps then makes the configuration available, globally or by host.

At this point debops-defaults is very useful: It displays all available configuration parameters for currently active roles. If you want to add something to the configuration of a service on a host, this command first gives you a general overview; you can then proceed to configure the settings as described.

You will find some real gems in the DebOps playbooks: You can quickly set up the entire LAMP stack, for example, with DebOps and PostgreSQL. Lesser known tools like Elasticsearch, Logstack, and Kibana (Figure 3) are no more a problem than Samba, Postfix, or SNMPD.

Secure Repository

The developers maintain the DebOps tools, as well as the corresponding playbooks, on GitHub [8]. The DebOps team initially consisted of just a handful of people, and because they all had some kind of relation to Debian, they were quite open to collaboration with other developers.

The developers were well aware of the need for security: If you want to check your own code into the GitHub directories, you first need to sign it using GPG. Only appropriately signed code can move into the GitHub directory. The project developers closely monitor the GPG keyring in which the keys are listed.

Basically, the creators have copied the workflow that Debian uses for its own packages: With Debian, only developers whose GPG key is included in the Debian keyring are allowed to upload new packages to the official Debian archive. To join up as a Debian developer or as a package maintainer, you need to add your own GPG key to the corresponding keyring.

The reliability of DebOps is thus similar to that of Debian: An attacker who gains access to the GPG key of an approved developer could theoretically cause a large amount of damage on the target systems, but if you rely on Debian or Ubuntu anyway, DebOps does not increase the risk.

Conclusions

DebOps takes a clever approach: After you install a system with the DebOps tools, you can quickly configure it to host a database, web server, or monitoring server. The prefabricated Ansible roles are extremely useful because they save you from the need to invest a large amount of time in writing your own roles.

The fact that DebOps is not totally intuitive spoils some of the fun. Another issue is, if you have never dealt with Ansible, you will need to plow through many pages of documentation to understand DebOps. DebOps provides good start-up support, but it ultimately does not avoid the need for administrators to establish their own Ansible expertise.

If you are looking for fast automation for Debian or Ubuntu, give the DebOps automation system a try.