Principled

Configuration management is the term used to describe the most automated and uniform management of systems possible with standardized tools. By means of special configuration languages, you feed your tool of choice instructions designed to convert target systems to the desired state and keep them there, taking the burden of implementing system management off the system administrator's shoulders.

Installing and configuring software, starting processes and services, creating users and groups, setting file permissions – the system administrator defines all these tasks once only and then lets the tool do the work. If a system deviates from the defined standard, configuration management straightens it out again. In this way, even small teams can automate the process of managing large system landscapes.

The most frequently used automation tools of this type include the well-known solutions Puppet and Chef, as well as Ansible, SaltStack, and, to a certain extent, Terraform. CFEngine 3 is less well known, although its predecessor was considered the first of a genre.

Credit Where Credit Is Due

For a long time, system administrators wrote scripts to manage a large number of systems uniformly. CFEngine was the first to set the principle of centralized and standardized configuration in stone as a software product. As early as 1993, Mark Burgess at the University of Oslo described CFEngine as a tool that enabled the administration of numerous systems with the help of an abstraction and a kind of configuration language. Growing use eventually resulted in the far more mature version 2 in 2002, which because of its reliability is still in use in some companies today.

As with other open source projects, the CFEngine project became increasingly commercialized with wider enterprise use, leading to the formation of a support and consulting services company in 2008. One year later, CFEngine 3 [1] was released, partially abandoning backward compatibility with its predecessor and available for the first time in a community edition and in an enterprise edition that focuses on enterprise customers with features such as reporting and a web interface. However, the free version also contains all the important functions for daily configuration management. The commercial edition is made more palatable because you can try out the software and manage up to 25 systems free of charge.

Design Principles

The CFEngine design is based on several principles that run like a thread through the structure and use of the tool. The system administrator uses a declarative approach to define the desired end state of a system in a configuration language. How this is achieved is usually not given in detail. The instructions are deposited on a central system from which the computers to be configured can themselves download the latest versions (pull principle).

CFEngine automatically executes all necessary steps locally, thereby abstracting (i.e., converting) the concrete instructions for the system description at the command line. An example of a command would be something like: Make sure the Nginx web server is installed and started. The maintainer does not have to give a specific command like yum install nginx because CFEngine maintains a library of commands for each supported operating system, which it uses automatically when implementing the instructions.

Promises

The most important principle of CFEngine is the Promise Theory [2] and is intended to solve the problem of system administrators not always being sure whether a target system is currently in the desired state. When in doubt, the admin feels compelled to log in manually to view the state and correct it if necessary. After some time, repeating this time-consuming undertaking becomes a real risk: Other users could have (involuntarily) changed the system so that it no longer reliably fulfills its original purpose.

The Promise Theory model now describes how this permanent uncertainty can be resolved efficiently by equipping each system with an agent that is controlled by a central server and from which it obtains instructions, or promises. According to the creators of the software, pretty much everything in CFEngine 3 is a promise. A promise means that an autonomous system announces its intention to change itself into the desired state on the basis of the instructions received – even if, for example, the central server with the instructions cannot be reached at the moment. Of course, the promise does not contain a certainty of success, but only the intention to get as close as possible to the desired state (if necessary, after several iterations).

What sounds quite weird at first has tangible benefits. In addition to the description of the state, the system administrator can store instructions on what should happen if the desired state is not in place at the moment or cannot even be achieved in principle. In the best case, the CFEngine agent repairs the system and restores the desired state. In the worst case, it cannot sort out the system and resorts to the instructions the administrator has stored for such cases.

Of Promises and Policies

The smallest self-contained and executable unit in CFEngine is the promise, which contains at least one concrete statement (e.g., make sure an account is created). Several of these statements are then bundled into a policy.

The name reveals how CFEngine 3 ticks: Since version 1, the CFEngine makers have considered systems to be entities that are supposed to comply with enterprise policies. Logically, a group of instructions (policies) then enforces the individual measures (promises). If you need to bundle multiple policies, you can use the next higher grouping form, which is conveniently named bundle.

Toolkit

CFEngine 3 does the usual work that you might expect from a configuration tool. It can obtain information about a target system and its state and modify text files in various formats. It can set file permissions and ownership, as well as POSIX access control lists (ACLs).

Users, groups, firewall entries, processes, services, launching third party programs – everything needed for system administration under Linux (CentOS/RHEL, Debian, Ubuntu, SLES), Unix (AIX, HP-UX, Solaris), and Windows (Enterprise variant only) is included in CFEngine 3 by default. Additionally, it supports numerous advanced operations such as querying and modifying databases. Thanks to a connection to VMware, KVM, Xen, and VirtualBox, virtual machines can also be managed by the standard statements in promises.

The declarative approach makes CFEngine policies idempotent; they can therefore be executed as often as required and always achieve the same results. Permanent monitoring of the local system by the CFEngine agent ensures that changed states are detected and corrected.

Setting Up CFEngine 3

To run CFEngine 3 you need to install the central server (Policy Server or even Policy Hub) and at least one agent; these components [3] can run on the same system for test purposes. In regular operation, the policy server distributes its policy files to agents running on different systems. (Figure 1).

Because I always use Linux, I set up a small test environment for this article (Table 1) and carried out the installation manually in line with the official instructions for the community edition [4]. The linuxmag account was set up and given Sudo rights during the operating system installation on all systems.

The necessary software packages are available directly from the manufacturer's website [5]. An installation from the distribution repositories is not recommended because they often contain outdated software versions. The CFEngine 3 package in the Ubuntu 20.04 repositories, for example, was incomplete and poorly maintained at the time of testing.

CFEngine v3.15.3 was installed on all participating test systems (Listings 1-3). Corresponding packages are available for download, even though the online documentation of CFEngine 3 does not list Ubuntu 20.04 or CentOS 8 in the list of supported platforms, probably because the online documentation was simply not adapted after its release; the respective previous versions of both distributions can be found in the list.

$ sudo wget https://cfengine-package-repos.s3.amazonaws.com/community_binaries/Community-3.15.3/agent_ubuntu18_x86_64/cfengine-community_3.15.3-1.ubuntu18_amd64.deb
 **
$ sudo apt install ./cfengine-community_3.15.3-1.ubuntu18_amd64.deb
 **
$ sudo cf-agent --bootstrap 192.168.38.131
R: Bootstrapping from host '192.168.38.131' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This host assumes the role of policy server
R: Updated local policy from policy server
R: Triggered an initial run of the policy
R: Restarted systemd unit cfengine3
  notice: Bootstrap to '192.168.38.131' completed successfully!

$ sudo wget https://cfengine-package-repos.s3.amazonaws.com/community_binaries/Community-3.15.3/agent_ubuntu18_x86_64/cfengine-community_3.15.3-1.ubuntu18_amd64.deb
$ sudo apt install ./cfengine-community_3.15.3-1.ubuntu18_amd64.deb
$ sudo cf-agent --bootstrap 192.168.38.131
  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
  notice: Trusting new key: MD5=d67ad40160db5f79a616eea18bb9073c
R: Bootstrapping from host '192.168.38.131' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
R: Updated local policy from policy server
R: Triggered an initial run of the policy
R: Restarted systemd unit cfengine3
  notice: Bootstrap to '192.168.38.131' completed successfully!

$ wget https://cfengine-package-repos.s3.amazonaws.com/community_binaries/Community-3.15.3/agent_rhel8_x86_64/cfengine-community-3.15.3-1.el8.x86_64.rpm
 **
$ sudo yum localinstall cfengine-community-3.15.3-1.el8.x86_64.rpm
 **
$ sudo /var/cfengine/bin/cf-agent --bootstrap 192.168.38.131
  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
  notice: Trusting new key: MD5=d67ad40160db5f79a616eea18bb9073c
R: Bootstrapping from host '192.168.38.131' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
R: Updated local policy from policy server
R: Triggered an initial run of the policy
R: Restarted systemd unit cfengine3
  notice: Bootstrap to '192.168.38.131' completed successfully!
 **
### Open port 5308/TCP on the local firewall
$ sudo firewall-cmd --zone=public --add-service=cfengine

After installing the Policy Hub and agents on the lab systems, I logged in as the linuxmag user on host cf3-ubsrv and carried out a short connection test (Figure 2) with the command:

# /var/cfengine/bin/cf-net -H 192.168.38.131,192.168.38.132, 192.168.38.133 connect

CFEngine 3 best practices dictate executing all the commands with the root account, which the listings that follow take into account. CFEngine 3 starts several processes per system, all of which perform different tasks. A short overview can be found in the official documentation [6].

Getting Started

The initial setup of CFEngine 3 took place with the bootstrapping section at the end of the listings for installation on the various systems. Among other things, sample configurations were copied from the installed package, and important directory structures were created under /var/cfengine/. The predefined contents in detail are described in the online documentation [7].

On Policy Hub cf3-ubsrv, I stored several of my own CFEngine files with instructions in /var/cfengine/masterfiles/. The folder there already contained files that CFEngine created automatically. For the sake of order, CFEngine 3 wants you to store DIY-created policy files in /var/cfengine/masterfiles/services/. If you want a policy file to be executed automatically, as I did in this article, you need store it in /var/cfengine/masterfiles/services/autorun/.

All files stored on the central Policy Hub at /var/cfengine/masterfiles/ (Figure 3) download the agents at five-minute intervals by default and make them available locally at /var/cfengine/inputs/. Following the principle of infrastructure as a code, the developers recommend that the content of /var/cfengine/masterfiles/ be placed under version control through SVN or Git.

To illustrate how CFEngine 3 works, I introduced a simple policy that installs the lightweight Nginx web server on the target systems and enables the associated systemd service, starting it at the same time. To do this, I launched a text editor on Policy Hub to create the /var/cfengine/masterfiles/services/autorun/webserver.cf file and filled it with life (Listing 4). The "webserver.cf in Detail" box provides more information. The command

# cf-promises -f /var/cfengine/masterfiles/services/autorun/webserver.cf

bundle agent nginx_installed_running {
  meta:
    "tags" slist => { "autorun" };
  vars:
    "package_list" slist => { "nginx" };
  packages:
    "$(package_list)"
      package_policy => "add",
      package_method => generic;
  services:
    "$(package_list)"
      service_policy => "start";
}

checks the content of the file for errors.

To enable automatic execution of your own policy files on the Policy Hub, create the def.json file under /var/cfengine/masterfiles/ to configure the services_autorun option (or, more precisely, the CFEngine class):

{
  "classes":
  {
    "services_autorun": [ "any" ]
  }
}

The comments in the /var/cfengine/masterfiles/promises.cf file show its effect.

In the default configuration, the CFEngine agent on the target systems implements the promises every five minutes. The command

cf-agent --dry-run --inform

shows what it will trigger the next time it is executed automatically (Figure 4).

After the next automatic execution, the installed nginx package and the matching service, enabled and running, can be found on the target systems (Figure 5), but on the cf3-centcli lab system, systemd seems to have failed to start Nginx because of an Nginx configuration problem, which I did not investigate further. CFEngine 3 will always try to start Nginx on the CentOS system in the future.

If you want to delve deeper, you can now add instructions to the webserver.cf file about what to do in case of unfulfillable promises. Also, Nginx can only be reached from other systems by opening the local firewall. Configuring this for two different distributions by promise is not so easy and another possible exercise for the future.

Documentation and Community

In this article, custom policy files were included and executed by the autorun method. However, sys admins have other ways to activate bundles and policies [10].

The use of CFEngine 3 gets more exciting with the use of preconfigured bundles from the creators and the community. You can find these several places on the web – the CFEngine Design Center [11] with its Sketches being one of the best known. Ready-made bundles are intended to facilitate the tasks for newcomers and act as templates for your own configuration snippets.

Somewhat confusingly the CFEngine documentation appears to be distributed several places. The official website http://cfengine.com has a Quickstart Guide with mini-documentation, whereas http://docs.cfengine.com has the official documentation. An archive [12] also houses a hodgepodge of old manuals, some of which are easier to read than the current documentation.

If you want to go further into CFEngine 3, you have to allocate plenty of time. The approach seems a bit unwieldy in places, and the comparatively small community makes it difficult to get good learning resources or even help online. However, you can find a few good books. The members who are involved in the community seem to have been around for a long time and know all the tricks. In researching for this article, I often stumbled across the same names I did in 2012 and 2013 when researching previous articles.

Strengths and Weaknesses

CFEngine shows its strengths wherever admins need to manage a large number of heterogeneous systems with the smallest possible footprint. According to statements from the community, environments with a million or more agents can be realized, which is also attributable to the ability to cascade the central Policy Hub (e.g., with one hub per data center).

Once it's up and running, CFEngine 3 proves to be a powerful and lean machine that takes very little performance away from the applications on the target systems. Unlike Puppet or SaltStack, for example, the CFEngine agent also assumes virtually no dependencies. The ability to manage Linux, Unix, and Windows (in the enterprise variant) with a common tool proves to be an advantage in many environments.

The weaknesses of CFEngine 3 reveal themselves if getting started needs to be a fast and simple process, which would presuppose a large number of up-to-date libraries with prefabricated configuration snippets. Moreover, CFEngine 3 lacks the popularity of Puppet and the like, which will slow down the influx of new users and, in turn, new cash flow to fund development.

However, if you choose to get involved with CFEngine 3 and give the tool some time, you can use it to build a solid and functional configuration management system. If you are eager to experiment, you can also take a look at a SlideShare presentation [13], in which the CFEngine makers demonstrate the orchestration of Docker containers with CFEngine 3.