Robot Admin

Ansible and SaltStack, with their ability to create entire OpenStack and Ceph setups on their own, have become hot topics in automation and are the focus of major distributors. Just a few years ago, neither were on the radar; instead, Puppet and Chef were vying for the admin's favor.

However, they are far from history: Puppet and Chef continue to enjoy a large fan base, and both tools are still under active development. The new 2.0 version of Chef Automate, for example, officially celebrated its premiere in front of an audience at the Chef Conference in May 2018.

As usual, the manufacturer is not stingy with its promises: Chef Automate 2.0 vows to make everything better, faster, more convenient, and, of course, more automatic than its predecessors. In this article, I take a close look at the innovations in version 2.0 of Chef Automate.

What Is Chef Automate?

Many admins may have already met Chef in one form or another, and most think intuitively of Chef as the tool responsible for automation. In fact, Chef is still the core of the solution – but Chef Automate includes significantly more components.

First introduced in mid-2016, Chef Automate was intended to add various automation functions, with the manufacturer thus promoting the platform as a continuous automation tool; the obvious implication of continuous integration and continuous delivery (CI/CD) is no coincidence.

Automate now comprises three tools. All are open source software and, in principle, handle their tasks independently of each other. InSpec was added to Chef Automate to check compliance measures on systems. Automate also includes Habitat, a CI/CD framework that allows developers to develop, build, and distribute their applications fully automatically. Along with Chef, these components result in a complete automation toolchain.

Version 2.0 adds another component, Chef Workstation, which is intended to bring the many functions of the major Chef solution to the local computer and make life easier for both developers and admins.

Additional components include various dashboards and analysis tools that glue everything together, graphically display the state of the entire environment, and show what is going on at a glance. Remarkably, it is this glue that sets Automate apart from the simple combination of the three components described above, and it is precisely this part that is undergoing fundamental restructuring in version 2.0.

Chef developers noticed that DevOps practitioners have a vested interest in shifting new features and functions into production as quickly as possible, this being a central premise of DevOps. Instead of developing a new function behind closed doors for weeks and months and rolling it out as a large release with troubles to match, DevOps follows the principle of taking small steps and rolling out features to production as soon as they are available and tested. Many companies that follow the DevOps paradigm prioritize this goal.

The scope for interpretation between theory and practice becomes apparent every day in IT departments. In many cases in DevOps practice, very little remains of the noble goal of fast releases. The end result is an operating mode that has certain agile components, but to a large extent does not offer the flexibility that the stakeholders hoped for during implementation. Chef supports this claim with concrete figures, showing that many companies that have introduced DevOps principles have had only partial success [1].

From System to App

Chef attributes some of this failure to thrive to the toolchains companies use for their projects. Especially when it comes to Chef, you need to clean up at home first. Chef has always assumed the system to be the smallest unit to be maintained. The dashboards available in Chef Automate, for example, provided information on the status of individual systems. However, it was not possible to monitor the roll-out of individual applications.

Chef Automate was also not particularly well interwoven with other solutions from the CI/CD environment. Historically, when Chief development began several years ago, the maintainers focused on finding automation solutions for those tedious day-to-day system administration tasks that, up to that point, had been done manually in batches or pieced together in scripts.

The idea that automation could play a role in the CI/CD context did not develop until the advent of Docker and the like, when containers became the focus of interest. In such scenarios, you no longer need to roll out the application using an automation tool. Instead, you can use the automation tool to provision the application as a container. However, Chef offered very few benefits in these cases, although in version 2.0 of the software, this is expressly intended to change.

Complete Redesign

Chef Automate 2.0, say its developers, has been completely rewritten and now has a Go-based architecture. Also, version 2.0 now relies on a microarchitecture in which several components interact, which means that the individual components are now – in the best cloud style – RESTful API interfaces that can be controlled externally through standardized protocols. At the same time, the administrator will be able to manage all the processes of the various Automate components in a uniform interface.

The numbers Chef handles in this context are quite impressive: Chef Automate 2.0 is said to be able to control and process several tens of thousands of nodes, and it will be completely irrelevant where they run in the future. Whether on bare metal or as virtual machines (VMs) in a public cloud, Chef Automate 2.0 knows how to handle all the layouts.

The completely redesigned GUI that most users will use for their daily work with Automate 2.0 is a web interface (Figure 1). A kind of news ticker informs quickly and clearly which actions are taking place on the platform. If something goes wrong, a clear warning appears. Trending graphs and a separate query language are also included in Automate 2.0.

Chef Automate 2.0 also includes various compliance functions, which I discuss later in this article. In the future, Chef Automate will help find compliance violations as early as possible through a "detect, correct, automate" process under a single plane of control.

Applications to the Fore

When considering the various changes to the architecture already described, especially the switch to REST APIs, you might wonder about the reasons behind these changes. The manufacturer gives a clear answer: Whereas Chef Automate was previously a tool for managing large server farms, version 2.0 focuses on the application.

With the open, clearly defined APIs, Chef Automate will connect far more easily with external services such as Jenkins or GitHub. For example, if you push code to GitHub, you can specify in Chef Automate 2.0 that it automatically builds new container images, which can even be rolled out automatically, if required. Life cycle management thus goes far beyond the limits of the operating system, which in previous versions of Automate was the logical limit of Chef's responsibility.

Meanwhile, the developers have also made Chef Automate fit for working with cloud environments, especially the top dogs Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

Chef Workstation

A product named Chef Workstation joins Chef Automate 2.0 for the first time. Until now, anyone who wanted to use Chef Automate usually started by setting up a Chef server – that is, a separate piece of hardware, whose only task was to be the master instance for the many clients in the setup. Moreover, it was necessary to roll out Chef clients on the target systems, because Chef is based on the server-agent principle of communication. The whole hullabaloo required a huge amount of planning and took quite a while.

Chef Workstation puts an end to this: The solution is a kind of mini-Chef server with all the required tools that can be rolled out on the local admin system. Up to now, Chef relied on the Chef agent described previously running on the target systems; this setup is now replaced by chef-target. As long as you have an option for talking to the target systems directly (e.g., via SSH), you can also run Chef code on the systems directly. A whiff of Ansible is discernible at this point, which might confuse some die-hard Chef admins.

In the end, however, the effort pays off: If you want to try out Chef or are looking for a quick way to roll out Chef in a small setup, you will reach your goal quickly with Workstation. What's more, systems that have been rolled out by Chef Workstation can then be easily taken under the wings of Chef Automate. This tool is primarily aimed at DevOps developers who want to set up an appropriate environment and later put it into regular operation.

A View to Compliance

For some time now, the focus at Chef has been increasingly on compliance. This is particularly important in large companies: Admins often find a rigid set of rules based partly on public standards and partly on in-house regulations.

However, automation offers a very good springboard for not only enabling compliance retrospectively, but making it an integral part of the entire deployment scenario, whether for the maintenance of physical machines or the roll-out of applications in the form of containers. This is probably why InSpec, a tool originally built by VulcanoSec [2] to monitor compliance standards automatically, was acquired by Chef.

Earlier versions of InSpec discussed in ADMIN [3] proved to be extremely powerful. With the use of a special syntax, you define the criteria that must be fulfilled on a system. At the same time, you assign scores for deviations from the standard in the InSpec configuration. InSpec then automatically checks whether the condition is met and sounds an alarm if a certain score is exceeded (Figure 2).

Chef incorporated InSpec because InSpec is an excellent addition to a complete automation solution like Chef Automate. The goal is clear: Compliance violations and security issues need to be addressed at the automation solution level before they are rolled out to production systems. If an admin checks in a configuration file or code that violates applicable compliance rules to a Git directory, Chef Automate refuses point blank to start the rollout.

InSpec also monitors the running systems. For example, if a newcomer to administration makes a manual change to the configuration on a server that might expose the system to security risks, the red light on the Chef Automate dashboard lights up (Figure 3).

Compliance for the Cloud

In Chef Automate 2.0, the developers have continuously expanded the interaction of their platform with InSpec. Probably the biggest change is that InSpec can now also check cloud environments and the configuration made by the admin for compliance problems. Previously, only local systems could be tested with InSpec, but now the service offers a configuration option for the access credentials for AWS or Azure.

If you enter the access credentials, InSpec logs directly into the public cloud and examines the environment it finds there according to the defined compliance criteria. Corresponding functions for GCP are also available, although they are still listed as beta in the current InSpec version.

At the same time, the InSpec developers have significantly expanded the functionality of the solution. A resource in InSpec is a kind of prebuilt check for various criteria, such as the configuration of the Apache web server. More than 30 new resources have been added to InSpec in Chef Automate 2.0, such as support for Cisco IOS (originally Internetwork Operating System) devices. On top of that, the developers have cleaned up InSpec and now promise far quicker execution of the tests.

What is impressive about Chef Automate 2.0 is how seamlessly InSpec is integrated into the various work steps of the platform. Depending on the configuration, Automate uses InSpec to check every single step of a process; if you point the tool at a Linux system, it automatically tests whether all prescribed rules have been implemented there.

If a developer uses Automate to build an application instead, InSpec can check and interrupt each step of container creation if a non-compliant container is created. In fact, the combination of Automate and InSpec forces developers to comply with applicable rules. If they do not follow the rules, no application is created in the first place.

Prebuilt Tests

If you combine Chef Automate 2.0 and InSpec in your setup, you can benefit from many prebuilt tests included with Chef Automate. Standard compliance tests from several recognized compliance organizations can be performed on common operating systems and thus serve as a basis for your own compliance requirements.

Happily, Chef is exemplary in version 2.0 of Automate, as well: The entire InSpec source code is still freely available on GitHub, so that even for those users who do not want to use Chef Automate, InSpec is and remains usable.

Habitat Now Available Locally

Do not forget the new version of Habitat, a framework for application release management in Chef Automate 2.0. Here, application does not take on the typical definition, but rather refers to cloud microservices: The tool is designed to help companies transform existing environments into a microarchitecture (Figure 4), providing a whole box of tools.

One important key to Habitat's success is its great flexibility: On the one hand, it receives input in the form of Git directories; on the other hand, it outputs finished images of containers and can roll them out in a Kubernetes cluster (Figure 5).

Accordingly, Habitat is the spearhead of an app-centric automation drive. In the new version of Automate, its developers emphasize two Habitat functions in particular: Habitat Builder can now also be run at the customer's data center, making the solution attractive for those customers who are not allowed access to cloud services for compliance reasons. Habitat also now comes with far better integration with other services. The broker for rolling out applications in Kubernetes has seen several updates. Additionally, you have the option to roll out directly in Azure, as well as an interface to the open service broker (OSB).

Interaction

If you put the four components that belong to Chef Automate 2.0 into a larger context, you get a solid overall impression: In fact, in the new version, the company has made the leap from a solution only for infrastructure-level automation to a paradigm in which physical and virtual systems and the roll-out of packaged applications have equal rights.

Of course, people who want to continue using Chef Automate as a platform for automating large server farms can continue to do so and still benefit from the many innovations in terms of usability and user friendliness that are part of Automate 2.0.

If you want to embrace the principles of DevOps more strongly in your company, Chef Automate 2.0 is also a useful partner: Seamless integration of InSpec 2.0 or Habitat, for example, makes it seem irrelevant whether the target is a Linux system or the creation of an application in a container format. Both processes provide the same tools for automation and implicit compliance testing.

Moreover, do not underestimate Chef Workstation: For the first time, this product offers the opportunity to enter the Chef world quickly and directly without having to worry about the setup and the services previously required. The fact that it is no problem to transfer the systems that were rolled out by Workstation later is just the icing on the cake.

Price-Less

What will all this fun cost? The manufacturer is not saying, which is common practice today: An individual offer can be requested from their website [5], but a rough overview of the prices is not to be found anywhere on the Chef website.

However, you can try Chef Automate in a trial phase and view the extensive online documentation, including a workshop and prepared VMs, to see how easy it is to work with Chef Automate 2.0.