Containers and Virtualization FAI.me Lead image: Lead Image © Nataliya Hora, 123RF.com

Build operating system images on demand

Assembly Line

If you are looking for a way to build images quickly and easily, FAI.me is the place to go. By Martin Loschwitz

In popular clouds, the providers usually roll out standard distribution images. SUSE, Red Hat, and Canonical offer these explicitly, and there is no reason why you should not use them. However, these images may have one or two annoying features, such as missing packages, wrong configurations, or other everyday difficulties.

Changing a finished image is not trivial. Instead, many admins start rebuilding from the source and, sooner or later, give up. In most cases it is not possible to achieve the same image quality as that of the distributors. Either the DIY images are bulky and far too big or they don't work well.

This problem is exactly what FAI.me addresses: The tool is an extension of the Fully Automated Installer (FAI; see also the "FAI Review" box) [1] that builds operating system (OS) images on demand, for both bare metal and use in the cloud. In this article, I introduce FAI.me and explain what happens in the background.

FAI Review

A short review of FAI will help you understand FAI.me. Although FAI is not new, the author Thomas Lange is continuously adding new features. Moreover, a small but hard-working community has gathered around the tool, keeping it up to date and ensuring that it can install Ubuntu and CentOS in addition to Debian.

The original purpose of FAI was clearly defined: After unpacking, new servers install autonomously to the extent possible and without too much manual intervention. Quite remarkably, FAI was created back in the late 1990s, long before automation tools such as Puppet or Ansible existed.

FAI offered the ability to roll out an OS automatically at an early stage. In the standard configuration, it combines a number of different protocols. A DHCP server is supported by a TFTP server. Clients use the PXE protocol to obtain an IP and then load a bootloader via TFTP. A kernel, usually one from the installation routine, is responsible for taking care of the rest.

The program needs PXE to boot into a custom environment where it can roll out its various operating systems. Lange and volunteer FAI developers have implemented many features for this purpose. Scripts can be executed at different stages of the installation, which then implement certain functions not provided out of the box in FAI. All FAI components can be loaded from a central network server for this purpose.

The highlight is that FAI does not depend on the installation routine of an installer. If you want to implement automation for SUSE, CentOS, and Debian, you would theoretically have to create three boot environments: for AutoYaST, Kickstart, and preseeding.

FAI offers a mostly generic interface. Only local modifications, such as the selection of the packages to be installed automatically, add pitfalls because not all packages for the same components have identical names across distributions.

Lange recognized early on that dependence on the network can be quite disadvantageous. Conceivably a DHCP server may exist, but it then takes some time to integrate with FAI – or DHCP is not allowed at all. Maybe the systems you want to install just can't use PXE – not all network cards come with support for this protocol. However, the network boot in FAI will not work without a network either.

For many years, FAI has supported the possibility of generating a static image from a precomposed FAI configuration, which you can then burn to a CD-ROM or DVD or write to a USB stick. The local boot medium then behaves exactly as a network-based FAI, but with a few system-related limitations: If you change the FAI configuration, you have to create new images afterward.

In the first years of FAI's existence, this function was limited to generating images for bare metal, but now FAI also provides functions for building images for cloud environments. Taking Debian as an example, the command

fai-diskimage -u cloudhost -S900M -cDEFAULT,DEBIAN,AMD64,FAIBASE,DEMO,GRUB_PC,CLOUD /tmp/disk

creates an image of an installed Debian system built for the AMD64 architecture in /tmp/disk; it contains the GRUB bootloader and needs 900MB of storage space (Figure 1). If you have fast Internet access on the system on which you call the fai-diskimage command, the process is also quick. It hardly takes a minute for the finished image to become available. Debian is happy to have the tool, because, among other things, the project uses it to create its official images for the cloud [2].

Figure 1: fai-diskimage creates a cloud image based on an existing FAI configuration.

Instant Images

FAI.me provides the functionality of FAI without the kind of tinkering that's otherwise necessary. Basically, it's not much more than a graphical web-based interface for fai-diskimage, which assembles bootable OS images on demand. Images for bare metal installations as well as for clouds are included, but FAI.me offers a whole host of extremely practical functions.

Naturally, the bootable FAI images for bare metal differ significantly from the cloud images. The one contains the normal FAI installer, which starts its work after launching from the boot medium, whereas the cloud version comes with a pre-installed operating system. To take this into account, Lange has implemented FAI.me on two subpages on the FAI website for cloud images [3] and bare metal [4]. Both pages are quite straightforward.

Clouds

If you look at the cloud page, you only have a few – really important – parameters to set. At the very top of the form, for example, you need to enter both the target size of the image and its format. The background to this is that if you build an image for AWS, it needs a different format than for KVM, which usually wants the QCOW2 format for hard drives.

You can define the hostname, but it is usually overwritten by the software-defined networks in clouds and their name resolution. Practically, if you add your public SSH key to a cloud image, you don't have to specify it when starting the virtual machine.

If you want to set a password for root, you can, but I strongly advise against it. Leaving the field empty is one less attack vector, and it doesn't mean having to do without root rights thanks to sudo. If you then set the desired language and the release you want to use, you are virtually ready to start.

FAI.me just wants to know which packages it should integrate into the image. Best be economical here: As a rule, clouds are connected by fast lines, so it is advisable to keep the basic image as small as possible and load the rest off the network or a local mirror as needed.

The big distributors demonstrate this vividly. The Ubuntu images, for example, which Canonical makes available for use in OpenStack, have managed with around 260MB for years.

Bare Metal

If you want to build an image for use on bare metal instead, the effort is not much greater. Although a separate option displays the advanced settings, it only takes you to the settings for the root password and lets you add a public SSH key. FAI assumes by default that it will create a user with a password, who then becomes root with sudo.

You can specify the partition scheme in a drop-down menu. FAI.me provides several suggestions for the use of the Logical Volume Manager or /home on your own partition. The remaining settings largely correspond to those of the cloud variant.

Push-Button Image

Whether you want an image for bare metal or for the cloud, at the end of the process, pressing the button at bottom left for creating the image is all it takes to start the automatic image building process (Figures 2 and 3). After a short wait, the browser then starts downloading the image, which can then be used on a USB stick, on a CD/DVD, or in the cloud.

Figure 2: FAI.me creates images for the cloud in QCOW2 or AWS format or …

Figure 3: … installation images that equip a physical host with an operating system.

As mentioned, no hocus-pocus is taking place in the background; instead, the web interface calls fai-cd and fai-mirror or fai-diskimage behind the scenes and creates a matching image on the fly. Therefore, you can be absolutely sure that you always get the packages for the latest Debian GNU/Linux.

Unlike the big distributors, you decide when to build the image, although it means not using an official image, but one you build yourself with FAI.me. What Lange originally intended as a showcase for FAI and to give users an understanding of FAI's range of functions is itself a very practical tool.

Your Own Image Factory

To recap, FAI.me has virtually no functionality of its own. The tool uses a preconfigured FAI installation in the background to build images on demand in line with FAI standards, which is the solution to a problem that many cloud providers face. Prebuilt cloud images are fine, but sometimes you need local modifications. If you offer special hardware in your cloud and want to pass it through to your users, you find yourself regularly building your own images.

As explained in the "Images or Automation?" box, this question is not trivial, especially if you don't have the right toolset at hand. FAI and FAI.me, on the other hand, have proven to be very useful tools that can quickly form the basis of a local image factory that automatically outputs state-of-the-art disk images with special local modifications.

Images or Automation?

On the basis of my own experience, FAI.me triggers two reactions that could hardly be more different. On the one hand, enthusiastic admins have needed a tool like this and had not yet found it. On the other hand, more conventional admins with backgrounds in automation have turned up their noses.

A conflict comes to light that plays an important role in contemporary IT. Does it make more sense to work with operating system images, or should you instead rely on the vendor's installation tools and use automation to make the required adjustments? Although this discussion is undoubtedly still in full sway, many assumptions and fears are based on obsolete knowledge.

Admins are absolutely right when they warn against monster images that cannot be regenerated when you need them. Companies commonly find that a golden master image for the installation of new systems has "grown historically": It works, but nobody in the company knows exactly what it contains. When a new image has to be built, it often involves massive overhead and consumes a huge amount of time.

The same applies to images you can pick up from alternative "black box" sources from the Internet. One thing you do not want in your data center is a pre-owned image with a built-in Bitcoin miner, although this is mostly discovered in the context of container images. However, the same caveat naturally also applies to images of entire operating systems.

By the way, when many admins think of images, they think of bare metal deployments. Because the local variance in this area is much higher than in defined environments such as KVM or VMware, many people in the past believed that monster images were legitimate or even necessary.

Like with a pendulum, a countermovement of tinkerers categorically reject OS images. Instead, its proponents say you should install Linux on your hard disk with AutoYaST, Kickstart, the Debian preseeding method, or whatever your distribution uses as an automatic installation tool. According to this narrative, then, the automation engineer handles the rest of the work.

However, this problem is easy to work around: Continuous integration and continuous delivery/deployment (CI/CD) environments based on Jenkins offer the ability to build OS images completely automatically. Of course, FAI.me is also an approach to circumventing precisely the problem described. If you use FAI.me to build your images, you can understand the process in detail, and if you so desire, you can also run FAI.me in an instance of its own, which then contains local modifications – but in a comprehensible way.

The images built with FAI.me can just as easily be frugal operating system images that simply prepare a host for use with Puppet, Ansible, or some other automation system. By the way, this is more elegant by several orders of magnitude than the automation structures that some administrators build themselves with scripting in Kickstart or AutoYaST or by preseeding.

One thing should be clear by now: Nothing works without operating system images. They are essential in clouds because virtual instances cannot be built and started without them. Installers from distributions are simply not viable alternatives, because the current clouds do not support the PXE boot functionality required in the first place.

In the end, as is often the case, a whole range of gray tones exist, and those admins who find a perfect mix of images on the one hand and automation on the other, will have a pleasing result. FAI.me is a promising and well-proven component in such a context.

How It Works

To begin, you first set up FAI as if you wanted to use it for the live installation of nodes. Factors like DHCP can be ignored – the purpose is to create bootable media. After that, you already have the option to create your own images with fai-cd and fai-diskimage. But that's only half the battle. Users actually want to have this file embedded in a CI/CD process to ensure that images are automatically built when changes are made to the FAI configuration and that the images are then available for download from a central location.

Therefore, connecting FAI to a CI/CD tool such as Jenkins is a good idea, and this is exactly what the Debian project does. It stores its FAI configuration in Debian GitLab and uses hooks to wire it to an FAI installation in such a way that the described mechanism is implemented. When a commit ends up in the master branch of the repository, GitLab then ensures that new images are created automatically.

If you prefer not to overwrite the old images automatically, the recommendation is to encode the date in the name. The example with GitLab, in particular, is not difficult to set up if you make sure that GitLab has a virtual machine on which FAI is executable and that can access the GitLab repository itself to build images according to FAI rules.

Instead of laboriously developing an image factory yourself, it could be a good idea to turn to FAI, especially if the target system is Debian, with which FAI is particularly connected through its author.

Conclusions

For many admins, building operating system images is an unnecessarily complicated exercise that requires a huge amount of preparation. FAI shows another way: By combining the appropriate parameters for fai-diskimage or fai-cd and fai-mirror, it builds generic disk images at the command line in a very short time.

However, FAI itself cannot be set up easily and quickly. Anyone planning to install dozens, hundreds, or even thousands of systems automatically with this solution will be happy to put up with the overhead of the initial FAI installation: It's guaranteed to pay dividends. Each new server that is installed in this way then reduces the total overhead and pays for itself.

If you just want a sample of the FAI atmosphere, FAI.me is the right place to start. In a very short time, you can build disk images for Debian that still offer some leeway for local customizations. FAI.me is therefore a very useful extension to FAI itself and worth exploring.