Micro Cures

As anyone who has spent a couple of years as a Linux admin will note: IT has changed more in the past decade than ever before. The cloud has been shaking up the industry since 2010, and containers have increasingly been the focus of interest since 2015, at the latest. Container automation is being taken one step further in the form of micro operating systems (OSs), which have nothing to do with microkernels; rather, they refer to the software set that comes with the product. Today, the first premise of container automation is to reduce to the max the software needed on systems. The intent is to avoid maintenance overhead, because where nothing is installed, nothing needs updating and maintaining.

In this article, I explain the basic principle of the approach and then introduce k3OS [1] and Flatcar [2], two microdistributions that are seen as alternatives to systems by the established vendors.

Microdistributions

The major vendors have consistently trained administrators over decades to cope with increasingly complex systems. A distribution is nothing more than a compiled Linux kernel, which – together with a huge bunch of administrative tools – finds its way onto a server hard drive.

Current systems in particular come with considerable complexity on board. One typical example is the package manager: To avoid admins having to compile every program themselves, the distributors deliver many programs in packaged form. To be able to update them, package managers use a lot of tricks to avoid trouble (Figure 1). Debian, for example, makes many compromises in Dpkg to prevent packages from overwriting changes in /etc.

However, every admin knows from experience that, whatever magic the package manager has up its sleeve, it will not help you when worst comes to worst. Updates from one major version to another remain a challenge. If something goes wrong, the system can be offline – for minutes, hours, or sometimes even days. Having high availability (HA) as part of the standard scope of common distributions is of little consolation, because the complexity introduced by HA software is itself massive.

The problems do not stop with the package manager. If you compare, for example, the way Linux configured its systems a few years ago with NetworkManager and others, you can't help noticing that manufacturers have paid for the ever-increasing variety of functions with complexity that the admin has to handle at the end of the day.

New Possibilities

Part of the complexity is difficult to eliminate. If systems have several bonding interfaces (e.g., on which VLANs are then located that are controlled over a network bridge), configuring the network on the server is inevitably going to be complex. However, in other areas you can start taking things off the list – and microdistributions show how this can be done.

When Docker finally made container virtualization under Linux socially acceptable, the distributors switched to gold rush mode. Docker offered a realistic perspective at the time for saying goodbye to parts of package management. Instead of loading software in the form of RPM packages onto the system, admins would simply load and start the appropriate container. Because each container is a self-contained system, all dependencies of the respective application are included. Actual user data is transferred to the container at runtime by mount. If an update is pending, the user loads the new container with the new program version, detaches the volume from the old container, attaches it to the new one, and launches – all done.

In this workflow, however, the rules of the game change dramatically: A system running on Linux mutates from a complex environment to a simple tool for operating containers. You don't need much in the way of software on the system side: the kernel, a few basic components like systemd or a cluster consensus tool like Etcd or Consul, and the runtime environment for the containers themselves (nowadays either Docker or Podman) – that's all. That the distributors are following exactly this path can be seen by both Red Hat and Canonical now having package managers that are based on container principles. Snaps in Ubuntu (Figure 2) and Flatpaks in Fedora are nothing more than tools to load and manage container images like traditional packages.

Container Orchestration Help

The truth is that operating concepts in which the admin manually runs individual applications in container form on systems are possible with these solutions. However, the distributors' focus is clearly on container orchestrators and, above all, Kubernetes.

The ideal scenario then looks like this: A basic installation of the OS only includes the runtime environment for container and kubelet (i.e., the control agent in Kubernetes). As soon as a node goes online, it automatically becomes part of the fleet that Kubernetes manages. All changes to the system, as well as starting and stopping services and containers, are handled through Kubernetes, so administrators do not have to do anything manually on the system.

This process again reduces the infrastructure overhead per node for the admin considerably, because the container orchestrator is running anyway. Without Kubernetes or a corresponding alternative, operating a container is not only inefficient, but ultimately futile.

Similar Methods, Different Targets

Now that the role and function of microdistributions are clear, I will discuss two representatives of this category in detail. Most admins probably will have heard of JeOS (Figure 3) or CoreOS, because these are the microdistributions from established vendors SUSE and Red Hat.

k3OS and Flatcar, on the other hand, are the underdogs and probably unknown to many administrators. However, the following description does not intend to compare the products, because they are based on different factors and partly address different target groups. Rather, it is a brief overview of the product itself.

k3OS for Kubernetes Lovers

k3OS is especially for users who want to use K3s [3]. If you don't know K3s yet but regularly work in the Kubernetes (K8s) context, you have probably already guessed that the abbreviation has something to do with Kubernetes. K3s is a fully Kubernetes compatible distribution of the orchestrator, which is easier to use, smaller, and comes with fewer dependencies.

K3s also comes with packet filtering rules out of the box, which, according to the developers, improved the security of the entire installation. Last but not least, K3s helps admins where they really need it. One of the most annoying features of Kubernetes is setting up a port share on each Kubernetes worker (kubelet) so that the Kubernetes manager can talk to the kubelets.

K3s exposes the API interface over a websocket tunnel instead, so port sharing is no longer a problem. Anyone in the enterprise who has to deal with firewall constructs and compliance rules will like this feature.

Under the Hood

Additionally, K3s bundles various Kubernetes extensions so that they are ready for use on an ad hoc basis. Containerd and runC serve as runtime environments for the containers, and Flannel takes care of the network in between. CoreDNS, Helm, and Kine are also on board by default. With tools you program in-house, K3s makes it easier to maintain required SSL certificates and manage Etcd as a consensus algorithm.

K3s, however, has had to dump some features to become a lighter weight Kubernetes. The storage drivers, which K8s includes out the box, are missing, as is the functionality to communicate with cloud providers. However, because Kubernetes itself is introducing tweaks and replacing the existing functions with new approaches, this point should not be a problem for most admins.

k3OS is not a fork of an existing Linux distribution, which is quite unusual in itself. Most projects that build microdistributions today take an existing distribution and remove the components that are not necessary from the view of the maintainer. k3OS is basically based on a mixture of an Alpine userland and the Ubuntu 18.04 kernel. The developers have assembled the individual components such that they harmonize as well as possible for k3OS.

However, one thing is evident: The evaluation criteria that apply to classical distributions only play a minor role in microdistributions. Basically, the system just needs to support all the server's hardware, and a recent hardware enablement (HWE) kernel by Ubuntu is certainly not a bad idea for that purpose.

Automation Is Strange

What is a bit more adventurous is the way k3OS finds, or should find according to the developers, its way onto your hard disk. The k3OS makers do not use a powerful automation framework like Preseedint, Anaconda, or AutoYaST.

Instead, the command line that accompanies the kernel at startup defines the type of installation desired. Moreover, k3OS can install itself directly from a running Linux system; to do this, it changes the partitioning of the disk and then the GRUB configuration. However, the developers also provide for a legacy installation with an ISO file.

The admin will search in vain for tools to configure IP addresses and similar settings. A host running k3OS gets all these details automatically from the appropriate operator in K3s. As soon as the kubelet agent is running on the host, it learns how to do the configuration. In the usual style for clouds, protocols such as DHCP are also used for this purpose.

Basically, the following is true: k3OS (Figure 4) is so closely interconnected with and designed for K3s that it is hardly worthwhile taking a closer look at the system without being interested in K3s. In this case, you get a largely non-extendable basic Linux. Even for use with other Kubernetes distributions, K3s is not a useful choice because it does not adopt part of their standard configuration. Conversely, if you do want to use K3s, you should definitely take a closer look at k3OS.

Red Hats on a Flatcar

The second microdistribution for review is not a new project, but rather an old acquaintance. Red Hat has a proven track record of buying open source companies and making them big. Ceph, for example, has become more or less a standard solution for software-defined storage since its takeover by Red Hat.

The manufacturer might have had a similarly lucky hand with CoreOS, which has belonged to Red Hat since 2018. The CoreOS microdistribution was considered a pioneer in the entire industry. The OS is the work of a company with the same name, which also invented a number of other components that are practically indispensable in the container context today. Examples include the Etcd cluster consensus algorithm, Flannel, and Prometheus.

From Red Hat's point of view, it was logical – almost necessary – to buy CoreOS. On the one hand, Red Hat was able to secure its open source properties, on the other hand, the CoreOS products fit perfectly into Red Hat's Kubernetes portfolio (which it had just turned on its head with OpenShift). Red Hat did what was expected immediately after the acquisition: It made CoreOS the centerpiece of new products. CoreOS became Red Hat CoreOS (RHCOS) and Fedora CoreOS.

Noses Turned Up

In the course of development, CoreOS changed continuously; in the meantime, it was no longer compatible with the original version in many ways. Therefore, Red Hat officially discontinued the old standalone version of CoreOS; by the end of October 2020, at the latest, everything will be done and dusted. However, not everyone is happy: Critics complain about Red Hat having too much influence and fear that CoreOS will not be usable without Red Hat's blessing.

Add to that the frustration of having to switch to a new OS and adapt existing workloads. People who already have complete lifecycle management for their container fleet do not see any value in the change. The same is true for third-party vendors who develop applications for CoreOS and who would also have to put in additional work because of Red Hat's move, without seeing any technical reasons for doing so.

Berlin-based consulting and development company Kinvolk went on the offensive and forged CoreOS from the last stable version. This project is known as Flatcar Linux. Kinvolk promises to maintain Flatcar as an unofficial successor to CoreOS for the foreseeable future and to keep it compatible with CoreOS. Also important for corporate customers is that the company offers commercial support for the system.

High Time to Migrate

On the project's website, the team behind Flatcar provides ISO images for the install. Alternatively, the microdistribution can be rolled out with a network-based deployment tool. Moreover, a migration from CoreOS to Flatcar Linux is possible or, you might say, necessary. If the vendor is no longer supplying updates for the old system from November onward, migration already takes the highest priority.

A migration from CoreOS to Flatcar Linux should not be imagined in the way you would expect with a classic Linux distribution after a major update. As I mentioned earlier, the project is making sure that Flatcar Linux stays compatible with CoreOS. You can therefore migrate a system on the fly without problems. If you have implemented working automation for server lifecycle management, you can alternatively migrate existing servers without content and reinstall them.

Which of these approaches will be faster and more convenient depends on the respective application scenario. However, practically, you can install Flatcar Linux with Matchbox and Ignition, as was the case with CoreOS.

One-to-One Successor

Once rolled out, Flatcar Linux keeps the promise of being a one-to-one successor to CoreOS. All services known from CoreOS are also included, such as the familiar command-line environment. Etcd can be established in a few seconds as a central configuration repository in cluster mode between the different Flatcar instances. Additionally, the micro-updates for the individual CoreOS components, which can be easily installed during operation, have been kept.

Briefly summarized: If you install Flatcar Linux, you basically get CoreOS with an update promise that can then serve, for example, as a generic basis for Kubernetes. Flatcar does not tie the user into a certain Kubernetes distribution – Kinvolk even sells its own Kubernetes as an open source product named Lokomotive, which can be used on Flatcar systems.

The Kinvolk update service, which is basically centralized patch management for your own Flatcar Linux instances, also proves to be very practical and has a certain similarity to Ubuntu's Landscape, although it is based on the open source project Nebraska (Figure 5).

At least one small hair has landed in the Flatcar Linux soup: If you are already the maintainer of your own distribution or a distribution within the framework of a Linux project, you will be familiar with the resulting overhead. Naturally, the manpower required for microdistributions is less than for a full-grown Debian, but still significant. Whether Kinvolk will be able to cope with this overhead in the long term remains to be seen.

Kinvolk also has provided little information on whether and to what extent they will be providing new functions through updates. That they want to retain compatibility with the original CoreOS for the foreseeable future is certainly a great relief for existing products and workloads. Experience shows, however, that sooner or later, the desire for new features is unavoidable. Only time will tell how Kinvolk deals with this issue.

Warning Words

One thing is clear: Containers give admins various shortcuts and make maintaining and operating Linux systems easier than before. If you strictly rely on official containers and get them directly from the respective developers (i.e., Kubernetes Helm or as a Snap or Flatpak), you don't need to worry about the security issue – at least not any more than with packages. The distributors also do a good job with their container images and update them in step with the discovery of security vulnerabilities.

On the other hand, it is not a good idea to fish around and roll out arbitrary container images off the web. It doesn't matter if you choose k3OS, Flatcar, or any other microdistribution. Their advantages are usually ruined if you drag a black box in the form of an application into the house. Whereas containers for Docker and others are quickly built, "running in VMware on my MacBook Air" is not a functional benchmark for production.

Practically, you should build your own container images for the applications you want to run on Flatcar. However, this requires a functional continuous integration and continuous delivery or deployment (CI/CD) approach, at the end of which git commit ensures that a Git directory becomes an executable container. Setting up such a system is certainly complex, but the work can be easily automated.

In return, admins will benefit from lean systems and completely automated containers in which bugs can be fixed quickly in everyday life. This concept, also known as immutable underlay, should be the benchmark for new environments in the year 2020. In any case, the manufacturers of the large distributions should do their utmost to push their users in exactly this direction.

Conclusions

Microdistributions are the logical next step of automation in a world where containers are becoming increasingly relevant. Clearly, all major Linux distributors see the future of Linux servers in the container camp. Because a great deal of functionality that used to run on the system itself can now be bundled into containers, it is a wonderful opportunity to remove complexity from a setup.

This scenario is exactly what k3OS and Flatcar are aiming for: The running system should have as few loose parts as possible. The few components that are still needed are already included in both approaches. That k3OS specializes in the Kubernetes distribution K3s and that Flatcar prefers to target users of the previous CoreOS is only a side issue.

Admins should follow the container subject anyway, which is likely to become more necessary than ever in the future, and k3OS and Flatcar offer good approaches to understanding the principle. RHEL 8 now offers container-based package directories that integrate seamlessly with the management tools.

Quite possibly Podman and the like will eventually herald the end of RPM and Dpkg. From a distributor's point of view, programs that find their way to the users in container form offer an unbeatable advantage. Instead of maintaining a specific program for each version of a distribution, with containers, it is sufficient to have exactly one container with the appropriate program for all supported distributions. Canonical has already reaped the rewards on the desktop in Ubuntu, so applications like Chromium are only included as Snaps out of the box.

If the principle works – and that's the way it looks at the moment – the established manufacturers are likely to consider going exclusively down this path. The future remains exciting.