Fluent

The idea of populating all clouds with the same templates has remained a pipe dream until now. Although some services offer a compatibility interface for templates from Amazon Web Services (AWS), it being the industry leader, my experience suggests you should not rely on it. Admins are faced with the major dilemma of theoretically having to learn how to use the orchestration tools for various clouds and maintain their own templates (see "The Nebulous Cloud" box). Tools like Terraform (Figure 1) promise to avoid exactly that.

One relatively new player in the field of these tools is Pulumi. It differs significantly from other tools in that it uses known programming languages instead of its own declarative syntax. So, how does it work, and what does the admin get out of it?

Infrastructure as Code

Pulumi's central promise is infrastructure as code (IaC). Admins provide their virtual infrastructures in the form of code, and Pulumi makes sure that the virtual resources run as desired on the target platform. In itself, this is not a groundbreaking feature, because other multicloud orchestrators like Terraform can do this, too. What Pulumi advertises as a unique killer feature is that it has not invented a new scripting language but relies on proven programming languages. What exactly can admins look forward to?

If you take a look at other solutions (e.g., Terraform), you will see that they use a declarative scripting language. Usually, a markup language such as YAML or JSON lets you define the resources in a single file, which you then hand over to the orchestrator. The orchestrator evaluates the domain-specific language (DSL) file and interprets it accordingly. The disadvantage from the administrator's point of view is that you have to learn and understand the declarative scripting language to work with the program.

Pulumi takes a different tack and instead relies on the syntax of existing scripting languages. For example, if you are familiar with Python (Figure 2), you can package the infrastructure in a Python script. Pulumi provides the appropriate classes for the different cloud providers, which can be imported in the usual way. If you don't work with Python, you have a choice between TypeScript, JavaScript, Go (Figure 3), or C#. Pulumi supports all programming languages from the .NET framework, including VB and F#. Most developers will find a language in this potpourri that suits their taste.

To begin, you load the individual Pulumi modules into your IaC document according to the standard procedures for your choice of scripting language. Pulumi does not force admins to use a special environment for development work. Instead, your standard editor or standard development environment is used, with all the features that are otherwise available.

Nevertheless, Pulumi is a declarative tool. Once you have written your code, you call pulumi up at the command line to start the described infrastructure in the desired environment. In practice, Pulumi tries to balance the declarative approach of classic tools like Terraform with the well-known syntax of common programming languages.

Program, Project, Stack

If you are dealing with Pulumi for the first time, it is best to first familiarize yourself with its programming model. Here, three terms come to mind that need further clarification: the program, the project, and the stack. Each of these three elements has its own task within the Pulumi universe.

What Pulumi means when it refers to a program is more or less all the files that are written in a specific programming language and together form a logical unit. For example, if you write your own Python class for your Pulumi code, it would be part of the program in the Pulumi language.

The project is a little larger. It contains a program and the metadata needed for Pulumi. The pulumi up step assumes that working metadata for the respective program is available, so that Pulumi knows what to do.

Finally, the stack is a concrete incarnation of a project on a cloud platform. The stack state is reached when you have successfully started the infrastructure defined by Pulumi on a cloud platform and can use it.

Working with Pulumi

In theory, the idea behind Pulumi sounds great at first glance. Instead of learning a complex declarative script language, you draw on the skills you already have from working with other programming languages. Indeed, this is one advantage of Pulumi compared with other solutions such as Terraform. However, from the administrator's or developer's point of view, this advantage should not be rated too highly. Even though Pulumi supports the use of well-known programming and scripting languages, you will still need to read a mass of documentation.

The ability to use well-known scripting languages only makes your job easier in the sense that you can work with a familiar syntax. However, if you have a Python program that is intended to start an instance in AWS, you still need to call the appropriate functions to do so. Pulumi does not abstract the cloud APIs from the program so radically at this point that you could rely on uniform commands for different cloud platforms.

In concrete terms, this means that if you want to start a new instance in AWS, for example, you can do so in Go, Java, or one of the other supported scripting languages. However, you do have to deal with how the present task can be solved with the Pulumi modules for the various clouds, and to do so, you need to understand how the Python pulumi_aws module, which provides the corresponding functions, works. Because pulumi_aws is specific to Pulumi, you have to deal with its syntax, which is all the more true if different workloads are to be set up in different environments with Pulumi. Not only do you need to understand how Pulumi's AWS modules work, but also how to use the modules for the other clouds.

To avoid misunderstandings: The Pulumi feature set is clearly oriented to the standards of the market. Other solutions such as Terraform do not fully abstract the respective cloud platform but have different modules for different environments. You have to use these when defining the resources, just as in Pulumi. The only difference is that, in Pulumi, you can use a syntax you already know from your previous work. Whether this really saves as much time as the developers claim in their documentation seems questionable.

From Program to Project

What does working with Pulumi look like from the administrator's point of view? First, you need to install Pulumi, but this is not complicated: For Linux systems, the manufacturer offers a downloadable shell script [1] that you call locally. The only required binary, pulumi, is then stored in your home directory. The script automatically extends the PATH variable to include the location of the binary file.

Installing the program on other systems is a similarly relaxed experience. Pulumi is by no means Linux-specific. On macOS, it is included in the Homebrew collection [2], although the installation script for Linux that I mentioned earlier will also do the trick. For Windows, a separate installation wizard is available to help you install Pulumi on your system.

The Pulumi Service

At this point, Pulumi is almost ready for use – almost. Now you need to deal with a service that seems a bit strange at first glance: the Pulumi service app.pulumi.com, a centrally operated directory for Pulumi stacks of individual users (Figure 4). The manufacturer describes the Pulumi service as a kind of support for the administrator in managing their rolled-out setups: From here, various stack parameters can be changed or examined.

For the open source version of Pulumi, the manufacturer assumes that the app is centrally operated by the provider and remains free for personal use by the user. If you want to use the Pulumi service commercially, license costs are incurred, which I will discuss in detail later. In the Enterprise version, it is also possible to run a separate instance of the Pulumi service locally, which is a particularly useful alternative if you do not want to pass on certain information (e.g., details of your own stacks and where they are rolled out) to third parties.

The remainder of this article assumes private use, for which Pulumi does not charge. However, you do have to create an account, because the first time pulumi new is called, the command-line tool tries to connect to the service and create a corresponding entry there. Single sign-on for Pulumi's hosted app service will work with a variety of providers; for example, you can use a GitHub account. Once the Pulumi account has been created, it's time to get started with the actual project.

First Project

In a freshly created directory, run the pulumi new <Project> command. After prompting you for the login credentials for Pulumi's App Service, a wizard launches to guide you through the most important steps of the setup and prompt you for various parameters. You can decide whether this is a project for AWS, what the credentials for the AWS account are, or in which AWS region the rollout will take place. Now, with your preferred scripting language, define the desired virtual environment such that it contains all the necessary components.

Once the project meets your requirements, the next step is deployment. The

pulumi up

command handles all the necessary steps in the background. The App Service shows that the Pulumi service is far more powerful than it seems at first glance, because it, not the Pulumi binary on the developer's host, handles the communication with the respective cloud services in the background. Detouring by way of the Pulumi app does have some advantages: If a cloud vendor changes something in their APIs, end users do not necessarily have to update their binaries to use the new features.

As soon as the deployment of a stack from within a project is finished, Pulumi shows it as active, and you can use it. If the developer made a mistake or forgot something, the stack can be pushed into a black hole with

pulumi destroy

in the project's folder. All told, the Pulumi service proves to be a helpful tool in everyday development work.

Supported Target Platforms

How Pulumi takes applications into the cloud is explained in sufficient detail in this article, but what clouds can you manage with Pulumi? Clearly the big hyperscalers will be part of the package: Pulumi not only supports AWS, Azure, and Google Cloud Platform but offers a massive feature set. AWS, for example, is just a roof under which hundreds of services now reside. Pulumi cannot address them all, but in a normal working day with AWS, admins are unlikely to worry about missing functionality. The same applies to Microsoft's Azure and Google's Cloud.

Private solutions are far much less well supported. Pulumi has a resource provider (the name of the plugins that extend Pulumi with support for certain OpenStack features) for OpenStack. However, the provider is far removed from supporting all the features that modern OpenStack installations offer and is limited to rather basic support. Pulumi cannot handle the more complex tasks, perhaps because most users simply use Pulumi with the large providers and do not want to deal with additional problems that arise when using Pulumi with private clouds. For example, if you are using a private cloud that cannot be accessed over the Internet, you will inevitably need the Pulumi Enterprise version, so you can operate the Pulumi service locally. If you want to access the large public clouds, you will have no problems with the program; however, it is not suitable for private clouds.

Support for Kubernetes

In addition to genuine infrastructure-as-a-service offerings, Pulumi now also offers strong support for Kubernetes. Little wonder: Strictly speaking, Kubernetes is nothing more than a fleet orchestrator. However, the magic takes place one level higher than in the classic clouds. Kubernetes assumes that it has access to executable systems on which it can install its components and operate containers. A tool like Pulumi would be difficult to imagine without support for Kubernetes; therefore, the developers integrated precisely this function into their software. However, the Pulumi service must be able to communicate with the Kubernetes API for the deployment to work. If the Kubernetes instance can only be reached over a local connection, a local Pulumi service is again needed, which requires the Enterprise version of the product.

Policy as Code

The Pulumi developers draw particular attention to one feature in terms of security, and the product has a kind of unique selling point: Policy as code is the main focus of the Pulumi CrossGuard product.

The narrative goes something like this: Individual developers should not have to ask for resources or manual approval from the manager for every Kubernetes cluster they want to start, because that would lead to enormous administrative overhead. However, administrators should not be able to do what they want on all cloud accounts of a company. An Amazon AWS instance in the appropriate configuration tends to burden your wallet to the extent of $1,000 or more per month. If an administrator's Pulumi project runs amok and starts virtual machines without anyone noticing, a rude awakening threatens at the end of the month in the form of a massive bill.

Pulumi therefore has the option to map a set of rules that provides different permissions for different developers – with the Enterprise version, only, for which you again need to put your money on the table; however, it opens a multitude of possibilities. Admins then have the option to define Policy Packs (Figure 5), which contain a set of rules or several related sets of rules that assign different authorizations to different users.

In Policy Packs, granular permissions can be set at the API level of each cloud vendor down to the individual API call, including the parameters for each. For example, if an administrator is only to be allowed to start instances of a specific flavor on AWS, the Policy Pack provides a switch for this. Policy Packs also provide on-demand restrictions for where admins can start specific workloads. For example, if you want to run an unimportant workload on AWS, but only on cheap instances, it would be better to rule out Amazon's high-end data center in Frankfurt for the workload.

In addition to hard prohibitions, Policy Packs also provide warnings. If an admin uses certain services in a public cloud, Pulumi can display a note for the individual case that warns of financial or technical risks. Practically, the policies also use well-known programming languages, so they are implemented like all other programs in Pulumi. The downside is that you have to choose the corresponding descriptions and keywords for the individual scripting languages from the documentation and practice with them before you can create and use Policy Packs in a meaningful way.

Differences from Terraform

How does Pulumi perform compared with its direct competitor (and top dog) Terraform, and where do the similarities and differences lie?

First, you have to learn a declarative scripting language in Terraform: the HashiCorp configuration language (HCL), a domain-specific language (DSL). Second, certain constructs like loops, functions, and classes are not available. You will also look in vain for a counterpart to the Pulumi service in Terraform. Instead of communicating with a central service, Terraform creates local state files, which it leaves to the administrator to manage.

Pulumi offers various automation features that are completely missing in Terraform. If you use the Prometheus monitoring, alarming, and trending solution, for example, you can set up basic Pulumi integration right away. The tool integrates running stacks into the solution on request.

Pulumi Commercial

If you want to use Pulumi in a team, you need a Pulumi Team License. The Starter License is designed for three users, costs a moderate $50, and can be used to start 20 stacks. The Pro version for $75 provides access for up to 25 members, teams, and roles; unlimited stacks; and support during business hours.

If you choose the Enterprise package, the price is a matter of negotiation. The package then includes 24/7 support, policies-as-code, and an unlimited number of members in teams. As already mentioned, the Pulumi service can be operated locally if you don't want it in the cloud. A detailed list of the functions associated with each level is provided by the manufacturer on its website [3].

Conclusions

I like the idea behind Pulumi. Anyone who has ever worked with Terraform and rummaged through its complex syntax may have wished they already had the skills it requires. Pulumi offers this possibility, even if not all of the learning overhead is eliminated. The author of a project still has to adjust to how individual classes or functions are called in Pulumi.

What is less pleasing is the relatively hard-wired tie-in to the Pulumi service. Although the tool can theoretically be used without this service, the practical benefits of the Pulumi service are quite clear. In commercial use, however, this also means that you will have to pay; although it is open source software, it is only free for personal use.

At the end of the day, Pulumi's pricing looks reasonable. Because the single-user license costs nothing, the solution can be put through its paces before you pay a single penny to the manufacturer. If you are looking for a tool for multicloud orchestration, Pulumi is definitely worth a look.