Change of Course

If you believe what Kohsuke Kawaguchi, inventor of the open source automation server Jenkins [1], is saying, open source continuous integration software is stuck in a "local optimum." Although traditional users like Jenkins, it can't attract new users. Meanwhile, new competition is growing on Jenkins' stomping ground [2].

The continuous integration/continuous delivery (CI/CD) landscape has changed completely in recent years. CI/CD was a new concept when Jenkins first started, but it is a matter of course today, and often a central component of many services. Although Jenkins has written its own success story over the past 10 years, some problems have grown with it and have not been solved.

Users today are looking for more plugins, more workloads, and more availability from Jenkins, pushing the software to its limits. According to Kawaguchi, the software is too complex. The operation of a larger Jenkins instance involves too much overhead, and sometimes daily restarts are necessary. Problems include pipeline execution, processes going haywire, and memory requirements.

Upgrade Pain

Many admins hesitate to update Jenkins and its plugins. Even simply adjusting job settings can sometimes cause undesirable side effects, which draws a picture of test software that itself is not well tested. Moreover, the Jenkins project, instead of moving forward, has developers struggling with compatibility issues.

According to Kawaguchi, Jenkins traditionally follows the Lego principle: Admins pick the parts they need and assemble their solution. This approach is no longer appropriate; more plugins are not a solution. On the contrary, Jenkins has to be far easier to use and ready for use immediately, which would improve support for users and keep the developer community going.

Kawaguchi offers a solution. His company, CloudBees, is looking to tackle the problems in collaboration with the Jenkins project. To achieve this goal, the project needs to pursue two key objectives: launch a cloud-native Jenkins and increase the speed of development for Jenkins 2.

Cloud Native Jenkins

A generalizable CI/CD engine running on Kubernetes is the idea behind Cloud Native Jenkins (CNJ). Thanks to a fundamentally different architecture and a new expansion mechanism, in the future, CNJ will scale arbitrarily on the basis of Kubernetes. The Cloud Native Special Interest Group (SIG) is responsible for implementation.

According to the plans, CNJ will enable a serverless or Functions-as-a-Service (FaaS) build execution, which already partially exists thanks to Jenkinsfile Runner [3] (Jenkins pipeline execution packaged as a command-line tool), and offer certain functions as microservices. Additionally, the services will use custom resource definitions (CRDs) to talk to Kubernetes.

Developers could extend the storage back end beyond local file systems to cloud-based storage solutions. Here, too, scaling is an essential point. Jenkins Configuration as Code (JCasC) [4] is set to play a central role.

Jenkins doesn't have to be reinvented completely for the cloud. The Jenkins X project (Figure 1) [5], for example, is already preparing the software for use in the cloud – specifically, on Kubernetes. The developers could customize Jenkins X to become Cloud Native Jenkins, and CNJ could then act as an engine for Jenkins X in the development process for other Cloud Native Apps.

Jenkins Evergreen [6], the automatically updating rolling distribution system, is an attempt to get away from the Lego principle, or to prepare Jenkins so that users can get it up and running quickly. Finally, the project has to focus more on security for CNJ (Secure by Design).

Minimum Viable Product

Kawaguchi proposes first to develop a minimum viable product (MVP, i.e., a slimmed down prototype) that will consist of a FaaS-like build engine that Jenkins X could use. In this way, the MVP would bring together existing projects such as Pipeline (plugins that support implementing and integrating CD pipelines into Jenkins), Evergreen, Jenkinsfile Runner, and JCasC.

Specifically, a webhook receiver can receive webhooks from GitHub and trigger the build engine, which in turn would launch a pipeline execution with the help of the Jenkinsfile Runner, where Jenkinsfile Runner exists as a FaaS function. JCasC files take care of configuration and plugin management. In the form of Evergreen, however, Jenkins is looking to achieve a manageable number of combinable plugin variations and is also set to step in as a test environment for CNJ itself.

What the MVP lacks is a graphical user interface. From the MVP, the Jenkins project will further enhance the software, adding new services, revising the extension system, and making up for lost features.

Jenkins 2 Reloaded

Although it will not be usable at first and later only on certain platforms, Jenkins 2 will work in the usual way, with a few changes. To develop it faster in the future and achieve greater stability, however, the developers are willing to drop some components. At the same time, reorientation requires changes to the development model.

The new release model will be based on Jenkins Evergreen; it will inevitably "disappoint expectations," especially in terms of compatibility. Jenkins will no longer be "compatible forever." The Java project keeps the option open for disruptive changes. If this requires major versions more frequently, Kawaguchi is open to them. In doing so, he is orienting his actions on the modified development model for Java SE, which picked up speed after restructuring.

Of course, the inconvenience of upgrading Jenkins needs to pay off for users at the end of the day. The project will remain largely compatible to protect existing job definitions and freestyle jobs. However, this apparently only applies to existing versions of Jenkins.

Also Ran

Developers are also looking to focus more on Configuration as Code, improve the developer experience by automatically detecting project types, and further optimize the Jenkins pipeline. At the end of the rebuild, Jenkins 2 also should work well with CNJ.

The Jenkins makers also plan to reduce the "feature interface" and thus focus on the essential components. The project wants to discover which services users actually need and what competitors offer. Missing or dropped components could then be supplied by Jenkins.

In the long run, Cloud Native Jenkins will also have a less extensive graphical user interface thanks to the use of JCasC, but the project will probably want to implement this as a separate app and not as a plugin. The existing Blue Ocean user interface [7] could serve as a template for the new user interface.