Handmade

Whether you are new to continuous integration/continuous deployment (CI/CD) pipelines and the world of DevOps or fully familiar with such practices, one name always comes to the fore when discussing the automation of processes: Jenkins [1]. The highly popular automation server is open source, and although capable of fulfilling many types of complex tasks, it is equally accessible to novice users through its straightforward user interface (UI). In this article, I look at how I solved a recent headache on a project in which I was running Jenkins as a Docker container for the demonstration of a vendor-supplied Jenkins plugin.

Although I haven't used Jenkins much in the past, the quandary I faced and, most importantly, resolved could affect novice and advanced users alike. The problem was that once I had figured out how to update Jenkins plugins online, I couldn't connect a plugin to remote software as a service (SaaS) because it was using self-signed security certificates that were not recognized by Jenkins.

The Version

According to my relatively cursory inspection of the provided Docker containers, the first thing to note about Jenkins is that the version you use can be the difference between documentation immediately solving a problem or compounding it further. From the container perspective, I opted to look at two versions in particular: the very latest and greatest weekly release called jenkins/jenkins:latest and the jenkins/jenkins:lts, where lts stands for the long-term support stable version.

The reason the version seems to matter so much is that Jenkins is developed at great speed and is constantly evolving. Admirable and fantastic as that is, every now and again trying to solve an issue on a version that hasn't been written about much online is tricky at best. This situation applies to most modern software and definitely not just the pervasive-for-good-reason Jenkins.

Although Jenkins provides a carefully considered and slick UI, in this article I focus on the container side and a number of command-line interface (CLI) options. Again with reference to the importance of versioning, the issue I encountered when trying to figure out UI options was that several version changes ago a number of menu options were deprecated. Nonetheless, with some trial and error I settled with the LTS version of the container image and the plugin working as hoped. The results can be reproduced easily to accommodate other online services that Jenkins doesn't immediately work with out of the box because the certificate authority isn't recognized.

The Installation

I prefer to save complex Docker run commands in a small script, as shown in the run_jenkins.sh file shown in Listing 1. (Remember to run

chmod +x run_jenkins.sh

to make it executable.)

docker run -d \
    --name jenkins-lts \
    --user root \
    -p 0.0.0.0:8080:8080 \
    -p 0.0.0.0:8443:8443 \
    -p 0.0.0.0:50000:50000 \
    --env JENKINS_OPTS="--httpPort=8080" \
    --env JAVA_OPTS=-"Djavax.net.ssl.trustStore=/var/jenkins_home/keystore/cacerts" \
    -v jenkins_config:/var/jenkins_home \
    jenkins/jenkins:lts

Rather than just blindly running that script, which you can do without breaking anything in a laboratory environment, you might first want to read through to the end of this article to understand why --user root is present, along with the JAVA_OPTS environment variable.

Importantly, before creating a container with the script in Listing 1, I will ensure that changes to configuration data persist between stops and starts by creating and checking a volume called jenkins_config:

$ docker volume create jenkins_config
$ docker volume ls | grep jenkins
local jenkins_config

To run the script, proceed with creating a container:

$ ./run_jenkins.sh

If you run $ docker ps and no errors are present, you should see it running.

The Plugins Quandary

The more minor of my connectivity issues were that I couldn't update plugins to get the most secure versions. Jenkins uses an online repository to store and serve its incredible library of exceptionally useful plugins. It seems, however, that under some scenarios, even when a proxy isn't used to connect to the Internet, Jenkins struggles to connect over HTTPS.

After hunting online and trying a few different fixes without success, what worked for me was a remarkably simple URL edit (Figure 1) that let me continue updating and installing the plugins required for the project demo I was preparing. To get to that UI option, navigate to Manage Jenkins | Manage Plugins, select the Advanced tab, and scroll to the bottom of the page.

Simply alter https to http in the URL and then click the Submit button. You can now promptly navigate to the Check Now button, which is on the bottom right of that same page. Then scroll back to the top of the page and click the Updates tab to check that the fix worked. If for some reason that doesn't work, go into your browser settings and see if you are using a proxy to access the Internet. If you are, at the top of the same page as shown in Figure 1, you can enter your proxy information manually (e.g., web.proxy.org:8080), along with login details if they are used. If it works, update all your plugins as required.

If you are keen to use HTTPS for your plugin connectivity (which is obviously recommended for the benefit of security outside of a lab environment), you can potentially use the next solution, or (armed with the knowledge that JUC stands for Jenkins Update Center) you could create your own plugin repository locally, according to instructions online [2]. This step could also be useful if you want to limit which plugins developers have access to for security reasons.

The Self-Signed Certificate Palaver

After investigating various online fixes, I was eventually able to connect the proprietary Jenkins plugin I was using to an online service even though a certificate authority didn't recognize its self-signed certificate. The solution came about through distilling some of the instructions found on Stack Overflow [3]. The salient bits are as follows:

• Enter the container as the root user (just ensure the --user root \ entry is on a line near the top of Listing 1) and run a keytool command to add your online service's certificate to the cacerts file. Next, copy that certificate to your persistent volume.
• With the use of an environment variable when the container is spawned, ensure that Jenkins uses the correct cacerts file on your volume to prevent rebuilds and reboots from failing in the future.

The following workflow shows the simple steps needed to add a recognized self-signed certificate to Jenkins.

Step 1

Enter the Jenkins container by running an exec command to access the filesystem of the persistent volume created. Find the hash ID of the Jenkins container – in case you didn't name it with the -name option in Listing 1. Use the hash ID to enter the container to run the desired commands:

$ hash=$(docker ps | grep jenkins | awk '{print $1}')
bf66cc1b2916
$ docker exec -it ${hash} bash
root@bf66cc1b2916:/#

The first command will only work if one instance of Jenkins is running; in in the unlikely event you are running two Jenkins instances, edit the command accordingly.

Now you should be able to enter your Jenkins container. You have a root prompt inside your container because you left the --user root \ line in Listing 1. You need to be the root user to run the keytool command, but you should remove that line later for much more security. The container should run as the jenkins user by default. Line 47 in an online example that shows the Dockerfile used to create the LTS image should convince you [4].

Step 2

Visit the website that you want Jenkins to trust (e.g., with Google Chrome on Linux, Figure 2 – or you can use openssl) and save the certificate to a location by accessing the website and clicking on the address bar padlock to download the certificate file locally. The file you are after usually will end in .pem – or .cer on Windows.

Once you have clicked the Certificate field highlighted in Figure 2, click the Details tab at the top of the next dialog box and then Export at the bottom (Figure 3).

Once Export has been clicked, you can follow the prompts to save the certificate locally. On Linux Mint, I just pressed Save As; by default, cloudnativesecurity.cc is the suggested name because that's the website I visited in my browser, so I adjusted the name slightly to cloudnativesecurity.pem. If you open your PEM file, you can see it's a standard certificate file that will look something like:

$ cat cloudnativesecurity.cc
-----BEGIN CERTIFICATE-----
MIIF1zCCBL+gAwIBAgIRAK7AdUDa5C4Y1o6SSOX4aC0wDQYJKoZIhvcNAQEL
...

Step 3

Now that you have the certificate you want to trust saved locally, copy it with a Docker cp command to the /tmp directory in the Jenkins volume, so you can later move it to a more relevant location:

$ docker cp cloudnativesecurity.pem ${hash}:/tmp/cloudnativesecurity.pem
$ docker exec -it ${hash} ls /tmp

Although a relatively simple step, take care with the syntax of this first command, which reuses the hash variable and the filename chosen earlier for the .pem file. The last command checks to make sure the file made it to its new location, as hoped. (On my version of Docker, the cp command does not seem to show errors if it fails to copy.) If you see your file in the listing, it worked and you can proceed.

Step 4

Now that you have copied the file into your container, you can use your trusty keytool command to add it to your trusted certificates by adding the certificate to the sites that Jenkins trusts. This excellent command can be used to import certificates into the cacerts file that Jenkins uses in its keystore, and with a single command you will be asked to confirm whether to trust the self-signed certificate that you have just imported.

Before you do that (as I discovered after much reading), the Java virtual machine paths seem to differ in some Jenkins versions, so you need to know precisely where the file resides. For this step, you need to be inside the container (as per Step 1). Incidentally, as noted, I am entering this container as the root user on purpose, which is required to run keytool. Use the --user root \ option when you are performing these steps, before you complete the process and need to switch from the jenkins user (just delete that --user root \ line in Listing 1 to do so). Now, check where the existing cacerts file is from inside the container:

root@bf66cc1b2916:/# find / -name cacerts
/usr/local/openjdk-8/jre/lib/security/cacerts

Found! The next step is updating the local cacerts file and then copying it into your persistent volume, having made it readable by the jenkins user. The keytool command refers to the /tmp path from earlier (Listing 2).

root@bf66cc1b2916:/# keytool -import -alias CNS-cert -keystore /usr/local/openjdk-8/jre/lib/security/cacerts -file /tmp/cloudnativesecurity.pem
...
#9: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: E2 B9 A7 59 F6 11 B4 00   3B 76 56 1F 29 5D CF 91  ...Y....;vV.)]..
0010: EA AB 17 F6                                        ....
]
]
Trust this certificate? [no]:

After you enter your password (the default appears to be – all lowercase – changeit), look at the end of the output where it asks whether you want to trust the certificate. At this point, you can type yes to proceed. You are then offered the confirmation Certificate was added to keystore.

Now that you have updated the cacerts file in your container (so that it survives reboots) and the container receiving the docker rm command, copy it to your persistent volume and make sure the jenkins user can read from it correctly:

$ mkdir /var/jenkins_home/keystore
$ cp /usr/local/openjdk-8/jre/lib/security/cacerts /var/jenkins_home/keystore/cacerts
$ chown /var/jenkins_home/keystore/cacerts

Step 5

Almost finished! You just need to ensure at startup that Jenkins is looking in the correct place for the cacerts file to which you have just added your certificate. Add an environment variable that points at your persistent volume path, which the container will use when spawned, by adding the line

--env "JENKINS_OPT=-Djavax.net.ssl.trustStore=/var/jenkins_home/keystore/cacerts"

to Listing 1. This step ensures that Jenkins is using the correct cacerts file.

Step 6

Restart Jenkins and ensure that changes persist. Because the data is now safe on the persistent volume, stop and delete the older Jenkins container with the commands

$ docker stop $(hash}
$ docker rm $(hash}

Having made the appropriate changes to the Jenkins start-up script (removing the --user root line) and ensuring the addition of the environment variable for the path, you can now execute your run_jenkins.sh script again to start up your container once more.

Once you have patiently waited a couple of minutes for the UI to start up, any plugins that interact with https://cloudnativesecurity.cc should do so without generating any errors. Rinse and repeat for other online services to which you need to connect.

The End

Having also tried the Skip Certificate Check plugin [5] for Jenkins updates, I was relieved that adjusting the cacerts file worked. I tested a number of apparent fixes, such as adding a /etc/default/jenkins.xml file.

The benefits of getting this setup working is that, first, your Jenkins instance is able to confirm that it is connecting to the correct online service (and that no imposters are involved). Second, your traffic is transmitted in an encrypted format. I trust this will help you, too, at some point in the future when using Jenkins.