Jenkins Environment with Containers

Jenkins Logo

This blob post is about a special setup of Jenkins for multiple projects with different technology stacks, different environments, and other specialties. It helps to minimize resource usage, introduces strict isolation, and reduces maintenance effort.

This blog post assumes that you have basic knowledge of Jenkins, containers and source code management.

Common Scenario

A normal, default Jenkins setup is done on a single physical node or virtual machine. Jenkins builds, tests, and packages on this one main node. This is OK as long as only one technology stack is used. If there is a second stack emerging, usually an additional node with a Jenkins Agent is setup and the needed builds, tests, and packaging for this stack is delegated there by node labels.

In my current assignment, there are a lot of different stacks and technologies to handle. Which are (not complete):

  • Embedded firmware with C/C++ build with CMake
  • Java builds with Maven (different Java versions for compatibility and during migration phases)
  • Amazon AWS builds with Node and Python tools
  • Python projects
  • R projects
  • special environment to build and test Amazon AWS cloud setups

As described in the first paragraph, this let to the exact setup with multiple VMs to handle all this technology stacks and also parallel builds. There were about 8 VMs as build agents for the technologies mentioned about. (Not in scope here, but there are additional agents for hardware testing on Raspberry Pis and Windows VMs.)

All before mentioned 8 VMs consume in total about 40 vCPUs, more than 64GB of Memory and about 1TB of disk space. Everything runs on only two potent physical hosts.

Common Issues

No isolation on a node

Some projects need additional resources like databases in order to conduct integration tests. These potentially shared resources cause concurrency issues as parallel builds may alter the resource. Especially, when the environment needs to be cleaned up, a database is cleared. This also holds true for changes on the file system.

Waste of resources

A lot of VMs for different technology stacks, environments, and versions are created which use up a lot of resource like memory and disk space. In the example above, wouldn’t it be better to host only two potent VMs on the two hosts? One VM per host with, let’s say 24GB of RAM, 200GB of disk space, and 16 vCPUs? These two VMs would then have 4 executors who could handle all builds with 8 executors in total. It would reduce the overall resource consumption and the number of VMs would go down from 8 to 3 (1 Jenkins main node + 2 container build nodes).

Artifact accumulation

Artifacts are accumulating on VMs during builds. Especially tools like Maven and Node’s NPM download dependencies and do not clear the local caches. Over time these artifacts accumulate when versions are upgraded in the projects and own versions are released.

Not as worse as downloaded dependencies, but also log file may accumulate.

Easier support for different versions

When migrations need to be done from one version of a library or technology to another and not all branches can be migrated at first or different projects need different versions of something, it is very cumbersome and also a waste of resources to always setup a new VM for this special environment.

During compatibility testing against dependencies or environments. this is becoming a much more important issue. A low effort, low cost, but easy to setup solution is needed.

Solution: Containers

The idea to solve all the before mentioned issues is to provide the technology stacks, the needed environments, and tool versions via container images. The containers can be started at the beginning of a pipeline build and then run all the dedicated build steps inside the container.

The advantages are obvious:

  • Full Isolation and therefore parallel builds with same dependencies can be run what increases build stability dramatically.
  • Build environments are inside containers and one VM can build projects on all of those what helps load balancing, but also limits the overall amount of builds to not slow them all down too much.
  • As containers contain a whole environment, different environment settings can be applied to different projects increasing stability.
  • Clean containers are started from scratch from pre-defined images and the containers are removed afterwards cleaning up the space used during build. This assures always a clean starting point which increases stability.
  • Container images are much cheaper than VMs and it is much more easy to create multiple container images for different versions of an environment.
  • Container images are created ones and can be re-used for different projects what reduces maintenance effort.

Container image creation and Jenkins setup

In the next sections, I explain the needed setups of the containers and Jenkins itself. The focus is here on the basic setup and configuration. There are more blog posts planned which provide additional tips and trick about how to create multiple containers with similar content without duplicated code in a convenient way. Additionally, one blog post is planned about how to run Docker inside these containers for frameworks like AWS SAM or Localstack.

Container images for Jenkins builds

The container images are build with Buildah and run with Podman. For these tools I already wrote a small review in my post “Podman and Buildah for Containers“. Docker would probably work, too, but when Jenkins agents are run with a non-root user, the calls need to be wrapped into sudo calls or the permissions for Docker need to be changed what could lead to security risks.

In this post we go solely with Podman what is preferred over Docker (see the post above). I also explain the container image creation with Containerfiles here. The scripted approach for Buildah will work, too.

We also assume that the only user inside the container is the root user what makes the build commands easier.

In order to achieve our goals, we need to get some things worked out:

  1. How do we use the Jenkins checked out branch from the host? Jenkins is still managing the GIT checkouts on the host and this is something we want to keep. (If you work with another SCM than GIT, this fact is still true, but you may need to adapt some details to your tool.)
  2. How do we handle credentials? It is not a good idea to package passwords, license keys, or application secrets into the container images. As soon as these escape into the wild, the security implications are potentially catastrophic.
  3. As containers are process based, how do we manage that containers stay alive during build? And, how do we implement a hung-build killer in case something goes wrong? A container started, but not stopped, would keep resources allocated, what is not wanted.
  4. How do we assure that the container names are unique that builds can run in parallel? The automatically created names are good for uniqueness, but the name is not so easy to be retrieved, but is needed to run the correct commands in the correct container.
  5. How do we manage that container which are stopped are completely removed?
  6. How do we handle additional services to run inside the container during build like databases or web servers?

Using a work directory for GIT checkout

For the GIT checkout, simply add a work directory to the container. It could be called /work for instance. In the container file, just add a VOLUME:

VOLUME /work

In Jenkins the workspace folder can be mapped via the environment variable $WORKSPACE.

Credentials for builds

The credentials or other secrets must not, under any circumstance be copied into container images. The simplest way is to use additional VOLUMEs to provide a mechanism to hook them into the container from the secured host if needed.

For instance, you could add volumes like:

VOLUME /root/.aws
VOLUME /root/.gnupg

During container startup, the appropriate folders can be mapped.

Container lifetime and Hung-Build Killer

Containers are process based. The container stays alive as long as a dedicated process is still running. As soon as the process finishes, the container is stopped.

The simple trick is here to use the sleep command in order to keep the container alive for a given amount of time. The nice side effect is that the hung-build killer comes for free, as the hung-build will timeout automatically with the sleep command finishing.

Inside the container file add the following command:

CMD sleep $CONTAINER_TIMEOUT

The environment variable $CONTAINER_TIMEOUT can be set during container start.

Unique container names

The container names I use are concatenated names with a name defined in the Jenkins pipeline and the build number of Jenkins given in the environment variable $BUILD_NUMBER.

The project name can also be used, but based on the setup it may contain invalid characters which are not supported by Podman like whitespace. Therefore, I would not recommend it as a generic approach. But, it can be valid in special setups with strict rules.

Automated container removal

Podman has an option --rm to define that the container is completely removed with all its content when stopped.

Additional services to run

In case additional commands are to be run in order to start services, just chain them into the command provided in the Containerfile:

CMD runuser -l postgres -c postgres & sleep $CONTAINER_TIMEOUT

This is an example how to start a PostgreSQL database as postgres user in the background before running the sleep explained above.

Putting it all together in the Jenkins Pipeline

For this blog post a declarative pipeline Jenkinsfile is used to reduce the complexity. The scripted version works the same way.

Environment variables and options for convenience

The following snipped of a beginning of a declarative Jenkinsfile shows the basic configuration and initialization of the pipeline

pipeline {
    agent {
        node {
            label 'container'
        }
    }
    // [...]
    options {
        timeout(time: 15, unit: 'MINUTES')
        // [...]
    }
    environment {
        CONTAINER_NAME = "projectname-${BUILD_NUMBER}"
        EXEC = "podman exec -w /work ${CONTAINER_NAME}"
    }
    stages {
        // [...] steps are discussed below
    }
    // [...]
}

Important is to select the correct node with the agent section. As example the nodes providing container builds are labeled with ‘container‘.

Additionally, Jenkins shall get a timeout on its own to abort the build and free the executor.

For the environment we do two things:

  1. We build a container name and put it into the environment variable CONTAINER_NAME for later re-use. As mentioned above, we define here a project name on our own and use the environment variable $BUILD_NUMBER by Jenkins to create a globally unique container name.
  2. A second environment variable is created for convenience: It helps to run the build commands inside the started container. For details, read on.

Important: The parameter -w /work defines the default working directory when the container is started. It sets the directory to the Git repository. This is the exact same location where Jenkins puts you when it starts the shell commands in the pipeline. We do the same here inside the container, so the calls for the builds stay mostly the same, but are just run inside the container.

Start the container

The following snippet shows the startup of the container and the cleanup of the environment before:

pipeline {
    // [...]	
    stages {
        stage('Initialize') {
            steps {
                sh 'git clean -d -f -x'		   
            }
        }
        stage('Start Container') {
            steps {
                sh 'podman run --rm --privileged --volume $WORKSPACE:/work --env CONTAINER_TIMEOUT=900 --name ${CONTAINER_NAME} localhost/puresol-technologies/build-container:1.0.0 &'
            }
        }
        // [...]
    }
    // [...]
}

First step is here to call a git clean to clear up the whole checkout done by Jenkins. As there might be still some artifacts inside the checkout, these are clear out. We have a clean environment afterwards what helps to stabilize the builds.

Afterwards Podman is used to run the container. Tge parameters and their meaning are:

  • --rm tells Podman to remove the whole container instance after it is stopped. podman ps -a will not show this container after the run.
  • --privileged is maybe too lacy here, but it runs the container unrestricted. If we would not have a container, there would also not be additional security and the container is short lived. So, I think it is fine and it helps to get a stable environment.
  • --volume $WORKSPACE:/work mounts the current Jenkins workspace aka. Git repository directory into the container to the former mention /work directory. In there, we find the project to build, test, and package. All artifacts which need to be accessible after the container stopped need to be put in there.
  • --env CONTAINER_TIMEOUT=900 sets the environment variable CONTAINER_BUILD inside the container. This is the environment variable used by the sleep command which controls the container lifetime. The setting is in seconds and should be in the same range like the Jenkins timeout setting.
  • –name ${CONTAINER_NAME} give the container the pre-defined name as mentioned above.
  • localhost/puresol-technologies/build-container:1.0.0 is the container image to be run. Here the image is taken from the local repository from the host where Buildah created the image. When localhost/ is removed, it will be taken from the defined container image repositories.
  • & at the end decouples the call from the current terminal and moves the process into the background. The shell command will immediately return and let Jenkins run the next steps in the pipeline. The container is running in the background.

If credentials are needed as mentioned above, additional --volume parameter should be added.

Run build commands inside the container

The actual call for running a build command could look like this:

pipeline {
    // [...]	
    stages {
        // [...]
        stage('Build') {
            steps {
                sh '$EXEC make -j'
            }
        }
        // [...]
    }
    // [...]
}

In most circumstances, the original build command is used and just called by using the pre-defined $EXEC environment variable which contains the Podman call to execute a command. In our example it is the make -j command to compile C/C++ code with maximum parallelism.

Stopping the container after build

In order to stop the container after a build (either successful or failed), add a post section at the end of the Jenkinsfile to stop the container in an orderly manner and let it get removed:

pipeline {
    // [...] 
    post {
        always {
            // [...]
            sh 'podman stop ${CONTAINER_NAME}'
            // [...]
        }
    }
}

Debugging

Some words on debugging, because there is always a chance that something goes wrong.

There are two simple options how to start:

  1. Setup the Jenkinsfile with increased CONTAINER_TIMEOUT value and comment the postman stop command. With that you can have a look into the still running container after the build has failed to check the content.
  2. You can check the Jenkins console output and look for the podman run command and start the container again on the command line on the VMs. Maybe, the CONTAINER_TIMEOUT variable can be increased.

When the container is still running then a bash can be started inside the container for debugging with something like:

$ podman exec -it <container name> bash

Conclusions

The described setup is running very well and it can build Java projects, NodeJS projects, and others very well. As there are only two VMs involved which can handle all projects, the load is nicely distributed over the physical hosts and not as before handle by one VM on one physical host per technology. It indeed saves resources and increases build times, as the load balancing of this generic build VMs reduces the wait time for executors for builds.