Efficient Container Creation with Make and C Preprocessor

Posted on by Rick-Rainer Ludwig

During the writing of “Jenkins Environment with Containers“, I was faced with some inefficiencies during the creation of all the needed containers. How to create and handle a lot of different containers in an efficient. automated way is described below.

Example and Issue

When a lot of different containers with different content is to be created and some containers shall share the same components, then some copy and paste code appears.

Let’s work with a small example:

  • Root OS Container: Contains a a base image with local, default settings to a local repository
    • Java Container: Contains additional a JDK environment to build Java projects
    • NodeJS Container: Contains additional NodeJS environment to build JavaScript projects
    • Web Application Container: Shall contain Java and NodeJS to build a SpringBoot based application with a web frontend in ReactJS

The Root OS Container is a base container for all other containers inside the project. The reason for it is to have a clear, generalized starting point for all other containers. This container uses an image from CentOS, OpenSuSE, or something similar and adds standard configuration for a local repository cache for instance to save internet bandwidth and to speed up container deployment when distribution content is to be installed. In the examples. I will call this image puresoltechnologies/opensuse:15.4.

The other containers are the actual functional containers. The functional containers should follow one simple rule:

Container Rule: Only the essential tools shall be inside containers. No additional tools shall be used unnoticed. Therefore, clear environmental dependencies are provided.

The creation of a Java container for the Jenkins Container Environment with a clearly defined Java version could look like this:

FROM puresoltechnologies/opensuse:15.4
LABEL com.puresoltechnologies.image.author="Rick-Rainer Ludwig <ludwig@puresol-technologies.com>"

RUN zypper install -y curl tar gzip

# GraphViz for JavaDoc to create graphs
RUN zypper install -y graphviz graphviz-gd

# Installing additional fonts for GraphViz needed by PlantUML and SchemaSpy
RUN zypper install -y intlfonts-type1-fonts ghostscript-fonts ghostscript-fonts-other ghostscript-fonts-std ghostscript-fonts-std-converted gnu-unifont-bitmap-fonts google-opensans-fonts xorg-x11-fonts

RUN cd /opt && curl "https://aka.ms/download-jdk/microsoft-jdk-11.0.15-linux-x64.tar.gz" -L | tar -zxvh
RUN cd /opt && ln -s jdk-11.0.15* jdk
ENV PATH=$PATH:/opt/jdk/bin
ENV JAVA_HOME=/opt/jdk

RUN zypper clean --all

VOLUME /work

CMD sleep $CONTAINER_TIMEOUT</code>

An equivalent for Node could look like this:

FROM puresoltechnologies/opensuse:15.4
LABEL com.puresoltechnologies.image.author="Rick-Rainer Ludwig &lt;ludwig@puresol-technologies.com>"

RUN zypper install -y curl tar xz
RUN cd /opt &amp;&amp; curl "https://nodejs.org/dist/v16.16.0/node-v16.16.0-linux-x64.tar.xz" | tar -Jxvh
RUN cd /opt &amp;&amp; ln -s "node-v16.16.0-linux-x64" node
ENV PATH=$PATH:/opt/node/bin
ENV NODE_HOME=/opt/node

# Registr Nexus as repository
RUN npm config set registry "http://nexus.puresol-technologies.net/repository/npm"
RUN npm config set email "ludwig@puresol-technologies.com"
RUN npm config set _auth "XXXXXXXX"

# Git installation to get npm version working which needs git command present
RUN zypper install -y git
RUN git config --global user.email "jenkins@puresol-technologies.com"
RUN git config --global user.name "Jenkins"

RUN zypper clean --all

VOLUME /work

CMD sleep $CONTAINER_TIMEOUT</code>

These two containers are in general straight forward, but of course, as we use the original downloads from the internet and not the distribution packages, there is some code involved.

Now, what about the last container? In this project NodeJS is needed in combination with Java. As there is no way to just ‘merge’ the first two containers to have a third one containing all tools, what is to be done?

We could use the Java container and add NodeJS there or we could use the NodeJS container and add Java there. Nevertheless, there would be copy and paste code. As we all know:

Copy and Paste code is evil! 🙂

So what can be done?

In the past, I was developing automation tools based on C/C++. There were tools which I liked a lot. Not only for C/C++ development, but for the general purpose (mis-)use capabilities: C Preprocessor (cpp) and GNU Make (make).

Solution to avoid duplicate code in Containerfiles

In order avoid duplicated code, the duplications are to be extracted into snippets which can be included with a pre-processor #include statement.

There are two changes involved in order to get it work:

Rule 1: Comments are started with a double slash // and not with a sharp #.

Rule 2: URLs (and all other strings containing a double slash) need to be quoted.

The java.inc file to be included for Java development looks then like this:

RUN zypper install -y curl tar gzip

# GraphViz for JavaDoc to create graphs
RUN zypper install -y graphviz graphviz-gd

# Installing additional fonts for GraphViz needed by PlantUML and SchemaSpy
RUN zypper install -y intlfonts-type1-fonts ghostscript-fonts ghostscript-fonts-other ghostscript-fonts-std ghostscript-fonts-std-converted gnu-unifont-bitmap-fonts google-opensans-fonts xorg-x11-fonts

RUN cd /opt &amp;&amp; curl "https://aka.ms/download-jdk/microsoft-jdk-11.0.15-linux-x64.tar.gz" -L | tar -zxvh
RUN cd /opt &amp;&amp; ln -s jdk-11.0.15* jdk
ENV PATH=$PATH:/opt/jdk/bin
ENV JAVA_HOME=/opt/jdk</code>

Equivalent, the node.inc is looks after the adaptions like this:

RUN zypper install -y curl tar xz
RUN cd /opt &amp;amp;&amp;amp; curl "https://nodejs.org/dist/v16.16.0/node-v16.16.0-linux-x64.tar.xz" | tar -Jxvh
RUN cd /opt &amp;amp;&amp;amp; ln -s "node-v16.16.0-linux-x64" node
ENV PATH=$PATH:/opt/node/bin
ENV NODE_HOME=/opt/node

# Registr Nexus as repository
RUN npm config set registry "http://nexus.puresol-technologies.net/repository/npm"
RUN npm config set email "ludwig@puresol-technologies.com"
RUN npm config set _auth "XXXXXXXX"

# Git installation to get npm version working which needs git command present
RUN zypper install -y git
RUN git config --global user.email "jenkins@puresol-technologies.com"
RUN git config --global user.name "Jenkins"

The two container file for Java and NodeJS get now pretty simple:

FROM puresoltechnologies/opensuse:15.4
LABEL com.puresoltechnologies.image.author="Rick-Rainer Ludwig &lt;ludwig@puresol-technologies.com>"

#include "java.inc"

RUN zypper clean --all

VOLUME /work

CMD sleep $CONTAINER_TIMEOUT
FROM puresoltechnologies/opensuse:15.4
LABEL com.puresoltechnologies.image.author="Rick-Rainer Ludwig &lt;ludwig@puresol-technologies.com>"

#include "node.inc"

RUN zypper clean --all

VOLUME /work

CMD sleep $CONTAINER_TIMEOUT

The third Containerfile for the combined Java and NodeJS becomes as simple as the other two and does not contain duplicated code:

FROM puresoltechnologies/opensuse:15.4
LABEL com.puresoltechnologies.image.author="Rick-Rainer Ludwig &lt;ludwig@puresol-technologies.com>"

#include "java.inc"
#include "node.inc"

RUN zypper clean --all

VOLUME /work

CMD sleep $CONTAINER_TIMEOUT

When include files are used to put content into the containers, pay attention to make the include files self-contained. Each single include file should work on its own and also the order of includes shall not have any influence of the final result.

Efficient creation of Containerfile files with Make

In order to use the C Preprocessor it is helpful to use another tool: make.

First, the Containerfile files with #include statements are renamed to Containerfile.in. These files are then processed with cpp into valid Containerfile files and then processed with buildah.

A Makefile looks like this:

Containerfile: Containerfile.in java.inc
	cpp $< -o $@

build: Containerfile
.created: Containerfile
	buildah bud -t puresoltechnologies/java:11.0.15
	touch $@

.DEFAULT_GOAL := build
build: .created

inspect:
	podman run --rm -it localhost/puresoltechnologies/java:11.0.15 bash

clean:
	rm .created
	rm Containerfile

There are some interesting points here:

  • The .created file is used to allow Make to judge when a rebuild is needed by comparing the modification timestamps of this file and the Containerfile.
  • The dependencies for Containerfile are Containerfile.in and java.inc. This allows Make to rebuild the Containerfile when either of the dependencies was modified.
  • An additional inspect target was added for convenience to start the newly built container for debugging.
  • A clean target was added to clean the folder.