GitLab Runners for CI/CD
If a GitLab repository contains a continuous integration pipeline, its jobs will be executed via a GitLab Runner. A GitLab Runner is a daemon running on an other server, waiting to be contacted by the central GitLab server to execute CI pipelines.
You can find more detailed information about GitLab Runners in the GitLab Documentation.
There are two different ways of using a GitLab runner:
individual runners, installed on local machines, remote clusters, or cloud-based systems by individual users (without support from MPCDF)
shared runners, which are offered by MPCDF
If you don’t want to install and configure an individual GitLab runner, you can execute your CI pipelines on the shared runners offered by MPCDF. If you don’t add one or more tags to your CI file .gitlab-ci.yml, your pipeline will automatically executed by a shared runner.
Docker images for CI with MPCDF environment modules
To provide the developers of HPC applications with a familiar and comprehensive software environment also within GitLab-based continuous integration (CI) pipelines, the MPCDF is offering special Docker images. These images use environment modules to make software accessible, in a very similar way to how software is managed on the HPC systems. Hence, e.g. build scripts will work on both the HPC systems and the CI cloud runners in a consistent way.
Image tagging-and-purging strategy
latest and the calendar year
To limit the individual growth of these Docker images over time, we put the following tagging-and-purging strategy in place:
Essentially, all images are tagged using
latest and/or the calendar year. In
the course of a year, say 2024, the images tagged with
latest and the year
2024) are identical and receive regular updates and additions of software.
With the beginning of the new year, all images tagged with the previous year
stay unchanged (frozen). The newly created images for 2025, say, will start out
in early January again in a slim state and will be tagged
latest. Users can
then choose to migrate to the more recent images (tagged
our example) or stick with the older (but static!) images (tagged
2024) for a
In case a user opts for using the tag
latest, please be warned that the
software environment will likely change at the beginning of each year.
Non versioned images tagged
Moreover, we provide special images without an explicit version number pointing
to the respective most recent compiler and MPI in the MPCDF software stack. For
the compilers, for example, we offer the images
similarly for depending images containing Intel- or OpenMPI.
See the page on the
latest image tag
for an overview.
A typical use case for these images is a user’s code which should be built and
tested with the newest available compiler. Only the tag
latest exist for these
non-versioned images. In order to seamlessly receive updates, the
command should also omit the compiler’s and MPI’s version number in this case
(for example, using
module load gcc instead of
module load gcc/13).
Case Study: Set up a CI job using a recent Intel C++ compiler
This section shows how to set up a CI job to compile and test some C++ software using the Intel compiler. The required steps are:
Identify the Docker image that provides the required software by checking the CI Docker image website. Copy the image tag, in our example we’re using
.gitlab-ci.ymlfile and paste the image tag after
Select proper tags for the shared runners you’re intending to use.
.gitlab-ci.ymlfile might look as follows:
- module load intel/2023.1.0.x
- module load gcc/13
- module load cmake
- icpx --version
# compile C++ code and perform tests ...
Migrating from the legacy
module-image to the new CI module images
Users of the previous
module-image are encouraged to migrate to the new CI
images now, report potential issues and request additional software modules via
the helpdesk, if necessary. As shown in the previous section, the
module-image can simply be replaced in the user’s
.gitlab-ci.yml file with
one of the new images that provides the desired software stack for the
respective CI job.
The new CI module images were first introduced in the December 2023 edition of the Bits and Bytes.