HPC Software and Applications
How and where can I install software myself?
On the HPC systems and clusters, users can install software into their home directory where they have write permissions. For typical software installation no root privileges are necessary.
Make sure to set the installation directory accordingly as shown in the following examples for different build systems:
# GNU build system
./configure --prefix=$HOME/soft/package_name ...
make && make install
# CMAKE example
cmake -DCMAKE_INSTALL_PREFIX:PATH=$HOME/soft/package_name ...
make && make install
# Python example, installing into '~/.local'
python setup.py install --user
# Python/pip example, installing into '~/.local'
pip install --user package_name
Details on the build and install procedure are usually given in the documentation that comes with the source files. In case of problems users may request support via the helpdesk.
The MPCDF uses modules to adapt the user environment for working with
software installed at various locations in the file system or for switching
between different software versions. The user is no longer required to
explicitly specify paths for different executable versions, or keep track of
MANPATH and related environment variables. With the modules
approach, users simply ‘load’ and ‘unload’ modules to control their
environment. Note that since 2018, HPC systems as well as the increasing
number of dedicated clusters all use hierarchical environment modules.
How can I use environment modules interactively?
Here is a list of the most important commands (see https://modules.readthedocs.io/en/latest/module.html for a complete reference):
module help: lists module subcommands and switches
module avail: lists available software packages and versions which can be enabled (“loaded”) with the module command
module help <package>/<version>: provides brief documentation for the specified module.
module load <package>/<version>: “loads” the module, i.e. modifies the user’s environment (
module unload <package>/<version>: “unloads” the module
module list: lists all modules which are currently loaded in the user’s environment
How can I use environment modules in scripts?
Environment modules are provided by default only in interactive login shells. For non-interactive shells (e.g. in shell scripts or batch scripts) the following system profiles must be explicitly sourced (for Linux, specifically SLES):
/etc/profile.d/modules.shfor sh-derivative shells: sh, bash
/etc/profile.d/modules.cshfor csh-derivative shells: csh, tcsh
How can I use environment modules to avoid absolute paths in my scripts?
Instead of absolute paths to libraries, binaries etc. the environment
variables set by the modulefile should be used in scripts, makefiles etc. By
convention, an MPCDF modulefile sets an environment variable named
<PKG>_HOME (where PKG is the name of the package, for example:
which points to the root directory of the installation path (see below for
example usage). Information about additional, package-specific environment
variables can be obtained with the commands
module help <package>/<version>
module show <package>/<version>.
Interactive session on the command line
$ module load intel mkl
$ ifort -I$MKL_HOME/include example.F -L$MKL_HOME/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
$(FC) -I$(MKL_HOME)/include test.F -L$(MKL_HOME)/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
How can I read the long output from
module avail page by page?
# sh/bash shell
( module avail ) 2>&1 | less
# c/tcsh shell
( module avail ) | & less
How do the hierarchical environment modules work?
To manage the plethora of software packages resulting from all the relevant combinations of compilers and MPI libraries, we have decided to organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the uppermost level, depending libraries (e.g., MPI) on the second level, more depending libraries on a third level. This means that not all the modules are visible initially: only after loading a compiler module, will the modules depending on this become available. Similarly, loading an MPI module in addition will make the modules depending on the MPI library available.
To start at the root of the environment modules hierarchy, issue
For example, the FFTW library compiled with the default Intel compiler and the default Intel MPI library can be loaded as follows: First, load the default Intel compiler module using the command
module load intel
second, the default Intel MPI module with
module load impi
and, finally, the FFTW module fitting exactly to the compiler and MPI library via
module load fftw-mpi
You may check using the command
that after the first and second steps the depending environment modules become visible, in the present example impi and fftw-mpi. Moreover, note that the environment modules can be loaded via a single ‘module load’ statement as long as the order given by the hierarchy is correct, e.g., ‘module load intel impi fftw-mpi’.
How do I quickly find a module?
In case you know the name of the module you wish to load, but you are not sure about the available versions or what dependencies need to be loaded first, you can try to use the ‘find-module’ command. This tool searches for the MODULENAME string through a list of all installed modules
You can then choose the desired module version, use the output of the command to determine the correct order to load dependencies, and finally load the module itself, e.g.
$ find-module horovod
horovod/cpu/0.13.11 (after loading anaconda/3/2019.03 tensorflow/cpu/1.14.0)
$ module load anaconda/3/2019.03 tensorflow/cpu/1.14.0 horovod/cpu/0.13.11
It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the uppermost hierarchy. Note that a hierarchy exists for Python modules with the ‘anaconda’ module files on the top level.
How can I disable the “MPCDF specific note” for
On recent installations, the
module avail command prints a “MPCDF specific note”
that describes how to find nested environment modules in the hierarchy. To disable
that note, set the environment variable ‘MPCDF_DISABLE_MODULE_AVAIL_HINT’ to ‘1’, e.g.,
by adding the line
export MPCDF_DISABLE_MODULE_AVAIL_HINT=1 to your ‘.bashrc’ file.
Why there are no BLAS/LAPACK modules available on the clusters?
Intel’s Math Kernel Library (MKL) library provides highly optimized replacements for the BLAS, Sparse BLAS, LAPACK, ScaLAPACK, PBLAS, and BLACS libraries as well as other math routines. For more information please refer to our MKL guide.
Which CMake version should I use?
CMake is backwards compatible. This means you can load the newest version and still run old CMakeLists.txt files (>3.0). However, since CMake increases support for newer compilers and libraries with every version you should load the newest version available to get best possible support for the environment.
Policy CMPXXXX is not set - Warning
The behavior for something you are using has changed between the CMake version specified in the CMakeLists.txt file and the currently loaded CMake executable version. As a user you do not have to worry. CMake will act according to the behavior specified in the CMakeLists.txt file version. As a developer look at the changed behavior and check if you code is compatible with it. Then set the policy explicitly to the expected behavior. (https://cmake.org/cmake/help/latest/manual/cmake-policies.7.html)
CMake cannot find library XXX
CMake checks all standard installation directories for requested libraries. If it cannot
find the library there or you want to enforce a different install location use the
XXX_ROOT (case-sensitive!) environment variable and point it to the installation directory.
We try to set this variable automatically for all packages loaded on the cluster. However, not
all modules do this at the moment. If you encounter a module that does not set the variable,
please drop us a note.
C/C++ and Fortran
Which compilers are supported?
The C/C++ and Fortran compilers from Intel Parallel Studio XE and from the GNU Compiler Collection are supported. For these compilers, MPI bindings and a plethora of libraries are provided via the hierarchical environment modules. In addition, other compilers such as clang may be provided via the modules system but without support and specific library builds.
The Intel compiler doesn’t recognize STL, it says <<error: namespace “std” has no member named …>>. What is going on?
The Intel C/C++ compiler does support all modern C++ features (i.e. C++14 and
newer standards). However, it relies on the header files and standard library
libstdc++) provided by the operating system — which themselves may not
cover the later features. To access a set of up-to-date header files as well as
a modern C++ standard library, simply load a recent GCC environment module (i.e.
module load gcc/10) immediately prior to compilation, after loading the
intel module and other library modules (this ensures that libraries other than
libstdc++ have been compiled with the Intel compiler).
Please see the documentation on compilers for more information.
How do I set the rpath?
To enable your binaries pick up shared objects from a certain directory at runtime,
pass the following (third) flag to the linker. The example assumes a library
to be installed in the directory
export LDFLAGS="-lfoobar -L/path/to/library -Wl,-rpath,/path/to/library"
This will write the necessary information directly into the binary.
How do I set the LD_LIBRARY_PATH and why should I avoid it?
The environment variable
LD_LIBRARY_PATH serves a similar purpose as the rpath,
however, it needs to be set at runtime, e.g.
A major disadvantage is that your code will have a dependency on the environment, i.e., the execution fails if the variable is not set or wrongly set. The solution via the rpath is therefore to be preferred.
Debugging of C/C++ and Fortran codes
How do I enable the writing of core dumps?
Core dumps are controlled by the shell. In order to enable writing of core
ulimit -c unlimited (bash),
limit coredumpsize unlimited (tcsh), or
alike has to be specified in the interactive shell or in the batch script.
Codes compiled with the Intel Fortran compiler (ifort) in addition require
export FOR_DUMP_CORE_FILE=1 (bash). Please enable writing of
core dumps only with caution, as writing hundreds or thousands of
(typically large) core files from a large parallel job can significantly
impact the performance of the file system for all users.
On MPCDF systems, a Python environment for scientific computing is provided
via the Anaconda distribution which includes many optimized scientific
packages such as NumPy, SciPy, Cython, Numba, and more. It is available on
the systems as an environment module. You can get a list of the available
module available anaconda.
In addition, there is a plain Python interpreter installed with the operating system. It is provided ‘as is’ and can be used for basic scripting tasks.
How can I install Python packages?
If the package is not available in the default anaconda distribution, we recommend to try installing the desired package using pip into your home directory. First, load the anaconda module using
module load anaconda
then install the desired package using
pip install --user package_name
Please note the flag
--user which will instruct pip to install the package
into your home folder. If this flag is not specified, the installation will
fail because you as a user are not allowed to write to system directories.
We recommend setting up virtual environments as described here!
How can I use Conda environments?
Conda environments created by users cannot be supported by the MPCDF,
nevertheless, users may use them at their own risk and responsibility.
Note that in case the required Python packages are installable via
should prefer a
pip-based installation instead of bootstrapping a complete
Before considering to use Conda, please be aware of the following potentially
harmful side effects: The command
conda init changes the user’s local
environment by appending code to shell rc files (such as
.bashrc). As a
consequence, its installation location (i.e. the one of the anaconda
installation in use) gets effectively hardcoded which conflicts with the
environment modules that are used at MPCDF to provide software in a flexible
In case of issues with your software environment after using Conda, please
remember to clean the files
.condarc, and re-login.
Instead of running
conda init better use an
eval command as shown below
bash shell to enable the use of the
conda executable for the current
interactive session or batch job in a safe and non-intrusive way:
module load anaconda/3/2023.03
# The following command replaces `conda init` for the current session
# without touching the .bashrc file:
eval "$(conda shell.bash hook)"
conda create -n my_env python=3.11
conda activate my_env
As Conda packages downloaded from the Internet are not specifically built for the operating systems and architectures used at the MPCDF it might happen that unexpected errors occur, e.g. issues related to the GLIBC version that is expected by the precompiled binaries.
Moreover, most likely the performance will be inferior to the one of natively compiled executables with specific optimization flags, so software provided by the MPCDF and/or natively compiled software should be preferred.
How can I perform parallel computation using Python?
To speed up your calculations and to exploit the capacity of the HPC systems, it is advisable to perform computations in parallel. Besides running independent jobs in parallel, single jobs can also be parallelized with different methods. These methods can be split according to their parallelization into shared-memory methods (implicit threading, multiprocessing) that will only work on a single node and distributed-memory methods (mpi4py) that will also work across multiple nodes.
Implicit threading via NumPy
Many NumPy and SciPy methods rely on linear algebra methods. Anaconda python uses highly optimized linear algebra functions from Intel MKL (math kernel library) that already provide multithreading. So if your code heavily relies on these linear algebra methods, it might already profit from this implicit parallelization.
The number of threads used by MKL can be controlled with the environment variable
MKL_NUM_THREADS. You can find out how much your code profits from this
parallelization by running it once with
export MKL_NUM_THREADS=1 and once without
(by default MKL will use all available cores). This will show you how much
faster the code executes when using multithreaded MKL functions.
If your code does not profit enough from the implicit threading (which is often
the case), you can use the
multiprocessing package to use several python
processes that can work on tasks independently.
In the simplest example, you can use the
Pool class to execute a function
for different parameters in parallel. This is illustrated by the following
example that was slightly adapted from the python documentation:
from multiprocessing import Pool
number_of_processes = 10
with Pool(number_of_processes) as p:
This will distribute the computation to 10 different processes.
This approach, however, will only work on a single node because the communication between processes only works on the same node in this approach.
The standard approach for parallel HPC applications is to use MPI (message
passing interface) for communication between processes with independent memory.
In python, an interface to MPI is available via the
mpi4py package. On MPCDF systems,
it is available after loading the anaconda module by using
module load mpi4py.
For more information on how to use mpi4py, please look at its documentation. Inside your slurm script, you can start a python program using mpi4py with
srun python application.py
which will take care of starting the application on the resources specified in the script.
On MPCDF systems, R is provided via the ‘R’ environment modules. These R installations are built from the official vanilla sources and are linked against Intel MKL for better performance. In addition, there is the environment module ‘r_anaconda’ that comprises R plus various additional packages, conceptually similar to the Anaconda Python Distribution.
How can I install R packages?
R offers a plethora of additional packages that are available on the CRAN repository.
For example, to install the package ‘mypkg’, issue the command
from an R prompt and select to use a local directory for the installation when prompted, in order to have the package installed locally into your homedirectory.
If you get an error message similar to
package '***' is not available (for R version ***)
ERROR: dependency '***' is not available for package '***'
the reason may be that the version of R you are using is too old. Try using a
newer R version by unloading the present R environment module and loading a
On MPCDF systems, Julia is provided via the ‘julia’ environment modules, where the installations are based on the regular release builds.
The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Find more information on https://jupyter.org/.
Recent MATLAB versions are provided via environment modules on the HPC and cluster
systems. Use the command
module available matlab to get a list of the available
versions, and then load the corresponding module.
How can I run the MATLAB GUI on the login nodes of HPC and cluster systems?
The simplest way to run the MATLAB GUI is to use SSH X forwarding and launch MATLAB on the login node of a cluster, as shown in the following example for Raven:
# 1) on your local machine
ssh -C -Y YOUR_MPCDF_USER_ID@gatezero.mpcdf.mpg.de
# 2) on gatezero
ssh -C -Y raven.mpcdf.mpg.de
# 3) on raven
module load matlab
This will work from a Linux client. For Windows and Mac clients, please make
sure that you have a local X server running, otherwise the GUI will not be displayed and
you will get an error message. Make sure to use the
-C flag with SSH to increase the
performance of the remote GUI.
A more efficient way to run the MATLAB GUI is from within a VNC session. You can either launch VNC manually, or use the VNC sessions available via the remote visualization service.
How can I run MATLAB code via SLURM?
In batch jobs, MATLAB must be run without any graphical window. The following
script shows how to run the sequential MATLAB code
program.m via SLURM:
#SBATCH -J MATLAB # job name
#SBATCH -o ./job.out.%j # standard out file
#SBATCH -e ./job.err.%j # standard err file
#SBATCH -D ./ # work directory
#SBATCH --ntasks=1 # launch job on a single core
#SBATCH --cpus-per-task=1 # on a shared node
#SBATCH --mem=2000MB # memory limit for the job
#SBATCH --time=0:09:59 # run time, up to 24h
module load matlab
srun matlab -nodisplay -nosplash -nodesktop -noFigureWindows -r 'run("program.m")'
How can I run parallel MATLAB code?
Parallel MATLAB code can be run on up to a full compute node. It is important
to inform MATLAB about the number of cores it may use, otherwise it will use
an internal non-optimal default. For example, to run a
parfor loop with as
many workers as you have requested via SLURM, please see the following code
snippet which uses SLURM’s environment variable
ncpus = str2num(getenv('SLURM_CPUS_PER_TASK'))
n = 200;
A = 500;
a = zeros(1,n);
parfor i = 1:n
a(i) = max(abs(eig(rand(A))));
The following Slurm submit script can be used to run the parallel example:
#SBATCH -J MATLAB # job name
#SBATCH -o ./job.out.%j # standard out file
#SBATCH -e ./job.err.%j # standard err file
#SBATCH -D ./ # work directory
#SBATCH --ntasks=1 # launch job on
#SBATCH --cpus-per-task=8 # 8 cores of a shared node
#SBATCH --mem=16000MB # memory limit for the job
#SBATCH --time=01:00:00 # run time, up to 24h
module load matlab
srun matlab -nodisplay -nosplash -nodesktop -noFigureWindows -r 'run("parallel_for_ncpus.m")'
Please make sure that your MATLAB code actually benefits from that many cores, and request less resources if this is not the case.
For some advanced use cases it may be necessary to define and use a cluster profile via MATLAB’s Cluster Profile Manager. Please see the MATLAB documentation for details.
Message Passing Interface (MPI) Libraries and Runtimes
Which MPI implementations are supported?
The MPI library from Intel Parallel Studio XE is supported. In addition, OpenMPI is also provided.
I cannot launch my MPI code using
mpirun, what am I doing wrong?
On SLURM-based HPC clusters the launcher
srun must be used for MPI codes.
Production runs and large tests must be run via batch scripts. From a login
node, small test runs can be started interactively, e.g. as follows:
srun --time=00:05:00 --mem=1G --ntasks=2 ./mpi_application
How can I combine many image files (JPG, PNG) into a movie (MP4)?
To combine individual frames named input_0001.png, input_0002.png, etc. (i.e., jpg or png files that use a numbering scheme in their file names) to a movie (mp4 video file), the ffmpeg software can be used. The following example creates a video that can be played on nearly any device.
module load ffmpeg
ffmpeg -start_number 1 -i input_%04d.png -c:v libx264 -vf "fps=30,format=yuv420p" output.mp4
The frame rate, compression, and other parameters can be adjusted, see the ffmpeg documentation for more details.
Visual Studio Code (in combination with the Remote-SSH extension) does not work any more on certain HPC clusters
Starting with version 1.86 VSCode requires more recent versions of fundamental libraries that are not provided on older OS such as SLES12 and CentOS7. As a result, users of recent versions of VSCode are not able to connect to systems running one of these OS any more, notably Cobra and some older Linux clusters. Note that this does also affect the Remote-SSH extension even if the GUI is run locally on a Laptop.
As a workaround, please use the latest VSCode version 1.85.2 with automatic updates disabled, for the time being.
Why do certain GUI applications not work (any more) on the login nodes of HPC clusters? What are the alternatives?
Some desktop applications are built on top of frameworks that use web technologies to render their graphical user interface (GUI). Examples include web browsers such as Firefox and Chromium, but also text editors and IDEs such as Spyder (using the Qt web framework), VSCode (using the Electron framework based on the Chromium engine), and potentially others.
These frameworks typically use sandboxing techniques based on Linux user namespaces to potentially increase application security at runtime. Unfortunately, some of these user namespace features are and have been suffering from security issues repeatedly such that they got disabled in mid May 2023 on MPCDF clusters.
To continue using these applications via X forwarding or via VNC, users can try to disable sandboxing:
QT-based applications, for example certain Spyder versions, can be launched as follows:
$ export QTWEBENGINE_DISABLE_SANDBOX=1
$ spyder &
Users of Electron-based apps might want to try the
--no-sandbox flag when
launching the application.
Firefox users can set
export MOZ_DISABLE_CONTENT_SANDBOX=1 MOZ_DISABLE_GMP_SANDBOX=1 MOZ_DISABLE_NPAPI_SANDBOX=1 MOZ_DISABLE_GPU_SANDBOX=1 MOZ_DISABLE_RDD_SANDBOX=1 MOZ_DISABLE_SOCKET_PROCESS_SANDBOX=1
An environment module
nosandbox/1.0 is provided which sets the aforementioned environment variables,
so load that before launching Firefox or Spyder.
However, a more efficient option would be to avoid running such heavyweight GUI
applications remotely and rather launch them locally, and
access remote files via
sshfs or via builtin features such as the “Remote -
SSH” extension for VSCode. Similarly, web browsers can be launched locally and
configured to access remote services (such as manually launched Jupyter
sessions) via SSH tunnels or the SOCKS5 feature of SSH.