HPC Software and Applications

General questions

How and where can I install software myself?

On the HPC systems and clusters, users can install software into their home directory where they have write permissions. For typical software installation no root privileges are necessary.

Make sure to set the installation directory accordingly as shown in the following examples for different build systems:

# GNU build system
./configure --prefix=$HOME/soft/package_name ...
make && make install

# CMAKE example
cmake -DCMAKE_INSTALL_PREFIX:PATH=$HOME/soft/package_name ...
make && make install

# Python example, installing into '~/.local'
python setup.py install --user

# Python/pip example, installing into '~/.local'
pip install --user package_name

Details on the build and install procedure are usually given in the documentation that comes with the source files. In case of problems users may request support via the helpdesk.

Environment modules

The MPCDF uses modules to adapt the user environment for working with software installed at various locations in the file system or for switching between different software versions. The user is no longer required to explicitly specify paths for different executable versions, or keep track of PATH, MANPATH and related environment variables. With the modules approach, users simply ‘load’ and ‘unload’ modules to control their environment. Note that since 2018, HPC systems as well as the increasing number of dedicated clusters all use hierarchical environment modules.

How can I use environment modules interactively?

Here is a list of the most important commands (see https://modules.readthedocs.io/en/latest/module.html for a complete reference):

  • module help: lists module subcommands and switches

  • module avail: lists available software packages and versions which can be enabled (“loaded”) with the module command

  • module help <package>/<version>: provides brief documentation for the specified module.

  • module load <package>/<version>: “loads” the module, i.e. modifies the user’s environment ($PATH, $MANPATH, etc.)

  • module unload <package>/<version>: “unloads” the module

  • module list: lists all modules which are currently loaded in the user’s environment

How can I use environment modules in scripts?

Environment modules are provided by default only in interactive login shells. For non-interactive shells (e.g. in shell scripts or batch scripts) the following system profiles must be explicitly sourced (for Linux, specifically SLES):

  • /etc/profile.d/modules.sh for sh-derivative shells: sh, bash

  • /etc/profile.d/modules.csh for csh-derivative shells: csh, tcsh

How can I use environment modules to avoid absolute paths in my scripts?

Instead of absolute paths to libraries, binaries etc. the environment variables set by the modulefile should be used in scripts, makefiles etc. By convention, an MPCDF modulefile sets an environment variable named <PKG>_HOME (where PKG is the name of the package, for example: MKL_HOME) which points to the root directory of the installation path (see below for example usage). Information about additional, package-specific environment variables can be obtained with the commands module help <package>/<version> and module show <package>/<version>.

Examples

Interactive session on the command line

$ module load intel mkl
$ ifort -I$MKL_HOME/include example.F -L$MKL_HOME/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core

Makefile (fragment)

FC=ifort

example: example.F
        $(FC) -I$(MKL_HOME)/include test.F -L$(MKL_HOME)/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core

How can I read the long output from module avail page by page?

# sh/bash shell
( module avail ) 2>&1 | less

# c/tcsh shell
( module avail ) | & less

How do the hierarchical environment modules work?

To manage the plethora of software packages resulting from all the relevant combinations of compilers and MPI libraries, we have decided to organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the uppermost level, depending libraries (e.g., MPI) on the second level, more depending libraries on a third level. This means that not all the modules are visible initially: only after loading a compiler module, will the modules depending on this become available. Similarly, loading an MPI module in addition will make the modules depending on the MPI library available.

To start at the root of the environment modules hierarchy, issue module purge.

For example, the FFTW library compiled with the default Intel compiler and the default Intel MPI library can be loaded as follows: First, load the default Intel compiler module using the command

module load intel

second, the default Intel MPI module with

module load impi

and, finally, the FFTW module fitting exactly to the compiler and MPI library via

module load fftw-mpi

You may check using the command

module available

that after the first and second steps the depending environment modules become visible, in the present example impi and fftw-mpi. Moreover, note that the environment modules can be loaded via a single ‘module load’ statement as long as the order given by the hierarchy is correct, e.g., ‘module load intel impi fftw-mpi’.

How do I quickly find a module?

In case you know the name of the module you wish to load, but you are not sure about the available versions or what dependencies need to be loaded first, you can try to use the ‘find-module’ command. This tool searches for the MODULENAME string through a list of all installed modules

find-module MODULENAME

You can then choose the desired module version, use the output of the command to determine the correct order to load dependencies, and finally load the module itself, e.g.

$ find-module horovod
horovod/cpu/0.13.11    (after loading anaconda/3/2019.03 tensorflow/cpu/1.14.0)
$ module load anaconda/3/2019.03 tensorflow/cpu/1.14.0 horovod/cpu/0.13.11

It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the uppermost hierarchy. Note that a hierarchy exists for Python modules with the ‘anaconda’ module files on the top level.

How can I disable the “MPCDF specific note” for module avail?

On recent installations, the module avail command prints a “MPCDF specific note” that describes how to find nested environment modules in the hierarchy. To disable that note, set the environment variable ‘MPCDF_DISABLE_MODULE_AVAIL_HINT’ to ‘1’, e.g., by adding the line export MPCDF_DISABLE_MODULE_AVAIL_HINT=1 to your ‘.bashrc’ file.

Why there are no BLAS/LAPACK modules available on the clusters?

Intel’s Math Kernel Library (MKL) library provides highly optimized replacements for the BLAS, Sparse BLAS, LAPACK, ScaLAPACK, PBLAS, and BLACS libraries as well as other math routines. For more information please refer to our MKL guide.

Compiled Languages

CMake

Which CMake version should I use?

CMake is backwards compatible. This means you can load the newest version and still run old CMakeLists.txt files (>3.0). However, since CMake increases support for newer compilers and libraries with every version you should load the newest version available to get best possible support for the environment.

Policy CMPXXXX is not set - Warning

The behavior for something you are using has changed between the CMake version specified in the CMakeLists.txt file and the currently loaded CMake executable version. As a user you do not have to worry. CMake will act according to the behavior specified in the CMakeLists.txt file version. As a developer look at the changed behavior and check if you code is compatible with it. Then set the policy explicitly to the expected behavior. (https://cmake.org/cmake/help/latest/manual/cmake-policies.7.html)

CMake cannot find library XXX

CMake checks all standard installation directories for requested libraries. If it cannot find the library there or you want to enforce a different install location use the XXX_ROOT (case-sensitive!) environment variable and point it to the installation directory. We try to set this variable automatically for all packages loaded on the cluster. However, not all modules do this at the moment. If you encounter a module that does not set the variable, please drop us a note.

C/C++ and Fortran

Which compilers are supported?

The C/C++ and Fortran compilers from Intel Parallel Studio XE and from the GNU Compiler Collection are supported. For these compilers, MPI bindings and a plethora of libraries are provided via the hierarchical environment modules. In addition, other compilers such as clang may be provided via the modules system but without support and specific library builds.

How do I make sure that my executable will find the necessary libraries (shared objects) at runtime?

Dynamic libraries (shared objects, *.so) are found at runtime only at system locations, by default. In case you have linked shared objects in other directories (e.g., provided by environment modules or by your custom installation), information on where to find those libraries needs to be available at runtime. Two options can be used to solve this problem.

The Intel compiler doesn’t recognize STL, it says <<error: namespace “std” has no member named …>>. What is going on?

The Intel C/C++ compiler does support all modern C++ features (i.e. C++14 and newer standards). However, it relies on the header files and standard library (libstdc++) provided by the operating system — which themselves may not cover the later features. To access a set of up-to-date header files as well as a modern C++ standard library, simply load a recent GCC environment module (i.e. module load gcc/10) immediately prior to compilation, after loading the intel module and other library modules (this ensures that libraries other than libstdc++ have been compiled with the Intel compiler).

Please see the documentation on compilers for more information.

How do I set the rpath?

To enable your binaries pick up shared objects from a certain directory at runtime, pass the following (third) flag to the linker. The example assumes a library libfoobar.so to be installed in the directory /path/to/library:

export LDFLAGS="-lfoobar -L/path/to/library -Wl,-rpath,/path/to/library"

This will write the necessary information directly into the binary.

With CMake you can control the global rpath settings via CMAKE_BUILD_RPATH and CMAKE_INSTALL_RPATH or the per target settings via BUILD_RPATH and INSTALL_RPATH.

How do I set the LD_LIBRARY_PATH and why should I avoid it?

The environment variable LD_LIBRARY_PATH serves a similar purpose as the rpath, however, it needs to be set at runtime, e.g.

export LD_LIBRARY_PATH=/path/to/library

A major disadvantage is that your code will have a dependency on the environment, i.e., the execution fails if the variable is not set or wrongly set. The solution via the rpath is therefore to be preferred.

Debugging of C/C++ and Fortran codes

How do I enable the writing of core dumps?

Core dumps are controlled by the shell. In order to enable writing of core dumps ulimit -c unlimited (bash), limit coredumpsize unlimited (tcsh), or alike has to be specified in the interactive shell or in the batch script. Codes compiled with the Intel Fortran compiler (ifort) in addition require the setting export FOR_DUMP_CORE_FILE=1 (bash). Please enable writing of core dumps only with caution, as writing hundreds or thousands of (typically large) core files from a large parallel job can significantly impact the performance of the file system for all users.

How can I use the Address Sanitizer (ASAN) in combination with CUDA code?

The Address Sanitizer (ASAN) is a powerful tool to detect host memory errors in C/C++ codes. It is integrated into the GCC (flag -fsanitize=address) and Clang compilers. However, when the host parts of CUDA codes are compiled with the address sanitizer enabled, the program will not be able to access any GPU at runtime, reporting the error code ‘cudaErrorNoDevice’ which is due to an incompatibility between the NVIDIA driver and ASAN. To mitigate the issue, set the environment variable export ASAN_OPTIONS="protect_shadow_gap=0" before running the ASAN-instrumented CUDA code.

Interpreted Languages

Python

On MPCDF systems, a Python environment for scientific computing is provided via the Anaconda distribution which includes many optimized scientific packages such as NumPy, SciPy, Cython, Numba, and more. It is available on the systems as an environment module. You can get a list of the available versions with module available anaconda.

In addition, there is a plain Python interpreter installed with the operating system. It is provided ‘as is’ and can be used for basic scripting tasks.

How can I install Python packages?

If the package is not available in the default anaconda distribution, we recommend to try installing the desired package using pip into your home directory. First, load the anaconda module using

module load anaconda

then install the desired package using

pip install --user package_name

Please note the flag --user which will instruct pip to install the package into your home folder. If this flag is not specified, the installation will fail because you as a user are not allowed to write to system directories.

We recommend setting up virtual environments as described here!

How can I use Conda environments?

Conda environments created by users cannot be supported by the MPCDF, nevertheless, users may use them at their own risk and responsibility. Note that in case the required Python packages are installable via pip users should prefer a pip-based installation instead of bootstrapping a complete Conda environment.

If possible, use the MPCDF open source tool Condainer to create portable standalone conda environments using compressed images, avoiding most of the issues described below.

Before considering to use Conda, please be aware of the following potentially harmful side effects: The command conda init changes the user’s local environment by appending code to shell rc files (such as .bashrc). As a consequence, its installation location (i.e. the one of the anaconda installation in use) gets effectively hardcoded which conflicts with the environment modules that are used at MPCDF to provide software in a flexible way. In case of issues with your software environment after using Conda, please remember to clean the files .bashrc and .condarc, and re-login.

Instead of running conda init better use an eval command as shown below for the bash shell to enable the use of the conda executable for the current interactive session or batch job in a safe and non-intrusive way:

module purge
module load anaconda/3/2023.03

# The following command replaces `conda init` for the current session
# without touching the .bashrc file:
eval "$(conda shell.bash hook)"

conda create -n my_env python=3.11
conda activate my_env

As Conda packages downloaded from the Internet are not specifically built for the operating systems and architectures used at the MPCDF it might happen that unexpected errors occur, e.g. issues related to the GLIBC version that is expected by the precompiled binaries.

Moreover, most likely the performance will be inferior to the one of natively compiled executables with specific optimization flags, so software provided by the MPCDF and/or natively compiled software should be preferred.

How can I perform parallel computation using Python?

To speed up your calculations and to exploit the capacity of the HPC systems, it is advisable to perform computations in parallel. Besides running independent jobs in parallel, single jobs can also be parallelized with different methods. These methods can be split according to their parallelization into shared-memory methods (implicit threading, multiprocessing) that will only work on a single node and distributed-memory methods (mpi4py) that will also work across multiple nodes.

Implicit threading via NumPy

Many NumPy and SciPy methods rely on linear algebra methods. Anaconda python uses highly optimized linear algebra functions from Intel MKL (math kernel library) that already provide multithreading. So if your code heavily relies on these linear algebra methods, it might already profit from this implicit parallelization.

The number of threads used by MKL can be controlled with the environment variable MKL_NUM_THREADS. You can find out how much your code profits from this parallelization by running it once with export MKL_NUM_THREADS=1 and once without (by default MKL will use all available cores). This will show you how much faster the code executes when using multithreaded MKL functions.

Multiprocessing

If your code does not profit enough from the implicit threading (which is often the case), you can use the multiprocessing package to use several python processes that can work on tasks independently.

In the simplest example, you can use the Pool class to execute a function for different parameters in parallel. This is illustrated by the following example that was slightly adapted from the python documentation:

from multiprocessing import Pool

def f(x):
    return x*x

number_of_processes = 10
with Pool(number_of_processes) as p:
    print(p.map(f, list(range(100))))

This will distribute the computation to 10 different processes.

This approach, however, will only work on a single node because the communication between processes only works on the same node in this approach.

mpi4py

The standard approach for parallel HPC applications is to use MPI (message passing interface) for communication between processes with independent memory. In python, an interface to MPI is available via the mpi4py package. On MPCDF systems, it is available after loading the anaconda module by using module load mpi4py.

For more information on how to use mpi4py, please look at its documentation. Inside your slurm script, you can start a python program using mpi4py with

srun python application.py

which will take care of starting the application on the resources specified in the script.

R

On MPCDF systems, R is provided via the ‘R’ environment modules. These R installations are built from the official vanilla sources and are linked against Intel MKL for better performance. In addition, there is the environment module ‘r_anaconda’ that comprises R plus various additional packages, conceptually similar to the Anaconda Python Distribution.

How can I install R packages?

R offers a plethora of additional packages that are available on the CRAN repository.

For example, to install the package ‘mypkg’, issue the command

install.packages("mypkg")

from an R prompt and select to use a local directory for the installation when prompted, in order to have the package installed locally into your homedirectory.

If you get an error message similar to package '***' is not available (for R version ***) or ERROR: dependency '***' is not available for package '***' the reason may be that the version of R you are using is too old. Try using a newer R version by unloading the present R environment module and loading a newer one.

Julia

On MPCDF systems, Julia is provided via the ‘julia’ environment modules, where the installations are based on the regular release builds.

How can I install Julia packages?

Julia offers a plethora of additional packages that are available on the Julia package registry.

For example, to install the package ‘mypkg’, issue the command

using Pkg
Pkg.add("mypkg")

from a Julia prompt.

Jupyter Notebooks

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Find more information on https://jupyter.org/.

How can I launch Jupyter notebooks on the HPC systems?

On the HPC systems, Jupyter notebooks can be launched and accessed conveniently via the RVS web service at https://rvs.mpcdf.mpg.de/.

MATLAB

Recent MATLAB versions are provided via environment modules on the HPC and cluster systems. Use the command module available matlab to get a list of the available versions, and then load the corresponding module.

How can I run the MATLAB GUI on the login nodes of HPC and cluster systems?

The simplest way to run the MATLAB GUI is to use SSH X forwarding and launch MATLAB on the login node of a cluster, as shown in the following example for Raven:

# 1) on your local machine
ssh -C -Y YOUR_MPCDF_USER_ID@gate1.mpcdf.mpg.de

# 2) on gate1
ssh -C -Y raven.mpcdf.mpg.de

# 3) on raven
module load matlab
matlab &

This will work from a Linux client. For Windows and Mac clients, please make sure that you have a local X server running, otherwise the GUI will not be displayed and you will get an error message. Make sure to use the -C flag with SSH to increase the performance of the remote GUI.

A more efficient way to run the MATLAB GUI is from within a VNC session. You can either launch VNC manually, or use the VNC sessions available via the remote visualization service.

How can I run MATLAB code via SLURM?

In batch jobs, MATLAB must be run without any graphical window. The following script shows how to run the sequential MATLAB code program.m via SLURM:

#!/bin/bash -l
#SBATCH -J MATLAB          # job name
#SBATCH -o ./job.out.%j    # standard out file
#SBATCH -e ./job.err.%j    # standard err file
#SBATCH -D ./              # work directory
#SBATCH --ntasks=1         # launch job on a single core
#SBATCH --cpus-per-task=1  #   on a shared node
#SBATCH --mem=2000MB       # memory limit for the job
#SBATCH --time=0:09:59     # run time, up to 24h
module load matlab
srun matlab -nodisplay -nosplash -nodesktop -noFigureWindows -r 'run("program.m")'

How can I run parallel MATLAB code?

Parallel MATLAB code can be run on up to a full compute node. It is important to inform MATLAB about the number of cores it may use, otherwise it will use an internal non-optimal default. For example, to run a parfor loop with as many workers as you have requested via SLURM, please see the following code snippet which uses SLURM’s environment variable SLURM_CPUS_PER_TASK:

% parallel_for_ncpus.m
ncpus = str2num(getenv('SLURM_CPUS_PER_TASK'))
parpool('local', ncpus)

n = 200;
A = 500;
a = zeros(1,n);
parfor i = 1:n
    a(i) = max(abs(eig(rand(A))));
end
exit

The following Slurm submit script can be used to run the parallel example:

#!/bin/bash -l
#SBATCH -J MATLAB          # job name
#SBATCH -o ./job.out.%j    # standard out file
#SBATCH -e ./job.err.%j    # standard err file
#SBATCH -D ./              # work directory
#SBATCH --ntasks=1         # launch job on
#SBATCH --cpus-per-task=8  #   8 cores of a shared node
#SBATCH --mem=16000MB      # memory limit for the job
#SBATCH --time=01:00:00     # run time, up to 24h
module load matlab
srun matlab -nodisplay -nosplash -nodesktop -noFigureWindows -r 'run("parallel_for_ncpus.m")'

Please make sure that your MATLAB code actually benefits from that many cores, and request less resources if this is not the case.

For some advanced use cases it may be necessary to define and use a cluster profile via MATLAB’s Cluster Profile Manager. Please see the MATLAB documentation for details.

Message Passing Interface (MPI) Libraries and Runtimes

Which MPI implementations are supported?

The MPI library from Intel Parallel Studio XE is supported. In addition, OpenMPI is also provided.

I cannot launch my MPI code using mpirun, what am I doing wrong?

On SLURM-based HPC clusters the launcher srun must be used for MPI codes. Production runs and large tests must be run via batch scripts. From a login node, small test runs can be started interactively, e.g. as follows:

srun --time=00:05:00 --mem=1G --ntasks=2 ./mpi_application

Visualization

How can I combine many image files (JPG, PNG) into a movie (MP4)?

To combine individual frames named input_0001.png, input_0002.png, etc. (i.e., jpg or png files that use a numbering scheme in their file names) to a movie (mp4 video file), the ffmpeg software can be used. The following example creates a video that can be played on nearly any device.

module load ffmpeg
ffmpeg -start_number 1 -i input_%04d.png -c:v libx264 -vf "fps=30,format=yuv420p" output.mp4

The frame rate, compression, and other parameters can be adjusted, see the ffmpeg documentation for more details.

How can I install additional packages for TeX/LaTeX locally?

Comprehensive LaTeX environments are provided via the environment modules texlive. In case additional packages are required, users can install them locally as follows.

module purge
module load texlive/2021
# initialize a local usertree for a specific texlive version, once
tlmgr init-usertree
tlmgr --usermode option repository https://ftp.tu-chemnitz.de/pub/tug/historic/systems/texlive/2021/tlnet-final/
# install the package (here 'PACKAGE') locally
tlmgr --usermode install PACKAGE

Make sure to specify the version of the environment module and the repository consistently (here 2021).

GUI applications

Visual Studio Code (in combination with the Remote-SSH extension) does not work any more on certain HPC clusters

Starting with version 1.86 VSCode requires more recent versions of fundamental libraries that are not provided on older OS such as SLES12 and CentOS7. As a result, users of recent versions of VSCode are not able to connect to systems running one of these OS any more, notably Cobra and some older Linux clusters. Note that this does also affect the Remote-SSH extension even if the GUI is run locally on a Laptop.

As a workaround, please use the latest VSCode version 1.85.2 with automatic updates disabled, for the time being.

Why do certain GUI applications not work (any more) on the login nodes of HPC clusters? What are the alternatives?

Some desktop applications are built on top of frameworks that use web technologies to render their graphical user interface (GUI). Examples include web browsers such as Firefox and Chromium, but also text editors and IDEs such as Spyder (using the Qt web framework), VSCode (using the Electron framework based on the Chromium engine), and potentially others.

These frameworks typically use sandboxing techniques based on Linux user namespaces to potentially increase application security at runtime. Unfortunately, some of these user namespace features are and have been suffering from security issues repeatedly such that they got disabled in mid May 2023 on MPCDF clusters.

To continue using these applications via X forwarding or via VNC, users can try to disable sandboxing:

QT-based applications, for example certain Spyder versions, can be launched as follows:

$ export QTWEBENGINE_DISABLE_SANDBOX=1
$ spyder &

Users of Electron-based apps might want to try the --no-sandbox flag when launching the application.

Firefox users can set

export MOZ_DISABLE_CONTENT_SANDBOX=1 MOZ_DISABLE_GMP_SANDBOX=1 MOZ_DISABLE_NPAPI_SANDBOX=1 MOZ_DISABLE_GPU_SANDBOX=1 MOZ_DISABLE_RDD_SANDBOX=1 MOZ_DISABLE_SOCKET_PROCESS_SANDBOX=1

before launching firefox.

An environment module nosandbox/1.0 is provided which sets the aforementioned environment variables, so load that before launching Firefox or Spyder.

However, a more efficient option would be to avoid running such heavyweight GUI applications remotely and rather launch them locally, and access remote files via sshfs or via builtin features such as the “Remote - SSH” extension for VSCode. Similarly, web browsers can be launched locally and configured to access remote services (such as manually launched Jupyter sessions) via SSH tunnels or the SOCKS5 feature of SSH.