Compilers and languages
Intel C/C++ Compiler for Linux
The name of the C compiler executable is
icc, the name of the C++
compiler executable is
Compilation and linking of a C program (source file
myprog.c) is done
icc -o myprog myprog.c
To get an overview on the available command line options use the
More information is provided by the manual page
Extensive documentation and e.g. information on code optimization strategies is provided by the official Intel C/C++ Compiler Documentation. For documentation of compatibility, updates and changes of different versions see the Intel Composer release notes
To compile and link MPI codes, use the wrappers
C++ header files and standard library
While the Intel C/C++ compiler does support modern C++ features (C++11, C++14, and newer), the header files and standard library (‘libstdc++’) that come with the operating system may not support such features yet. To access more recent header files and, similarly, a more recent standard library required for modern C++ codes, simply load a recent GCC environment module (‘module load gcc’) before loading the Intel compiler module.
Intel Fortran Compiler for Linux
The name of the Intel Fortran Compiler executable is
Compilation and linking of a Fortran program (source file
myprog.f90) is done as follows:
ifort -o myprog myprog.f90
To get an overview on the available command line options use the
More information is provided by the manual page
Extensive documentation and e.g. information on code optimization strategies is provided by the official Intel Fortran Compiler Documentation. For documentation of compatibility, updates and changes of different versions see the Intel Composer release notes.
To compile and link MPI codes, use the wrapper
How to get access to the Intel Compilers
On the Cobra and Raven supercomputers, user need to load and specify a version of the Intel compiler explicitly, similarly for Intel MPI. No default versions exist for the Intel compiler and MPI modules, and no default versions are loaded at login.
To get a list of all available Intel compilers, enter
module avail intel.
To get access to a specific Intel compiler, load the module by
module load intel/<version>.
Intel Compiler for Linux Optimization Flags
Compiler optimization flags
Compiler optimization flags have strong influence on the performance of the executable. Some important flags are given below. First, different optimization levels are available:
-O2: Standard optimization. Default.
-O3: Aggressive optimization. Use it with care and check the results against a less optimized binary.
-O0: Disables all optimization. Useful for fast compilation and to check if unexpected behavior results from a higher compiler optimization level.
-O1: Very conservative optimization.
is key to achieving good floating point performance on modern CPUs. For
detailed information on how to specify the instruction set level during
compilation please consult the
Intel Compiler Documentation
In particular, the switches
-m are relevant, for example:
-xCORE-AVX512 -qopt-zmm-usage=high: Enable AVX512 vectorization for Intel Skylake, CascadeLake, IceLake processors. These flags are recommended on cobra and raven.
-xCORE-AVX2: Enable AVX2 vectorization for Intel Haswell and Broadwell CPUs.
-ipo: Enable interprocedural optimizations beyond individual source files.
The meta switch
-fast is not supported on MPCDF systems because it
forces the static linking of all libraries (i.e. it implies the switch
-static) which is not possible with certain system libraries.
[*] To obtain information about the features of the Linux host CPU
issue the command
cat /proc/cpuinfo | grep flags | head -1.
Instruction-set-related keywords are, among others, avx, avx2, avx512.
The list of all supported switches and extensive information is covered by the official Intel Compiler Documentation.
Floating point accuracy
Intel compilers tend to adopt increasingly more aggressive defaults for
the optimization of floating-point semantics. The default is
-fp-model fast=1. We recommend to double check the accuracy of
simulation results by using more conservative settings (which might come
at the expense of computational performance) like
-fp-model precise (recommended) or even
See the compiler man pages for more details.
The GNU Compiler Collection provides – among others – front ends for C, C++, and Fortran.
A default version of GCC comes with the operating system. More recent versions suitable for HPC can be accessed via environment modules.
To compile and link MPI codes using the GNU compilers, use the commands
mpigfortran in combination with Intel MPI.
Find the full documentation at https://gcc.gnu.org/.
The following packages are provided on the HPC clusters to enable users develop applications for NVIDIA GPUs.
NVIDIA CUDA Toolkit
The NVIDIA CUDA Toolkit provides a
development environment for the programming of NVIDIA GPUs. It includes the CUDA
C++ compiler (
nvcc), optimized libraries, debuggers and profilers, among
module avail cuda to get an up-to-date list of the CUDA versions
available on a system.
NVIDIA HPC SDK
The NVIDIA HPC SDK provides a C, C++,
and Fortran compiler for the programming of NVIDIA GPUs and multi-core CPUs. It
is the successor product of the PGI compiler suite.
In addition, the NVIDIA HPC SDK comprises a copy of the CUDA toolkit and various
libraries for numerical computation, deep learning and AI, and communication.
module avail nvhpcsdk to get an up-to-date list of
the versions available on a system.
Kokkos C++ Performance Portability Library
The Kokkos performance portability framework enables the development of applications that achieve consistent good performance accross all relevant modern HPC platforms based on a single-source C++ implementation. It provides abstractions for parallel computation and data management, and supports several backends such as OpenMP and CUDA, among others.
The high-level Python programming language can be extended with modules written in plain C++/CUDA to leverage GPU computing. In this case, the interfaces may be created with Cython or pybind11 in a comparably easy way. Alternatively, the PyCUDA module offers a straight forward way to embed CUDA code into Python modules. Codes that make heavy use of NumPy may compile such costly expressions to CPU or GPU machine code using the Numba package. Note that Numba is non-intrusive as it only uses decorators. It is part of the Anaconda Python distribution.
Usually we have the latest NAG Fortran compiler installed. To see all
available NAG compilers on UNIX, enter
module avail nagf95.
To use a specific NAG compiler, load the module by
module load nagf95/<$version>.
The compiler command is
To use the compiler on windows, follow the instructions given in
for versions equal or later rel5.3. For access a valid AFS-Token for the cell ipp-garching.mpg.de is necessary.
More information about the NAG Fortran Compiler can be found in the documentation of NAG at NAG Fortran Compiler Documentation.
At the MPCDF, Python including a plethora of scientific packages for numerical computing and data science (NumPy, SciPy, matplotlib, Cython, Numba, Pandas, etc.) is provided via the Anaconda Python Distribution.
Obtain an up-to-date list of the installed Anaconda releases via the following command:
module avail anaconda
Python for HPC
Being an interpreted and dynamically-typed language, plain Python is not a language suitable per-se to achieve high performance. Nevertheless, with the appropriate packages, tools, and techniques the Python programming language can be used to perform numerical computation in a very efficient manner, covering both aspects, the program’s efficiency and the programmer’s efficiency. The aim of this article is to provide some advice and orientation to the reader in order to use Python correctly on the HPC systems and to take first steps towards basic Python code optimization.
The key to achieve good performance with Python is to move notably expensive computation from the interpreted code layer down to a compiled layer which may consist of compiled libraries, code written and compiled by the user, or just-in-time compiled code. Below, three packages are discussed for these use cases.
NumPy is the Python module that provides arrays of native datatypes (float32, float64, int64, etc.) and mathematical operations and functions on them. Typically, mathematical equations (in particular, vector and matrix arithmetic) can be written with NumPy expressions in a very readable and elegant way, which brings several advantages: NumPy expressions avoid explicit, slow loops in Python. In addition, NumPy uses compiled code and optimized mathematical libraries internally, e.g. Intel MKL on MPCDF systems, which enables vectorization and other optimizations. Parts of these libraries use thread-parallelization in a very efficient way by default, e.g. to perform matrix multiplications. In summary, NumPy provides the de-facto standard for numerical array-based computations and serves as the basis for a multitude of additional packages.
Cython is a Python language extension that makes it relatively easy to create compiled Python modules written in Cython, C or C++. It integrates well with NumPy arrays and can be used to implement time-critical parts of an algorithm. Moreover, Cython is very useful to create interfaces to C or C++ code, such as legacy libraries or native CUDA code. Technically, the Cython source code is translated by the Cython compiler to intermediate C code which is then compiled to machine code by a regular C compiler like GCC or ICC.
Numba is a just-in-time compiler based on the LLVM framework. It compiles Python functions at runtime for the datatypes these functions are being called with. Moreover, Numba implements a subset of NumPy’s functions, i.e. it is able to compile NumPy expressions. Functions are declared via a simple decorator-syntax to be suitable for jit-compilation, hence, Numba is only little intrusive on existing code bases.
While Python does implement threads as part of the standard library, these cannot be used to accelerate computation on more than one core in parallel due to cPython’s global interpreter lock. Nevertheless, Python is suitable for parallel computation. In the following, two important packages for intra-node and inter-node parallelism are addressed.
The multiprocessing package is part of the Python standard library. It implements building blocks such as pools of workers and communication queues that can be used to parallelize data-parallel workloads. Technically, multiprocessing forks subprocesses from the main Python process that can run in parallel on multiple cores of a shared-memory machine. Note that some overhead is associated with the inter-process communication. It is, however, possible to access shared memory from several processes simultaneously. A typical use case would be large NumPy arrays.
Access to the Message Passing Interface (MPI) is available via the module mpi4py. It enables parallel computation on distributed-memory computers where the processes communicate via messages with each other. In particular, the mpi4py package supports the communication of NumPy arrays without additional overhead. On MPCDF systems, the environment module mpi4py provides an optimized build based on the default Intel MPI library.
NumPy implements efficient binary IO for array data that is useful, e.g., for temporary files. A better choice with respect to portability and long-term compatibility are HDF5 files. HDF5 is accessible via the h5py Python package and offers an easy-to-use dictionary-style interface. For parallel codes, a special build of h5py with support for MPI-parallel IO is provided via the environment module h5py-mpi.
The Python software ecosystem
In addition to the packages discussed up to now, there is a plethora of solid and well-proven packages for scientific computation and data science available, covering, e.g., numerical libraries (SciPy), visualization (matplotlib, seaborn), data analysis (pandas), and machine learning (TensorFlow, pytorch), to name only a few.
Often, users need to install special Python packages for their
scientific domain. In most cases, the easiest and quickest way is to
create an installation local to the user’s homedirectory. After loading
the Anaconda environment module, the command
pip install --user PACKAGE_NAME would download and install a package
from the Python package index (PyPI), or similarly, the command
python setup.py install --user would install a package from an
unpacked source tarball. In both cases, the resulting installation is
located below “~/.local” where Python will find it by default.
The software recommended in this article is available via the Anaconda Python Distribution (environment module “anaconda/3”) on MPCDF systems. Note that for some packages (mpi4py, h5py-mpi), the hierarchical environment modules matter, i.e., it is necessary to load a compiler (gcc, intel) and an MPI module (impi) in addition to Anaconda in order to get access to these depending environment modules.
The application group at the MPCDF has developed an in-depth course on “Python for HPC” which covers all the topics touched in this article in more detail on two days. It is taught one to two times per year and announced via the MPCDF web page.
Finally, it should be pointed out that Python 2 reaches its official end-of-life on January 1, 2020. Consequently, new Python modules and updates to existing ones will not take Python 2 compatibility into account in the future. Users still running legacy code are strongly encouraged to migrate to Python 3.