FHI / MPIE / MPIP


Name of the machine learning cluster:

TALOS

Institutions:

Fritz Haber Institute of the Max Planck Society

Max Planck Institute for Iron Research

Max Planck Institute for Polymer Research

Access:

  • talos01.bc.rzg.mpg.de

Hardare Configuration:

Login nodes talos01 :

  • CPUs Model:  Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz

  • 2 sockets

  • 20 cores per socket

  • hyper-threading (2 threads per core)

  • 188 GB RAM

  • 2 x Tesla Volta V100 32GB

84 execution nodes talos[001-084] :

  • CPUs Model:  Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz

  • 2 sockets

  • 20 cores per socket

  • hyper-threading (2 threads per core)

  • 188 GB RAM

  • 2 x Tesla Volta V100 32GB

Node interconnect is based on Intel Omni-Path Fabric (Speed: 100Gb/s)

Filesystems:

/u shared home filesystem; GPFS-based; user quotas (currently 1TB, 500K files) enforced quota can be checked with ‘/usr/lpp/mmfs/bin/mmlsquota’.

/talos/scratch shared scratch filesystem (852 TB); GPFS-based; no quotas enforced NO BACKUPS!

Compilers and Libraries:

The “module” subsystem is implemented on TALOS. Please use ‘module available’ to see all available modules.

  • Intel compilers (-> ‘module load intel’): icc, icpc, ifort

  • GNU compilers (-> ‘module load gcc’): gcc, g++, gfortran

  • Intel MKL (-> ‘module load mkl’): $MKL_HOME defined; libraries found in $MKL_HOME/lib/intel64

  • Intel MPI (-> ‘module load impi’): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, …´

  • Python (-> ‘module load anaconda’): python

  • CUDA (-> ‘module load cuda’)

Software for Data Analytics:

List of supported software can be found here.

For examples on how to use the provided machine learning software please have a look at data analytics home page.

Batch system based on Slurm:

The batch system on TALOS is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, …) can be found on the Cobra home page. For more detailed information, see the Slurm handbook. See also the sample batch scripts which must be modified for TALOS cluster (partition must be changed).

Current Slurm configuration on TALOS:

  • default max. run time (wallclock): 24 hours

  • qos for long running jobs: 2d, 3d, 4d for up to 2, 3 or 4 days respectively

  • two partitions: p.talos (default) and s.talos

  • p.talos partition: for parallel MPI or hybrid MPI/OpenMP jobs. Resources are exclusively allocated on nodes

  • s.talos partition: for serial or OpenMP jobs. Nodes are shared. Jobs are limited to use CPUs only on one node

Useful tips:

By default run time limit used for jobs that don’t specify a value is 24 hours. Use --time option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 96 hours. To run jobs longer than 1 day use QoS 2d for up to 2 days, 3d for up to 3 days and 4d for jobs up to 4 days. For instance --qos=2d together with --time option.

By default jobs use all memory on nodes in p.talos partition.

In s.talos partition default allocated memory per job is 60000 MB. To grant the job access to use more or less memory on each node use --mem option for sbatch/srun

The OpenMP codes require a variable OMP_NUM_THREADS to be set. This can be obtained from the Slurm environment variable $SLURM_CPUS_PER_TASK which is set when --cpus-per-task is specified in a sbatch script (an example is on help information page)

To use GPUs add in your slurm scripts *–gres* option and choose how many GPUs to allocate: #SBATCH --gres=gpu:1 or #SBATCH --gres=gpu:2
Valid gres options are: gpu[[:type]:count]
where
type is a type of gpu (v100)
count is a number of resources (1 or 2)

GPU cards are in default compute mode.

Support:

For support please create a trouble ticket at the MPCDF helpdesk.