FHI / MPIE / MPIP
- Name of the machine learning cluster:
TALOS
- Institutions:
Fritz Haber Institute of the Max Planck Society
Max Planck Institute for Iron Research
Max Planck Institute for Polymer Research
Access:
talos01.bc.rzg.mpg.de
Hardare Configuration:
Login nodes talos01 :
CPUs Model: Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
2 sockets
20 cores per socket
hyper-threading (2 threads per core)
188 GB RAM
2 x Tesla Volta V100 32GB
84 execution nodes talos[001-084] :
CPUs Model: Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
2 sockets
20 cores per socket
hyper-threading (2 threads per core)
188 GB RAM
2 x Tesla Volta V100 32GB
Node interconnect is based on Intel Omni-Path Fabric (Speed: 100Gb/s)
Filesystems:
/u shared home filesystem; GPFS-based; user quotas (currently 1TB, 500K files) enforced quota can be checked with ‘/usr/lpp/mmfs/bin/mmlsquota’.
/talos/scratch shared scratch filesystem (852 TB); GPFS-based; no quotas enforced NO BACKUPS!
Compilers and Libraries:
The “module” subsystem is implemented on TALOS. Please use ‘module available’ to see all available modules.
Intel compilers (-> ‘module load intel’): icc, icpc, ifort
GNU compilers (-> ‘module load gcc’): gcc, g++, gfortran
Intel MKL (-> ‘module load mkl’): $MKL_HOME defined; libraries found in $MKL_HOME/lib/intel64
Intel MPI (-> ‘module load impi’): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, …´
Python (-> ‘module load anaconda’): python
CUDA (-> ‘module load cuda’)
Software for Data Analytics:
List of supported software can be found here.
For examples on how to use the provided machine learning software please have a look at data analytics home page.
Batch system based on Slurm:
The batch system on TALOS is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, …) can be found on the Cobra home page. For more detailed information, see the Slurm handbook. See also the sample batch scripts which must be modified for TALOS cluster (partition must be changed).
Current Slurm configuration on TALOS:
default max. run time (wallclock): 24 hours
qos for long running jobs: 2d, 3d, 4d for up to 2, 3 or 4 days respectively
two partitions: p.talos (default) and s.talos
p.talos partition: for parallel MPI or hybrid MPI/OpenMP jobs. Resources are exclusively allocated on nodes
s.talos partition: for serial or OpenMP jobs. Nodes are shared. Jobs are limited to use CPUs only on one node
Useful tips:
By default run time limit used for jobs that don’t specify a value is 24 hours. Use --time option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 96 hours. To run jobs longer than 1 day use QoS 2d for up to 2 days, 3d for up to 3 days and 4d for jobs up to 4 days. For instance --qos=2d together with --time option.
By default jobs use all memory on nodes in p.talos partition.
In s.talos partition default allocated memory per job is 60000 MB. To grant the job access to use more or less memory on each node use --mem option for sbatch/srun
The OpenMP codes require a variable OMP_NUM_THREADS to be set. This can be obtained from the Slurm environment variable $SLURM_CPUS_PER_TASK which is set when --cpus-per-task is specified in a sbatch script (an example is on help information page)
GPU cards are in default compute mode.
Support:
For support please create a trouble ticket at the MPCDF helpdesk.