Polymer Research

Name of the cluster:

OTTER

Institution:

Max Planck Institute for Polymer Research

Login nodes:

  • otter01.bc.mpcdf.mpg.de

  • otter02.bc.mpcdf.mpg.de

Hardware Configuration:

2 login node otter[01-02]
2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz
64 cores per node
hyper-threading enabled
770 GB RAM
4 x NVIDIA L40s GPUs per node
41 execution nodes otterg[001-041]
2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz
64 cores per node
hyper-threading enabled
770 GB RAM
4 x NVIDIA L40S GPUs per node
node interconnect

Ethernet (Speed: 25Gb)

Filesystems:

GPFS-based with total size of 2 PB.

/u

shared home filesystem; GPFS-based; user quotas (100 GB data, 1M files/) enforced; quota can be checked with ‘/usr/lpp/mmfs/bin/mmlsquota’. NO BACKUPS YET

/otter/ptmp

shared scratch filesystem; no quotas enforced. NO BACKUPS !

/nexus/posix0/bmm

ONLY available on login nodes!

Compilers and Libraries:

Hierarchical environment modules are used at MPCDF to provide software packages and enable switching between different software versions. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the module avail command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the find-module command.

Batch system based on Slurm

A brief introduction into the basic commands (srun, sbatch, squeue, scancel, …) can be found on the Raven home page. For more detailed information, see the Slurm handbook. See also the sample batch scripts which must be modified for OTTER cluster.

Current Slurm configuration on OTTER:

  • default turnaround time: 2 hours

  • current max. turnaround time (wallclock): 24 hours

  • p.otter partition include all batch nodes in exclusive usage and is default

  • s.otter partition can be used for serial jobs and can be shared

Useful tips:

By default run time limit used for jobs that don’t specify a value is 2 hours. Use --time option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours.

Default memory per node in the shared partition is 94000 MB, maximum per allocated node per job is 770000 MB. To grant the job access to all of the memory on each node use --mem=0 option for sbatch/srun.

The OpenMP codes require a variable OMP_NUM_THREADS to be set. This can be obtained from the Slurm environment variable $SLURM_CPUS_PER_TASK which is set when --cpus-per-task is specified in a sbatch script (an example is on help information page).

To use GPUs add in your slurm scripts --gres option and choose how many GPUs and/or which model of them to have: #SBATCH --gres=gpu:l40s:X, where X is a number of resources from 1 up to 4 (l40s).

Support:

For support please create a trouble ticket at the MPCDF helpdesk