Astronomy

Name of the cluster:

VERA

Institution:

Max Planck Institute for Astronomy

Login nodes:

  • vera01.bc.rzg.mpg.de

  • vera02.bc.rzg.mpg.de

Their SHA256 ssh host key fingerprint is:

BCiQsLifb24aMoVJ0yNDIxHIhNfaztAE5DH+wNkn9ZQ (ED25519)

Hardware Configuration:

  • VERA is built on top of Intel Xeon Platinum 8360Y CPUs (36 cores at 2.40GHz), each node is equipped with two 8360Y CPUs

  • Contrary to RAVEN, VERA is operated with Hyper-Threading disabled, though

  • login nodes vera[01-02] (500 GB RAM each)

  • 72 execution nodes vera[001-072] (250 GB RAM each)

  • 36 execution nodes vera[101-136] (500 GB RAM each)

  • 2 execution nodes vera[201-202] (2 TB RAM each)

  • 3 execution nodes verag[001-003] (500 GB RAM and 4 Nvidia A100-40GB GPUs each)

  • node interconnect is based on Mellanox/Nvidia Infiniband HDR-100 technology (Speed: 100 Gb/s)

Filesystems:

/u

  • shared home filesystem

  • user quotas (1 TB of data; 400k files/directories) enforced

  • quota can be checked with ‘/usr/lpp/mmfs/bin/mmlsquota’.

/vera/ptmp

  • shared scratch filesystem (2.0 PB)

  • user quotas enforced (default 5 TB)

  • quota can be checked with ‘/usr/lpp/mmfs/bin/mmlsquota’

  • organized in folders apex, gc and psf - new users should contact their group leader to get a directory

  • NO BACKUPS!

Compilers and Libraries:

Hierarchical environment modules are used at MPCDF to provide software packages and enable switching between different software versions. There are no modules preloaded on VERA. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the module avail command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the find-module command.

Batch system based on Slurm:

  • a brief introduction into the basic commands (srun, sbatch, squeue, scancel, sinfo, s*…) can be found on the Raven home page or on the Slurm handbook

  • four partitions: p.vera (default), p.large, p.huge and p.gpu

  • current max. run time (wallclock): p.vera (2 days), p.large (2 days), p.huge ( 1 day), p.gpu (1 day, default runtime is 12 hours)

  • maximum memory per node for jobs: p.vera (250000 MB), p.large (500000 MB), p.huge (2048000 MB), p.gpu (500000 MB)

  • p.vera partition: nodes are exclusively allocated to users

  • p.large, p.huge, p.gpu partitions: resources on the nodes may be shared between jobs

  • p.gpu partition: to access GPU resources --gres parameter must be explicitly set for jobs

  • sample batch scripts can be found on Raven home page (must be modified for VERA)

Useful tips

Use --time option for sbatch/srun to set a limit on the total run time of the job allocation.

The OpenMP codes require a variable OMP_NUM_THREADS to be set. This can be obtained from the Slurm environment variable $SLURM_CPUS_PER_TASK which is set when --cpus-per-task is specified in a sbatch script.

Nvidia Ampere GPUs are available in p.gpu partition. Type of gpu must be explicitly set, i.e. --gres=gpu:a100:X, where X is between 1 and 4

GPU cards are in default compute mode.

Nodes in p.gpu partition are in shared mode i.e. jobs allocate only requested resources. Default memory per job is 125000 MB. Use --mem parameter to set necessary amount of RAM for jobs.

Nodes in p.large and p.huge partitions are in shared mode i.e. jobs allocate only requested resources. By default jobs allocate all memory on nodes. This means that to share node between other jobs --mem parameter is required for jobs.

Support:

For support please create a trouble ticket at the MPCDF helpdesk