Geoanthropology
- Name of the cluster:
GEANY
- Institution:
Max Planck Institute of Geoanthropology
Login nodes:
|
|
Hardware Configuration:
- 2 login node geany[01-02]:
- 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz64 cores per nodehyper-threading enabled - 2 threads per core770 GB RAM
- 70 compute nodes geany[001-070]:
- 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz64 cores per nodehyper-threading enabled - 2 threads per core770 GB RAM
- 4 compute nodes geany[101-104]:
- 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz64 cores per nodehyper-threading enabled - 2 threads per core1.5 TB RAM
- 2 compute nodes geanyg[001-002]:
- 2 x AMD EPYC 9554 48-Core Processor @ 3.10 GHz48 cores per nodehyper-threading disabled - 2 threads per core770 GB RAM8 x Nvidia L40S GPUs per node
- 1 compute node geanyg101:
- 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz64 cores per nodehyper-threading disabled - 2 threads per core770 GB RAM4 x Nvidia H100 GPUs per node
- 1 compute node geanyg201:
- 2 x AMD EPYC 9555 64-Core Processor @ 3.20 GHz64 cores per nodehyper-threading disabled - 2 threads per core770 GB RAM4 x Nvidia RTX6000 Blackwell Server Edition GPUs per node
- node interconnect:
based on Mellanox Technologies InfiniBand fabric (Speed: 200Gb)
Filesystems:
GPFS-based with total size of 4.7 PB and independent inode space for the following filesets:
- /u
shared home filesystem; GPFS-based; user quotas (100 GB data, 1M files/) enforced; quota can be checked with ‘/usr/lpp/mmfs/bin/mmlsquota’.
- /geany/fs
shared filesystem with project and individual user scratch area; GPFS-based; no quotas enforced. ONLY SELECTED PARTS ARE BEING BACKED UP !
Compilers and Libraries:
Hierarchical environment modules are used at MPCDF to provide software packages and enable switching between different software versions. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the module avail command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the find-module command.
Batch system based on Slurm
The batch system on GEANY is the Slurm Workload Manager. A brief introduction to the basic commands (srun, sbatch, squeue, scancel, …) can be found in the Raven user guide. For more detailed information, see the Slurm handbook. See also the sample batch scripts which must be modified for GEANY cluster.
Current Slurm configuration on GEANY:
default run time limit: 2 hours
current max. run time (wallclock): 168 hours
p.geany partition include all batch nodes in exclusive usage
s.geany partition can be used for serial jobs and can be shared, CPU only, default partition
s.geany.gpu partition can be used for serial jobs and can be shared, incl. GPUs
Useful tips:
By default run time limit used for jobs that don’t specify a value is 2 hours. Use --time option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 168 hours
Default memory per node in the shared partition is 47000 MB, maximum per allocated node per job is 360000 MB. To grant the job access to all of the memory on each node use --mem=0 option for sbatch/srun
The OpenMP codes require a variable OMP_NUM_THREADS to be set. This can be obtained from the Slurm environment variable $SLURM_CPUS_PER_TASK which is set when --cpus-per-task is specified in an sbatch script (an example is on help information page)
To use GPUs add in your slurm scripts --gres option and choose how many GPUs and/or which model of them to have: #SBATCH --gres=gpu:l40s:X, where X is a number of resources from 1 up to 4 (h100) or 8 (l40s)
GPU cards are in default compute mode.
Support:
For support please create a trouble ticket at the MPCDF helpdesk