Last updated: 2026-09-25T16:37:23Z (git commit 0f9112f4) .. MPCDF Technical Documentation root document and entry point for Sphinx, it must contain the root `toctree` directive. MPCDF Technical Documentation ============================= This technical documentation is a reference for the users of the MPCDF services. In case of questions that go beyond the scope of the documentation please contact `MPCDF support `_. Introductory online tutorials on how to use the MPCDF services are `given regularly `_. Frequently Asked Questions -------------------------- .. toctree:: :maxdepth: 2 faq/index.rst.txt Documentation ------------- .. toctree:: :maxdepth: 3 doc/index.rst.txt Bits and Bytes -------------- .. toctree:: :maxdepth: 2 bnb/index.rst.txt .. INDEX currently disabled Index ~~~~~ :ref:`genindex` .. note:: LLM-friendly versions of this documentation are available: * `llms.txt `_: index with links to the plain-text source of each page * `llms-full.txt `_: full text in a single file Frequently Asked Questions -------------------------- .. toctree:: :maxdepth: 2 :glob: account.md.txt connecting.md.txt 2fa.md.txt selfservice.md.txt hpc_software.md.txt hpc_systems.md.txt tricks.md.txt help.md.txt # Account Registration To apply for an MPCDF user account, fill out the application form on our website: [New Users - Computing at MPCDF](https://www.mpcdf.mpg.de/userspace/new-users). The Terms of Use for different account types are also available on that page: | Account Type | Access Provided | |--------------|-----------------| | **Full user accounts** | Access to Unix systems | | **Guest accounts** | Access to DataShare and GitLab only | # Connecting to MPCDF Systems ## How do I log in to MPCDF machines? All MPCDF compute resources run Linux. Access is provided exclusively via SSH with two-factor authentication (2FA). To connect from the public internet, first log in to a gateway machine, then connect to your target system from there. The standard command-line client is OpenSSH, available on Linux, macOS, and Windows. Use the `ssh` command to connect, and tools like `ssh-keygen` and `ssh-add` to manage your SSH keys. For convenience, create a configuration file at `~/.ssh/config` to define host aliases and connection parameters. For more details, see our [SSH configuration guide](ssh_config). ### What are the SSH gateway machines? The following gateway machines are available for SSH connections from the internet: | Gateway | Reboot Schedule | |---------|-----------------| | `gate1.mpcdf.mpg.de` | Tuesdays at 3:45 AM (German local time) | | `gate2.mpcdf.mpg.de` | Saturdays at 3:45 AM (German local time) | These machines provide a small home directory and serve as jump hosts to the HPC systems. **Note:** Gateway machines are rebooted weekly, so user sessions are not persistent. For more details, including host key fingerprints, see the [gateway machines documentation](../doc/computing/gateways). ## How can I tunnel through the gateway machines? Configure an SSH tunnel to connect directly to a target machine through a gateway. The recommended method is `ProxyJump` (for OpenSSH 7.3 and newer). For older versions, use `ProxyCommand`. **Using the `ProxyJump` flag:** ```bash ssh -J YOUR_USERNAME@gate1.mpcdf.mpg.de YOUR_USERNAME@viper.mpcdf.mpg.de ``` **Using your SSH `config` file:** Simplify the connection by adding the following to your `~/.ssh/config` file: ``` Host gate Hostname gate1.mpcdf.mpg.de User YOUR_USERNAME ServerAliveInterval 120 Host viper Hostname viper.mpcdf.mpg.de User YOUR_USERNAME ProxyJump gate ``` With this configuration, connect to Viper by simply running: ```bash ssh viper ``` ## How can I avoid repeatedly typing my password? Use SSH's `ControlMaster` feature to establish a connection once and reuse it for subsequent logins. This is particularly useful when combined with `ProxyJump`. **Note:** `ControlMaster` is limited to 10 sessions by default. Here is an example configuration for your `~/.ssh/config` file that sets up `ControlMaster` for the `gate` and `raven` hosts: ``` Host gate Hostname gate.mpcdf.mpg.de User YOUR_USERNAME ServerAliveInterval 120 ControlMaster auto ControlPersist 12h ControlPath ~/.ssh/master-%C Host raven Hostname raven.mpcdf.mpg.de # or a specific login node like raven02i.mpcdf.mpg.de User YOUR_USERNAME ControlMaster auto ControlPersist 12h ControlPath ~/.ssh/master-%C # For OpenSSH 7.3 and newer: ProxyJump gate # For older OpenSSH versions: # ProxyCommand ssh -W %h:%p gate ``` After adding this configuration, connect to Raven with: ```bash ssh raven ``` You will only need to enter your password and OTP once. Subsequent connections, including `scp` and `rsync`, will reuse the existing connection. **Note:** This configuration works on Linux and macOS. For Windows, use PuTTY's "Share SSH connections if possible" feature to achieve a similar result. ## How can I connect to HPC systems with Visual Studio Code (VSCode)? Connect to our HPC systems using the [Remote - SSH](https://code.visualstudio.com/docs/remote/ssh) extension of VSCode. To avoid repeated authentication prompts and resource limits, follow these steps: 1. **Configure SSH:** Set up `ProxyJump` and `ControlMaster` in your `~/.ssh/config` as described above. Use a specific login node (e.g., `raven03i` or `viper03i`) rather than the generic hostname. 2. **Open the master SSH connection:** Log in once in a terminal (e.g., `ssh raven`) and enter your password and OTP. Keep this session open. VSCode reuses the connection and does not prompt for your credentials again. 3. **Install the extension:** Install "Remote - SSH" from the VSCode Marketplace. 4. **Configure the extension:** Set `remote.SSH.useExecServer: false`; set `remote.SSH.useLocalServer: false` on Linux and MacOS 5. **Connect:** Run "Remote-SSH: Connect to Host..." from the command palette (`F1`) or via the panel on the left and select your host alias (e.g., `raven`). **If the connection fails**, try the following steps one at a time: * If you enter your password and OTP within VSCode, increase `remote.SSH.connectTimeout` (e.g., to `60` seconds) and enable `remote.SSH.showLoginTerminal`. * Run "Remote-SSH: Kill VS Code Server on Host..." from the command palette and reconnect. **Windows users:** `ControlMaster` is not available on Windows. Add `"remote.SSH.useLocalServer": true` directly to your `settings.json` (it cannot be set in the settings dialog on Windows). All VSCode windows then share a single SSH connection, and you only need to authenticate once. **Note:** Due to the wide variety of user configurations, we cannot provide support for VSCode beyond these instructions. ## How do I connect from a Windows machine? The `ProxyJump` configuration described above is compatible with the VSCode Remote-SSH extension and the OpenSSH client in PowerShell. However, `ControlMaster` is not supported on Windows. For VSCode, see the [hints for Windows users above](#how-can-i-connect-to-hpc-systems-with-visual-studio-code-vscode). For detailed instructions on using PuTTY and WinSCP, see our step-by-step guides: * [PuTTY guide](steps/putty) * [WinSCP guide](steps/winscp) ### What if my connection fails with "Corrupted MAC on input"? This error can occur on Windows with native OpenSSH clients due to stricter encryption algorithms on our gateway systems. To resolve this, specify a different MAC (Message Authentication Code) algorithm. **On the command line:** ```bash ssh -m hmac-sha2-256-etm@openssh.com YOUR_USERNAME@gate1.mpcdf.mpg.de ``` **In your `~/.ssh/config` file:** ``` Host gate Hostname gate1.mpcdf.mpg.de User YOUR_USERNAME MACs hmac-sha2-256-etm@openssh.com ``` ### Is two-factor authentication (2FA) required? Yes, 2FA is mandatory for all connections. For more information, see the [2FA FAQ](2fa). ### Are SSH keys supported for login? No, SSH key-based login is not supported on the gateway machines or any of the HPC systems. ### What should I do if I see an SSH host key warning? If you receive a host key warning, it may be due to a recent maintenance operation. We announce host key changes via email and on this documentation page. You can verify the current host keys in the [gateway machines documentation](../doc/computing/gateways). If you are unsure, contact the [MPCDF helpdesk](help#how-can-i-get-help-and-support) for assistance. ## How can I run GUI applications on MPCDF systems? You can run applications with graphical user interfaces (GUIs) on our systems using one of the following methods. ### X11 Forwarding You can forward X11 displays to your local machine via SSH. To connect with X11 forwarding: ```bash ssh -C -Y YOUR_USERNAME@gate1.mpcdf.mpg.de ``` Or, add the following to your `~/.ssh/config` file: ```xorg.conf Host gate Hostname gate1.mpcdf.mpg.de User YOUR_USERNAME Compression yes ForwardX11 yes ForwardX11Trusted yes ``` The `-C` flag enables compression, which can improve performance. While most Linux systems have a built-in X server, macOS and Windows users will need to install one, such as [XQuartz](https://www.xquartz.org/) (for macOS) or [Xming](https://sourceforge.net/projects/xming/) (for Windows). For graphically intensive applications, we recommend using VNC. ### VNC You can run a persistent VNC server on the login nodes of the HPC systems. This allows you to run GUI applications without X11 forwarding. Alternatively, you can use our web-based [remote visualization service](../doc/visualization/index#remote-visualization-and-jupyter-notebook-services) to launch VNC sessions on dedicated resources (with a time limit). ### Remote Visualization Service For applications that require hardware-accelerated OpenGL rendering, our web-based [remote visualization service](../doc/visualization/index#remote-visualization-and-jupyter-notebook-services) provides access to GPUs on select HPC systems. # Two-factor authentication (2FA) ## General information about 2FA ### Do I need to enable 2FA? You must enable 2FA if you use any of the following services: * The SSH gateway machines (gate1, gate2) * The HPC systems via SSH or VNC * The remote visualization service * The MPCDF instance of GitLab at [https://gitlab.mpcdf.mpg.de](https://gitlab.mpcdf.mpg.de) * The MPCDF DataShare instance at [https://datashare.mpcdf.mpg.de](https://datashare.mpcdf.mpg.de) (after 09/13/2025) If you do not use any of these services, you are not required to enable 2FA at this time. Please note that this list may be updated in the future. ### How can I check if 2FA is activated on my account? You can easily check if 2FA is active for your account: * When you log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de), you will be prompted for a One-Time Password (OTP) if 2FA is enabled. * If you are already logged in, navigate to "My Account > Security > Configure 2FA". If a token is listed, 2FA is enabled. ### Why is 2FA enforced? Enforcing 2FA is a crucial security measure to protect our systems and your account. Here's why: * Cyberattacks are increasingly common. In early 2020, a major attack on European research institutions exploited leaked login credentials, highlighting the need for stronger security. * Even if your account doesn't contain sensitive data, a compromised account can be used to disrupt services and harm other users. * 2FA adds a second layer of security. It requires you to provide both something you know (your password) and something you have (a token). This means that even if an attacker steals your password, they cannot access your account without your token. ### What are tokens, OTPs, and seeds? These are the core components of 2FA: * **OTP (One-Time Password):** A password that is valid for only one login session or transaction. The most common type is a Time-based One-Time Password (TOTP), which automatically regenerates every 30 seconds. * **Token:** A device or application that generates OTPs. This can be a dedicated hardware device, or an app on your smartphone. * **Seed:** A secret key shared between the token and the authentication server. The seed is used to generate the same sequence of OTPs on both your token and the server. It is critical to keep your seed secure, just like a password. ### What kinds of tokens are available? We offer two types of tokens: * **Primary Tokens:** You can have only one primary token at a time. * **App Token:** An application on your smartphone or tablet. * **Hardware Token:** A dedicated physical device. * **Secondary Tokens (Backup):** You can have multiple secondary tokens of different types. These are for backup purposes. * **TAN List:** A list of single-use passwords. * **SMS:** A code sent to your mobile phone (this method is being phased out and is not recommended). **Important:** SMS tokens are less secure and should only be used as a backup if you cannot access your primary token. ### When will I be asked for an OTP? You will be prompted for an OTP in the following situations: * **Logging into the SelfService Portal:** If 2FA is enabled for your account, you will be asked for an OTP. You can find your OTP in the authenticator app on your phone. Look for an entry with your account name and a serial number starting with "TOTP...". * **Accessing SSH Servers:** 2FA is mandatory for all SSH access. You will always be asked for an OTP, so you must have a token enrolled to access these systems. **What if I've lost my token?** If you have uninstalled your authenticator app or changed your phone, you will need to use the token recovery process to regain access to your account. ## Activation of 2FA ### How do I enable 2FA? To enable 2FA on your account, follow these steps: 1. Log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de). 2. Navigate to "My Account > Security" from the top menu. 3. Select "Configure 2FA" and enter your password. 4. Choose a primary token type to enroll. 5. **Validate your token by entering a valid OTP from it.** 6. (Optional) Choose a secondary token type for backup. **Important:** You must validate your primary token to complete the 2FA setup. Without validation, you will not be able to use it for authentication. ### How do I enroll and use an app token? You can use any smartphone or tablet with a compatible OTP app. While tablets work, we recommend using a smartphone for convenience. **Important:** OTP generation is time-sensitive. Ensure your device's clock is set accurately. For information on using devices with a deliberate time offset, see "[How to use 2FA on a phone with a time shift](#how-to-use-2fa-on-a-phone-with-a-time-shift)". To enroll an app token: 1. Log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de). 2. Navigate to "My Account > Security" and select "Configure 2FA". You will be prompted for your password. 3. If you have an existing token, click "Replace existing or enroll additional token". 4. Click "App token". (This option is only available if you do not have a hardware token). 5. Open your OTP app and scan the QR code. You may need to tap a "+" icon to add a new token. 6. If your app requires manual configuration, use these settings: * **Type:** TOTP * **Algorithm:** SHA-1 * **Timestep:** 30 seconds * **Digits:** 6 7. A new token entry will appear in your app, named with your username and a serial number starting with "TOTP". 8. The app will generate a new 6-digit OTP every 30 seconds. 9. **Activate your token:** Click the validation button in the SelfService portal and enter the current OTP from your app. **This step is mandatory.** ### Which authenticator app should I use? You can use a wide variety of authenticator apps. We recommend open-source applications for transparency and security. While there are fewer open-source options for iOS, several excellent choices are available for Android. You can also use popular closed-source apps like Google Authenticator, Microsoft Authenticator, or Authy if you prefer. Here are some recommended open-source apps: App | OS | Source | Features ------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | ---------------------------------------------------- Aegis Authenticator | [Android](https://play.google.com/store/apps/details?id=com.beemdevelopment.aegis&hl=en_US) | [Github](https://github.com/beemdevelopment/Aegis) | Backup, Encryption, Authentication with fingerprint andOTP | [Android](https://play.google.com/store/apps/details?id=org.shadowice.flocke.andotp&hl=en_US) | [Github](https://github.com/andOTP/andOTP) | Backup, Encryption PrivacyIDEA Authenticator | [Android](https://play.google.com/store/apps/details?id=it.netknights.piauthenticator&hl=en_US), [iOS](https://apps.apple.com/us/app/privacyidea-authenticator/id1445401301) | [Github](https://github.com/privacyidea/privacyidea-authenticator) | Push tokens (not yet supported by us) For Android users who prefer not to use the Google Play Store, most of these apps are also available on [F-Droid](https://f-droid.org), an alternative app repository for open-source software. ### How do I register an existing hardware token? If you already own a hardware token, you can register it with our services. **Important:** We only support TOTP tokens (time-based, with OTPs changing every 30 or 60 seconds). HOTP tokens are not supported. To register your token: 1. Log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de). 2. Navigate to "My Account > Security" and select "Configure 2FA". 3. If you have an existing token, click "Replace existing or enroll additional token". 4. Click "Register token". 5. Enter your token's details. All information must be exact, except for the serial number, which is for your reference. 6. If you do not know your token's seed, please contact your supplier. 7. Submit the form and confirm the registration by entering a valid OTP from your token. ### How do I enroll and use a secondary (backup) token? To enroll a secondary token for backup purposes: 1. Log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de). 2. Navigate to "My Account > Security" and select "Configure 2FA". 3. If you do not have a primary token, you must enroll one first. 4. Click "Replace existing or enroll additional token". 5. Choose a secondary token type (e.g., TAN list, SMS). **Using SMS Tokens:** * You will be prompted to provide your mobile number if we do not have it on file. * Once enrolled, you can request an SMS OTP from the SelfService login page if you do not have access to your primary token. **Using TAN Lists:** * Store your TAN list in a secure location, such as a locked cabinet or an encrypted file. * Do not store your TAN list in a password manager, as this would compromise your security if your manager is breached. * Each TAN can be used once, in any order. ### Why can't I have an app token and a hardware token simultaneously? This is due to a technical limitation and our security policy. Both app and hardware tokens are of the same type ("TOTP"), and our system permits only one token of each type to be active at a time. Therefore, you must choose between an app token or a hardware token as your primary 2FA method. ## 2FA Tips and Tricks ### Do I have to type in an OTP every time I access the secured systems? For the systems you access via SSH you can configure a ControlMaster setup. Allowing you to conveniently type in your password and OTP only once a day. After that an SSH tunnel will be kept open for the day that can be used without having to retype your credentials. #### Linux and MacOS Please find an example of a ControlMaster setup for Linux and MacOS [here](tricks#how-can-i-avoid-having-to-type-my-password-repeatedly-how-can-i-tunnel-through-the-gateway-machines). #### Windows Unlike on Linux and MacOS, the OpenSSH client on Windows doesn't support ControlMaster setups. You'll need to use one of the graphical clients to avoid having to retype your password and OTP. Two examples are given in the following. ##### PuTTY To avoid retyping your credentials with PuTTY, you can configure connection sharing. For detailed instructions on using PuTTY, please refer to the [official user manual](https://the.earth.li/~sgtatham/putty/0.74/htmldoc/) or our [step-by-step guide](steps/putty). **Connecting via a Gateway:** 1. Create a new session for the gateway machine (e.g., `gate1.mpcdf.mpg.de`). 2. Save the session with a descriptive name (e.g., `raven.mpcdf.mpg.de`). 3. Load the session and go to "Connection > SSH > Tunnels". 4. Enter `22` as the "Source port". 5. Enter the destination hostname and port (e.g., `raven.mpcdf.mpg.de:22`) as the "Destination". 6. Click "Add". 7. Return to the "Session" settings, and save the session again. 8. Click "Open" to connect. **Enabling Connection Sharing:** 1. Go to "Connection > SSH". 2. Enable the "Share SSH connections if possible" option. 3. As long as the initial SSH session is active, new PuTTY windows will reuse the connection without requiring you to re-enter your credentials. ##### MobaXterm For guidance on using MobaXterm, refer to the [official documentation](https://mobaxterm.mobatek.net/documentation.html). To enable 2FA support in MobaXterm: 1. Go to "Settings > SSH". 2. Check the "Use 2-factor authentication for SSH gateways" box. To configure a session for connecting via a gateway: 1. Create a new session by clicking "Session > SSH". 2. Enter the remote hostname (e.g., `raven.mpcdf.mpg.de`) and your username. 3. Go to the "Network settings" tab and select "SSH gateway (jump host)". 4. Enter the gateway hostname (e.g., `gate1.mpcdf.mpg.de`) and your username. 5. Click "OK". You will be prompted for your password and OTP. MobaXterm automatically reuses connections, so you will not need to enter your OTP again for subsequent connections. #### Weekly Reboots For security reasons, our SSH gateway machines are rebooted weekly. This action terminates any long-standing SSH tunnels. ## 2FA Troubleshooting ### What should I do if I need to factory-reset my phone? Before factory-resetting your phone, ensure you have a secondary token enrolled. If you don't, please [enroll one first](#how-do-i-enroll-and-use-a-secondary-backup-token). After the reset, you can re-enroll your app token: 1. Log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de) using your secondary token. 2. Navigate to "My Account > Security > Configure 2FA > Replace existing or enroll additional token". 3. Click "App token" to create a new token. 4. Scan the QR code with the authenticator app on your newly reset phone. 5. Validate the new token by entering a valid OTP. This process will replace your old token with the new one. Alternatively, some authenticator apps (like Aegis Authenticator) support backup and restore. You can back up your tokens before resetting your phone and restore them afterward. Be sure to save the backup file to an external location (e.g., an SD card or cloud storage). ### How do I transfer my token to a new phone? To transfer your token to a new phone, you will need access to your old phone or a secondary token. 1. Log in to the [SelfService portal](https://selfservice.mpcdf.mpg.de) using your old phone's token or a secondary token. 2. Navigate to "My Account > Security > Configure 2FA > Replace existing or enroll additional token". 3. Click "App token" to create a new token. 4. Scan the QR code with the authenticator app on your new phone. 5. Validate the new token by entering a valid OTP. This will replace your old token with the new one. You can then safely delete the token from your old phone. Alternatively, some authenticator apps (like Aegis Authenticator) support backup and restore. You can create a backup on your old phone, transfer it to your new phone, and restore it in the app. ### What if I can't validate my token ("Wrong OTP" error)? If you receive a "Wrong OTP" error during token validation, please try the following: * **Check the code:** Ensure you are entering the 6-digit code displayed by your authenticator app. Some apps require you to tap the entry to reveal the code. * **Enter the current code:** OTPs are time-sensitive. Enter the code while it is still active. * **Check your device's clock:** The OTP algorithm relies on an accurate clock. Make sure your device's time is correct. * **Try a different device:** If possible, try setting up the token on a different device. * **Contact support:** If you still can't validate your token, please [contact support](help). 2FA will not be enabled on your account until you have an active token. ### I can't log in to the SelfService anymore #### Troubleshooting a Rejected Password * **Check other services:** Try logging in to other MPCDF services to verify your password. * **Wait and retry:** If you have made several failed attempts, your account may be temporarily locked. Wait 10 minutes and try again. * **Contact support:** If you still cannot log in, your account may be suspended or your password may have expired. Please [contact support](help). #### Troubleshooting a Rejected OTP ("Wrong OTP") * **Check the code and timing:** Ensure you are entering the correct 6-digit code while it is still active. * **Verify the token:** Make sure you are using the correct token from your authenticator app. The token's serial number in the app should match the one listed in the SelfService portal under "My Account > Security > Configure 2FA". * **Check your device's clock:** Ensure your device's time is accurate. * **Resync your token:** You can try to resynchronize your token by clicking "Resync token" in the SelfService portal. * **Use a backup token:** If you have a backup token enrolled, try using it to log in. * **Contact support:** If you are still unable to log in, please [contact support](help) from your registered email address. #### What to Do If You Lose Your Token * **If you have a backup token:** In the SelfService portal, click “Lost Token” to receive a one-time password via SMS (if you have an SMS token enrolled) or use a token from your backup TAN list. * **If you don't have a backup token:** * If the email address registered with your MPCDF account is signed with a personal S/MIME certificate, you can contact support from your registered email address. Please specify that you wish to authenticate through your email certificate. * Enter your username and password into SelfService and then request a token reset. For security reasons, we will send an automated message to the contact person/administrator of your MPCDF account. They need to confirm your request via a web link in that email. * **Hardware token loss:** If you have lost a hardware token, please [notify support](help) immediately so we can disable it. ### What if I can't log in to a gate machine via SSH? If you are having trouble logging in to a gate machine via SSH, try these steps: * **Simplify your connection:** Attempt to log in without any extra local configuration. * **Verify your password:** Try logging in to another MPCDF service to confirm your password is correct. * **Check token activation:** Ensure you have an active 2FA token by logging in to the SelfService portal. You should be prompted for an OTP. * **Isolate the issue:** Try logging in to a different gate machine or the SelfService portal to determine if the problem is with a specific machine. * **Troubleshoot the OTP:** If you suspect the OTP is being rejected, follow the steps in the [Troubleshooting a Rejected OTP](#troubleshooting-a-rejected-otp-wrong-otp) section. ### Why can't I access HPC clusters via VNC? When using `vncviewer` with the `-via` option to connect to an HPC machine through a gate machine, you must provide an OTP. Ensure you have enrolled a 2FA token by following the instructions in ["How do I enable 2FA?"](#how-do-i-enable-2fa). To avoid entering an OTP for each connection, you can configure a ControlMaster setup as described in our [tips and tricks section](tricks#how-can-i-avoid-having-to-type-my-password-repeatedly-how-can-i-tunnel-through-the-gateway-machines). ## Hardware and Client Support ### How can I use GUI tools (sshfs, rsync, scp, sftp) with 2FA? While some GUI applications support 2FA natively, many do not. **Applications that support 2FA:** * FileZilla (see [How can I use FileZilla with 2FA?](#how-can-i-use-filezilla-with-2fa)) * WinSCP (see our [step-by-step guide](steps/winscp)) **Applications that do not support 2FA:** * KDE Dolphin * Gnome Nautilus If your client does not support 2FA, you have three options: 1. Use a client that supports 2FA. 2. Use [MPCDF DataShare](https://datashare.mpcdf.mpg.de) for your data. 3. Create an SSH tunnel and forward the remote port to your local machine. For example: ```bash ssh -L 2002:raven.mpcdf.mpg.de:22 USER@gate1.mpcdf.mpg.de ``` You can then connect to `sftp://USER@localhost:2002/u/USER`. ### How can I use FileZilla with 2FA? In FileZilla, set the "Logon Type" to "Interactive". You will be prompted for your password and OTP. We also recommend enabling "Limit number of simultaneous connections" under "Transfer Settings" and setting it to 1. ### How do I use 2FA on a phone with a custom time shift? Our system can accommodate phones with a deliberate time offset (this is different from a time zone shift, which requires no special configuration). During token validation, you will have the option to synchronize your token. You will be prompted to enter two consecutive OTPs to complete the synchronization. ### Do you support FIDO2, U2F, or YubiKeys? You can use a YubiKey (NEO, 4, 5, and FIPS series) by enrolling it as an app token with the [Yubico Authenticator](https://www.yubico.com/products/yubico-authenticator/) app. The app will generate an OTP when the YubiKey is touched or tapped. We are exploring support for other FIDO/U2F mechanisms, but there is no implementation date at this time. ## Security ### How are token seeds secured on the server? Token seeds are stored in an AES-encrypted database. Access to this database is strictly limited to system administrators, who must use a separate 2FA system and dedicated administrator accounts. In the event of a database leak, the seeds would remain encrypted and unreadable. ### Who provides the hardware tokens, and do they know the seeds? We source our hardware tokens directly from the manufacturer, [Feitian Technologies](https://www.ftsafe.com/). We generate a unique seed for each token and program it ourselves via NFC. The vendor does not have access to the seeds. # SelfService Portal General FAQ and troubleshooting for the MPCDF SelfService platform. ## Accepting a Guest invitation MPCDF users invite guests to let them gain access to specific services provided by the MPCDF in order to let them participate in projects or share files. Guests use the credentials they chose when registering on this platform to log in to those backend services. Access to the services may be granted or revoked on a per-service basis at any point. Once you received an invitation from an MPCDF user you have 7 days to go to the provided link and register as a guest user. After 7 days the link expires and your data is deleted from the database. The link also expires after a successful registration. If you have lost your invitation mail please ask your inviter to resend it. On the registration page you have the chance to correct any spelling mistakes in your name. You are provided with a preliminary username that you can change to your liking. Note that any username you choose will always start with 'g-'. For the username you may use any Latin character and Arabic numeral; for your full name, a range of European characters and dashes is allowed. You will need to set a password for your account. Please consult our [password policy](https://selfservice.mpcdf.mpg.de/index.php?r=site%2Fpasswordpolicy) for password requirements and make sure you use a password that can't be guessed easily. Also note that reusing existing passwords is not allowed since this makes your account very vulnerable to password reuse attacks. After you have accepted the Terms of Use you can activate your account. Please note it can take up to 20 minutes for your account to be synchronized to the backend services. After this period you will be able to log in to the services you were invited to use. ## Two-Factor Authentication (2FA) For all questions regarding the setup and use of two-factor authentication, please refer to the [2FA Page](/faq/2fa.html). ## Login Issues **I recently changed my password and cannot log in.** Your password is likely locked temporarily due to multiple failed attempts by background services (like email clients) using your old credentials. Wait at least 5 minutes and ensure all devices are updated. **I haven't changed my password recently or My Password/Account has expired** Your password or account may have expired. Please request a password reset (account restoration), see below: * **Users:** Please request a [Password Reset](https://selfservice.mpcdf.mpg.de/index.php?r=site/request-password-reset). * **Guests:** Please ask your inviter to extend your account validity. ## Password Changes **The "Change password" button is not clickable.** The button remains disabled until the new password meets all security requirements. - Look for the **red X symbols** next to the input field. - The password is only accepted once all symbols turn into **green checkmarks**. - **Browser Compatibility:** Ensure you are using a modern browser (Firefox 55+, Chrome 58+, Edge 16+, Safari 12.1+). - **JavaScript:** Ensure JavaScript is enabled; otherwise, the validation logic cannot run. **My current password is not accepted.** Ensure you have selected the correct account from the dropdown menu. If you are changing a **secondary/functional account**, the password is not the same as your primary login. Use the "eye" icon to check for typos. ## Locked Accounts If there are too many failed login attempts, your account is locked **temporarily**. This sometimes happens after a password change and is often caused by client software that still has the old password saved. When your password is locked you won't be able to authenticate or change your password. Please check all your programs, apps, and devices and delete the old password everywhere. Such software could be: * Email clients (Thunderbird, Outlook) * The Email app on your phone * SFTP/SCP file transfer clients * Automated scripts * Calendar apps that sync with DataShare Note that programs like Thunderbird have an internal password store, and you might have to delete the password from there. Please see the documentation for the according program on how to do this. Please delete the old password from these services before attempting to log in again. After all old passwords are removed you'll need to wait 5 minutes for the password to become unlocked again. ## Windows Password Sync If you see an error updating your Windows password, your Kerberos and Windows passwords may be out of sync. Please request a [Password Reset](https://selfservice.mpcdf.mpg.de/index.php?r=site/request-password-reset) to re-sync them. ## Forgotten Passwords If you have forgotten your password, please request a [Password Reset](https://selfservice.mpcdf.mpg.de/index.php?r=site/request-password-reset). * **With 2FA:** You will be prompted for an OTP. * **Without OTP:** If your token is unavailable, click "Token unavailable." Note that this requires manual verification therefore the process might take longer. ## Cannot access backend services (DataShare, GitLab, etc.) * **Users:** Please check if you are actually subscribed to the service you can't access by logging in to the SelfService and navigating to **My Account > Services**. You can grant yourself access to the services there. Note that it can take up to 20 minutes for the changes to become effective. If you also can't log in to this platform or any other MPCDF service then your password or account may have expired. If you find that you are already subscribed to the service you are trying to log in to please contact support.
* **Guests:** Your account may have expired or have been deactivated by your inviter. Your inviter can also withdraw access on a per-service basis so you might still have access to other services. Please check with your inviter whether your account has the necessary access rights. Note that it can take up to 20 minutes for the access rights to be updated after a change. It is also possible that your inviter's user account has been deactivated. In this case you will need to find an MPCDF user that is willing to "adopt" your orphaned account. ## My Inviter's Account is locked For Guest users, if your inviter's account is locked, please contact support to have your guest account transferred to a different "inviter". ## Export Control Administration of Accounts In case of questions, please contact [support@mpcdf.mpg.de](mailto:support\@mpcdf.mpg.de) ## Other FAQs **My regular user account is about to be closed and I still need access to some resources** You can have another regular MPCDF user invite you as a guest with a private email address. However, note that it is not possible to create a guest account with an email address that used to be associated with your regular user account. You will also not be able to access any data associated with your regular user account from your guest account. **I don't continuously need an MPCDF account but always need access to certain resources like a git repository** It is not possible to grant access to resources that belong to a locked user. Therefore, to ensure that a person always has access to certain resources even if the person's regular account is locked please create a dedicated guest account with the appropriate access rights. Please note that transferring resources between the two accounts or accessing one from the other is not possible due to security considerations. Please do not ask support for exceptions as requests will not be considered. # HPC Software and Applications ## General Questions ### How can I install my own software? You can install software in your home directory, where you have write permissions. Root privileges are not required for most software installations. When installing software, be sure to specify an installation directory within your home directory. Here are some examples for common build systems: **GNU Build System:** ```bash ./configure --prefix=$HOME/soft/my_package make && make install ``` **CMake:** ```bash cmake -DCMAKE_INSTALL_PREFIX:PATH=$HOME/soft/my_package . make && make install ``` **Python:** To install a Python package to your local user directory (`~/.local`): ```bash # Using pip pip install --user package_name # Using a setup.py file python setup.py install --user ``` For detailed instructions, please refer to the documentation provided with the software. If you encounter any issues, you can request support from our [helpdesk](help#how-can-i-get-help-and-support). ## Environment modules The MPCDF uses modules to adapt the user environment for working with software installed at various locations in the file system or for switching between different software versions. The user is no longer required to explicitly specify paths for different executable versions, or keep track of `PATH`, `MANPATH` and related environment variables. With the modules approach, users simply 'load' and 'unload' modules to control their environment. Note that since 2018, HPC systems as well as the increasing number of dedicated clusters all use [hierarchical environment modules](#how-do-the-hierarchical-environment-modules-work). ### How do I use environment modules interactively? Here are the most common `module` commands. For a complete reference, see the [official documentation](https://modules.readthedocs.io/en/latest/module.html). * **`module help`**: Lists all `module` subcommands. * **`module avail`**: Lists all available software packages. * **`module help /`**: Shows help for a specific package. * **`module load /`**: Loads a package into your environment. * **`module unload /`**: Removes a package from your environment. * **`module list`**: Lists all currently loaded packages. ### How do I use environment modules in scripts? Environment modules are not loaded by default in non-interactive shells (e.g., shell scripts or batch scripts). To use them, you must first source the appropriate profile script: * **For `bash` or `sh` shells:** ```bash source /etc/profile.d/modules.sh ``` * **For `csh` or `tcsh` shells:** ```csh source /etc/profile.d/modules.csh ``` ### How can I avoid using absolute paths in my scripts? When you load a module, it sets environment variables that you can use in your scripts and makefiles. For example, most modules set a `_HOME` variable (e.g., `MKL_HOME`) that points to the package's installation directory. To see all environment variables set by a module, use `module show /`. For more information, use `module help /`. ### Examples #### Interactive Session ```bash $ module load intel mkl $ ifort -I$MKL_HOME/include example.F -L$MKL_HOME/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core ``` #### Makefile ```make FC = ifort example: example.F $(FC) -I$(MKL_HOME)/include test.F -L$(MKL_HOME)/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core ``` #### Paging through `module avail` output To view the output of `module avail` one page at a time, you can pipe it to `less`: **bash/sh:** ```bash module avail 2>&1 | less ``` **csh/tcsh:** ```csh ( module avail ) |& less ``` ### How do hierarchical environment modules work? To manage the large number of software packages and their dependencies, we use a hierarchical module system. Compilers (e.g., `gcc`, `intel`) are at the top level, followed by MPI libraries, and then other libraries. This means that you must load a compiler module before you can see the modules that depend on it. Similarly, you must load an MPI module to see the modules that depend on that MPI implementation. To reset your environment and return to the root of the module hierarchy, use `module purge`. For example, to load the FFTW library compiled with the Intel compiler and Intel MPI, you would load the modules in order: ```bash module load intel module load impi module load fftw-mpi ``` After loading `intel`, `module avail` will show the available MPI libraries. After loading `impi`, you will see the available FFTW libraries. You can also load all required modules in a single command, as long as you maintain the correct hierarchical order: ```bash module load intel impi fftw-mpi ``` ### How do I quickly find a module? If you know the name of a module but are unsure of its version or dependencies, you can use the `find-module` command: ```bash find-module ``` The output will show you the available versions and any required dependencies. You can then load the module and its dependencies in the correct order. **Example:** ```bash $ find-module horovod horovod/cpu/0.13.11 (after loading anaconda/3/2019.03 tensorflow/cpu/1.14.0) $ module load anaconda/3/2019.03 tensorflow/cpu/1.14.0 horovod/cpu/0.13.11 ``` Note that many applications and tools (e.g., `git`, `cmake`, `matlab`) are not part of the hierarchical module system and are available at the top level. ### How can I disable the "MPCDF specific note" for `module avail`? The `module avail` command displays a note about our hierarchical module system. To disable this note, set the following environment variable in your `~/.bashrc` file: ```bash export MPCDF_DISABLE_MODULE_AVAIL_HINT=1 ``` ### Why are there no BLAS/LAPACK modules? We provide Intel's Math Kernel Library (MKL), which includes highly optimized versions of BLAS, LAPACK, and other linear algebra libraries. For more information, please see our [MKL guide](../doc/computing/software/libraries#intel-math-kernel-library). ## Compiled Languages ### CMake #### Which CMake version should I use? We recommend using the newest available version of CMake. CMake is backward-compatible, so you can use a newer version with older `CMakeLists.txt` files (version 3.0 and later). Newer versions of CMake provide better support for the latest compilers and libraries. #### What does the "Policy CMPXXXX is not set" warning mean? This warning indicates that the behavior of a CMake feature has changed between the version specified in your `CMakeLists.txt` file and the version you are currently using. * **As a user:** You can generally ignore this warning. CMake will use the behavior defined in the `CMakeLists.txt` file. * **As a developer:** You should review the policy change and update your `CMakeLists.txt` file to explicitly set the desired behavior. For more information, see the [CMake policies documentation](https://cmake.org/cmake/help/latest/manual/cmake-policies.7.html). #### What if CMake cannot find a library? If CMake cannot find a library, you can help it by setting the `_ROOT` environment variable (e.g., `BOOST_ROOT`) to the library's installation directory. We strive to set these variables automatically when you load a module, but not all modules currently do so. If you find a module that is missing a `_ROOT` variable, please let us know. ### C/C++ and Fortran #### Which compilers are supported? We support the Intel and GNU compilers for C/C++, and Fortran. We provide MPI bindings and a wide range of libraries for these compilers through our hierarchical module system. Other compilers, such as Clang, may be available through the module system but are not officially supported. #### How do I ensure my executable finds shared libraries at runtime? If your application depends on shared libraries (`.so` files) that are not in a standard system directory, you may encounter an error like `cannot open shared object file: No such file or directory`. To resolve this, you can either set the `rpath` when you compile your application (recommended) or set the `LD_LIBRARY_PATH` environment variable at runtime. ##### Setting the `rpath` (Recommended) The `rpath` embeds the library path directly into your executable. You can set it at link time using the `-rpath` linker flag. For example, to link against `libfoobar.so` located in `/path/to/library`: ```bash export LDFLAGS="-lfoobar -L/path/to/library -Wl,-rpath,/path/to/library" ``` In CMake, you can control the `rpath` using the `CMAKE_INSTALL_RPATH` variable. ##### Setting the `LD_LIBRARY_PATH` Alternatively, you can set the `LD_LIBRARY_PATH` environment variable at runtime: ```bash export LD_LIBRARY_PATH=/path/to/library:$LD_LIBRARY_PATH ``` However, we do not recommend this approach, as it can create dependencies on a specific environment and may cause your application to fail if the variable is not set correctly. #### Why do I get C++ standard library errors with the Intel compiler? The Intel C++ compiler supports modern C++ standards but relies on the system's C++ standard library (`libstdc++`), which may not be up-to-date. To resolve this, load a recent GCC module (e.g., `module load gcc/13`) *after* loading the Intel compiler and any other library modules. This will provide a modern `libstdc++` while ensuring that other libraries are linked against the Intel compiler. For more information, see our [compilers documentation](../doc/computing/software/compilers_languages). ### Debugging C/C++ and Fortran Codes #### How can I use the Address Sanitizer (ASAN) with CUDA? [ASAN](https://github.com/google/sanitizers/wiki/AddressSanitizer) is a memory error detector for C/C++ codes, available in the GCC and Clang compilers (via the `-fsanitize=address` flag). When using ASAN with CUDA code, you may encounter a `cudaErrorNoDevice` error due to an incompatibility with the NVIDIA driver. To work around this, set the following environment variable before running your application: ```bash export ASAN_OPTIONS="protect_shadow_gap=0" ``` #### How do I debug a GPU memory error on MI300A (Viper-GPU)? To debug a GPU memory error on Viper-GPU, follow these steps: 1. **Enable XNACK:** ```bash export HSA_XNACK=1 ``` 2. **Build your HIP code with debug symbols:** ```bash hipcc -g -ggdb -O0 ... ``` 3. **Launch your application with `rocgdb`:** ```bash rocgdb --args ``` 4. **Configure `rocgdb`:** ``` set pagination off set amdgpu precise-memory on b abort set non-stop on ``` 5. **Set a breakpoint at your kernel.** If the kernel name is mangled, you can demangle it with `c++filt`. ``` b ``` 6. **Run the application and inspect the threads.** ``` r info threads thread set scheduler-locking step ``` This will allow you to step through the kernel code and inspect variables. To turn off scheduler locking, use `set scheduler-locking off`. ## Interpreted Languages ### Python **Update 2024:** Due to license changes, we can no longer provide recent versions of Anaconda Python. For more details, please see our article in [Bits and Bytes issue 216](../bnb/216). We provide Python software stacks for scientific computing that include optimized packages like NumPy, SciPy, and Numba. You can see the available Python modules with `module avail python-waterboa` and `module available anaconda` (legacy). A basic system Python is also available for simple scripting tasks. #### How do I install Python packages? If a package is not available in our default installations, you can install it in your home directory using `pip`. First, load a Python module: ```bash module load python-waterboa ``` Then, install the package with the `--user` flag: ```bash pip install --user ``` The `--user` flag tells `pip` to install the package in your local user directory (`~/.local/`). Without this flag, the installation will fail due to a lack of write permissions in the system directories. For managing multiple projects, we strongly recommend using [virtual environments](../doc/computing/software/data_analytics-machine_learning#how-to-install-additional-python-packages). #### How do I use Conda environments? **Update 2024:** Due to licensing restrictions, you may only use free channels like `conda-forge` and `bioconda`. The `main`, `anaconda`, `r`, and `msys2` channels from `repo.anaconda.com` are not permitted. To enforce this, add the following to your `environment.yml` file: ```yaml channels: - conda-forge - nodefaults ``` If you have a local `.condarc` file, you must update it to include these lines: ```yaml channels: - conda-forge - nodefaults channel_priority: strict custom_channels: main: null r: null anaconda: null msys2: null ``` **Disclaimer:** We do not provide support for user-created Conda environments. If your required packages are available via `pip`, we recommend using a `pip`-based virtual environment instead. For a more robust solution, consider using our open-source tool, [Condainer](../bnb/214#move-conda-environments-into-compressed-image-files), which creates portable, compressed Conda environments. **Important:** Do not use `conda init`. This command modifies your `.bashrc` file and can interfere with our module system. If you have already run `conda init`, you will need to manually clean up your `.bashrc` and `.condarc` files. To use `conda` in a non-intrusive way, use `eval` instead: ```bash module purge module load python-waterboa/2024.06 eval "$(conda shell.bash hook)" conda create -n my_env python=3.11 conda activate my_env ``` Be aware that Conda packages are not optimized for our systems and may have performance or compatibility issues. We recommend using our provided modules whenever possible. #### How can I write parallel Python code? There are several ways to parallelize your Python code on our systems. The best method depends on your specific needs. ##### Implicit Threading (NumPy/SciPy) Many NumPy and SciPy functions are backed by Intel's Math Kernel Library (MKL), which is automatically parallelized. If your code relies heavily on linear algebra operations, it may already be taking advantage of multiple cores. You can control the number of threads used by MKL with the `MKL_NUM_THREADS` environment variable. To test the performance benefits, you can compare a run with `export MKL_NUM_THREADS=1` to a run where the variable is unset (which will default to using all available cores). ##### `multiprocessing` For tasks that are not automatically parallelized, you can use Python's `multiprocessing` package to manually distribute work across multiple processes on a single node. The `Pool` class is a convenient way to apply a function to a sequence of inputs in parallel: ```python from multiprocessing import Pool def f(x): return x * x if __name__ == '__main__': with Pool(10) as p: print(p.map(f, range(100))) ``` This example will distribute the work across 10 processes. Note that `multiprocessing` is limited to a single node. ##### `mpi4py` For distributed-memory parallelism across multiple nodes, you can use `mpi4py`, the Python interface to MPI. To use `mpi4py`, load the `mpi4py` module after loading a Python module: ```bash module load python-waterboa module load mpi4py ``` In your Slurm script, you can launch an `mpi4py` application with `srun`: ```bash srun python my_application.py ``` `srun` will automatically handle the distribution of processes according to the resources you have requested in your Slurm script. For more information, see the [`mpi4py` documentation](https://mpi4py.readthedocs.io/en/stable/). ### R We provide R through the `R` environment module. Our R installations are built from source and linked against Intel MKL for improved performance. #### How do I install R packages? You can install packages from the [CRAN repository](https://cran.r-project.org/web/packages/) directly from the R prompt. To install a package, use the `install.packages()` function: ```R install.packages("my_package") ``` When prompted, choose to install the package to a local directory. This will install the package in your home directory. If you encounter an error that a package is not available for your version of R, try loading a newer R module. ### Julia We provide Julia through the `julia` environment module. #### How do I install Julia packages? You can install packages from the [Julia package registry](https://github.com/JuliaRegistries/General) using the built-in package manager. From the Julia prompt, use the `Pkg` manager to add a package: ```julia using Pkg Pkg.add("my_package") ``` ### Jupyter Notebooks The Jupyter Notebook is an open-source web application for creating and sharing documents with live code, equations, and visualizations. For more information, see the [Jupyter website](https://jupyter.org/). #### How do I launch a Jupyter Notebook on an HPC system? You can launch Jupyter Notebooks on our HPC systems through our remote visualization service at . ### MATLAB We provide recent versions of MATLAB through the `matlab` environment module. You can see the available versions with `module avail matlab`. Starting with version R2024aU2, our MATLAB installations are containerized, but this does not affect how you use the software. #### How do I run the MATLAB GUI? There are several ways to run the MATLAB GUI on our systems. ##### Remote Visualization System (RVS) - Recommended The most efficient way to run the MATLAB GUI is through a "Remote Desktop" session on the [RVS](../doc/visualization/index). This provides access to dedicated resources and longer session times. ##### VNC You can run the MATLAB GUI in a VNC session on a login node. For instructions, see our [VNC documentation](../doc/computing/software/vnc). ##### X11 Forwarding You can also use X11 forwarding, but this is the least efficient method. 1. Connect to a gateway machine with X11 forwarding enabled: ```bash ssh -C -Y YOUR_USERNAME@gate1.mpcdf.mpg.de ``` 2. From the gateway, connect to a login node: ```bash ssh -C -Y raven.mpcdf.mpg.de ``` 3. Load the MATLAB module and launch the GUI: ```bash module load matlab matlab & ``` **Note:** macOS and Windows users will need to install an X server (e.g., XQuartz or Xming). #### How do I run MATLAB code in a batch job? To run a MATLAB script in a batch job, you must run it in non-graphical mode. Here is an example Slurm script for a sequential MATLAB job: ```bash #!/bin/bash -l #SBATCH -J MATLAB #SBATCH -o ./job.out.%j #SBATCH --ntasks=1 #SBATCH --mem=2000MB #SBATCH --time=00:10:00 module purge module load matlab srun matlab -singleCompThread -nodisplay -r "run('my_program.m')" ``` #### How do I run parallel MATLAB code? You can run parallel MATLAB code on up to a full compute node. You must tell MATLAB how many cores to use. Here is an example of a `parfor` loop that uses the number of cores requested from Slurm: ```matlab % my_parallel_program.m ncpus = str2num(getenv('SLURM_CPUS_PER_TASK')); parpool('local', ncpus); n = 200; A = 500; a = zeros(1, n); parfor i = 1:n a(i) = max(abs(eig(rand(A)))); end disp(a(n)); disp('OK!'); exit ``` And the corresponding Slurm script: ```bash #!/bin/bash -l #SBATCH -J MATLAB_parallel #SBATCH -o ./job.out.%j #SBATCH --ntasks=1 #SBATCH --cpus-per-task=8 #SBATCH --mem=16000MB #SBATCH --time=01:00:00 module purge module load matlab srun matlab -nodisplay -r "run('my_parallel_program.m')" ``` Please request only the resources your code can effectively use. For advanced use cases, you may need to use MATLAB's Cluster Profile Manager. ## Message Passing Interface (MPI) ### Which MPI implementations are supported? We support the Intel MPI library and OpenMPI. ### How do I compile and link an MPI application? After loading a compiler and an MPI module, use the appropriate MPI wrapper script to compile your code. These wrappers automatically include the necessary flags for compiling and linking your application. * **Intel compilers:** `mpiicx`, `mpiicpx`, `mpiifx` * **GNU compilers:** `mpicc`, `mpicxx`, `mpifort` To see the underlying compiler command, you can use the `-show` flag with the wrapper script. ### What if CMake cannot find MPI? If CMake has trouble finding your MPI installation (a common issue with Intel MPI), you can explicitly specify the MPI compiler wrappers. **For Intel compilers:** ```bash module load intel/... module load impi/2021.x ... cmake -DMPI_C_COMPILER=mpiicx \ -DMPI_CXX_COMPILER=mpiicpx \ -DMPI_Fortran_COMPILER=mpiifx \ ... ``` **For GNU compilers:** ```bash module load gcc/... module load impi/2021.x ... cmake -DMPI_C_COMPILER=mpicc \ -DMPI_CXX_COMPILER=mpicxx \ -DMPI_Fortran_COMPILER=mpifort \ ... ``` ### Why can't I use `mpirun` to launch my MPI code? On our Slurm-based clusters, you must use `srun` to launch MPI applications. For production jobs, you should always submit a batch script. For small, interactive tests, you can use `srun` on a login node: ```bash srun --time=00:05:00 --mem=1G --ntasks=2 ./my_mpi_application ``` ## Visualization ### How do I create a movie from a sequence of images? You can use `ffmpeg` to create a movie from a sequence of images (e.g., `input_0001.png`, `input_0002.png`, etc.). First, load the `ffmpeg` module: ```bash module load ffmpeg ``` Then, run `ffmpeg` with your input files and desired options. This example creates a 30fps MP4 video: ```bash ffmpeg -start_number 1 -i input_%04d.png -c:v libx264 -vf "fps=30,format=yuv420p" output.mp4 ``` For more information on the available options, see the [`ffmpeg` documentation](https://ffmpeg.org/ffmpeg.html). ### How do I install additional TeX/LaTeX packages? We provide comprehensive LaTeX environments through the `texlive` module. If you need a package that is not included, you can install it locally using the TeX Live Manager (`tlmgr`). 1. Load the `texlive` module: ```bash module load texlive/2021 ``` 2. Initialize a local user tree (you only need to do this once): ```bash tlmgr init-usertree ``` 3. Set the repository to match your `texlive` version: ```bash tlmgr --usermode option repository https://ftp.tu-chemnitz.de/pub/tug/historic/systems/texlive/2021/tlnet-final/ ``` 4. Install the package: ```bash tlmgr --usermode install ``` Be sure to use a repository that matches the version of `texlive` you have loaded. ## GUI Applications ### Why does VSCode fail to connect to an older Linux system? Since version 1.86, the [VSCode server requires `glibc` 2.28 or newer](https://code.visualstudio.com/docs/remote/linux) on the remote host. All current MPCDF HPC systems meet this requirement. If you need to connect to an older Linux system, you can use [VSCode version 1.85.2](https://code.visualstudio.com/updates/v1_85) and disable automatic updates. ### Why are some GUI applications not working on the login nodes? Some GUI applications (e.g., Firefox, Spyder, VSCode) use sandboxing features that are not compatible with our security policies. As a workaround, you can disable sandboxing by loading the `nosandbox` module before launching the application: ```bash module load nosandbox/1.0 spyder & ``` Alternatively, you can set the appropriate environment variables yourself: * **For Qt-based applications (e.g., Spyder):** ```bash export QTWEBENGINE_DISABLE_SANDBOX=1 ``` * **For Electron-based applications (e.g., VSCode):** Use the `--no-sandbox` flag. * **For Firefox:** ```bash export MOZ_DISABLE_CONTENT_SANDBOX=1 export MOZ_DISABLE_GMP_SANDBOX=1 ... ``` For better performance, we recommend running heavyweight GUI applications on your local machine and accessing remote files via `sshfs` or VSCode's Remote-SSH extension. ## AI Coding Agents Command-line AI coding agents such as Claude Code, OpenAI Codex, GitHub Copilot CLI, and Google Gemini CLI can read and modify source files, run commands, and execute tests largely on their own. They can be useful for developing scientific software, refactoring code, or exploring an unfamiliar codebase. Because such an agent runs with your full user permissions, MPCDF provides a containerized environment that confines it to the project you are working on. ### How can I run AI coding agents responsibly on MPCDF systems? By default, an AI agent inherits all your permissions: it can read and write any file you can, run shell commands, and send file contents to external cloud services. A misunderstood instruction or a leaked credential can therefore do real damage, so running an agent with unrestricted access to your account is strongly discouraged. To reduce this risk, MPCDF provides an [Apptainer](https://apptainer.org/)-based container that makes only two things visible to the agent: your **current working directory** (the project) and a dedicated **fake home directory** that holds the agents' own configuration and credentials. The rest of your `$HOME` and all other data remain invisible, and launching directly from `$HOME` is blocked. On the SLES- and RHEL-based HPC systems (e.g. Raven and Viper) the MPCDF software tree is bind-mounted read-only, so `module load` works exactly as on the login node. Set it up once (make sure `~/bin` is in your `PATH`): ```bash git clone https://gitlab.mpcdf.mpg.de/mpcdf/ai-cli-agents-container.git cd ai-cli-agents-container ./build-containers.sh # builds the container image (auto-detects the OS flavour) ./install-agents-launcher.sh # creates the ~/bin/agents launcher ``` Then, from within your project directory: ```bash cd /path/to/your/project agents ``` This drops you into a shell inside the container, indicated by an `[Agents]` prompt, from which you can start a pre-installed agent such as `claude`, `codex`, `gemini`, or `copilot`. Exit the shell to leave the sandbox. For full details, see the `README.md` in the [AI agents container repository](https://gitlab.mpcdf.mpg.de/mpcdf/ai-cli-agents-container). The container reduces risk considerably but is **not** an impenetrable security boundary. You remain fully responsible for the agents you run and for assessing the associated security and compliance risks. ### What should I keep in mind when using AI coding agents? * **Always start in your project directory, never in `$HOME`.** The isolation is only as good as the directory you open. * **Mind confidentiality and data protection.** Prompts and file contents are sent to external cloud services operated by the respective providers. Do not expose personal data, sensitive research data, or credentials, and check that using a given service is compatible with the terms under which your data was obtained. * **Review everything the agent produces.** AI agents make mistakes and can "hallucinate" plausible-looking but incorrect code. Read the diffs, run your tests, and keep your work under version control so that unwanted changes can be reverted. * **Use the batch system from outside the container.** Slurm client tools are not available inside the container. Submit and manage jobs (`sbatch`, `srun`, `squeue`) from a regular login shell; the agent can then analyse the resulting output files. * **Keep an eye on cost and quotas.** Agents can issue many API calls in a short time. Be aware of the usage limits and billing associated with your chosen provider and account. # HPC Systems and Services ## Raven ### What are the recommended compiler flags for Raven? Raven is based on Intel Xeon "Ice Lake" SP processors and NVIDIA A100 GPUs. Here are the recommended compiler flags for optimal performance. **For the CPU (Intel Ice Lake):** * **Intel:** `-O3 -xICELAKE-SERVER -qopt-zmm-usage=high` * **GNU:** `-O3 -march=icelake-server` * **NVIDIA HPC SDK:** `-O3 -tp=skylake` **For the GPU (NVIDIA A100):** * **NVIDIA CUDA (nvcc):** `-O3 -arch=sm_80` * **NVIDIA HPC SDK (OpenACC):** `-O3 -tp=skylake -acc=gpu -gpu=cc80` For more information, please consult the compiler documentation or the [Raven User Guide](../doc/computing/raven-user-guide). ## Viper ### What are the recommended compiler flags for Viper? The Viper HPC system has two variants: Viper-CPU and Viper-GPU. **Viper-CPU (AMD EPYC "Genoa"):** * **Intel:** `-O3 -march=znver4` (for AVX512) or `-O3 -march=core-avx2` (for AVX2) * **GNU:** `-O3 -march=znver4` (for AVX512) or `-O3 -march=znver4 -mprefer-vector-width=256` (for AVX2) Whether AVX512 provides a performance benefit depends on your application. For more information, see the [Viper User Guide](../doc/computing/viper-user-guide). **Viper-GPU (AMD Instinct MI300A):** The CPU cores on Viper-GPU are the same as on Viper-CPU. To target the GPU, use the following flags: * **HIP:** `hipcc --offload-arch=gfx942` * **OpenMP:** `amdclang++ -O3 -fopenmp --offload-arch=gfx942` For more information, see the [Viper-GPU User Guide](../doc/computing/viper-gpu-user-guide). ## Slurm Batch System ### How do I submit a job to Slurm? To submit a job, you first need to create a submission script (e.g., `my_job.sh`) that specifies the resources your job requires and the commands to be executed. You can find example scripts in the documentation for each HPC system. Once you have a script, submit it with `sbatch`: ```bash sbatch my_job.sh ``` `sbatch` will return a job ID that you can use to track your job. ### Can I submit jobs that run longer than 24 hours? No, jobs on our HPC systems are limited to a 24-hour runtime. This policy ensures fair access and high system utilization. If your application needs to run for longer, it must support checkpointing. This allows your application to save its state and be restarted in a subsequent job. ### How do I launch an MPI application? To launch an MPI application, use `srun` in your Slurm submission script: ```bash srun my_application ``` `srun` will automatically distribute the processes according to the resources you have requested. ### What is the correct order of commands in a Slurm script? All `#SBATCH` directives must come before any executable commands in your script. Any `#SBATCH` directives that appear after the first command will be ignored. ```bash #!/bin/bash -l # SBATCH directives #SBATCH ... #SBATCH ... # Your commands module load ... srun ./my_executable ... ``` ### Can I run an interactive job for debugging? Yes, you can run short, interactive jobs on the login nodes for debugging and development. For example, to request an interactive session with 2 tasks for 5 minutes, you can use: ```bash srun --time=00:05:00 --mem=1G --ntasks=2 --pty /bin/bash ``` Once the resources are allocated, you will get a shell on a compute node where you can run your commands. ### How can I check the estimated start time of my job? Use `squeue` with the `--start` flag: ```bash squeue --start -j ``` ### How do I get detailed information about a running job? Use `scontrol show job`: ```bash scontrol show job -dd ``` ### How do I get information about a finished job? Use the `sacct` command to view information about your past jobs. To see information about a specific job: ```bash sacct -j ``` To see information about all of your recent jobs with custom formatting: ```bash sacct -u $USER --format=JobID,JobName,MaxRSS,Elapsed ``` ### What happens if a hardware failure occurs during my job? In the rare event of a hardware failure, Slurm will interrupt your job and you will see an error message like `srun: error: Node failure on...`. By default, your job will be automatically resubmitted to the queue and will run on a different set of nodes. If you do not want your job to be automatically requeued, you can use the `--no-requeue` flag with `sbatch`. ### How do I handle CPU pinning? CPU pinning is handled automatically by Slurm. To ensure correct pinning, please use `srun` to launch your application and refer to our example job scripts for MPI, OpenMP, and hybrid jobs. ## Parallel File Systems (GPFS) ### Which file systems are available and how should I use them? Each of our HPC systems has two main file systems: * **/u/$USER (home directory):** This file system is intended for source code, software installations, and smaller data files. **Do not run I/O-intensive applications from your home directory.** * **/ptmp/$USER (temporary storage):** This file system is optimized for large, streaming I/O, such as checkpoints and simulation output. **I/O-intensive applications must use this file system.** Please note that files on `/ptmp` are subject to a 12-week cleanup policy. For more details on backups, quotas, and cleanup policies, please see the documentation for the specific HPC system you are using. Remember that the file systems are a shared resource; improper use can affect all users. ### How can I improve my I/O performance? For best performance, use large, sequential I/O operations. Avoid random access patterns and creating a large number of small files, especially in a single directory. As the file systems are a shared resource, performance will vary depending on the overall system load. ### How do I share files with other users? You can use Access Control Lists (ACLs) to share files and directories with other users, even if they are not in your group. The primary tools for this are `getfacl` and `setfacl`. #### Granting Read Access To grant read access to a directory to user `jane`: ```bash setfacl -R -m user:jane:rx /u/my/directory setfacl -m user:jane:rx /u/my ``` The `-R` flag recursively applies the permissions. The `x` (execute) permission is required to traverse directories. You can also grant the access to a whole group of users. For example, similar to the example above if you want to share your files with the MPCDF support team, you can share your folder with the 'rzg' group as: ```bash setfacl -R -m group:rzg:rx /u/my/directory setfacl -m group:rzg:rx /u/my ``` #### Revoking Access To revoke access: ```bash setfacl -R -x user:jane /u/my/directory ``` #### Viewing ACLs To view the ACLs for a directory: ```bash getfacl /u/my/directory ``` #### Removing All ACLs To remove all ACLs from a directory and revert to standard Unix permissions: ```bash setfacl -b /u/my/directory ``` ## How do I transfer files to and from the HPC systems? There are several ways to transfer files: * **MPCDF DataShare:** Use the `ds` command-line client (`ds put`, `ds get`) for small to medium-sized files. * **`scp` and `rsync`:** Standard tools for transferring files and directories. `rsync` can resume interrupted transfers. * **`curl` and `wget`:** For downloading files from the web. * **`bbcp`:** For high-performance, parallel data transfers. * **Globus:** For large-scale, reliable data transfers. ## Performance Monitoring ### How can I check the performance of my jobs? We monitor all jobs and provide performance data as downloadable PDF reports at . These reports can help you identify performance issues that may require further investigation with a profiler. ### How do I disable the performance monitoring for my job? Our performance monitoring system can sometimes interfere with other profiling tools like VTUNE or `likwid`. To temporarily suspend it for a single job, use the `hpcmd_suspend` wrapper: ```bash srun hpcmd_suspend ./my_executable ``` The monitoring will be automatically re-enabled when your job finishes. Please do not suspend the monitoring unless you are performing your own measurements. ## GPU Computing ### How do I use the NVIDIA Multi-Process Service (MPS)? [NVIDIA MPS](https://docs.nvidia.com/deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf) allows multiple MPI processes to share a single GPU. This can be useful if you are running more MPI ranks on a node than there are GPUs. To enable MPS for your job on Raven, add the `--nvmps` flag to your `sbatch` command. ### How do I profile my GPU code? We provide the NVIDIA Nsight tools for GPU profiling. * **Nsight Systems:** Use `nsys` to generate a timeline of your application and identify bottlenecks. ```bash module load cuda/XX.Y nsight_systems/ZZZZ nsys profile -t cuda,cudnn srun ./my_application ``` You can view the resulting `.nsys-rep` file with the `nsys-ui` GUI. * **Nsight Compute:** Use `ncu` to perform an in-depth analysis of specific kernels. ### Are there dedicated resources for interactive GPU development? Yes, the `gpudev` partition on Raven is available for short, interactive GPU jobs. To use it, add the following to your Slurm script: ```bash #SBATCH --partition=gpudev #SBATCH --gres=gpu:a100:1 ``` The time limit for this partition is 15 minutes. You can also request an interactive session: ```bash srun --time=00:10:00 --partition=gpudev --gres=gpu:a100:1 --pty /bin/bash ``` ## Containers ### Can I run Docker containers? For security reasons, Docker is not directly supported on our HPC systems. However, you can convert Docker containers to Singularity or Charliecloud, which are supported. For more information, see our [container documentation](../doc/computing/software/containers). ## Remote Visualization ### How do I run GUI applications that use OpenGL? Applications that use OpenGL for hardware-accelerated rendering (e.g., VisIt, ParaView) must be run through our remote visualization service. After launching a remote visualization session, open a terminal and prefix your command with `vglrun`: ```bash module load visit vglrun visit ``` ### How do I access the remote visualization service? You can access the remote visualization service through your web browser at . No special client software is required. # Tips and Tricks ## How do I change my default shell? To change your default shell, please submit a request to the MPCDF helpdesk. We support `bash`, `tcsh`, and `csh`. We recommend using `bash`. Other shells, such as `zsh` or `ksh`, are available on our systems but must be invoked manually. ## How do I use `ssh-agent` to avoid typing my SSH key password? You can use `ssh-agent` to unlock your SSH key once per terminal session. This is particularly useful when working with Git repositories. To start the agent and add your key, run the following commands: ```bash eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_rsa ``` You will not be prompted for your key's password for the remainder of the session. # Getting Help, Support, and Training ## How can I get help and support? Before contacting us, check the MPCDF documentation and this FAQ to see if your question has already been answered. If you cannot find an answer, send an email to or open a ticket on our [helpdesk portal](https://helpdesk.mpcdf.mpg.de). To help us resolve your issue quickly, include the following information in your request: | Information | Description | |-------------|-------------| | **Subject line** | A descriptive subject line | | **Email address** | Your work email address | | **Problem description** | A detailed description, including steps to reproduce it | | **Technical details** | Cluster/machine name, `module list` output, input files, job submission scripts | | **Job information** | Job ID of any failing jobs, complete error messages | | **Web services** | Browser name and version for web-based issues | | **Login issues** | A copy of your terminal session, including commands and full output | **Tip:** Emails signed with a trusted S/MIME certificate help us verify your identity more efficiently. Modern operating systems typically include trusted Certificate Authority (CA) certificates in their truststore. As of 2026, European science and research organisations use the [GÉANT TCS service](https://security.geant.org/trusted-certificate-services/), offered in partnership with [HARICA](https://www.harica.gr/). ## Do you offer training on how to use MPCDF resources? Yes, we offer various training opportunities: | Training Type | Description | More Information | |---------------|-------------|------------------| | **Introductory Tutorials** | Online tutorials on using our services (twice a year) | [Introduction to MPCDF Services](https://www.mpcdf.mpg.de/training/introductiontompcdfservices) | | **Advanced Courses** | Courses on HPC, Python, and other topics | [Training web page](https://www.mpcdf.mpg.de/services/training) | Custom courses can also be arranged upon request. ## How should I acknowledge MPCDF in my publications? We appreciate acknowledgements of our services and resources in your publications. While there is no mandatory template, here are some examples: * "Computations were performed on the HPC system Raven at the Max Planck Computing and Data Facility." * "We acknowledge project support from the Max Planck Computing and Data Facility." If you have received substantial support from individuals at MPCDF, you may also wish to acknowledge them by name. Documentation ------------- .. toctree:: :maxdepth: 2 :glob: computing/index.rst.txt data/index.rst.txt cloud/index.rst.txt visualization/index.md.txt campus/index.rst.txt * Computing ========= This chapter provides comprehensive documentation on how to access and use HPC and cluster compute resources at the MPCDF. .. toctree:: :maxdepth: 1 overview.md.txt application-support.md.txt gateways.md.txt dais-user-guide.md.txt viper-user-guide.md.txt viper-gpu-user-guide.md.txt raven-user-guide.md.txt clusters/index.rst.txt software/index.rst.txt hpc-get-started.md.txt performance-monitoring.md.txt training.md.txt # Introduction ## Overview The MPCDF provides facilities for high-performance computing (HPC), capacity computing (Linux clusters) and remote visualization and supports general-purpose and dedicated systems. The operating systems used at the MPCDF are Linux-based. Users login to one of the [gateway machines](gateways) via `ssh`. From the gateway machines the login nodes of all compute facilities can be accessed. Note that two-factor authentication (2FA) is enforced which is [documented as part of the FAQ](../../faq/2fa.md). Personal settings (e.g. passwords, login shells) can be customized at [userspace/mympcdf](https://www.mpcdf.mpg.de/userspace/mympcdf) (login required). Information on the available software can be found [here](software/index). General support requests going beyond the information provided by the present documentation and [FAQ](../../faq/index) can be addressed to the [MPCDF helpdesk](../../faq/help.html#how-can-i-get-help-and-support). [Dedicated, high-level application support](application-support), e.g. for HPC code development and optimization, and machine-learning projects is offered to all Max-Planck researchers and their collaboration partners. ## Compute Facilities ### Gateway Systems - [Gateway machines](gateways) for accessing compute and data resources at the MPCDF ### High-Performance Computing - [MPG Supercomputer *viper* (since June 2024)](viper-user-guide) based on AMD EPYC Genoa processors - [MPG Supercomputer *viper-gpu* (since February 2025)](viper-gpu-user-guide) based on AMD MI300A APUs - [MPG Supercomputer *raven* (since September 2020)](raven-user-guide) based on Intel IceLake processors and Nvidia GPUs (A100) ### Linux Compute Clusters - [Dedicated Linux Compute Clusters](clusters/index) hosted for several Max Planck Institutes ### Interactive Data Analysis and Visualization - [Infrastructure and services for interactive remote data analysis and visualization](../visualization/index) - Jupyter notebooks, Remote visualization, Remote desktops enable convenient web-based access to the HPC systems and selected Linux Compute Clusters # Application Support for HPC, AI and HPDA The MPCDF provides consulting and dedicated high-level support for the development, optimization, analysis and visualization of high-performance-computing (HPC), high-performance data-analytics (HPDA) and artificial intelligence (AI) applications. This comprises the development and optimization of codes in collaboration with Max Planck scientists, scientific visualization of data from simulations and experiments, and technical consulting, e.g. on programming techniques and models such as the use of hardware accelerators (e.g. GPUs). A number of ongoing code-development and optimization projects and long-term collaborations with Max-Planck scientists can be found at the [HPC applications division pages](https://www.mpcdf.mpg.de/services/application-support) and at the [AI & HPDA division pages](https://www.mpcdf.mpg.de/services/data-analytics) Concerning requests for dedicated project support please contact [M. Rampp](mailto:markus.rampp@mpcdf.mpg.de?subject=HPC%20project%20support)(HPC), [K. Reuter](mailto:klaus.reuter@mpcdf.mpg.de?subject=HPC%20project%20support)(HPC) or [A. Marek](mailto:andreas.marek@mpcdf.mpg.de?subject=AI-HPDA%20project%20support)(AI&HPDA) In case of issues with HPC applications or systems please check the [FAQ](../../faq/index.md) first for potential solutions that are already documented. As a second step inquiries may be directed to the [MPCDF helpdesk](../../faq/help.html#how-can-i-get-help-and-support). # Gateway machines ## Login The gateway machines `gate1.mpcdf.mpg.de` and `gate2.mpcdf.mpg.de` provide ssh access to MPCDF computing resources. One should note that the home directory $HOME is local to each of those machines and very limited in size (`quota -vs` will tell your current usage). SHA256-based key exchange methods are supported exclusively; a more recent version of your favourite ssh/sftp client software might be required in case connection attempts fail. Note that all MPCDF gateway machines enforce 2 factor authentication (2FA). The ssh key fingerprints (SHA256) are: ```text gate1.mpcdf.mpg.de ( gate.mpcdf.mpg.de ) SHA256:mkNYGVYvBOMAUSZ+KBOcEKRY1kFYy336cAH0MKelwLA (ED25519) SHA256:7o7g1YLQmmcKRvZaHllxI2e5RvpD1g5akhLir4E2Vjc (RSA) gate2.mpcdf.mpg.de SHA256:VSjalFu2TI5LGonWDTSzSAz2ie9DFsXoLdbXNk3FoZY (ED25519) SHA256:1qiMie086Rc65y+rM7934Ml4suXn+HQk9N8Hru7+/+0 (RSA) ``` `gate1` will be rebooted each Tuesday, 3:45 am, and `gate2` each Saturday, 3:45 am, German local time; user sessions will thus persist no longer than 7 days on either gateway system. Please note further that both `gate1` and `gate2` support password and GSSAPI authentication methods only, an additional 2nd authentication method based on our OTP infrastructure is mandatory. If you intend to forward your Kerberos5 ticket from remote via GSSAPI, please ensure to pass 'GSSAPIDelegateCredentials=yes' to ssh. These gateway machines are for login only, not for compiling or running applications; the module environment is also not supported. Compilers and batch systems are available on the Linux clusters and on the HPC system. If necessary, please apply for an account on these systems via the [MPCDF helpdesk](../../faq/help.html#how-can-i-get-help-and-support). ## GSSAPI-based logins to MPCDF hosts If you want to login directly to an internal machine, here named 'TARGET' as user 'MPCDF-USERNAME', you can put following snippet into your '~/.ssh/config' file: ```text Host User ProxyCommand ssh -W %h:%p gate1.mpcdf.mpg.de 2>/dev/null GSSAPIAuthentication yes GSSAPIDelegateCredentials yes Host gate1.mpcdf.mpg.de User GSSAPIAuthentication yes GSSAPIDelegateCredentials yes ControlMaster auto ControlPath ~/.ssh/control:%h:%p:%r ``` This supports GSSAPI, so with a Kerberos5 ticket on your machine, you can login to TARGET without typing the password again. ## Tunneled access to MPCDF services Many MPCDF services and clusters are only available to internal MPG networks and are not visible from external institutes and/or a user's home network. To overcome this restriction ssh tunneling can be used to simplify access to these internal services. For example accessing the archive service from an external node can be achieved by creating a tunnel as follows ```bash ssh @gate1.mpcdf.mpg.de -L 2002:archive.mpcdf.mpg.de:22 -N ``` Once this tunnel has been established, SFTP/SCP can be used to access the archive as if it were on your local system (in this case point your sftp client to port 2002 on localhost). This means that you can use file transfer tools such as FileZilla by just setting up the tunnel and configuring the FileZilla remote SFTP/SCP connection to use localhost and port 2002. When using FileZilla the 2FA may cause some problems (login requests can occur on each file transfer). To overcome this, change the Login Type to interactive and set the Max number of connections to 1 in the Site Manager configuration. Note that for Windows systems WinSCP is also capable of using the gate node as a proxy. Simply configure WinSCP to use an ssh tunnel in the Advanced Options section using gate.mpcdf.mpg.de as the hostname and your usual MPCDF user name and password. To simplify direct access from Linux-based systems the ssh ProxyJump option can be used. To access the archive (or any cluster login node) ```bash sftp -o 'ProxyJump @gate1.mpcdf.mpg.de' @archive.mpcdf.mpg.de: ``` or alternatively: ```bash sftp -J @gate1.mpcdf.mpg.de @archive.mpcdf.mpg.de: ``` Note: This will also work for ssh connections and rsync via ssh ```bash ssh -J @gate1.mpcdf.mpg.de @raven.mpcdf.mpg.de rsync -av -e 'ssh -J @gate1.mpcdf.mpg.de' source-dir @archive.mpcdf.mpg.de: ``` # Dais User Guide --- ```{eval-rst} .. note:: The filesystem details, including the quota values, are not yet settled and are thus subject to change. ``` ```{contents} Contents :local: :depth: 2 ``` **Name of the cluster:** - **DAIS** **Institution:** - **Selected MPG Departments** ## How to get Access permissions Access can only be granted to members of institutes who procured the system. If you do not already have an account at MPCDF fill out the [registration form](https://selfservice.mpcdf.mpg.de/index.php?r=registration). If you do already have an account at MPCDF but you cannot access DAIS, please request access via our ticket system. Note that access to DAIS can only be granted after successfully passing the export control which is done by the respective export control officer of your institute. The export control officer will be automatically notified as soon as you request an account for DAIS. ## Access ### Login For security reasons, direct login to the HPC system DAIS is allowed only from within some MPG networks. Users from other locations have to log in to one of our [gateway systems](gateways) first. ### Login nodes - **dais11.mpcdf.mpg.de**, **dais12.mpcdf.mpg.de** Dais' ssh key fingerprints (SHA256) are: ```text ijGSRMd1K3bq14gUaKnI0rODsx5hgCVvtAzQoHC/sy0 (RSA) Ke44kG2tm/IRqYFg9iUGapSFCLQKIiSUERez5eSsT9Y (ED25519) ``` ## Hardware Configuration **2 login nodes dais[11-12]**: - 2 x INTEL(R) XEON(R) PLATINUM 8568Y+ 48-Core Processor @ 2.3 GHz - 96 cores per node - hyper-threading enabled - 2 threads per core - 500 GB RAM **17 execution nodes daisg[101-117]**: - 2 x INTEL(R) XEON(R) PLATINUM 8568Y+ 48-Core Processor @ 2.3 GHz - 96 cores per node - hyper-threading enabled - 2 threads per core - 2.0 TB RAM - 8 x NVIDIA H200 GPUs (with 141GB HBM each) per node **16 execution nodes daisg[201-216]**: - 2 x INTEL(R) XEON(R) PLATINUM 8568Y+ 48-Core Processor @ 2.3 GHz - 96 cores per node - hyper-threading enabled - 2 threads per core - 2.0 TB RAM - 8 x NVIDIA B200 GPUs (with 180GB HBM each) per node **2 execution nodes daisg[301-302]**: - 2 x AMD EPYC 9555 64-Core Processor @ 3.2 GHz - 128 cores per node - hyper-threading enabled - 2 threads per core - 1.5 TB RAM - 4 x NVIDIA RTX PRO 6000 GPUs (with 96GB HBM each) per node **Node interconnect**: - based on Mellanox Technologies InfiniBand fabric (Speed: 8*200Gb per GPU Node) ## Filesystems #### Filesystem `/u` - shared home filesystem - quoted to 500k of files and 1TB of data #### Filesystem `/dais/fs/scratch` - shared scratch filesystem, 200TB - quoted to 8M of files and 8TB of data - NO BACKUPS #### Filesystem `/viper/ptmp2` The file system /viper/ptmp2 is designed for batch job I/O (12 PB, no system backups). Files in /viper/ptmp2 that have not been accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary (with prior notice). As a current policy, no quotas are applied on /viper/ptmp2. This gives users the freedom to manage their data according to their actual needs without administrative overhead. This liberal policy presumes a fair usage of the common file space. So, please do a regular housekeeping of your data and archive/remove files that are not currently in use. #### Filesystem `/nexus/posix0` Additional storage space can be rented - see also [Nexus Posix](https://docs.mpcdf.mpg.de/doc/cloud/technical/storage.html#nexusposix) ## Compilers and Libraries [Hierarchical environment modules](./software/environment-modules.html#hierarchical-environment-modules) are used at MPCDF to provide software packages and enable switching between different software versions. Users have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **`module avail`** command; for some, users first need to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **`find-module`** command. ## Batch system based on Slurm The batch system on DAIS is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the [Raven home page](./raven-user-guide). For more detailed information, see the [Slurm handbook](https://slurm.schedmd.com). For example batch scripts see below. ## Current Slurm configuration on DAIS - default turnaround time: 2 hours - current max. turnaround time (wallclock): 24 hours - gpu partition: exclusive usage of compute nodes (with 8 GPUs each); default - gpu1 partition: shared usage of compute nodes; for jobs with less than 4 GPUs - gpudev partition: to debug codes; shared usage of compute nodes; one job per user; max 15 min, 4 GPUs in total - small partition: cpu only jobs; shared usage of daisg[301-302] nodes; max 6 hours; access to half of the node cores and memory ## Useful tips By default run time limit used for jobs that don't specify a value is 2 hours. Use **`--time`** option for sbatch/srun to set a limit on the total run time of the job allocation, but not longer than 24 hours. Default memory per node in the shared partition is 250000 MB, maximum per allocated node per job is 2000000 MB. To grant the job access to all of the memory on each node use **`--mem=0`** option for sbatch/srun. On nodes with RTX PRO 6000 gpus default memory per job is 375000 MB and nodes cannot be used exclusively. The OpenMP codes require a variable **OMP_NUM_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM_CPUS_PER_TASK** which is set when **`--cpus-per-task`** is specified in a sbatch script (an example is on [help information](./clusters/aux/migration-from-sge-to-slurm) page) To use GPUs add in your slurm scripts **`--gres`** option and choose how many GPUs and/or which model of them to have: **`#SBATCH --gres=gpu:GPUTYPE:X`**, where **`X`** is a number of resources from 1 up to 8, and **`GPUTYPE`** is one of `h200`, `b200`, `rtx_pro_6000`. If no gpu type is specified, the job will be scheduled to arbitrary nodes. Only one `rtx_pro_6000` gpu is allowed per job. GPU cards are in default compute mode. ## Slurm example batch scripts The following sections present some general examples of submission scripts for the DAIS system. For more examples with specific frameworks and containerized setups, or comparisons with other HPCS systems, refer to the [ai_containers](https://gitlab.mpcdf.mpg.de/dataanalytics-public/ai_containers) repository. ### Single-GPU job on a shared node The following example launches a Python program on one GPU (NVIDIA H200, B200, or RTX PRO 6000) on a shared node. ```bash #!/bin/bash -l # # Initial working directory: #SBATCH -D ./ # # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # # Job name #SBATCH -J test_1gpu # # Time limit #SBATCH --time=0-00:10:00 # wall-clock D-HH:MM:SS (here: 10 minutes) # #SBATCH --nodes=1 # request 1 node. #SBATCH --partition="gpu1" # request a shared node. #SBATCH --ntasks-per-node=1 # request 1 task on that node. # # --- default case: use a single H200 on a shared node --- #SBATCH --gres=gpu:h200:1 # use 1 H200. #SBATCH --cpus-per-task=12 # request 1/8 of available CPUs on a H200 node. #SBATCH --mem=250000 # grant the job access to 1/8 of the memory on a H200 node. # # --- uncomment to use a single B200 on a shared node --- # #SBATCH --gres=gpu:b200:1 # use 1 B200 # #SBATCH --cpus-per-task=12 # request 1/8 of available CPUs on a B200 node. # #SBATCH --mem=250000 # grant the job access to 1/8 of the memory on a B200 node # # --- uncomment to use a single RTX PRO 6000 on a shared node --- # #SBATCH --gres=gpu:rtx_pro_6000:1 # use 1 RTX PRO 6000 # #SBATCH --cpus-per-task=32 # request 1/4 of available CPUs on a RTX node. # #SBATCH --mem=375000 # grant the job access to 1/4 of the memory on a RTX node. # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de ###### Environment ###### module purge module load apptainer/1.4.3 CONTAINER="YOUR_CONTAINER" ###### Run the program: srun apptainer exec --nv $CONTAINER python3 ./your_python_executable ``` ### Multi-GPU job on a shared node The script below launches a distributed Python program in a single-node, multi-GPUs setting. The example illustrates a distributed PyTorch workflow. Note that different frameworks might require different SLURM settings (e.g. 1 task per node instead of 1 task per GPU). For additional examples refer to the [ai_containers](https://gitlab.mpcdf.mpg.de/dataanalytics-public/ai_containers) repository. ```bash #!/bin/bash -l # # Initial working directory: #SBATCH -D ./ # # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # # Job name #SBATCH -J test_gpu # # Time limit #SBATCH --time=0-00:10:00 # wall-clock D-HH:MM:SS (here: 10 minutes) # # Number of nodes, GPUs and MPI tasks per node: #SBATCH --nodes=1 # request 1 node. #SBATCH --partition="gpu1" # request a shared node. # # --- use 2 GPUs on a shared node --- #SBATCH --gres=gpu:h200:2 # use 2 GPU on a shared node. #SBATCH --ntasks-per-node=2 # request 2 tasks on that node (1 per gpu). #SBATCH --cpus-per-task=12 # request 1/8 of available CPUs on the node *per task*. #SBATCH --mem=500000 # grant the job access to 2/8 of the memory on the node. # # --- uncomment to use 3 GPUs on a shared node --- # #SBATCH --gres=gpu:h200:3 # use 3 GPU on a shared node. # #SBATCH --ntasks-per-node=3 # request 3 tasks on that node (1 per gpu). # #SBATCH --cpus-per-task=12 # request 1/8 of available CPUs on the node *per task*. # #SBATCH --mem=750000 # grant the job access to 3/8 of the memory on the node. # # --- uncomment to use 4 GPUs on a shared node --- # #SBATCH --gres=gpu:h200:4 # use 4 GPU on a shared node. # #SBATCH --ntasks-per-node=4 # request 4 tasks on that node (1 per gpu). # #SBATCH --cpus-per-task=12 # request 1/8 of available CPUs on the node *per task*. # #SBATCH --mem=1000000 # grant the job access to 4/8 of the memory on the node. # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de ###### Environment ###### module purge module load apptainer/1.4.3 CONTAINER="YOUR_PYTORCH_CONTAINER" ###### PyTorch distributed variables ###### export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1) export APPTAINERENV_MASTER_PORT=$(expr 10000 + $(echo -n $SLURM_JOBID | tail -c 4)) export WORLD_SIZE=$(($SLURM_NNODES * $SLURM_NTASKS_PER_NODE)) ###### Run the program: srun apptainer exec --nv $CONTAINER \ bash -c "RANK=\${SLURM_PROCID} python3 ./your_python_executable" ``` **B200:** B200 and H200 nodes share the same CPU layout. So B200 GPUs can be requested by only replacing `h200` by `b200` in the `--gres` line in the above script: ```bash # --- B200 version --- #SBATCH --gres=gpu:b200:x # x is the number of GPUs you need ``` ```{eval-rst} .. note:: - Requesting more than half of a node’s resources (for example, > 4 GPUs) triggers a full‑node allocation, meaning the node is reserved exclusively for the job. - The RTX PRO 6000 nodes do not allow multi-GPU jobs. ``` ### Multi-node job The script below runs a Python program on 16 GPUs spread over two nodes, illustrating a distributed PyTorch workflow (additional examples are available in the [ai_containers](https://gitlab.mpcdf.mpg.de/dataanalytics-public/ai_containers) repository). ```bash #!/bin/bash -l # # Initial working directory: #SBATCH -D ./ # # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # # Job name #SBATCH -J test_gpu # # Time limit #SBATCH --time=0-00:10:00 # wall-clock D-HH:MM:SS (here: 10 minutes) # # Number of nodes, GPUs and MPI tasks per node: #SBATCH --nodes=2 # request 2 or more full nodes #SBATCH --partition="gpu" # request an exclusive node # #SBATCH --gres=gpu:h200:8 # use 8 H200 on each node. # #SBATCH --gres=gpu:b200:8 # use 8 B200 on each node. #SBATCH --ntasks-per-node=8 # request 8 tasks on each node (1 per gpu). #SBATCH --cpus-per-task=12 # request 1/8 of available CPUs per task #SBATCH --mem=0 # grant the job access to all of the memory on each node # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de ###### Environment ###### module purge module load apptainer/1.4.3 CONTAINER="YOUR_PYTORCH_CONTAINER" ###### PyTorch distributed variables ###### export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1) export APPTAINERENV_MASTER_PORT=$(expr 10000 + $(echo -n $SLURM_JOBID | tail -c 4)) export WORLD_SIZE=$(($SLURM_NNODES * $SLURM_NTASKS_PER_NODE)) ###### Run the program: srun apptainer exec --nv $CONTAINER \ bash -c "RANK=\${SLURM_PROCID} python3 ./your_python_executable" ``` ## Support For support please create a trouble ticket at the [MPCDF helpdesk](https://helpdesk.mpcdf.mpg.de/mpcdf/index.html) # Viper-CPU User Guide ```{contents} Contents :local: :depth: 2 ``` ## System overview The HPC system Viper, deployed during 2024/2025, consists of two parts, [Viper-CPU](viper-user-guide.md) and [Viper-GPU](viper-gpu-user-guide.md) which are operated as two logically separate clusters. **Viper-CPU** has been operational since June 2024 and comprises 768 compute nodes with [AMD EPYC Genoa 9554](https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series.html) CPUs with 128 cores and at least 512 GB RAM per node. A subset of 609 nodes is equipped with 512 GB RAM (16 memory channels), 90 nodes with 768 GB RAM (24 memory channels), 66 nodes with 1024 GB RAM (16 memory channels), and 3 nodes with 2304 GB RAM (24 memory channels). There are 6 login nodes, and I/O subsystems that serve approximately 12 PB of disk-based storage with direct HSM access. Summary: 768 CPU compute nodes, 98304 CPU cores, 432 TB RAM (DDR5), 4.9 PFlop/s theoretical peak performance (FP64). ![MPCDF Viper Deployment](_static/viper-cpu-2024.jpg "MPCDF Viper") ## Access ### Login For security reasons, direct login to the HPC system Viper is allowed only from within certain MPG networks. Users from other locations have to log in to one of our [gateway systems](gateways) first. ```bash ssh @gate.mpcdf.mpg.de ``` Use `ssh` to connect to one of the six available Viper login nodes (`viper[01-06]i`), e.g.: ```bash ssh @viper01i.mpcdf.mpg.de ``` To log in, you must provide your (Kerberos) password and an OTP on the Viper login nodes. SSH keys are not allowed. Secure copy (scp) can be used to transfer data to or from `viper[01-06]i.mpcdf.mpg.de`. Viper's (all login/interactive nodes) SSH key fingerprints (SHA256) are: ```text SHA256:0DhkRt6Qom1GA0SmvEjnlKWpIg1+kMPDjOUSqJ8ceyQ (ED25519) SHA256:pM1fS+4YXGuIaLk0bLSt1sOlS1TpLrZYmFMRY9UjBKo (RSA) ``` ### Resource limits The login nodes `viper01i.mpcdf.mpg.de` and `viper02i.mpcdf.mpg.de` are intended only for editing, compiling and submitting parallel programs. Running parallel programs interactively on the login nodes is not allowed. Per user, the CPU resources are restricted to an equivalent of two physical CPU cores, memory usage to at most 10% of the available memory, and the number of tasks (including threads) to a maximum of 768. The login nodes `viper03i.mpcdf.mpg.de`, `viper04i.mpcdf.mpg.de`, `viper05i.mpcdf.mpg.de` and `viper06i.mpcdf.mpg.de` are also primarily intended only for editing, compiling and submitting parallel programs - but here, per user, the CPU resources are restricted to an equivalent of six physical CPU cores, memory usage to at most 20% of the available memory, and the number of tasks (including threads) to a maximum of 1536. Jobs have to be submitted to the Slurm batch system which reserves and allocates the resources (e.g. compute nodes) required for your job. Further information on the batch system is provided [below](#slurm-batch-system). ### Interactive (debug) runs To test or debug your code you may run your code interactively by using the Slurm partition "interactive" (2 hours at most) with the command: ```bash srun -n N_TASKS -p interactive --time=TIME_LESS_THAN_2HOURS --mem=MEMORY_LESS_THAN_32000M ./EXECUTABLE ``` It is not allowed to use more than 8 cores in total and to request more than 32 GB of main memory. ### Internet access Connections to the Internet are only permitted from the login nodes in outgoing direction; Internet access from within batch jobs is not possible. ### Data transfers To download source code or other data, command line tools such as `wget`, `curl`, `rsync`, `scp`, `pip`, `git`, or similar may be used interactively on the login nodes. In case the transfer is expected to take a long time it is recommended to run it as a Slurm job in the Slurm partition "datatransfer". Datatransfer jobs can occupy up to 8 cores and can take up to 6 hours. ## Hardware configuration ### Compute nodes #### CPU nodes - 768 compute nodes, 1536 CPUs - Processor type: [AMD EPYC Genoa 9554](https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9554.html) - Processor base frequency: 3.1 GHz - Cores per node: 128 (each with 2 "hyper-threads" (SMT), i.e. 256 logical CPUs per node) - Main memory (DDR5 RAM) per node: 512 GB (609 nodes), 768 GB (90 nodes), 1024 GB (66 nodes), 2304 GB (3 nodes) - *Theoretical* peak performance per node (FP64, "double precision"): 3.1 GHz \* 16 FP64 Flops/cycle \* 128 = 6350 GFlop/s - *Theoretical* memory bandwidth per node: 920 GB/s (768 GB nodes, 2304 GB nodes), 610 GB/s (512 GB nodes, 1024 GB nodes) - 2x4 NUMA domains per node each with 16 physical cores ### Login and interactive nodes - 6 nodes for login and code compilation (DNS names `viper[01-06]i.mpcdf.mpg.de`) - Processor type: [AMD EPYC Genoa 9554](https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9554.html) - Cores per node: 128 physical CPUs (256 logical CPUs) - Main memory (RAM) per node: login nodes: 512 GB, interactive nodes: 512 GB ### Interconnect Viper-CPU uses a Mellanox InfiniBand NDR network with a non-blocking fat-tree topology with a per-node bandwidth of 200 Gb/s (NDR200). ### I/O subsystem Approx. 12 PB of online disk space are available. ### Additional hardware details Additional details on the Viper hardware are given on a [separate documentation page](viper-details.md). ## File systems **Important:** The HPC systems Viper-CPU and Viper-GPU have *separate* file systems, i.e., each of the two HPC systems has its own local file systems at `/u` and `/ptmp`. To access data from Viper-CPU from Viper-GPU and vice versa, the data has to be copied. The data can be copied directly on either of the login nodes, as those mount all 4 filesystems. You can find the Viper-GPU filesystems there as `/viper/u2` and `/viper/ptmp2`. ### $HOME Your home directory is located in the GPFS file system `/u` (see below). ### GPFS There are two global, parallel file systems of type [GPFS](https://www.ibm.com/products/spectrum-scale) (`/u` and `/ptmp`), symmetrically accessible from all Viper cluster nodes, plus the migrating file system `/r` interfacing to the HPSS archive system. #### File system `/u` The file system `/u` (a symbolic link to `/viper/u`) is designed for permanent user data (source files, config files, etc.). The size of `/u` is 1.2 PB. Note that *no system backups* are performed. Your home directory is in `/u`. The default disk quota in `/u` is 1.0 TB, the file quota is 256K files. You can check your disk quota in `/u` with the command: ```bash /usr/lpp/mmfs/bin/mmlsquota viper_u1 ``` #### File system `/ptmp` The file system `/ptmp` (a symbolic link to /viper/ptmp) is designed for batch job I/O (12 PB, **no system backups**). Files in `/ptmp` that have not been accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary (with prior notice). As a current policy, no quotas are applied on `/ptmp`. This gives users the freedom to manage their data according to their current needs without administrative overhead. This liberal policy presumes fair usage of the common file space. So, please do a regular housekeeping of your data and archive/remove files that are not currently in use. Archiving data from the GPFS file systems to tape can be done using the migrating file system `/r` (see below). #### File system `/r` The `/r` file system (a symbolic link to `/ghi/r`) stages archive data. It is available only on the login nodes `viper[01-06]i.mpcdf.mpg.de`. Each user has a subdirectory `/r//` to store data. For efficiency, files should be packed to tar files (with a size of about 1 GB to 1 TB) before archiving them in `/r`, i.e., please avoid archiving small files. When the file system `/r` gets filled above a certain value, files will be transferred from disk to tape, beginning with the largest files which have not been used for the longest time. For documentation on how to use the MPCDF archive system, please see the [backup and archive section](../data/backup-archive/index.md). #### /tmp and node-local storage Please don't use the file system `/tmp` or `$TMPDIR` for scratch data. Instead, use `/ptmp` which is accessible from all Viper cluster nodes. In cases where an application really depends on node-local storage, please use the directories from the environment variables `JOB_TMPDIR` and `JOB_SHMTMPDIR`, which are set individually for each Slurm job. ## Software ### Access to software via environment modules Environment modules are used at MPCDF to provide software packages and enable easy switching between different software versions. Use the command ```bash module avail ``` to list the available software packages on the HPC system. Note that you can search for a certain module by using the `find-module` tool (see below). Use the command ```bash module load package_name/version ``` to actually load a software package at a specific version. Further information on the environment modules on Viper and their hierarchical organization is given below. Information on the software packages provided by the MPCDF is available [here](software/index.md). ### Recommended compiler and MPI software stack on Viper As explained below, **no defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login**. We currently recommend using the following versions on Viper: ```bash module load intel/2026.0 impi/2021.18 ``` or ```bash module load gcc/16 impi/2021.18 ``` Previously recommended versions are: ```bash module load intel/2024.0 impi/2021.11 # until 2026/08 module load gcc/13 impi/2021.11 # until 2026/08 ``` Specific optimizing compiler flags for the AMD EPYC CPU are given further below. ### Hierarchical module environment To manage the plethora of software packages resulting from all the relevant combinations of compilers and MPI libraries, we organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the uppermost level, dependent libraries (e.g., MPI) on the second level, more dependent libraries on a third level. This means that not all the modules are visible initially: Only after loading a compiler module, the modules depending on this will become available. And similarly, loading an MPI module in addition will make the modules depending on the MPI library available. No defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login. This forces users to specify explicit versions for those modules during compilation and in the batch scripts to ensure that the same MPI library is loaded. This also means that users can decide themselves when they use newer compiler and MPI versions for their code which avoids compatibility problems when changing defaults centrally. For example, the FFTW library compiled with the Intel compiler and the Intel MPI library can be loaded as follows: First, load the Intel compiler module using the command ```bash module load intel/2026.0 ``` second, the Intel MPI module with ```bash module load impi/2021.18 ``` and, finally, the FFTW module fitting exactly to the compiler and MPI library via ```bash module load fftw-mpi ``` You may check by using the command ```bash module avail ``` that after the first and second steps the dependent environment modules become visible, in the present example impi and fftw-mpi. Moreover, note that the environment modules can be loaded via a single 'module load' statement as long as the order given by the hierarchy is correct, e.g., ```bash module load intel/2026.0 impi/2021.18 fftw-mpi ``` It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the uppermost hierarchy. Note that a hierarchy exists for dependent Python modules via the 'python-waterboa' module files on the top level. Because of the hierarchy, some modules only appear after other modules (such as compiler and MPI) have been loaded. One can search all available combinations of a certain software (e.g. fftw-mpi) by using ```bash find-module fftw-mpi ``` Further information on using environment modules is given [here](software/environment-modules.md). ## Slurm batch system The batch system used on the HPC cluster Viper is the open-source workload manager [Slurm (Simple Linux Utility for Resource management)](https://slurm.schedmd.com). To run test or production jobs, submit a job script (see below) to Slurm, which will allocate the resources required for your job (e.g. the compute nodes to run your job on). By default, the job run limit is set to 8 on Viper, the default job submit limit is 300. If your batch jobs can't run independently from each other, please use job steps. There are mainly two types of batch jobs: - Exclusive, where all resources on the nodes are allocated to the job - Shared, where several jobs share the resources of one node. In this case it is necessary that the number of CPUs and the amount of memory are specified for each job. The AMD processors on Viper support simultaneous multithreading (SMT) which **potentially** increases the performance of an application by up to 20%. To use SMT, you have to increase the product of the number of MPI tasks per node and the number of threads per MPI task from 128 to 256 in your job script. Please be aware that when doubling the number of MPI tasks per node each task only gets half of the memory compared to the non-SMT job. If you need more memory, you have to specify this in your job script (see the example batch scripts). Overview of the available per-job resources on Viper: ```text Job type Max. CPUs Number of GPUs Max. Memory Number Max. Run per Node per node per Node [MB] of Nodes Time ============================================================================================= shared cpu 64 / 128 in HT mode 250000 < 1 24:00:00 --------------------------------------------------------------------------------------------- 480000 1-256 24:00:00 exclusive cpu 128 / 256 in HT mode 730000 1-64 24:00:00 980000 1-64 24:00:00 2250000 1-3 24:00:00 --------------------------------------------------------------------------------------------- ``` If an application needs more than 480000 MB per node, the required amount of memory has to be specified in the Slurm submit script, e.g. with the following options: ```text #SBATCH --mem=730000 # to request up to 730000 MB or #SBATCH --mem=2250000 # to request up to 2250000 MB ``` To submit an application to exactly one of the four possible node types w.r.t. memory, a job feature --constraint has to be specified with one of the following values: normalmem (480000 MB), mediummem (730000 MB), largemem (980000 MB), hugemem (2250000 MB). A job submit filter will automatically choose the right partition and job parameters from the resource specification. Interactive testing and debugging is possible by using the command: ```bash srun -n N_TASKS -p interactive --time=TIME_LESS_THAN_2HOURS --mem=MEMORY_LESS_THAN_32000M ./EXECUTABLE ``` Interactive jobs are limited to 8 cores, 32000M memory and 2 hours runtime. For detailed information about the Slurm batch system, please see [Slurm Workload Manager](https://slurm.schedmd.com/). The most important Slurm commands are - `sbatch ` Submit a job script for execution - `squeue` Check the status of your job(s) - `scancel ` Cancel a job - `sinfo` List the available batch queues (partitions). Do not run Slurm client commands from loops in shell scripts or other programs. Ensure that programs limit calls to these commands to the minimum necessary for the information you are trying to gather. Sample Batch job scripts can be found below. Notes on job scripts: - The directive ```text #SBATCH --nodes= ``` in your job script specifies the number of compute nodes that your program will use. - The directive ```text #SBATCH --ntasks-per-node= ``` specifies the number of MPI processes for the job. The parameter ntasks-per-node cannot be greater than 128 because one compute node on Viper has 128 physical cores (with 2 threads each and thus 256 logical CPUs in SMT mode). - The directive ```text #SBATCH --cpus-per-task= ``` specifies the number of threads per MPI process if you are using OpenMP. - The expression ```text ntasks-per-node * cpus-per-task ``` may not exceed 256. - The expression ```text nodes * ntasks-per-node * cpus-per-task ``` gives the total number of CPUs that your job will use. - Jobs that need less than half a compute node have to specify a reasonable memory limit so that they can share a node! - A job submit filter will automatically choose the right partition/queue from the resource specification. ## Slurm example batch scripts ### MPI and MPI/OpenMP batch scripts #### MPI batch job ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=128 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 # Run the program: srun ./myprog > prog.out ``` #### Hybrid MPI/OpenMP batch job ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job_hybrid.out.%j #SBATCH -e ./job_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=8 # for OpenMP: #SBATCH --cpus-per-task=16 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly: export OMP_PLACES=cores # Run the program: srun ./myprog > prog.out ``` #### Hybrid MPI/OpenMP batch job in simultaneous multithreading (SMT) mode ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job_hybrid.out.%j #SBATCH -e ./job_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=32 #SBATCH --ntasks-per-node=8 # Enable SMT: #SBATCH --ntasks-per-core=2 # for OpenMP: #SBATCH --cpus-per-task=32 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock Limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly: export OMP_PLACES=threads # Run the program: srun ./myprog > prog.out ``` #### Small MPI batch job on a shared node ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of MPI Tasks, e.g. 8: #SBATCH --ntasks=8 # Memory usage [MB] of the job is required, e.g. 3000 MB per task: #SBATCH --mem=24000 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 # Run the program: srun ./myprog > prog.out ``` ### Batch jobs with dependencies The following script generates a sequence of jobs, each job running the given job script. The start of each individual job depends on its dependency, where possible values for the `--dependency` flag are, e.g. - `afterany:job_id` This job starts after the previous job has terminated - `afterok:job_id` This job starts after previous job has successfully executed ```bash #!/bin/bash # Submit a sequence of batch jobs with dependencies # # Number of jobs to submit: NR_OF_JOBS=6 # Batch job script: JOB_SCRIPT=./my_batch_script echo "Submitting job chain of ${NR_OF_JOBS} jobs for batch script ${JOB_SCRIPT}:" JOBID=$(sbatch ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} I=1 while [ ${I} -lt ${NR_OF_JOBS} ]; do JOBID=$(sbatch --dependency=afterany:${JOBID} ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} let I=${I}+1 done ``` ### Batch job using a job array ```bash #!/bin/bash -l # specify the indexes (max. 30000) of the job array elements (max. 300 - the default job submit limit per user) #SBATCH --array=1-20 # Standard output and error: #SBATCH -o job_%A_%a.out # Standard output, %A = job ID, %a = job array index #SBATCH -e job_%A_%a.err # Standard error, %A = job ID, %a = job array index # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_array # # Number of nodes and MPI tasks per node: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=128 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 # Run the program: # the environment variable $SLURM_ARRAY_TASK_ID holds the index of the job array and # can be used to discriminate between individual elements of the job array srun ./myprog > prog.out ``` ### Single-node example job scripts for sequential programs, plain-OpenMP cases, Python, Julia, Matlab In the following, example job scripts are given for jobs that use at maximum one full node. Use cases are sequential programs, threaded programs using OpenMP or similar models, and programs written in languages such as Python, Julia, Matlab, etc. The Python example programs referred to below are available for [download](https://datashare.mpcdf.mpg.de/s/KCEtd0tP3zLypq4). #### Single-core job ```bash #!/bin/bash -l # # Single-core example job script for MPCDF Viper. # In addition to the Python example shown here, the script # is valid for any single-threaded program, including # sequential Matlab, Mathematica, Julia, and similar cases. # #SBATCH -J PYTHON_SEQ #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH --ntasks=1 # launch job on a single core #SBATCH --cpus-per-task=1 # on a shared node #SBATCH --mem=2000MB # memory limit for the job #SBATCH --time=0:10:00 module purge module load python-waterboa/2025.06 # Set number of OMP threads to fit the number of available cpus, if applicable. export OMP_NUM_THREADS=1 # Run single-core program srun python3 ./python_sequential.py ``` #### Small job with multithreading, applicable to Python, Julia and Matlab, plain OpenMP, or any threaded application ```bash #!/bin/bash -l # # Multithreading example job script for MPCDF Viper. # In addition to the Python example shown here, the script # is valid for any multi-threaded program, including # Matlab, Mathematica, Julia, and similar cases. # #SBATCH -J PYTHON_MT #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH --ntasks=1 # launch job on #SBATCH --cpus-per-task=8 # 8 cores on a shared node #SBATCH --mem=16000MB # memory limit for the job #SBATCH --time=0:10:00 module purge module load python-waterboa/2025.06 # Set number of OMP threads to fit the number of available cpus, if applicable. export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun python3 ./python_multithreading.py ``` #### Python/NumPy multithreading, applicable to Julia and Matlab, plain-OpenMP, or any threaded application ```bash #!/bin/bash -l # # Multithreading example job script for MPCDF Viper. # In addition to the Python example shown here, the script # is valid for any multi-threaded program, including # parallel Matlab, Julia, and similar cases. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J PY_MULTITHREADING #SBATCH --nodes=1 # request a full node #SBATCH --ntasks-per-node=1 # only start 1 task via srun because Python multiprocessing starts more tasks internally #SBATCH --cpus-per-task=128 # assign all the cores to that first task to make room for multithreading #SBATCH --time=00:10:00 module purge module load python-waterboa/2025.06 # set number of OMP threads *per process* export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun python3 ./python_multithreading.py ``` #### Python multiprocessing ```bash #!/bin/bash -l # # Python multiprocessing example job script for MPCDF Viper. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J PYTHON_MP #SBATCH --nodes=1 # request a full node #SBATCH --ntasks-per-node=1 # only start 1 task via srun because Python multiprocessing starts more tasks internally #SBATCH --cpus-per-task=128 # assign all the cores to that first task to make room for Python's multiprocessing tasks #SBATCH --time=00:10:00 module purge module load python-waterboa/2025.06 # Important: # Set the number of OMP threads *per process* to avoid overloading of the node! export OMP_NUM_THREADS=1 # Use the environment variable SLURM_CPUS_PER_TASK to have multiprocessing # spawn exactly as many processes as you have CPUs available. srun python3 ./python_multiprocessing.py $SLURM_CPUS_PER_TASK ``` #### Python mpi4py ```bash #!/bin/bash -l # # Python MPI4PY example job script for MPCDF Viper. # May use more than one node. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J MPI4PY #SBATCH --nodes=1 #SBATCH --ntasks-per-node=128 #SBATCH --time=00:10:00 module purge module load gcc/16 openmpi/5.0 module load python-waterboa/2025.06 module load mpi4py/4.1.1 # Important: # Set the number of OMP threads *per process* to avoid overloading of the node! export OMP_NUM_THREADS=1 srun python3 ./python_mpi4py.py ``` ## Migration guide for users coming from Intel-based HPC systems ### Application performance On Viper, sufficiently optimized (SIMD, parallel scalability) application codes can expect a performance boost by at least a factor of two per node in direct comparison to a [Raven](raven-user-guide) node. Users who experience relatively bad performance or need help with porting are encouraged to contact the MPCDF helpdesk for support. Information on important hardware details and software recommendations, including optimizing compiler flags, are given below. ### Software #### Compilers Both Intel and GNU compilers for Fortran, C, and C++ are known to generate well-optimized code for the AMD EPYC (zen4) CPU and are therefore recommended for use on Viper. The AMD EPYC (zen4) CPU has AVX2 (256 bit) execution units but can decode AVX512 instructions. Depending on the application, using AVX512 may or may not produce faster code (by up to approx. 15%). For **Intel compilers** (`ifx`, `icx`, `icpx`) the relevant microarchitecture-specific flags are `-march=znver4` (targeting AMD EPYC (zen4) CPUs including AVX512 vectorization, newly introduced with Intel oneAPI version 2024), and `-march=skylake-avx512` (AVX512) or `–march=core-avx2` (AVX2). The latter two flags are also supported and recommended for the legacy Intel Fortran compiler `ifort` that is still part of Intel oneAPI 2024. Users are advised to diagnose the degree of SIMD vectorization by employing the compiler option `-qopt-report=3`. In general, please note that _none_ of the Intel compiler flags starting with `-x` or `-ax` can be used on AMD CPUs because these flags enable code generation that exclusively targets Intel CPUs. In particular, using the well-known option `-xHost` to automatically select the target architecture of the compilation host does _not_ work for the AMD CPUs. For **GNU compilers** (`gfortran`, `gcc`, `g++`) the relevant architecture-specific flags are `-march=znver4` (AVX512) or `-march=znver4 -mprefer-vector-width=256` (AVX2). Users are advised to diagnose the degree of SIMD vectorization by using the compiler option `-fopt-info`. Vectorization of non-trivial SIMD loops may require the flags `-O3 -ffast-math` in addition. The AMD Optimizing C/C++ and Fortran Compilers (AOCC) are provided with limited support and mainly for experimental purposes, to be used standalone or together with the AMD Optimizing CPU Libraries (AOCL). More details about the features, the limitations, and known issues can be found on AMD's website for [AOCC](https://www.amd.com/en/developer/aocc.html) and [AOCL](https://www.amd.com/en/developer/aocl.html). #### Math libraries The Intel Math Kernel Library ([oneMKL](https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl.html)) is the recommended mathematical library on Viper. It provides, among others, highly tuned implementations of linear algebra operations and Fast Fourier Transforms (FFTs), which are exposed via the standard BLAS, LAPACK, and FFTW interfaces for C/C++ and Fortran. In addition, a broad selection of numerical libraries such as FFTW, PETSc, SLEPc is available. The AMD Optimizing CPU Libraries ([AOCL](https://www.amd.com/en/developer/aocl.html)) are provided for GCC and AMD (AOCC) compilers, with limited support and for experimental purposes. Users may evaluate them as an alternative to MKL, and careful performance benchmarking is needed. #### Performance tools Intel VTUNE can be used on Viper to identify hotspots, call stacks, and measure multithreading characteristics of an application. However, certain hardware-specific analysis types are not available on AMD CPUs. In addition, installations of AMD uProf are available, covering similar use cases. For low-level performance measurements, `perf` and the likwid toolsuite are provided. To perform lightweight profiling of MPI applications, Intel APS (for Intel MPI only, comes bundled with VTUNE) or mpitrace can be used. # Viper-GPU User Guide ```{contents} Contents :local: :depth: 2 ``` ## System overview The HPC system Viper, deployed during 2024/2025, consists of two parts, [Viper-GPU](viper-gpu-user-guide.md) and [Viper-CPU](viper-user-guide.md), which are operated as two logically separate clusters. **Viper-GPU** has been operational since February 2025 and in its final configuration provides 327 GPU compute nodes, with 2 [AMD Instinct MI300A](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300a.html) APUs and 128 GB of high-bandwidth memory (HBM3) per APU. The nodes are interconnected with a NVIDIA/Mellanox NDR InfiniBand network using a fat-tree topology (NDR, 400 Gb/s). There are 3 login nodes and an I/O subsystem that serves approx. 12 PB of disk-based storage with direct HSM access, plus approx. 500 TB of NVMe-based storage. ![MPCDF Viper-GPU Deployment](_static/viper-gpu-2025.jpg "MPCDF Viper-GPU") ## Access ### Login For security reasons, direct login to the HPC system Viper-GPU is allowed only from within certain MPG networks. Users from other locations have to log in to one of our [gateway systems](gateways) first. ```bash ssh @gate.mpcdf.mpg.de ``` Use `ssh` to connect to one of the three available Viper-GPU login nodes (`viper[11-13]i`), e.g.: ```bash ssh @viper11i.mpcdf.mpg.de ``` To log in, you must provide your (Kerberos) password and an OTP on the Viper-GPU login node. SSH keys are not allowed. Secure copy (scp) can be used to transfer data to or from `viper[11-13]i.mpcdf.mpg.de`. Viper-GPU's (all login/interactive nodes) SSH key fingerprints (SHA256) are: ```text SHA256:0DhkRt6Qom1GA0SmvEjnlKWpIg1+kMPDjOUSqJ8ceyQ (ED25519) SHA256:pM1fS+4YXGuIaLk0bLSt1sOlS1TpLrZYmFMRY9UjBKo (RSA) ``` ### Resource limits The login nodes `viper11i.mpcdf.mpg.de` and `viper12i.mpcdf.mpg.de` are intended only for editing, compiling and submitting parallel programs. Running parallel programs interactively on the login nodes is not allowed. Per user, the CPU resources are restricted to an equivalent of two physical CPU cores, memory usage to at most 10% of the available memory, and the number of tasks (including threads) to a maximum of 768. The login node `viper13i.mpcdf.mpg.de` is also primarily intended only for editing, compiling and submitting parallel programs - but here, per user, the CPU resources are restricted to an equivalent of six physical CPU cores, memory usage to at most 20% of the available memory, and the number of tasks (including threads) to a maximum of 1536. Jobs have to be submitted to the Slurm batch system which reserves and allocates the resources (e.g. compute nodes) required for your job. Further information on the batch system is provided [below](#slurm-batch-system). ### Interactive (debug) runs To test and optimize GPU codes one can use the "apudev" partition by specifying ```bash #SBATCH -p apudev ``` in the submit script. One node with two MI300A APUs is available in the "apudev" partition. The maximum wall clock time is 15 minutes. One or two APUs can be requested. ### Internet access Connections to the Internet are only permitted from the login nodes in the outgoing direction; Internet access from within batch jobs is not possible. ### Data transfers To download source code or other data, command line tools such as `wget`, `curl`, `rsync`, `scp`, `pip`, `git`, or similar may be used interactively on the login nodes. In case the transfer is expected to take a long time it is recommended to run it inside a `screen` or `tmux` session. ## Hardware configuration ### Compute nodes #### GPU nodes - 327 GPU nodes, 654 APUs - Processor type: [AMD Instinct MI300A](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300a.html) APU - Main memory (HBM3) per APU: 128 GB - 24 CPU cores per APU - 228 GPU compute units per APU - *Theoretical* peak performance per APU (FP64, "double precision"): 61 TFlop/s - *Theoretical* memory bandwidth per APU: 5.3 TB/s ### Login and interactive nodes - 3 nodes for login and code compilation (DNS names `viper[11-13]i.mpcdf.mpg.de`) - Processor type: [AMD EPYC Genoa 9554](https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9554.html) - Cores per node: 128 physical CPUs (256 logical CPUs) - Main memory (RAM) per node: login nodes: 512 GB ### Interconnect Viper-GPU uses a Mellanox InfiniBand NDR network with a non-blocking fat-tree topology with a per-node bandwidth of 400 Gb/s (NDR). ### I/O subsystem Approx. 12 PB of online disk space are available. ### Additional hardware details Additional details on the hardware are given on a [separate documentation page](viper-gpu-details.md). ## File systems **Important:** The HPC systems Viper-CPU and Viper-GPU have *separate* file systems, i.e., each of the two HPC systems has its own local file systems at `/u` and `/ptmp`. To access data from Viper-CPU from Viper-GPU and vice versa, the data has to be copied. The data can be copied directly on either of the login nodes, as those mount all 4 filesystems. You can find the Viper-CPU filesystems there as `/viper/u1` and `/viper/ptmp1`. ### $HOME Your home directory is located in the GPFS file system `/u` (see below). ### GPFS There are two global, parallel file systems of type [GPFS](https://www.ibm.com/products/spectrum-scale) (`/u` and `/ptmp`), symmetrically accessible from all Viper-GPU cluster nodes, plus the migrating file system `/r` interfacing to the HPSS archive system. #### File system `/u` The file system `/u` (a symbolic link to `/viper/u2`) is designed for permanent user data (source files, config files, etc.). The size of `/u` is 1.2 PB. Note that *no system backups* are performed. Your home directory is in `/u`. The default disk quota in `/u` is 1.0 TB, the file quota is 256K files. You can check your disk quota in `/u` with the command: ```bash /usr/lpp/mmfs/bin/mmlsquota viper_u2 ``` #### File system `/ptmp` The file system `/ptmp` (a symbolic link to `/viper/ptmp2`) is designed for batch job I/O (12 PB, **no system backups**). Files in `/ptmp` that have not been accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary (with prior notice). As a current policy, no quotas are applied on `/ptmp`. This gives users the freedom to manage their data according to their actual needs without administrative overhead. This liberal policy presumes a fair usage of the common file space. So, please do a regular housekeeping of your data and archive/remove files that are not currently in use. Archiving data from the GPFS file systems to tape can be done using the migrating file system `/r` (see below). #### File system `/r` The `/r` file system (a symbolic link to `/ghi/r`) stages archive data. It is available only on the login nodes `viper[11-13]i.mpcdf.mpg.de`. Each user has a subdirectory `/r//` to store data. For efficiency, files should be packed to tar files (with a size of about 1 GB to 1 TB) before archiving them in `/r`, i.e., please avoid archiving small files. When the file system `/r` gets filled above a certain value, files will be transferred from disk to tape, beginning with the largest files which have not been used for the longest time. For documentation on how to use the MPCDF archive system, please see the [backup and archive section](../data/backup-archive/index.md). #### /tmp and node-local storage Please don't use the file system `/tmp` or `$TMPDIR` for scratch data. Instead, use `/ptmp` which is accessible from all Viper-GPU cluster nodes. In cases where an application really depends on node-local storage, please use the directories from the environment variables `JOB_TMPDIR` and `JOB_SHMTMPDIR`, which are set individually for each Slurm job and cleaned afterwards. ## Software ### Access to software via environment modules Environment modules are used at MPCDF to provide software packages and enable easy switching between different software versions. Use the command ```bash module avail ``` to list the available software packages on the HPC system. Note that you can search for a certain module by using the `find-module` tool (see below). Use the command ```bash module load package_name/version ``` to actually load a software package at a specific version. Further information on the environment modules on Viper-GPU and their hierarchical organization is given below. Information on the software packages provided by the MPCDF is available [here](software/index.md). ### Recommended compiler and MPI software stack on Viper-GPU As explained below, **no defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login**. We currently recommend using the following versions on Viper-GPU: ```bash module load gcc/16 rocm/7.2 openmpi/5.0 ``` Specific optimizing compiler flags for the AMD EPYC CPU are given further below. If you want to use GPU-aware MPI, we recommend: ```bash module load gcc/16 rocm/7.2 openmpi_gpu/5.0 ``` Previously recommended versions are: ```bash module load gcc/14 rocm/6.3 openmpi/5.0 # until 2026/08 module load gcc/14 rocm/6.3 openmpi_gpu/5.0 # until 2026/08 ``` ### Hierarchical module environment To manage the plethora of software packages resulting from all the relevant combinations of compilers and MPI libraries, we organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the uppermost level, dependent libraries (e.g., MPI) on the second level, more dependent libraries on a third level. This means that not all the modules are visible initially: Only after loading a compiler module, the modules depending on this will become available. And similarly, loading an MPI module in addition will make the modules depending on the MPI library available. No defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login. This forces users to specify explicit versions for those modules during compilation and in the batch scripts to ensure that the same MPI library is loaded. This also means that users can decide themselves when they use newer compiler and MPI versions for their code which avoids compatibility problems when changing defaults centrally. For example, the FFTW library compiled with the GCC compiler and the OpenMPI library can be loaded as follows: First, load the compiler module using the command ```bash module load gcc/16 ``` second, the MPI module with ```bash module load openmpi/5.0 ``` and, finally, the FFTW module fitting exactly to the compiler and MPI library via ```bash module load fftw-mpi ``` You may check by using the command ```bash module avail ``` that after the first and second steps the dependent environment modules become visible, in the present example openmpi and fftw-mpi. Moreover, note that the environment modules can be loaded via a single 'module load' statement as long as the order given by the hierarchy is correct, e.g., ```bash module load gcc/16 openmpi/5.0 fftw-mpi ``` It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the uppermost hierarchy. Note that a hierarchy exists for dependent Python modules via the 'python-waterboa' module files on the top level. Because of the hierarchy, some modules only appear after other modules (such as compiler and MPI) have been loaded. One can search all available combinations of a certain software (e.g. fftw-mpi) by using ```bash find-module fftw-mpi ``` Further information on using environment modules is given [here](software/environment-modules.md). ## Slurm batch system The batch system used on the HPC cluster Viper-GPU is the open-source workload manager [Slurm (Simple Linux Utility for Resource management)](https://slurm.schedmd.com). To run test or production jobs, submit a job script (see below) to Slurm, which will allocate the resources required for your job (e.g. the compute nodes to run your job on). By default, the job run limit is set to 8 on Viper-GPU, the default job submit limit is 300. If your batch jobs can't run independently from each other, please use job steps. There are mainly two types of batch jobs: - Exclusive, where all resources on the nodes are allocated to the job - Shared, where two jobs may share the resources of one node. In this case, it is necessary that the number of CPUs and the amount of memory are specified for each job. The CPU cores on Viper-GPU support simultaneous multithreading (SMT) which **potentially** increases the performance of an application by up to 20%. To use SMT, you have to increase the product of the number of MPI tasks per node and the number of threads per MPI task from 48 to 96 in your job script. Please be aware that when doubling the number of MPI tasks per node each task only gets half of the memory compared to the non-SMT job. Overview of the available per-job resources on Viper-GPU: ```text Job type Max. CPUs Number of GPUs Max. Memory Number Max. Run per Node per node per Node [MB] of Nodes Time ============================================================================================= shared apu 24 / 48 in HT mode 1 110000 < 1 24:00:00 --------------------------------------------------------------------------------------------- exclusive apu 48 / 96 in HT mode 2 220000 1-128 24:00:00 --------------------------------------------------------------------------------------------- ``` A job submit filter will automatically choose the right partition and job parameters from the resource specification. For detailed information about the Slurm batch system, please see [Slurm Workload Manager](https://slurm.schedmd.com/). The most important Slurm commands are - `sbatch ` Submit a job script for execution - `squeue` Check the status of your job(s) - `scancel ` Cancel a job - `sinfo` List the available batch queues (partitions). Do not run Slurm client commands from loops in shell scripts or other programs. Ensure that programs limit calls to these commands to the minimum necessary for the information you are trying to gather. Sample Batch job scripts can be found below. Notes on job scripts: - The directive ```text #SBATCH --nodes= ``` in your job script specifies the number of compute nodes that your program will use. - The directive ```text #SBATCH --ntasks-per-node= ``` specifies the number of MPI processes for the job. The parameter ntasks-per-node cannot be greater than 48 because one apu compute node on Viper-GPU has 48 physical cores (with 2 threads each and thus 96 logical CPUs in SMT mode). - The directive ```text #SBATCH --cpus-per-task= ``` specifies the number of threads per MPI process if you are using OpenMP. - The expression ```text ntasks-per-node * cpus-per-task ``` may not exceed 96. - The expression ```text nodes * ntasks-per-node * cpus-per-task ``` gives the total number of CPUs that your job will use. - To select GPU nodes specify a job constraint as follows: ```text #SBATCH --constraint="apu" ``` - Jobs that need less than half a compute node have to specify a reasonable memory limit so that they can share a node! - A job submit filter will automatically choose the right partition/queue from the resource specification. ### Partitioning of MI300A GPU resources The MI300A APU allows its physical GPU compute resources to be divided into multiple logical devices, enabling more flexible workload scheduling and resource allocation. Details and instructions are given on a [separate documentation page](viper-gpu-apu-partitioning.md). ## Slurm example batch scripts ### Batch jobs using APUs Note that computing time on APU-accelerated nodes is accounted using a weighting factor of 2 relative to CPU-only jobs, corresponding to the additional computing power provided by the GPUs. Users are advised to check the performance reports of their jobs in order to monitor adequate utilization of the resources. #### GPU job using 1 or 2 APUs on a single node The following example job script launches a (potentially multithreaded) program to use one (or two) APU(s) on a single node. In case more than one GPU is requested, the user code must be able to utilize these additional APUs properly. ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_apu # #SBATCH --ntasks=1 #SBATCH --constraint="apu" # # --- default case: use a single APU on a shared node --- #SBATCH --gres=gpu:1 #SBATCH --cpus-per-task=24 #SBATCH --mem=110000 # # --- uncomment to use 2 APUs on a full node --- # #SBATCH --gres=gpu:2 # #SBATCH --cpus-per-task=48 # #SBATCH --mem=220000 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=12:00:00 module purge module load gcc/16 rocm/7.2 openmpi/5.0 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun ./apu_executable ``` #### Hybrid MPI/OpenMP job using one or more nodes with 2 APUs each The following example job script launches a hybrid MPI/OpenMP-code on one (or more) nodes running one task per APU. Note that the user code needs to attach its tasks to the different APUs based on some code-internal logic. ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_gpu # #SBATCH --nodes=1 # Request 1 or more full nodes #SBATCH --constraint="apu" # providing APUs. #SBATCH --gres=gpu:2 # Request 2 APUs per node. #SBATCH --ntasks-per-node=2 # Run one task per APU #SBATCH --cpus-per-task=24 # using 24 cores each. #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=12:00:00 module purge module load gcc/16 rocm/7.2 openmpi/5.0 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun ./mpi_openmp_apu_executable ``` ### Batch jobs with dependencies The following script generates a sequence of jobs, each job running the given job script. The start of each individual job depends on its dependency, where possible values for the `--dependency` flag are, e.g. - `afterany:job_id` This job starts after the previous job has terminated - `afterok:job_id` This job starts after previous job has successfully executed ```bash #!/bin/bash # Submit a sequence of batch jobs with dependencies # # Number of jobs to submit: NR_OF_JOBS=6 # Batch job script: JOB_SCRIPT=./my_batch_script echo "Submitting job chain of ${NR_OF_JOBS} jobs for batch script ${JOB_SCRIPT}:" JOBID=$(sbatch ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} I=1 while [ ${I} -lt ${NR_OF_JOBS} ]; do JOBID=$(sbatch --dependency=afterany:${JOBID} ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} let I=${I}+1 done ``` ### Batch job using a job array ```bash #!/bin/bash -l # specify the indexes (max. 30000) of the job array elements (max. 300 - the default job submit limit per user) #SBATCH --array=1-20 # Standard output and error: #SBATCH -o job_%A_%a.out # Standard output, %A = job ID, %a = job array index #SBATCH -e job_%A_%a.err # Standard error, %A = job ID, %a = job array index # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_array # # Number of nodes and MPI tasks per node: #SBATCH --nodes=1 #SBATCH --constraint="apu" # providing APUs. #SBATCH --gres=gpu:2 # Request 2 APUs per node. #SBATCH --ntasks-per-node=48 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load gcc/16 rocm/7.2 openmpi/5.0 # Run the program: # the environment variable $SLURM_ARRAY_TASK_ID holds the index of the job array and # can be used to discriminate between individual elements of the job array srun ./myprog > prog.out ``` ## Migration guide for users coming from Intel- and NVIDIA-based HPC systems Comprehensive general information on tools, techniques, and best practices necessary to target the AMD APUs is available in the slide decks from past training events: * [AMD GPU Workshop and Hackathon, November 2024](https://datashare.mpcdf.mpg.de/s/eBKu8221H6kwfcH) * [AMD GPU Workshop, November 2023](https://datashare.mpcdf.mpg.de/s/wFdwmYnLfwTeP5P) Below, crucial information to get started specifically on the Viper-GPU system at the MPCDF is given in condensed form. ### Application performance On a _single AMD MI300A APU_, sufficiently optimized GPU application codes should expect a performance improvement by at least a factor of two compared to a _single NVIDIA A100 GPU_. When comparing per-node performances, this corresponds to an application performance _per Viper-GPU node (with 2 MI300A APUs)_ being at least on par with (or better than) _a [Raven-GPU](raven-user-guide) node (with 4 NVIDIA A100 GPUs)_. Users who need help with improving performance or with porting their codes to Viper-GPU are encouraged to contact the MPCDF helpdesk for support. ### Placement of GPU-accelerated MPI tasks A Viper-GPU node has two distinct APU sockets with 24 CPU cores and one GPU, each. To avoid overhead due to inter-socket communication, MPI tasks (or processes) that use GPU acceleration must be placed physically (topologically) close to the actual GPU resource. For example, a job that runs two GPU-enabled MPI tasks per node should be launched with the following Slurm parameters: ``` #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=1 #SBATCH --gres=gpu:2 ``` This setup will place task 0 on core 0 of socket 0 (close to GPU 0), and task 1 on core 0 (24 in absolute numbering) of socket 1 (close to GPU 1). Similarly, for a hybrid OpenMP/MPI application that uses GPUs and multiple threads, the Slurm parameters ``` #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=24 #SBATCH --gres=gpu:2 ``` can be used to place task 0 on the 24 cores of socket 0 (close to GPU 0), and task 1 on the 24 cores of socket 1 (close to GPU 1). Please be aware that each MPI process has access to all GPUs locally available to the Slurm job, i.e. the application needs to handle the setting of the correct GPU affinity by itself. Moreover, please be aware that oversubscribing GPU resources (i.e. assigning a single GPU to multiple MPI tasks simultaneously) can significantly degrade performance on Viper-GPU. In general, one or at most two MPI tasks per GPU are recommended. This contrasts with NVIDIA GPUs (such as the A100 GPUs on Raven), where NVIDIA MPS is available to efficiently multiplex multiple tasks on a single GPU. It is recommended to perform performance measurements to identify the best performing configuration. ### Software In the following, information relevant to target the accelerators of Viper-GPU is provided. Moreover, the hints to compile and optimize the code parts that run on the CPU cores given for [Viper-CPU](viper-user-guide#application-performance) apply. #### Compilers Depending on the usage model of the GPUs/APUs, compilation is performed in different ways. ##### HIP/ROCm Using this model allows you to choose from several host compilers: gcc/gfortran, clang/flang, AMD or Intel compilers are possible. For your HIP (C++ *H*eterogeneous-Compute *I*nterface for *P*ortability) code, you have to use the `hipcc` compiler from the `rocm` module. We recommend loading the host compiler module first and then the `rocm` module. The architecture flags are set automatically for the MI300A on Viper-GPU (see below for more details and options). A typical compilation line reads ```bash module load gcc/16 rocm/7.2 hipcc -x hip --offload-arch=gfx942 -c -o my_gpu_code.o my_gpu_code.cxx ``` If your HIP code also contains MPI calls, you have to add the include path of the MPI library: ```bash module load gcc/16 rocm/7.2 openmpi/5.0 hipcc -x hip --offload-arch=gfx942 -c -o my_gpu_code.o -I${OPENMPI_HOME}/include my_gpu_code.cxx ``` You can build your CPU code with any of the aforementioned compilers, and finally link all object files with the `hipcc` command, where you then have to add the paths to the libraries needed (e.g. MPI, Fortran). ##### OpenMP TARGET directives If your code contains OpenMP TARGET directives to target the GPU, you have to use the `amdclang`, `amdclang++` or `amdflang` compiler from the `amd-llvm` module. It can be combined with `hipcc` from the `rocm` module, in this case you should load the `rocm` module first. The `amd-llvm` module always contains the latest versions of the compilers, which are currently under heavy development (especially the `amdflang` compiler). The `rocm` module contains the compilers bundled together with ROCm, which is updated less often. By first loading the `rocm` module and then the `amd-llvm` module, you can use the latest available LLVM compilers and ROCm version. A typical compilation line for an OpenMP TARGET code reads: ```bash module load amd-llvm/5.3 amdclang++ -fopenmp --offload-arch=gfx942 -c my_target_code.cxx -o my_target_code.o amdflang -fopenmp --offload-arch=gfx942 -c my_target_fortran_code.F90 -o my_target_fortran_code.o ``` For linking, you have to add the `libomptarget` library: ```bash module load amd-llvm/5.3 amdclang++ -fopenmp --offload-arch=gfx942 -o executable my_target_code.o -L${AMDLLVM_HOME}/lib/llvm/lib -lomptarget ``` Note that the `amd-llvm` module contains upstream LLVM with AMD GPU-specific additions (HIP compiler, OpenMP and debugging improvements etc.). This is not the same compiler as [AMD Optimizing C/C++ and Fortran Compilers (AOCC)](https://www.amd.com/en/developer/aocc.html), which adds AMD EPYC CPU-specific optimizations. ##### No defaults for the `--offload-arch` flag for MI300A Please be aware that no `--offload-arch` defaults are set for the HIP and AMD LLVM compilers. Previously set defaults had to be abandoned entirely due to several side effects. Please ensure that you specify at least `--offload-arch=gfx942` to generate proper code for the MI300A. Note that unified shared memory additionally requires setting the environment variable `HSA_XNACK=1` at runtime. #### Math libraries ##### ROCm libraries Similarly to CUDA, there are vendor-optimized ROCm (*R*adeon *O*pen *C*ompute Platfor*m*) implementations of commonly used math operations provided. After loading the `rocm` module, the headers are available in `${ROCM_HOME}/include` and the libraries can be used from `${ROCM_HOME}/lib`. Be aware that the directory structure of ROCm has changed in the releases starting with major version 6. They are now organized in subdirectories `rocblas`, `rocfft`, `rocrand`, ... The same applies to HIP, e.g., in your code you have to use the headers from e.g. `#include ` or `#include `. Most of the HIP libraries are drop-in replacements for the respective CUDA libraries: | HIP | CUDA | ROCm | | ------- | ------ | ------ | | hipblas | cublas | rocblas | | hipfft | cufft | rocfft | | hipsparse | cusparse | rocsparse | | hiprand | curand | rocrand | | hipsolver | cusolver | rocsolver | ##### Third-party libraries - [ginkgo](https://ginkgo-project.github.io/) (sparse linear solvers) - [magma](https://icl.utk.edu/magma/) (dense linear algebra: BLAS, LAPACK) - [heFFTe](https://icl.utk.edu/fft/) (distributed FFTs) #### Performance profilers ##### GPU kernel performance - AMD's [ROCm Compute Profiler](https://rocm.docs.amd.com/projects/rocprofiler-compute/en/latest/index.html) `rocprof-compute`, previously called Omniperf, is part of the `rocm` module starting from [the ROCm 6.3.0 release](https://rocm.docs.amd.com/en/docs-6.3.0/about/release-notes.html#rocm-compute-profiler-and-rocm-systems-profiler) - [ROCProfiler](https://rocm.docs.amd.com/projects/rocprofiler/en/latest/index.html) (command `rocprof`, is part of the ROCm installation in the `rocm` module) ##### Overall application performance - AMD's [ROCm Systems Profiler](https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/index.html) `rocprof-sys-*`, previously called Omnitrace, is part of the `rocm` module starting from [the ROCm 6.3.0 release](https://rocm.docs.amd.com/en/docs-6.3.0/about/release-notes.html#rocm-compute-profiler-and-rocm-systems-profiler) - Linaro MAP (load `linaro_ddt` module) # Raven User Guide ```{contents} Contents :local: :depth: 2 ``` ## System Overview The final expansion stage of the RAVEN HPC system was put into operation in June 2021 and comprises 1592 compute nodes with Intel Xeon IceLake-SP processors ([Platinum 8360Y](https://ark.intel.com/content/www/us/en/ark/products/212459/intel-xeon-platinum-8360y-processor-54m-cache-2-40-ghz.html)) with 72 cores and 256 GB RAM per node. A subset of 64 nodes is equipped with 512 GB RAM and 4 nodes with 2048 GB RAM. In addition, Raven provides 192 GPU-accelerated compute nodes, each with 4 Nvidia A100 GPUs (4 × 40 GB HBM2 memory per node and NVLink). The nodes are interconnected with a Mellanox HDR InfiniBand network (100 Gbit/s) using a pruned fat-tree topology with four non-blocking islands (720 CPU nodes with 256 GB RAM, 660 CPU nodes with 256 GB RAM, 192 GPU nodes plus 64 CPU nodes with 512 GB RAM and 4 CPU nodes with 2 TB RAM, 144 CPU nodes with 256 GB RAM). The GPU nodes are interconnected with at least 200 GBit/s. In addition, there are 2 login nodes and an I/O subsystem that serves 7 PB of disk storage with direct HSM access. Summary: 1592 CPU compute nodes, 114624 CPU cores, 421 TB DDR RAM, 8.8 PFlop/s theoretical peak performance (FP64), plus 192 GPU-accelerated compute nodes 768 GPUs, 30 TB HBM2, 14.6 PFlop/s theoretical peak performance (FP64). ![MPCDF Raven Deployment](_static/raven_final-2021.jpg "MPCDF Raven") ## Access ### Login For security reasons, direct login to the HPC system Raven is allowed only from within certain MPG networks. Users from other locations have to log in to one of our [gateway systems](gateways) first. ```bash ssh @gate.mpcdf.mpg.de ``` Use `ssh` to connect to Raven: ```bash ssh @raven01i.mpcdf.mpg.de ``` Next to `raven01i.mpcdf.mpg.de` there is also an identical login node `raven02i.mpcdf.mpg.de` available, as well as two more login nodes, `raven03i.mpcdf.mpg.de` and `raven04i.mpcdf.mpg.de`, which grant a bit more resources to users (see below). These are often less utilized, consider working there as well. You must provide your MPCDF password and an OTP on the Raven login nodes. SSH keys are not allowed. Secure copy (scp) can be used to transfer data to/from e.g. `raven01i.mpcdf.mpg.de` as well as the other mentioned nodes. Raven's (all login/interactive nodes) ssh key fingerprints (SHA256) are: ```text MrZnFLM64Zz+rZrNRxXdoTfN8lgppZnFdWo2XpRsQts (RSA) SRtUsiak+twYo1Ok9rd5AZATZT4Z5+9MqJHrxwss78g (ED25519) ``` ### Resource limits The login nodes `raven01i.mpcdf.mpg.de` and `raven02i.mpcdf.mpg.de` are intended only for editing, compiling and submitting parallel programs. Running parallel programs interactively on the login nodes is not allowed. Per user, the CPU resources are restricted to an equivalent of two physical CPU cores, memory usage to at most 10% of the available memory, and the number of tasks (including threads) to a maximum of 768. The login nodes `raven03i.mpcdf.mpg.de` and `raven04i.mpcdf.mpg.de` are also primarily intended only for editing, compiling and submitting parallel programs - but here, per user, the CPU resources are restricted to an equivalent of six physical CPU cores, memory usage to at most 20% of the available memory, and the number of tasks (including threads) to a maximum of 1536. Jobs have to be submitted to the Slurm batch system which reserves and allocates the resources (e.g. compute nodes) required for your job. Further information on the batch system is provided [below](#slurm-batch-system). ### Interactive (debug) runs To test or debug your code you may run your code interactively by using the Slurm partition “interactive” (2 hours at most) with the command: ```bash srun -n N_TASKS -p interactive --time=TIME_LESS_THAN_2HOURS --mem=MEMORY_LESS_THAN_32G ./EXECUTABLE ``` Users must ensure that the machine does not become overloaded. It is not allowed to use more than 8 cores in total and to request more than 32 GB of main memory. Ignoring these limits may cause a system crash or hangup! To test and optimize your GPU codes one can use the "gpudev" partition by specifying ```bash #SBATCH --partition=gpudev ``` in your submit script. Only one node with four A100 GPUs is available in the "gpudev" partition. The maximum execution time is 15 minutes. Between one and four GPUs can be requested like for the usual GPU jobs. ### Internet access Connections to the Internet are only permitted from the login nodes in outgoing direction; Internet access from within batch jobs is not possible. To download source code or other data, command line tools such as `wget`, `curl`, `rsync`, `scp`, `pip`, `git`, or similar may be used interactively on the login nodes. In case the transfer is expected to take a long time it is useful to run it inside a `screen` or `tmux` session. ## Hardware configuration ### Compute nodes CPU nodes: - 1592 compute nodes - Processor type: [Intel Xeon IceLake Platinum 8360Y](https://ark.intel.com/content/www/us/en/ark/products/212459/intel-xeon-platinum-8360y-processor-54m-cache-2-40-ghz.html) - Processor base frequency: 2.4 GHz - Cores per node: 72 (each with 2 hyperthreads, thus 144 logical CPUs per node) - Main memory (RAM) per node: 256 GB (1524 nodes), 512 GB (64 nodes), 2048 GB (4 nodes) - *Theoretical* peak performance per node (FP64, "double precision"): 2.4 GHz \* 32 DP Flops/cycle \* 72 = 5530 GFlop/s - 2 NUMA domains with 36 physical cores each GPU nodes: - 192 GPU-accelerated nodes (each hosting 4 Nvidia A100 GPUs, interlinked with NVlink 3) - GPU type: [Nvidia A100 NVlink](https://www.nvidia.com/en-us/data-center/a100/) 40 GB HBM2, CUDA compute capability 8.0 / Ampere - CPU host: Intel Xeon IceLake Platinum 8360Y with 72 CPU cores and 512 GB per node ### Login and interactive nodes - 2 nodes for login and code compilation (Hostnames `raven01i.mpcdf.mpg.de` and `raven02i.mpcdf.mpg.de`) - 2 nodes for interactive program development and testing (Hostnames `raven03i.mpcdf.mpg.de` and `raven04i.mpcdf.mpg.de`) - Processor type: [Intel Xeon IceLake Platinum 8360Y](https://ark.intel.com/content/www/us/en/ark/products/212459/intel-xeon-platinum-8360y-processor-54m-cache-2-40-ghz.html) - Cores per node: 72 (144 logical CPUs) - Main memory (RAM) per node: login nodes: 512 GB, interactive nodes: 256 GB ### Interconnect - Mellanox InfiniBand HDR network connecting all the nodes using a pruned fat-tree topology with four non-blocking islands - CPU nodes: 100 Gb/s (HDR100) - GPU nodes: 200 Gb/s (HDR200), 400 Gb/s (2x HDR200) for a subset of 32 nodes ### I/O subsystem - 7 PB of online disk space ### Additional hardware details Additional details on the Raven hardware are given on a [separate documentation page](raven-details.md). ## File systems ### $HOME Your home directory is in the GPFS file system `/u` (see below). ### GPFS There are two global, parallel file systems of type [GPFS](https://www.ibm.com/products/spectrum-scale) (`/u` and `/ptmp`), symmetrically accessible from all Raven cluster nodes, plus the migrating file system `/r` interfacing to the HPSS archive system. #### File system `/u` The file system `/u` (a symbolic link to `/raven/u`) is designed for permanent user data (source files, config files, etc.). The size of `/u` is 0.9 PB mirrored. Your home directory is in `/u`. The default disk quota in `/u` is 2.5 TB, the file quota is 1 mio files. You can check your disk quota in `/u` with the command: ```bash /usr/lpp/mmfs/bin/mmlsquota raven_u ``` #### File system `/ptmp` The file system `/ptmp` (a symbolic link to /raven/ptmp) is designed for batch job I/O. (12 PB, **no system backups**) Files in `/ptmp` that have not been accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary (with prior notice). As a current policy, no quotas are applied on `/ptmp`. This gives users the freedom to manage their data according to their actual needs without administrative overhead. This liberal policy presumes a fair usage of the common file space. So, please do a regular housekeeping of your data and archive/remove files that are not currently in use. Archiving data from the GPFS file systems to tape can be done using the migrating file system `/r` (see below). #### File system `/r` The `/r` file system (a symbolic link to `/ghi/r`) stages archive data. It is available only on the login nodes `raven.mpcdf.mpg.de` and on the interactive nodes `raven-i.mpcdf.mpg.de`. Each user has a subdirectory `/r//` to store data. For efficiency, files should be packed to tar files (with a size of about 1 GB to 1 TB) before archiving them in `/r`, i.e., please avoid archiving small files. When the file system `/r` gets filled above a certain value, files will be transferred from disk to tape, beginning with the largest files which have not been used for the longest time. For documentation on how to use the MPCDF archive system, please see the [backup and archive section](../data/backup-archive/index.md). #### /tmp Please don't use the file system `/tmp` or `$TMPDIR` for scratch data. Instead, use `/ptmp` which is accessible from all Raven cluster nodes. In cases where an application really depends on node-local storage, you can use the variables `JOB_TMPDIR` and `JOB_SHMTMPDIR`, which are set individually for each job. ## Software ### Access to software via environment modules Environment modules are used at MPCDF to provide software packages and enable switching between different software versions. Use the command ```bash module avail ``` to list the available software packages on the HPC system. Note that you can search for a certain module by using the `find-module` tool (see below). Use the command ```bash module load package_name/version ``` to actually load a software package at a specific version. Further information on the environment modules on Raven and their hierarchical organization is given below. Information on the software packages provided by the MPCDF is available [here](software/index.md). ### Recommended compiler and MPI stack on Raven As explained below, **no defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login**. We currently recommend using the following versions on Raven: ```bash module load intel/2026.0 impi/2021.18 ``` Previously recommended versions are: ```bash module load intel/21.4.0 impi/2021.4 # until 2026/08 ``` ```bash module load intel/21.2.0 impi/2021.2 # until 2022/01 ``` If you want to use GPU-aware MPI, we recommend the following OpenMPI-based software stack: ```bash module load gcc/15 cuda/13.2 openmpi_gpu/5.0 ``` Previously recommended versions are: ```bash module load gcc/13 cuda/12.6 openmpi_gpu/5.0 # until 2026/08 ``` ### Hierarchical module environment To manage the plethora of software packages resulting from all the relevant combinations of compilers and MPI libraries, we organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the uppermost level, dependent libraries (e.g., MPI) on the second level, and more dependent libraries on a third level. This means that not all the modules are visible initially: Only after loading a compiler module, the modules depending on this will become available. And similarly, loading an MPI module in addition will make the modules depending on the MPI library available. Starting with the HPC system Raven, no defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login. This forces users to specify explicit versions for those modules during compilation and in the batch scripts to ensure that the same MPI library is loaded. This also means that users can decide themselves when they use newer compiler and MPI versions for their code which avoids compatibility problems when changing defaults centrally. For example, the FFTW library compiled with the Intel compiler and the Intel MPI library can be loaded as follows: First, load the Intel compiler module using the command ```bash module load intel/2026.0 ``` second, the Intel MPI module with ```bash module load impi/2021.18 ``` and, finally, the FFTW module fitting exactly to the compiler and MPI library via ```bash module load fftw-mpi ``` You may check by using the command ```bash module avail ``` that after the first and second steps the depending environment modules become visible, in the present example impi and fftw-mpi. Moreover, note that the environment modules can be loaded via a single 'module load' statement as long as the order given by the hierarchy is correct, e.g., ```bash module load intel/2026.0 impi/2021.18 fftw-mpi ``` It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the uppermost hierarchy. Note that a hierarchy exists for depending Python modules via the 'anaconda' module files on the top level. Because of the hierarchy, some modules only appear after other modules (such as compiler and MPI) have been loaded. One can search all available combinations of a certain software (e.g. fftw-mpi) by using ```bash find-module fftw-mpi ``` Further information on using environment modules is given [here](software/environment-modules.md). ## Slurm batch system The batch system on the HPC cluster Raven is the open-source workload manager Slurm (Simple Linux Utility for Resource management). To run test or production jobs, submit a job script (see below) to Slurm, which will find and allocate the resources required for your job (e.g. the compute nodes to run your job on). By default, the job run limit is set to 8 on Raven, the default job submit limit is 300. If your batch jobs can't run independently from each other, please use job steps. There are mainly two types of batch jobs: - Exclusive, where all resources on the nodes are allocated to the job - Shared, where several jobs share the resources of one node. In this case it is necessary that the number of CPUs and the amount of memory are specified for each job. The Intel processors on Raven support hyperthreading which *might* increase the performance of your application by up to 20%. To use hyperthreading, you have to increase the product of the number of MPI tasks per node and the number of threads per MPI task from 72 to 144 in your job script. Please be aware that with 144 MPI tasks per node each process gets only half of the memory compared to the non-hyperthreading job by default. If you need more memory, you have to specify it in your job script (see the example batch scripts). Overview of the available per-job resources on Raven: ```text Job type Max. CPUs Number of GPUs Max. Memory Number Max. Run per Node per node per Node of Nodes Time ============================================================================================= shared cpu 36 / 72 in HT mode 120 GB < 1 24:00:00 ............................................................................................ 18 / 36 in HT mode 1 125 GB < 1 24:00:00 shared gpu 36 / 72 in HT mode 2 250 GB < 1 24:00:00 54 / 108 in HT mode 3 375 GB < 1 24:00:00 --------------------------------------------------------------------------------------------- 240 GB 1-360 24:00:00 exclusive cpu 72 / 144 in HT mode 500 GB 1-64 24:00:00 2048 GB 1-4 24:00:00 ............................................................................................ exclusive gpu 72 / 144 in HT mode 4 500 GB 1-80 24:00:00 exclusive gpu bw 72 / 144 in HT mode 4 500 GB 1-16 24:00:00 --------------------------------------------------------------------------------------------- ``` If an application needs more than 240 GB per node, the required amount of memory has to be specified in the Slurm submit script, e.g. with the following options: ```text #SBATCH --mem=500000 # to request up to 500 GB or #SBATCH --mem=2048000 # to request up to 2 TB ``` A job submit filter will automatically choose the right partition and job parameters from the resource specification. Interactive testing and debugging is possible on the nodes `raven-i.mpcdf.mpg.de` (`raven[03-06]i`) by using the command: ```bash srun -n N_TASKS -p interactive ./EXECUTABLE ``` Interactive jobs are limited to 8 cores, 256000M memory and 2 hours runtime. For detailed information about the Slurm batch system, please see [Slurm Workload Manager](https://slurm.schedmd.com/). The most important Slurm commands are - `sbatch ` Submit a job script for execution - `squeue` Check the status of your job(s) - `scancel ` Cancel a job - `sinfo` List the available batch queues (partitions). Do not run Slurm client commands from loops in shell scripts or other programs. Ensure that programs limit calls to these commands to the minimum necessary for the information you are trying to gather. Sample Batch job scripts can be found below. Notes on job scripts: - The directive ```text #SBATCH --nodes= ``` in your job script specifies the number of compute nodes that your program will use. - The directive ```text #SBATCH --ntasks-per-node= ``` specifies the number of MPI processes for the job. The parameter ntasks-per-node cannot be greater than 72 because one compute node on Raven has 72 cores with 2 threads each, thus 144 logical CPUs in hyperthreading mode. - The directive ```text #SBATCH --cpus-per-task= ``` specifies the number of threads per MPI process if you are using OpenMP. - The expression ```text ntasks-per-node * cpus-per-task ``` may not exceed 144. - The expression ```text nodes * ntasks-per-node * cpus-per-task ``` gives the total number of CPUs that your job will use. - To select either GPU nodes with standard (200 GBit/s) or with high-bandwidth (400 GBit/s) network interconnect, specify a job constraint as follows: ```text #SBATCH --constraint="gpu" # for gpu nodes with either 200 GBit/s or 400 GBit/s network connection # or #SBATCH --constraint="gpu-bw" # for gpu nodes with 400 GBit/s network connection # or #SBATCH --constraint="no-gpu-bw" # for gpu nodes with 200 GBit/s network connection ``` - For multi-process GPU jobs, [NVIDIA MPS](https://docs.nvidia.com/deploy/mps/index.html) can be launched using the command line flag `--nvmps` for `sbatch`. See the example scripts below for details. - Jobs that need less than a half compute node have to specify a reasonable memory limit so that they can share a node! - A job submit filter will automatically choose the right partition/queue from the resource specification. - Please note that setting the environment variable 'SLURM_HINT' in job scripts is not necessary and discouraged on Raven. ## Slurm example batch scripts ### MPI and MPI/OpenMP batch scripts #### MPI batch job without hyperthreading ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=72 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 # Run the program: srun ./myprog > prog.out ``` #### Hybrid MPI/OpenMP batch job without hyperthreading ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job_hybrid.out.%j #SBATCH -e ./job_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=4 # for OpenMP: #SBATCH --cpus-per-task=18 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly: export OMP_PLACES=cores # Run the program: srun ./myprog > prog.out ``` #### Hybrid MPI/OpenMP batch job in hyperthreading mode ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job_hybrid.out.%j #SBATCH -e ./job_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=32 #SBATCH --ntasks-per-node=4 # Enable Hyperthreading: #SBATCH --ntasks-per-core=2 # for OpenMP: #SBATCH --cpus-per-task=36 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock Limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly: export OMP_PLACES=threads # Run the program: srun ./myprog > prog.out ``` #### Small MPI batch job on a shared node ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of MPI Tasks, e.g. 8: #SBATCH --ntasks=8 # Memory usage [MB] of the job is required, e.g. 3000 MB per task: #SBATCH --mem=24000 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 # Run the program: srun ./myprog > prog.out ``` ### Batch jobs using GPUs Note that computing time on GPU-accelerated nodes is accounted using a weighting factor of 4 relative to CPU-only jobs, corresponding to the additional computing power provided by the GPUs. Users are advised to check the performance reports of their jobs in order to monitor adequate utilization of the resources. #### GPU job using 1, 2, or 4 GPUs on a single node The following example job script launches a (potentially multithreaded) CUDA program to use one (or more) GPU(s) on a single node. In case more than one GPUs are requested the user code must be able to utilize these additional GPUs properly. ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_gpu # #SBATCH --ntasks=1 #SBATCH --constraint="gpu" # # --- default case: use a single GPU on a shared node --- #SBATCH --gres=gpu:a100:1 #SBATCH --cpus-per-task=18 #SBATCH --mem=125000 # # --- uncomment to use 2 GPUs on a shared node --- # #SBATCH --gres=gpu:a100:2 # #SBATCH --cpus-per-task=36 # #SBATCH --mem=250000 # # --- uncomment to use 4 GPUs on a full node --- # #SBATCH --gres=gpu:a100:4 # #SBATCH --cpus-per-task=72 # #SBATCH --mem=500000 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=12:00:00 module purge module load gcc/15 cuda/13.2 openmpi_gpu/5.0 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun ./cuda_executable ``` #### Hybrid MPI/OpenMP job using one or more nodes with 4 GPUs each The following example job script launches a hybrid MPI/OpenMP-CUDA-code on one (or more) nodes running one task per GPU. Note that the user code needs to attach its tasks to the different GPUs based on some code-internal logic. In case more than one MPI task is accessing a GPU it is necessary to enable NVIDIA MPS using the flag `#SBATCH --nvmps` as shown in the plain MPI-CUDA example below. The flag `#SBATCH --constraint="gpu-bw"` may be used to request nodes with high-bandwidth network interconnect. ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_gpu # #SBATCH --nodes=1 # Request 1 or more full nodes #SBATCH --constraint="gpu" # providing GPUs. #SBATCH --gres=gpu:a100:4 # Request 4 GPUs per node. #SBATCH --ntasks-per-node=4 # Run one task per GPU #SBATCH --cpus-per-task=18 # using 18 cores each. #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=12:00:00 module purge module load gcc/15 cuda/13.2 openmpi_gpu/5.0 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun ./mpi_openmp_cuda_executable ``` #### Plain MPI job using GPUs The following example job script launches an MPI-CUDA-code on one (or more) nodes with one MPI task per CPU core. Note that the user code needs to attach its tasks across the different GPUs based on some code-internal logic. Moreover, note that it is necessary to launch NVIDIA MPS via the flag `#SBATCH --nvmps` to enable the MPI tasks access the GPUs in an efficient manner concurrently. The flag `#SBATCH --constraint="gpu-bw"` may be used to request nodes with high-bandwidth network interconnect. ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_slurm # #SBATCH --nodes=1 # Request 1 (or more) node(s) #SBATCH --constraint="gpu" # providing GPUs. #SBATCH --ntasks-per-node=72 # Launch 72 tasks per node #SBATCH --gres=gpu:a100:4 # Request all 4 GPUs of each node #SBATCH --nvmps # Launch NVIDIA MPS to enable concurrent access to the GPUs from multiple processes efficiently # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=12:00:00 module purge module load gcc/15 cuda/13.2 openmpi_gpu/5.0 srun ./mpi_cuda_executable ``` ### Batch jobs with dependencies The following script generates a sequence of jobs, each job running the given job script. The start of each individual job depends on its dependency, where possible values for the `--dependency` flag are, e.g. - `afterany:job_id` This job starts after the previous job has terminated - `afterok:job_id` This job starts after previous job has successfully executed ```bash #!/bin/bash # Submit a sequence of batch jobs with dependencies # # Number of jobs to submit: NR_OF_JOBS=6 # Batch job script: JOB_SCRIPT=./my_batch_script echo "Submitting job chain of ${NR_OF_JOBS} jobs for batch script ${JOB_SCRIPT}:" JOBID=$(sbatch ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} I=1 while [ ${I} -lt ${NR_OF_JOBS} ]; do JOBID=$(sbatch --dependency=afterany:${JOBID} ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} let I=${I}+1 done ``` ### Batch job using a job array ```bash #!/bin/bash -l # specify the indexes (max. 30000) of the job array elements (max. 300 - the default job submit limit per user) #SBATCH --array=1-20 # Standard output and error: #SBATCH -o job_%A_%a.out # Standard output, %A = job ID, %a = job array index #SBATCH -e job_%A_%a.err # Standard error, %A = job ID, %a = job array index # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_array # # Number of nodes and MPI tasks per node: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=72 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit (max. is 24 hours): #SBATCH --time=12:00:00 # Load compiler and MPI modules (must be the same as used for compiling the code) module purge module load intel/2026.0 impi/2021.18 # Run the program: # the environment variable $SLURM_ARRAY_TASK_ID holds the index of the job array and # can be used to discriminate between individual elements of the job array srun ./myprog > prog.out ``` ### Single-node example job scripts for sequential programs, plain-OpenMP cases, Python, Julia, Matlab In the following, example job scripts are given for jobs that use at maximum one full node. Use cases are sequential programs, threaded programs using OpenMP or similar models, and programs written in languages such as Python, Julia, Matlab, etc. The Python example programs referred to below are available for [download](https://datashare.mpcdf.mpg.de/s/KCEtd0tP3zLypq4). #### Single-core job ```bash #!/bin/bash -l # # Single-core example job script for MPCDF Raven. # In addition to the Python example shown here, the script # is valid for any single-threaded program, including # sequential Matlab, Mathematica, Julia, and similar cases. # #SBATCH -J PYTHON_SEQ #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH --ntasks=1 # launch job on a single core #SBATCH --cpus-per-task=1 # on a shared node #SBATCH --mem=2000MB # memory limit for the job #SBATCH --time=0:10:00 module purge module load gcc/16 impi/2021.18 module load python-waterboa/2025.06 # Set number of OMP threads to fit the number of available cpus, if applicable. export OMP_NUM_THREADS=1 # Run single-core program srun python3 ./python_sequential.py ``` #### Small job with multithreading, applicable to Python, Julia and Matlab, plain OpenMP, or any threaded application ```bash #!/bin/bash -l # # Multithreading example job script for MPCDF Raven. # In addition to the Python example shown here, the script # is valid for any multi-threaded program, including # Matlab, Mathematica, Julia, and similar cases. # #SBATCH -J PYTHON_MT #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH --ntasks=1 # launch job on #SBATCH --cpus-per-task=8 # 8 cores on a shared node #SBATCH --mem=16000MB # memory limit for the job #SBATCH --time=0:10:00 module purge module load gcc/16 impi/2021.18 module load python-waterboa/2025.06 # Set number of OMP threads to fit the number of available cpus, if applicable. export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun python3 ./python_multithreading.py ``` #### Python/NumPy multitheading, applicable to Julia and Matlab, plain-OpenMP, or any threaded application ```bash #!/bin/bash -l # # Multithreading example job script for MPCDF Raven. # In addition to the Python example shown here, the script # is valid for any multi-threaded program, including # parallel Matlab, Julia, and similar cases. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J PY_MULTITHREADING #SBATCH --nodes=1 # request a full node #SBATCH --ntasks-per-node=1 # only start 1 task via srun because Python multiprocessing starts more tasks internally #SBATCH --cpus-per-task=72 # assign all the cores to that first task to make room for multithreading #SBATCH --time=00:10:00 module purge module load gcc/16 impi/2021.18 module load python-waterboa/2025.06 # set number of OMP threads *per process* export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun python3 ./python_multithreading.py ``` #### Python multiprocessing ```bash #!/bin/bash -l # # Python multiprocessing example job script for MPCDF Raven. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J PYTHON_MP #SBATCH --nodes=1 # request a full node #SBATCH --ntasks-per-node=1 # only start 1 task via srun because Python multiprocessing starts more tasks internally #SBATCH --cpus-per-task=72 # assign all the cores to that first task to make room for Python's multiprocessing tasks #SBATCH --time=00:10:00 module purge module load gcc/16 impi/2021.18 module load python-waterboa/2025.06 # Important: # Set the number of OMP threads *per process* to avoid overloading of the node! export OMP_NUM_THREADS=1 # Use the environment variable SLURM_CPUS_PER_TASK to have multiprocessing # spawn exactly as many processes as you have CPUs available. srun python3 ./python_multiprocessing.py $SLURM_CPUS_PER_TASK ``` #### Python mpi4py ```bash #!/bin/bash -l # # Python MPI4PY example job script for MPCDF Raven. # May use more than one node. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J MPI4PY #SBATCH --nodes=1 #SBATCH --ntasks-per-node=72 #SBATCH --time=00:10:00 module purge module load gcc/16 impi/2021.18 module load python-waterboa/2025.06 module load mpi4py/4.1.1 # Important: # Set the number of OMP threads *per process* to avoid overloading of the node! export OMP_NUM_THREADS=1 srun python3 ./python_mpi4py.py ``` Dedicated clusters ================== .. Note: This is RST format, not markdown, because of the toctrees Dedicated Linux compute clusters are operated for more than 20 Max Planck Institutes and working groups. In order to facilitate migration between the central HPC systems and the dedicated Linux clusters the software stack of the dedicated clusters (including compilers, MPI, libraries) is kept as similar as possible to the software stack on the HPC systems. Contact us if you are considering purchasing a compute cluster for your Max Planck Institute. We are happy to support you each step of the way - from requirements gathering through procurement, set-up, and operations. Specific information is available for the clusters of the following Max Planck Institutes and organizations associated with the Max Planck Society: .. toctree:: :maxdepth: 1 :glob: systems/Astronomy.rst.txt systems/Astrophysics.rst.txt systems/Biochemistry.rst.txt systems/Biological_Cybernetics.rst.txt systems/Biological_Intelligence.rst.txt systems/Biophysics.rst.txt systems/Brain_Research.rst.txt systems/Chemical_Physics_Solids.rst.txt systems/Extraterrestrial_Physics.rst.txt systems/Geoanthropology.rst.txt systems/Gravitational_Physics.rst.txt systems/Gravitational_Physics_ACR.rst.txt systems/Gravitational_Physics_CRA.rst.txt systems/MPSD_PKS_ADA.rst.txt systems/Physics.rst.txt systems/Plasma_Physics.rst.txt systems/Polymer_Research.rst.txt systems/Psychiatry.rst.txt systems/Quantum_Optics.rst.txt systems/Radioastronomy.rst.txt systems/Science_of_Light.rst.txt systems/Sustainable_Materials.rst.txt .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Astronomy ========= ---- Name of the cluster: **VERA** Institution: **Max Planck Institute for Astronomy** Login nodes: ~~~~~~~~~~~~ * vera01.bc.rzg.mpg.de * vera02.bc.rzg.mpg.de Their SHA256 ssh host key fingerprint is: **BCiQsLifb24aMoVJ0yNDIxHIhNfaztAE5DH+wNkn9ZQ (ED25519)** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ * VERA is built on top of Intel Xeon Platinum 8360Y CPUs (36 cores at 2.40GHz), each node is equipped with two 8360Y CPUs * Contrary to RAVEN, VERA is operated with Hyper-Threading disabled, though .. * login nodes vera\[01-02\] (500 GB RAM each) * 72 execution nodes vera\[001-072\] (250 GB RAM each) * 36 execution nodes vera\[101-136\] (500 GB RAM each) * 2 execution nodes vera\[201-202\] (2 TB RAM each) * 3 execution nodes verag\[001-003\] (500 GB RAM and 4 Nvidia A100-40GB GPUs each) * node interconnect is based on Mellanox/Nvidia Infiniband HDR-100 technology (Speed: 100 Gb/s) Filesystems: ~~~~~~~~~~~~ /u - shared home filesystem - user quotas (1 TB of data; 400k files/directories) enforced - quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /vera/ptmp - shared scratch filesystem (2.0 PB) - user quotas enforced (default 5 TB) - quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota' - organized in folders apex, gc and psf - new users should contact their group leader to get a directory - NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. There are no modules preloaded on VERA. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * a brief introduction into the basic commands (srun, sbatch, squeue, scancel, sinfo, s\*...) can be found on the `Raven home page `__ or on the `Slurm handbook `__ * **four partitions:** p.vera (default), p.large, p.huge and p.gpu * **current max. run time (wallclock):** p.vera (2 days), p.large (2 days), p.huge ( 1 day), p.gpu (1 day, default runtime is 12 hours) * **maximum memory per node for jobs:** p.vera (250000 MB), p.large (500000 MB), p.huge (2048000 MB), p.gpu (500000 MB) * **p.vera partition:** nodes are exclusively allocated to users * **p.large, p.huge, p.gpu partitions:** resources on the nodes may be shared between jobs * **p.gpu partition:** to access GPU resources :bolditalic:`\-\-gres` parameter must be explicitly set for jobs * sample batch scripts can be found on `Raven home page `_ (must be modified for VERA) Useful tips ~~~~~~~~~~~ Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation. The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script. Nvidia Ampere GPUs are available in :bolditalic:`p.gpu` partition. Type of gpu must be explicitly set, i.e. :bolditalic:`\-\-gres=gpu:a100:X`, where X is between 1 and 4 GPU cards are in default compute mode. Nodes in :bolditalic:`p.gpu` partition are in shared mode i.e. jobs allocate only requested resources. Default memory per job is 125000 MB. Use :bolditalic:`\-\-mem` parameter to set necessary amount of RAM for jobs. Nodes in :bolditalic:`p.large` and :bolditalic:`p.huge` partitions are in shared mode i.e. jobs allocate only requested resources. By default jobs allocate all memory on nodes. This means that to share node between other jobs :bolditalic:`\-\-mem` parameter is required for jobs. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `_ .. ------------- .. authors: mykp .. ------------- Astrophysics ============ ---- Name of the clusters: .. toctree:: :maxdepth: 1 :glob: Astrophysics/MPA-FREYA.rst.txt Astrophysics/MPA-ORION.rst.txt Astrophysics/MPA-VIRGO.rst.txt Institution: **Max Planck Institute for Astrophysics** .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Astrophysics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Astrophysics FREYA ================== ---- Name of the cluster: **FREYA** Institution: **Max Planck Institute for Astrophysics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - freya01.bc.mpcdf.mpg.de - freya02.bc.mpcdf.mpg.de - freya03.bc.mpcdf.mpg.de - freya04.bc.mpcdf.mpg.de Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ - login nodes freya[01-04] : 2 x Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz; 40 cores per node; 384 GB RAM .. - 100 execution nodes freya[073-104,109-176] for parallel computing with a total amount of 6880 CPU cores; 2 x Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz; 192 GB RAM - 4 execution nodes freya[104-108] for parallel computing with a total amount of 160 CPU cores; 2 x Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz; 384 GB RAM - 8 execution nodes freyag[01-08] for parallel GPU computing with a total amount of 320 CPU cores; 2 x Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz; 384 GB RAM; 2 x Nvidia Tesla P100-PCIE-16GB GPUs per node - 4 execution nodes freyag[09-12] for parallel GPU computing with a total amount of 160 CPU cores; 2 x Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz; 384 GB RAM; 2 x Nvidia Tesla V100-PCIE-32GB GPUs per node - 11 execution nodes freyag[201-211] for parallel GPU computing with a total amount of 480 CPU cores; 2 x Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz; 384 GB RAM; 4 x Nvidia Tesla A100-PCIE-40GB GPUs per node .. - node interconnect is based on Intel Omni-Path Fabric (Speed: 100Gb/s) Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; GPFS-based; user quotas (currently 900 GB, 1M files) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /freya/ptmp shared scratch filesystem (1.7 PB); GPFS-based; no quotas enforced. NO BACKUPS! /virgotng shared scratch filesystem (8.0 PB); GPFS-based; no quotas enforced. NO BACKUPS Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on FREYA. Please use 'module available' to see all available modules. | Similar to the HPC systems, this module tree is `hierarchical `__. | To find a module and information about the available versions or what dependencies need to be loaded first one can use the ‘find-module’ command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - sbatch, srun, squeue, sinfo, scancel, scontrol, s\* - current max. turnaround time (wallclock): 24 hours - max. nodes limit per user: 92 - four partitions: p.24h (default), p.test, p.gpu & p.gpu.ampere - p.test partition: has 4 nodes with 2 Nvidia Pascal gpus and 30 min run time - sample batch scripts can be found on `Cobra home page `_ (must be modified for FREYA) Useful tips: ~~~~~~~~~~~~ Nodes in :bolditalic:`p.test` partition are in shared mode, default memory per job set to 9500 MB. To allocate necessary amount of memory use :bolditalic:`\-\-mem` parameter. Nvidia Pascal and Volta GPUs are available in :bolditalic:`p.gpu` partition. To use them add in your slurm scripts :bolditalic:`#SBATCH -p p.gpu` and choose how many GPUs to have :bolditalic:`#SBATCH \-\-gres=gpu:1` or :bolditalic:`#SBATCH \-\-gres=gpu:2` To use Volta or Pascal GPUs add type of GPUs into the :bolditalic:`\-\-gres` parameter: :bolditalic:`\-\-gres=gpu:p100:1` or :bolditalic:`\-\-gres=gpu:v100:2` Nodes in :bolditalic:`p.gpu` partition are in exclusive mode i.e. jobs allocate entire nodes. Nvidia Ampere GPUs are available in :bolditalic:`p.gpu.ampere` partition. Type of gpu must be explicitly set, i.e. :bolditalic:`\-\-gres=gpu:a100:X`, where X is between 1 and 4 Nodes in :bolditalic:`p.gpu.ampere` partition are in shared mode i.e. jobs allocate only requested resources. Default memory per job is 95000 MB. Use :bolditalic:`\-\-mem` parameter to set necessary amount of RAM for jobs. GPU cards are in default compute mode. To run code on nodes with different memory capacity (*mem192G; mem384G*) use :bolditalic:`\-\-constraint` option in a sbatch script: :bolditalic:`#SBATCH \-\-constraint=mem192G` or :bolditalic:`#SBATCH \-\-constraint=mem384G` To check node features, general resources and scheduling weight of nodes use :bolditalic:`sinfo -O nodelist,features:25,gres,weight` Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `_ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Astrophysics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Astrophysics ORION ================== ---- Name of the cluster: **ORION** Institution: **Max Planck Institute for Astrophysics** Login nodes: ~~~~~~~~~~~~ - **orion01.bc.mpcdf.mpg.de** - **orion02.bc.mpcdf.mpg.de** Their SHA256 ssh host key fingerprint is: **SHA256:1puEKbiOBuwsd4ak/5WQmL1NhfBysAUN1HJpnboghqE (ED25519)** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ - login nodes orion0[12] : 2 x Intel(R) Xeon(R) Platinum 8480+ CPU @ 2.00GHz; 112 cores per node; 2 threads per core; 1 TB RAM .. - 104 execution nodes orion[001-104] for parallel computing with a total amount of 11648 CPU cores; 2 x Intel(R) Xeon(R) Platinum 8480+ CPU @ 2.00GHz; 512 GB RAM; Turbo-mode is on .. - node interconnect is based on Infiniband Fabric (Speed: 200Gb/s) Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; GPFS-based; user quotas (currently 100 GB, 1M files) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /orion/ptmp shared scratch filesystem (5.0 PB); GPFS-based; user quotas (currently 20TB) enforced. NO BACKUPS! /virgotng shared scratch filesystem (8.0 PB); GPFS-based; no quotas enforced. NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on ORION. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icx, icpx, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL ('module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI 2024.0 ('module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ | Similar to the HPC systems, this module tree is `hierarchical `__. | To find a module and information about the available versions or what dependencies need to be loaded first one can use the ‘find-module’ command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - sbatch, srun, squeue, sinfo, scancel, scontrol, s\* - current max. turnaround time (wallclock): 24 hours - two partition: p.exclusive (default) and p.shared (6 nodes) for shared resources - max. nodes limit per user in p.exclusive partition: 30 - sample batch scripts can be found on `Raven home page `_ (must be modified for ORION) Useful tips: ~~~~~~~~~~~~ Nodes in :bolditalic:`p.shared` partition are in shared mode, default memory per job set to 32000 MB. To allocate necessary amount of memory use :bolditalic:`\-\-mem` parameter. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `_ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Astrophysics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Astrophysics VIRGO2024 ====================== ---- Name of the server: **VIRGO2024** Institution: **Max Planck Institute for Astrophysics** Login nodes: ~~~~~~~~~~~~ - **virgo2024.bc.mpcdf.mpg.de** Its SHA256 ssh host key fingerprint is: **SHA256:isyTpjgQgAQ15a3IAVziZLTOgYeTcJfW8NJGq44LcyM (ED25519)** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ - virgo2024 : 2 x Intel(R) Xeon(R) Platinum 8480+ @ 2.00GHz; 112 cores per node; 2 threads per core; 2 TB RAM Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; GPFS-based; user quotas (currently 900 GB, 1M files) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota freya_u'. /virgotng shared scratch filesystem (8.0 PB); GPFS-based; no quotas enforced. NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on ORION. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icx, icpx, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL ('module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI 2024.0 ('module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ | Similar to the HPC systems, this module tree is `hierarchical `__. | To find a module and information about the available versions or what dependencies need to be loaded first one can use the ‘find-module’ command. Batch system: ~~~~~~~~~~~~~ - no workload manager is available on VIRGO - limits in terms of CPU shares are set Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `_ .. ------------- .. authors: mykp .. ------------- Biochemistry ============ ---- Name of the clusters: .. toctree:: :maxdepth: 1 :glob: Biochemistry/Biochemistry-HPCL8.rst.txt Institution: **Max Planck Institute of Biochemistry** .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Biochemistry-HPCL8 .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biochemistry HPCL8 ================== ---- Name of the cluster: **HPCL8** Institution: **Max Planck Institute of Biochemistry** Login nodes: ~~~~~~~~~~~~ +-----------------------------+-----------------------------+----------------------------+ | - hpcl8001.bc.rzg.mpg.de | - hpcl8061.bc.rzg.mpg.de | - hpcl9001.bc.rzg.mpg.de | | - hpcl8002.bc.rzg.mpg.de | - hpcl8062.bc.rzg.mpg.de | - hpcl9002.bc.rzg.mpg.de | | - hpcl8003.bc.rzg.mpg.de | - hpcl8063.bc.rzg.mpg.de | | | - hpcl8004.bc.rzg.mpg.de | | | | - hpcl9301.bc.rzg.mpg.de | | | +-----------------------------+-----------------------------+----------------------------+ | Login nodes hpcl[8061-8063] are available for selected users, only (Dept. Conti) | Login nodes hpcl[9001-9002] are available for selected users, only (Dept. Briggs) Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :7 login nodes hpcl[8001-8004] & hpcl[8061-8063]: | 2 x Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz | 24 cores per node | hyper-threading disabled - 1 threads per core | 377 GB RAM; | 2 x RTX 5000 GPUs | **node interconnect**: based on 25 Gb/s ethernet :56 execution nodes hpcl[8005-8060] for parallel CPU/GPU computing: | total amount of 1344 CPU cores | 2 x Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz | 24 cores per node | hyper-threading disabled - 1 threads per core | 377 GB RAM | 2 x RTX 5000 GPUs | **node interconnect**: based on 25 Gb/s ethernet :2 login nodes hpcl[9001-9002]: | 2 x Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz | 72 cores per node | hyper-threading disabled - 1 threads per core | 1 TB RAM | 4 x NVIDIA A40 GPUs | **node interconnect**: based on 50 Gb/s ethernet :9 execution nodes hpcl[9003-9011] for parallel CPU/GPU computing: | total amount of 648 CPU cores | 2 x Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz | 72 cores per node | hyper-threading disabled - 1 threads per core | 1 TB RAM | 4 x NVIDIA A40 GPUs | **node interconnect**: based on 50 Gb/s ethernet :4 execution nodes hpcl[9101-9104] for parallel CPU/GPU computing: | total amount of 304 CPU cores | Intel(R) Xeon(R) Platinum 8368 CPU @ 2.40GHz | 76 cores per node | hyper-threading enabled - 2 threads per core | 1 TB RAM | 4 x NVIDIA H100 GPUs | **node interconnect**: based on 50 Gb/s ethernet :3 execution nodes hpcl[9201-9103] for parallel CPU: | total amount of 192 CPU cores | AMD EPYC 9374F 32-Core CPU @ 3.80GHz | 64 cores per node | hyper-threading enabled - 2 threads per core | 512 GB RAM | **node interconnect**: based on 50 Gb/s ethernet :1 login nodes hpcl9301: | 2 x AMD EPYC 9534 64-Core CPU @ 3.7GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 755 GB RAM | 4 x NVIDIA L40s GPUs | **node interconnect**: based on 50 Gb/s ethernet :19 execution nodes hpcl[9302-9320] for parallel CPU/GPU computing: | total amount of 2432 CPU cores | 2 x AMD EPYC 9534 64-Core CPU @ 3.7GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 755 GB RAM | 4 x NVIDIA L40s GPUs | **node interconnect**: based on 50 Gb/s ethernet Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on HPCL8. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (-> 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ - OpenMPI (-> 'module load openmpi'): mpicc, mpicxx, mpif77, mpif90, mpirun, mpiexec - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on HPCL8 is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for HPCL8 cluster (partition must be changed). Current Slurm configuration on HPCL8: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default run time: *24 hours* - current max. run time (wallclock): *21 days* - four partitions: *p.hpcl8* (default), *p.hpcl9* (Dept. Briggs only) *p.hpcl91* (b_borgwardt group only), *p.hpcl92* (b_mann & g_rz groups only) & *p.hpcl93* - nodes in *p.hpcl8* & *p.hpcl9* partitions are exclusively allocated to users. Multiple jobs may be run for the same user only - nodes in *p.hpcl91*, *p.hpcl92* & *p.hpcl93* can be shared by jobs - default memory size per job on node: *380000 MB* (*p.hpcl8* partition), *1000000 MB* (*p.hpcl9* partition), *40000 MB* (*p.hpcl91* partition), *32000 MB* (*p.hpcl92* partition) & *38000 MB* (*p.hpcl93* partition) - max submitted jobs per user: *2000* - max running jobs per user at one time: *200* Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 24 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 504 hours Memory is consumable resource. To run several jobs on one node use :bolditalic:`\-\-mem=` or :bolditalic:`\-\-mem-per-cpu=` options for sbatch/srun, where size should be less than default per node (380000 MB) The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page) To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs to allocate: :bolditalic:`#SBATCH \-\-gres=gpu:1` or :bolditalic:`#SBATCH \-\-gres=gpu:2` | Valid gres options are: **gpu[[:type]:count]** | where | **type** is a type of gpu (*rtx5000*, *a40*, *h100* or *l40s*) | **count** is a number of resources (1 or 2 in *p.hpcl8* partition and 1 - 4 in *p.hpcl9, p.hpcl91 & p.hpcl93* paritions) | GPU cards are in default compute mode. | GPU cards on hpcl[9101-9103] nodes are MIG-ed. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/biological-cybernetics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biological Cybernetics ====================== ---- Name of the cluster: **ERIS** Institution: **Max Planck Institute for Biological Cybernetics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 * eris01.bc.rzg.mpg.de * eris02.bc.rzg.mpg.de Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | | Login nodes | Compute nodes ( 1792 CPU cores) | Compute nodes (256 CPU cores) | Compute nodes (128 CPU cores) | + +---------------------------+---------------------------------+-------------------------------+--------------------------------+ | | 2 login nodes eris[01-02] | 28 compute nodes eris[001-028] | 4 compute nodes eris[101-104] | 2 compute nodes erisg[001-002] | +====================+===========================+=================================+===============================+================================+ | CPU | AMD EPYC 7452 | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | CPU(s) | 128 | 128 | 128 | 64 | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | Thread(s) per core | 2 | 2 | 2 | 2 | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | Core(s) per socket | 32 | 32 | 32 | 32 | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | Socket(s) | 2 | 2 | 2 | 1 | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | RAM | 250 GB | 500 GB | 1 TB | 250 GB | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ | GPU | -- | -- | -- | 4 x Quadro RTX 5000 | +--------------------+---------------------------+---------------------------------+-------------------------------+--------------------------------+ - Interconnect is based on 50Gb ethernet. Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 425 TB and independent inode space for the following filesets: /u shared home filesystem; GPFS-based; user quotas (currently default is 500 GB, 512K files) enforced quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /scratch shared scratch filesystem ; GPFS-based; no quotas enforced NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on ERIS. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (-> 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ... This module becomes visible and loadable only after a compiler module (Intel or GCC) has been loaded - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on ERIS is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Cobra home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for ERIS cluster. Current Slurm configuration on ERIS: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 7 days - two partitions: p.eris and s.eris (default) - p.eris partition: for parallel MPI or hybrid MPI/OpenMP jobs. Resources are exclusively allocated on nodes. Max. nodes per job is 28 - s.eris partition: for serial or OpenMP jobs. Nodes are shared. Jobs are limited to use CPUs only on one node. Default RAM per job is 128 GB Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 24 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours By default jobs use all memory on nodes in p.eris partition. In s.eris partition default allocated memory per job is 128 GB. To grant the job access to use more or less memory on each node use :bolditalic:`\-\-mem` or :bolditalic:`\-\-mem-per-cpu` options for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ \ page) To run code on nodes with different memory capacity (500GB; 1TB) use :bolditalic:`\-\-constraint=` option in a sbatch script: :bolditalic:`\-\-constraint=mem500G` or :bolditalic:`\-\-constraint=mem1T` To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:rtx\_5000:1` or :bolditalic:`#SBATCH \-\-gres=gpu:1` | Valid gres options are: **gpu[[:type]:count]** | where | **type** is a type of gpu (*rtx\_5000)* | **count** is a number of resources ( between 1 and 4) GPU cards are in default compute mode. To check node features use :bolditalic:`sinfo -O nodelist,features:30` Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biological Intelligence ======================= ---- Name of the clusters: **CAJAL** Institution: **Max Planck Institute for Biological Intelligence** Login nodes: ~~~~~~~~~~~~ .. list-table:: * - **cajalg001.wb.mpcdf.mpg.de** - **cajalg002.wb.mpcdf.mpg.de** SHA256 ssh host key fingerprints are: * **cajalg001: 3kIzXDB7ZDpMcrmFcxtxJ9c6qlHfPy42we6YsvD8x8o (ED25519)** * **cajalg002: /Y9+CyjtSf2MIeLWDmfrmarSWipp/RxQx3ddmABHe90 (ED25519)** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 Login node **cajalg[001-002]**: | total amount of 128 CPU cores | Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz | 64 cores per node | hyper-threading disabled - 1 threads per core | 1 TB RAM | 2 x NVIDIA A40 GPUs per node :51 execution nodes cajalg[003-053] for parallel computing: | total amount of 3264 CPU cores | Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz | 64 cores per node | hyper-threading disabled - 1 threads per core | 1 TB RAM | 2 x NVIDIA A40 GPUs per node :4 execution nodes cajalg[201-204] for parallel computing: | total amount of 256 CPU cores | Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz | 64 cores per node | hyper-threading disabled - 1 threads per core | 1 TB RAM | 8 x NVIDIA A40 GPUs per node :Node interconnect: | based on 25000Mb/s ethernet Filesystems: ~~~~~~~~~~~~ /u * shared GPFS-based home filesystem * user quotas (256GB of data; 500k files/directories) enforced /cajal/scratch/users/$USERNAME * shared GPFS-based scratch filesystem * quoted to 100TB of data and 1M files/directories * NO BACKUPS! /cajal/nvmescratch/users/$USERNAME * shared GPFS-based scratch filesystem * quoted to 5TB of data and 1M files/directories * NO BACKUPS! /wholebrain * shared GPFS-Based filesystem * will be converted to read-only end of 2022 * NO BACKUPS! Quota can be checked with :bolditalic:`/usr/lpp/mmfs/bin/mmlsquota` Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. There are no modules preloaded on CAJAL. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on CAJAL is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, sinfo, s\*...) can be found on the `Raven home page `__ or on the `Slurm handbook `__ Current Slurm configuration on CAJAL: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :three partitions: | **p.cajal** (for parallel/exclusive jobs), **p.share** (**default**, for serial jobs), **p.large** ( with 8 gpus per node) :run time (wallclock): | **2 days** (default), **7 days** (max. run time) :maximum memory per node for jobs: | 1024000 MB :default memory per node for jobs: | **1024000 MB** (p.cajal, p.large), **256000 MB** (p.share) :p.cajal partition: | nodes are exclusively allocated to users :p.share and p.large partitions: | resources on the nodes may be shared between jobs :all partitions: | to access GPU resources :bolditalic:`\-\-gres` parameter must be explicitly set for jobs Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 48 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 168 hours In shared partition **p.share** the default memory per node is 256000MB. To specify the real memory required per node use :bolditalic:`\-\-mem` option (see also :bolditalic:`\-\-mem-per-cpu` in case of multithreaded job). The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example can be found on `help information `__ \ page) | To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` options and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:a40:1` or :bolditalic:`#SBATCH \-\-gres=gpu:a40:2` | Valid gres options are: **gpu\[\[:type\]:count\]** | where | **type** is a type of gpu (*a40*) | **count** is a number of resources (*1<=N<=2* in **p.cajal** partition and *1<=N<=8* in **p.large** partition) GPU cards are in default compute mode. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. ------------- Biophysics ========== ---- Name of the clusters: .. toctree:: :maxdepth: 1 :glob: Biophysics/Biophysics-BIO.rst.txt Biophysics/Biophysics-CHEM.rst.txt Biophysics/Biophysics-CRYO.rst.txt Biophysics/Biophysics-LEO.rst.txt Biophysics/Biophysics-PHYS.rst.txt Institution: **Max Planck Institute of Biophysics** .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/biophysics-bio .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biophysics BIO ============== ---- Name of the cluster: **BIO** Institution: **Max Planck Institute of Biophysics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - bio01.tbc.biophys.mpg.de - bio02.tbc.biophys.mpg.de Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ +-------------------+--------------------------+--------------------------+ | | Login nodes | Compute nodes ( 2760 CPU | | | | cores) | +===================+==========================+==========================+ | | 2 login nodes bio[01-02] | 69 compute nodes | | | | bio[001-069] | +-------------------+--------------------------+--------------------------+ | CPU | Intel(R) Xeon(R) Gold | Intel(R) Xeon(R) Gold | | | 6248 CPU @ 2.50GHz | 6248 CPU @ 2.50GHz | +-------------------+--------------------------+--------------------------+ | CPU(s) | 40 | 40 | +-------------------+--------------------------+--------------------------+ | Thread(s) per core| 1 | 1 | +-------------------+--------------------------+--------------------------+ | Core(s) per socket| 20 | 20 | +-------------------+--------------------------+--------------------------+ | Socket(s) | 2 | 2 | +-------------------+--------------------------+--------------------------+ | RAM | 772 GB | 192 GB | +-------------------+--------------------------+--------------------------+ | GPU(s) | 2 x Quadro RTX 5000 | 2 x Quadro RTX 5000 | +-------------------+--------------------------+--------------------------+ .. - node interconnect is based on Mellanox Technologies InfiniBand fabric (Speed: 56Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 641 TB: /u shared home filesystem (641 TB) with user home directory in ``/u/``; GPFS-based; no quotas enforced. NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on BIO. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (- > 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ... This module becomes visible and loadable only after a compiler module (Intel or GCC) has been loaded - CUDA (-> 'module load cuda') - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on BIO is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel,...) can be found on the `Cobra home page `_. For more detailed information, see the `Slurm handbook `_. See also the `sample batch scripts `_ which must be modified for BIO cluster. Current Slurm configuration on BIO: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 2 hours - current max. turnaround time (wallclock): 24 hours - p.bio partition include all batch nodes in exclusive usage and is default - s.bio partition can be used for serial jobs and can be shared - l.bio shared partition for long running (up to 5 days) serial jobs (<=5 cores per job; <=160 cores in total) Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 2 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours Default memory per node is 9600 MB. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task**` is specified in a sbatch script (an example is on `help information `_ page). Exporting of **OMP\_PLACES=cores** also can be useful To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` options and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:rtx5000:1` or :bolditalic:`#SBATCH \-\-gres=gpu:rtx5000:2` | Valid gres options are: **gpu[[:type]:count]** | where | **type** is a type of gpu (rtx5000) | **count** is a number of resources (1 or 2) GPU cards are in default compute mode. .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biophysics CHEM =============== ---- Name of the cluster: **CHEM** Institution: **Max Planck Institute of Biophysics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - **chem11.bc.mpcdf.mpg.de** - **chem12.bc.mpcdf.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 login node chem[11-12]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 377 GB RAM | 4 x NVIDIA L40S GPUs per node :56 execution nodes chemg[101-156]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 377 GB RAM | 4 x NVIDIA L40S GPUs per node :2 execution nodes chemg[201-202]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 1.5 TB RAM | 4 x NVIDIA H200 GPUs per node :node interconnect: based on Mellanox Technologies InfiniBand fabric (Speed: 400Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 1.7 PB and independent inode space for the following filesets: /u shared home filesystem; GPFS-based; user quotas (100 GB data, 1M files/) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. NO BACKUPS YET /chem2/scratch shared scratch filesystem; GPFS-based; no quotas enforced. NO BACKUPS ! /phys see PHYS cluster documenation for its specs; its performance is limited on chem, so please use with care. Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on CHEM is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for CHEM cluster. Current Slurm configuration on CHEM: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 2 hours - current max. turnaround time (wallclock): 24 hours - p.chem partition include all batch nodes in exclusive usage and is default - s.chem partition can be used for serial jobs and can be shared - l.chem shared partition for long running (up to 5 days) serial jobs (<=10 cores per job; <=512 cores in total) - alpha qos for only AlphaFold jobs (up to 5 days) Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 2 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours Default memory per node in the shared partition is 47000 MB. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page) To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:l40s:X`, where :bolditalic:`X` is a number of resources (1, 2, 3 or 4) GPU cards are in default compute mode. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/biophysics-cryo .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biophysics CRYO =============== ---- Name of the cluster: **CRYO** Institution: **Max Planck Institute of Biophysics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - cryo101.bc.rzg.mpg.de - cryo102.bc.rzg.mpg.de Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ Login node cryo101 & cryo102 : - CPUs Model: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz - 2 sockets per node - 8 cores per socket - hyper-threading is on (2 threads per core) - 188 GB RAM - 2 x GeForce GTX 1080 Ti Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 1496 TB: /u shared home filesystem (214 TB) with user home directory in ``/u/``; GPFS-based; user quotas (currently 4 TB, 2M files) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /sbdata shared scratch filesystem (1.3 PB); no quotas enforced. NO BACKUPS! .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biophysics LEO =============== ---- Name of the cluster: **LEO** Institution: **Max Planck Institute of Biophysics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - **leo01.bc.mpcdf.mpg.de** - **leo02.bc.mpcdf.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 login node leo[01-02]: | 2 x AMD EPYC 9454 48-Core Processor @ 2.75 GHz | 96 cores per node | hyper-threading enabled - 2 threads per core | 755 GB RAM :10 execution nodes leo[001-010]: | 2 x AMD EPYC 9454 48-Core Processor @ 2.75 GHz | 96 cores per node | hyper-threading enabled - 2 threads per core | 755 GB RAM :8 execution nodes leog[101-108]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 755 GB RAM | 8 x NVIDIA L40S GPUs per node :3 execution nodes leog[201-203]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 1.5 TB RAM | 4 x NVIDIA H100 GPUs per node :5 execution nodes leog[301-305]: | 2 x AMD EPYC 9535 64-Core Processor @ 2.40 GHz | 128 cores per node | hyper-threading enabled - 2 threads per core | 1.5 TB RAM | 4 x NVIDIA H200 GPUs per node :node interconnect: based on Mellanox Technologies InfiniBand fabric (Speed: 200Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 7.5 PB and independent inode space for the following filesets: /u shared home filesystem; GPFS-based; user quotas (250 GB data, 512k files/) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /leo/work shared scratch filesystem; GPFS-based; no quotas enforced. NO BACKUPS ! /leo/data shared scratch filesystem; GPFS-based; no quotas enforced. /cryo/* | only on leo[01-02] | read-only Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Please also note that the environment modules found in */mpcdf/soft/eb/modules* are neither maintained nor supported by MPCDF. Please contact your local support representative in case of problems. Batch system based on Slurm ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on LEO is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for LEO cluster. Current Slurm configuration on LEO: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 24 hours - current max. turnaround time (wallclock): 96 hours - p.leo partition include all batch nodes in shared usage and is default Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 24 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 96 hours To use nodes exclusivly add in slurm scripts :bolditalic:`\-\-exclusive` option Default memory per node in the shared partition is 47000 MB. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page) To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:type:X`, where :bolditalic:`type` is either l40s or h100 and :bolditalic:`X` is a number of resources (1-8 for l40s and 1-4 for h100) GPU cards are in default compute mode. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/biophysics-phys .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Biophysics PHYS =============== ---- Name of the cluster: **PHYS** Institution: **Max Planck Institute of Biophysics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - **phys11.bc.mpcdf.mpg.de** - **phys12.bc.mpcdf.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 login node phys[11-12]: | 2 x Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz | 56 cores per node | hyper-threading enabled - 2 threads per core | 186 GB RAM | 3 x Quadro RTX 6000 GPUs per node :238 execution nodes physg[201-438]: | 2 x Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz | 56 cores per node | hyper-threading enabled - 2 threads per core | 186 GB RAM | 3 x Quadro RTX 6000 GPUs per node :node interconnect: based on Mellanox Technologies InfiniBand fabric (Speed: 100Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 820 TB and independent inode space for the following filesets: /u shared home filesystem; GPFS-based; user quotas (3 TB data, 500k files/) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /phys/ptmp shared filesystem for temporary files; GPFS-based; no quotas enforced. NO BACKUPS! /phys/scratch shared scratch filesystem; GPFS-based; no quotas enforced. NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on PHYS. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel/19.1.3'): icc, icpc, ifort - GNU compilers (-> 'module load gcc/10'): gcc, g++, gfortran - Intel MKL ('module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI 2019.9 ('module load impi/2019.9'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ - CUDA: module load cuda - Python (-> 'module load anaconda'): python To find a module and information about the available versions or what dependencies need to be loaded first one can use the ‘find-module’ command. Batch system based on Slurm ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on PHYS is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Cobra home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for PHYS cluster. Current Slurm configuration on PHYS: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 2 hours - current max. turnaround time (wallclock): 24 hours - p.phys partition include all batch nodes in exclusive usage and is default - s.phys partition can be used for serial jobs and can be shared - l.phys shared partition for long running (up to 5 days) serial jobs (<=5 cores per job; <=224 cores in total) Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 2 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours Default memory per node in the shared partition is 62000 MB. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page) To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:rtx6000:X`, where :bolditalic:`X` is a number of resources (1, 2 or 3) GPU cards are in default compute mode. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Brain%20Research .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Brain Research ============== ---- Name of the clusters: **GABA** Institution: **Max Planck Institute for Brain Research** Login nodes: ------------ .. hlist:: :columns: 200 - gaba[11-15].bc.mpcdf.mpg.de Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ Login nodes gaba[11-15]: - CPUs Model : AuthenticAMD EPYC 9554 64-Core Processor - 2 sockets per node; 64 cores per socket; hyper-threading is on - RAM : 768 GB - GPUs : 4 Nvidia L40s-48GB GPUs per node 31 execution nodes gabag[201-231] for parallel CPU and GPU computing (3968 CPU cores) : - CPUs Model : AuthenticAMD EPYC 9554 64-Core Processor - 2 sockets per node; 64 cores per socket; hyper-threading is on - RAM : 768 GB - GPUs : 4 Nvidia L40s-48GB GPUs per node Node interconnect is based on 25 Gb/s Ethernet Filesystems: ~~~~~~~~~~~~ /u shared home filesystem with user home directory in ``/u/``; user quotas enforced (200GB data, 512K files) /oldgaba former $HOME filesystem (formerly known as /gaba ), read-only, only available on login nodes. NO BACKUPS anylonger! /gabaghi filesystem for archive (1.3 PB); only available on login nodes /conndata (9 PB) shared filesystem filesystem; no quotas enforced. NO BACKUPS! /connscratch (9 PB) shared filesystem filesystem; no quotas enforced. NO BACKUPS! /tmpscratch (1.5 PB) shared filesystem filesystem; no quotas enforced. NO BACKUPS! /wKlive (220 TB) filesystem for live wKcubes project; only available on login nodes NO BACKUPS! /nexus/posix0 only available on gaba14 for data transfers to/from HPC systems Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on GABA. Please use 'module available' to see all available modules. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - login nodes: gaba[11-15]; batch nodes: gabag[201-229] - sbatch, srun, squeue, sinfo, scancel, scontrol, s\* - current default turnaround time (wallclock) is 24 hours and max. turnaround time is 169 hours Useful tips (slurm part of cluster): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 24 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 100 hours Default memory per node is 50G. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script. | To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:l40s:1` | Valid gres options are: **gpu[[:type]:count]** | where | **type** is a type of gpu (*l40s*) | **count** is a number of resources (*<=4*) To check node features, general resources and scheduling weight of nodes use :bolditalic:`sinfo -O nodelist,features,gres,weight` | For interactive jobs please use :bolditalic:`srun` command: | :bolditalic:`srun \-\-time=1-10 \-\-mem=32G \-\-gres=gpu:l40s:1 \-\-pty bash -i -l` | keep in mind that :bolditalic:`\-\-pty` option should be the last srun option. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Chemical%20Physics%20of%20Solids .. ------------- .. attention:: The MARS servers will be taken out of service on November 28, 2025. .. raw:: html .. role:: bolditalic :class: bolditalic Chemical Physics of Solids ========================== ---- Name of the cluster: **MARS** Institution: **Max Planck Institute for Chemical Physics of Solids** Login nodes: ~~~~~~~~~~~~ All nodes in cluster are login nodes - mars[1-5,8-9].opt.rzg.mpg.de Hardware-Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | | Login/Compute nodes | + +------+-------+-------+-------+-------+-------+-------+ | | mars1| mars2 | mars3 | mars4 | mars5 | mars8 | mars9 | +=====+=============================================+======+=======+=======+=======+=======+=======+=======+ | | Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz | | | | | | X | | + +---------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | | Intel(R) Xeon(R) CPU E7-8867 v3 @ 2.50GHz | | | | | | | X | + +---------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | | Intel(R) Xeon(R) CPU E7-8867 v4 @ 2.40GHz | X | | | | | | | + CPU:+---------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | | | X | | | | | + +---------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz | | X | | | | | | + +---------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | | Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz| | | | X | X | | | +-----+---------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | CPU(s) | 144 | 160 | 160 | 96 | 96 | 60 | 64 | +---------------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | Thread(s) per core | 2 | 2 | 2 | 1 | 1 | 1 | 1 | +---------------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | Core(s) per socket | 18 | 20 | 20 | 24 | 24 | 15 | 16 | +---------------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | Socket(s) | 4 | 4 | 4 | 4 | 4 | 4 | 4 | +---------------------------------------------------+------+-------+-------+-------+-------+-------+-------+ | RAM | 1.5T | 1.5T | 1.5T | 3.0T | 3.0T | 1.5T | 1.5T | +---------------------------------------------------+------+-------+-------+-------+-------+-------+-------+ Node interconnect is based on 1Gb/s ethernet Filesystems: ~~~~~~~~~~~~ Home directories are stored on AFS /batch3 shared home filesystem (44 TB); GPFS-based; no quotas enforced. NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on MARS. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL ('module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI 2017.4 ('module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - only for administrative system monitoring Useful tips: ~~~~~~~~~~~~ We recommend to submit jobs from /ptmp filesystem instead to use $HOME directories on network AFS The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ .. ------------- .. authors: mykp .. ------------- Extraterrestrial Physics ======================== ---- Name of the clusters: .. toctree:: :maxdepth: 1 :glob: ExtraterrestrialPhysics/MPE-EUCLID.rst.txt Institution: **Max Planck Institute for Extraterrestrial Physics** .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/mpe-euclid .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Extraterrestrial Physics EUCLID =============================== ---- Name of the cluster: **EUCLID** Institution: **Max Planck Institute for Extraterrestrial Physics** Support: ~~~~~~~~ The Euclid Linux Cluster is not operated by MPCDF. For support please create a trouble ticket as explained at `MPE Euclid SDC-DE Support `__ page. .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Geoanthropology =============== ---- Name of the cluster: **GEANY** Institution: **Max Planck Institute of Geoanthropology** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - **geany01.bc.mpcdf.mpg.de** - **geany02.bc.mpcdf.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 login node geany[01-02]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 64 cores per node | hyper-threading enabled - 2 threads per core | 770 GB RAM :70 execution nodes geany[001-070]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 64 cores per node | hyper-threading enabled - 2 threads per core | 770 GB RAM :4 execution nodes geany[101-104]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 64 cores per node | hyper-threading enabled - 2 threads per core | 1.5 TB RAM :2 execution nodes geanyg[001-002]: | 2 x AMD EPYC 9554 48-Core Processor @ 3.10 GHz | 48 cores per node | hyper-threading disabled - 2 threads per core | 770 GB RAM | 8 x NVIDIA L40S GPUs per node :1 execution node geanyg101: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 64 cores per node | hyper-threading disabled - 2 threads per core | 770 GB RAM | 4 x NVIDIA H100 GPUs per node :1 execution node geanyg201: | 2 x AMD EPYC 9555 64-Core Processor @ 3.20 GHz | 64 cores per node | hyper-threading disabled - 2 threads per core | 770 GB RAM | 4 x NVIDIA RTX6000 Blackwell Server Edition GPUs per node :node interconnect: based on Mellanox Technologies InfiniBand fabric (Speed: 200Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 4.7 PB and independent inode space for the following filesets: /u shared home filesystem; GPFS-based; user quotas (100 GB data, 1M files/) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /geany/fs shared filesystem with project and individual user scratch area; GPFS-based; no quotas enforced. ONLY SELECTED PARTS ARE BEING BACKED UP ! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on GEANY is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for GEANY cluster. Current Slurm configuration on GEANY: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 2 hours - current max. turnaround time (wallclock): 168 hours - p.geany partition include all batch nodes in exclusive usage - s.geany partition can be used for serial jobs and can be shared, CPU only, default partition - s.geany.gpu partition can be used for serial jobs and can be shared, incl. GPUs Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 2 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 168 hours Default memory per node in the shared partition is 47000 MB, maximum per allocated node per job is 360000 MB. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page) To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:l40s:X`, where :bolditalic:`X` is a number of resources from 1 up to 4 (:bolditalic:`h100`) or 8 (:bolditalic:`l40s`) GPU cards are in default compute mode. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Gravitational%20Physics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Gravitational Physics ===================== ---- Name of the cluster: **SAKURA** Institution: **Max Planck Institute for Gravitational Physics (Albert Einstein Institute)** Access: ~~~~~~~ .. hlist:: :columns: 200 - **sakura01.bc.rzg.mpg.de** - **sakura02.bc.rzg.mpg.de** Configuration: ~~~~~~~~~~~~~~ Login nodes sakura[01-02] : - CPU Model: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz - 2 sockets - 20 cores per socket - no hyper-threading (1 threads per core) - 376 GB RAM .. 362 execution nodes sakura[001-362] : - CPU Model: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz - 2 sockets - 20 cores per socket - no hyper-threading (1 threads per core) - 376 GB RAM Node interconnect is based on Intel Omni-Path Fabric (Speed: 100Gb/s) Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; GPFS-based; user quotas (currently 400GB, 1M files) enforced quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota' NO BACKUPS yet /sakura/ptmp shared scratch filesystem (1.3 PB); GPFS-based; no quotas enforced NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on SAKURA. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (-> 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on SAKURA is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Cobra home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for SAKURA cluster (partition must be changed). Current Slurm configuration on SAKURA: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default run time: 12 hours - current max. run time (wallclock): 1 days - only one partitions: p.sakura - default memory per node for jobs: p.sakura ( 380000 MB ) - nodes are exclusively allocated to jobs - max number of nodes each user is able to use: 160 Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 12 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ \ page) Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Gravitational%20Physics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Gravitational Physics - ACR =========================== ---- Name of the cluster: **URANIA** Institution: **Max Planck Institute for Gravitational Physics (Albert Einstein Institute)**: ACR department Access: ~~~~~~~ .. hlist:: :columns: 200 - **urania01.bc.mpcdf.mpg.de** - **urania02.bc.mpcdf.mpg.de** Configuration: ~~~~~~~~~~~~~~ Login nodes urania[01-02] : - CPU Model: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz - 2 sockets - 36 cores per socket - hyper-threading on (2 threads per core) - 512 GB RAM .. 84 execution nodes urania[001-084] : - CPU Model: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz - 2 sockets - 36 cores per socket - hyper-threading on (2 threads per core) - 256 GB RAM Node interconnect is based on Melanox/Nvidia Infiniband HDR-100 technology (Speed: 100Gb/s) Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; GPFS-based; user quotas (currently 100GB, 1M files) enforced quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota' /urania/ptmp shared scratch filesystem (1.1 PB); GPFS-based; no quotas enforced NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The `"module" subsystem `__ is implemented on URANIA. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (-> 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on URANIA is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for URANIA cluster (partition must be changed). Current Slurm configuration on URANIA: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - two partitions: p.urania (default), p.debug (2 nodes) - default run time: 24 hours (p.urania), 12 hours (p.debug) - current max. run time (wallclock): 1 days - default memory per node for jobs: p.urania ( 240000 MB ) - nodes are exclusively allocated to jobs in p.urania Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is default time limit per partition. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (check examples on `sample batch scripts `__ \ page) On login nodes to debug codes interactively with the native Intel MPI process managers (mpiexec/mpirun) use 'impi-interactive' which needs to be loaded after another 'impi' module has been loaded Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Gravitational%20Physics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Gravitational Physics - CRA =========================== ---- Name of the cluster: **MOMIJI** Institution: **Max Planck Institute for Gravitational Physics (Albert Einstein Institute)**: CRA department Access: ~~~~~~~ .. hlist:: :columns: 200 - **momiji01.bc.mpcdf.mpg.de** - **momiji02.bc.mpcdf.mpg.de** Configuration: ~~~~~~~~~~~~~~ Login nodes momiji[01-02] : - CPU Model: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz - 2 sockets - 36 cores per socket - hyper-threading on (2 threads per core) - 512 GB RAM .. 74 batch nodes momiji[001-084] : - CPU Model: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz - 2 sockets - 36 cores per socket - hyper-threading on (2 threads per core) - 256 GB RAM Node interconnect is based on Melanox/Nvidia Infiniband HDR-100 technology (Speed: 100Gb/s) Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; GPFS-based; user quotas (currently 100GB, 1M files) enforced quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota' /momiji/ptmp shared scratch filesystem (560 TB); GPFS-based; no quotas enforced NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The `"module" subsystem `__ is implemented on MOMIJI. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (-> 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on MOMIJI is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for MOMIJI cluster (partition must be changed). Current Slurm configuration on MOMIJI: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default run time: 12 hours - current max. run time (wallclock): 2 days - only one partitions: p.momiji - default memory per node for jobs: p.momiji ( 240000 MB ) - nodes are exclusively allocated to jobs Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is default time limit per partition. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 48 hours The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (check examples on `sample batch scripts `__ \ page) On login nodes to debug codes interactively with the native Intel MPI process managers (mpiexec/mpirun) use 'impi-interactive' which needs to be loaded after another 'impi' module has been loaded Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic MPSD / PKS ========== ---- Name of the cluster: **ADA** Institution: **Max Planck Institute for the Structure and Dynamics of Matter** **Max Planck Institute for the Physics of Complex Systems** Login nodes: ~~~~~~~~~~~~ * ada01.bc.rzg.mpg.de * ada02.bc.rzg.mpg.de Their SHA256 ssh host key fingerprint is: **Roiw24V2Yhw5a9MwghRWJYTyq9bPs2jYNKqWPJiaxuE (ED25519)** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ * ADA is built on top of Intel Xeon Platinum 8360Y CPUs (36 cores at 2.40GHz), each node is equipped with two 8360Y CPUs * As the HPC cluster RAVEN, ADA is operated with Hyper-Threading enabled .. * login nodes ada\[01-02\] (500 GB RAM each) * 72 execution nodes adag\[001-072\] (1 TB RAM each and 4 Nvidia A100-80GB GPUs each) * 2 execution nodes ada\[001-002\] (2 TB RAM each) * node interconnect is based on Mellanox/Nvidia Infiniband HDR-100 technology (Speed: 100 Gb/s) Filesystems: ~~~~~~~~~~~~ /u - shared home filesystem - user quotas (1 TB of data; 400k files/directories) enforced - quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /ada/ptmp {mpsd|pks} - shared scratch filesystem (3.5 PB) - NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. There are no modules preloaded on ADA. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * a brief introduction into the basic commands (srun, sbatch, squeue, scancel, sinfo, s\*...) can be found on the `Raven home page `__ or on the `Slurm handbook `__ * **two partitions:** p.ada (default), p.large * **current max. run time (wallclock):** p.ada (1 days), p.large (1 days) * **maximum memory per node for jobs:** p.ada (1024000 MB), p.large (2048000 MB) * **p.ada partition:** nodes are exclusively allocated to users * **p.large partition:** resources on the nodes may be shared between jobs * **p.ada partition:** to access GPU resources :bolditalic:`\-\-gres` parameter must be explicitly set for jobs Sample batch scripts ^^^^^^^^^^^^^^^^^^^^ You can find a set of sample batch scripts on the `Raven home page `_ that must be modified for Ada. Below you find a few examples that are adapted to Ada already so you can copy and paste them. Hybrid MPI/OpenMP job using one or more nodes with 4 GPUs each with CUDA-aware MPI """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" The following example job script launches a hybrid MPI/OpenMP-CUDA-code on one (or more) nodes running one task per GPU, using CUDA-aware MPI. You should use the same modules also for compiling your code. Note that the user code needs to attach its tasks to the different GPUs based on some code-internal logic. .. important:: For MPSD users: if you want to run octopus, please use this batch script! .. code-block:: bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_gpu # #SBATCH --nodes=1 # Request 1 or more full nodes #SBATCH --partition=p.ada # in the GPU partition #SBATCH --gres=gpu:a100:4 # Request 4 GPUs per node. #SBATCH --ntasks-per-node=4 # Run one task per GPU #SBATCH --cpus-per-task=18 # using 18 cores each. #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=24:00:00 module purge module load gcc/11 cuda/11.4 openmpi_gpu/4 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun ./mpi_openmp_cuda_executable Hybrid MPI/OpenMP job using one or more nodes with 4 GPUs each """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" The following example job script launches a hybrid MPI/OpenMP-CUDA-code on one (or more) nodes running one task per GPU. You should load the same modules in the slurm script as you did for compiling your code. Note that the user code needs to attach its tasks to the different GPUs based on some code-internal logic. .. code-block:: bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_gpu # #SBATCH --nodes=1 # Request 1 or more full nodes #SBATCH --partition=p.ada # in the GPU partition #SBATCH --gres=gpu:a100:4 # Request 4 GPUs per node. #SBATCH --ntasks-per-node=4 # Run one task per GPU #SBATCH --cpus-per-task=18 # using 18 cores each. #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=24:00:00 module purge module load intel/21.5.0 impi/2021.5 cuda/11.4 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun ./mpi_openmp_cuda_executable MPI job using one of the large-memory nodes """"""""""""""""""""""""""""""""""""""""""" The following example job script launches a MPI code on one large-memory node that has 2 TB of memory. .. code-block:: bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./job.out.%j #SBATCH -e ./job.err.%j # Initial working directory: #SBATCH -D ./ # Job name #SBATCH -J test_gpu # #SBATCH --nodes=1 # Request 1 or more full nodes #SBATCH --partition=p.large # in the large-mem partition #SBATCH --ntasks-per-node=72 # Run 72 tasks #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de #SBATCH --time=24:00:00 module purge module load intel/21.5.0 impi/2021.5 srun ./mpi_executable Useful tips ~~~~~~~~~~~ Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation. The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script. Nvidia Ampere GPUs are available in :bolditalic:`p.ada` partition. Type of gpu must be explicitly set, i.e. :bolditalic:`\-\-gres=gpu:a100:X`, where X is between 1 and 4 GPU cards are in default compute mode. Nodes in :bolditalic:`p.large` is in shared mode i.e. jobs allocate only requested resources. By default jobs allocate all memory on nodes. This means that to share a node between several jobs :bolditalic:`\-\-mem` parameter is required for jobs. For debugging, you can use the Quality-of-Service feature: by adding `--qos=debug`, your job will get a higher priority to start as soon as possible. The time limit is 15 minutes for such jobs. Profiling on GPUs ~~~~~~~~~~~~~~~~~ For profiling codes on GPUs, we provide packages for Nsight systems (``nsight_systems``) and Nsight compute (``nsight_compute``) from Nvidia. Nsight systems is great to profile the overall behavior of the code and will give you a timeline which can be used to see which parts are already executed on the GPU and where there are still gaps or also where in the code data is transfered between CPU and GPU or also directly between GPUs. To use it, you can run in your batch script: .. code-block:: bash module load nsight_systems nsys profile -t cuda,nvtx,mpi srun my_binary This will create a profile that you can open in nsys-ui. Be aware that this only works for single-node runs! Nsight compute is a great tool to analyze kernel behavior to optimize kernels. To profile a certain kernel, you would run: .. code-block:: bash module load nsight_compute ncu --kernel-id ::kernel_name:2 -o output ./my_binary This will profile the second invocation of the kernel named ``kernel_name`` and will give you a profile with a name starting with ``output``. You can open that profile with ``ncu-ui``. For this, it is enought to run the code on one GPU. Tips for MPSD users ~~~~~~~~~~~~~~~~~~~ The Octopus code is provided via the module system on the Ada cluster. You can load the most recent modules without needing to load a compiler or MPI module. We offer a build with GPU support with the module ``octopus-gpu``. You can see the available versions using ``module avail``. The versioning scheme includes the minor version number (e.g. ``octopus/12.0``). We also offer the version ``octopus/main`` which provides a build of the current ``main`` branch that is updated twice per month. To run on GPUs, the following code block is recommended: .. code-block:: bash module purge module load octopus-gpu/main export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun octopus This will let you use the main version of octopus compiled with support for CUDA-aware MPI which is especially important for domain-parallel runs. You can also compile Octopus yourself using the MPCDF build script shipped with the code as ``scripts/build/build_octopus_mpcdf.sh``. You can also find more information on Octopus at the `Octopus web page `__. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `_ .. raw:: html .. role:: bolditalic :class: bolditalic Physics ======= ---- Name of the cluster: **MPPMU** Institution: **Max Planck Institute for Physics** Documentation: `https://docs.t2.mpcdf.mpg.de/ `_ .. ------------- .. authors: mykp .. ------------- Plasma Physics ============== ---- Name of the clusters: **TOK** Institution: **Max Planck Institute of Plasma Physics** Support: ~~~~~~~~ The TOK Linux clusters are operated and maintained by MPCDF. An MPCDF account is **not** required to access TOK. Comprehensive documentation can be found at `IPP's Wiki `__. .. ------------- .. authors: mykp .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Polymer Research ================ ---- Name of the cluster: **OTTER** Institution: **Max Planck Institute for Polymer Research** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - **otter01.bc.mpcdf.mpg.de** - **otter02.bc.mpcdf.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 login node otter[01-02]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 64 cores per node | hyper-threading enabled | 770 GB RAM | 4 x NVIDIA L40s GPUs per node :41 execution nodes otterg[001-041]: | 2 x AMD EPYC 9554 64-Core Processor @ 3.10 GHz | 64 cores per node | hyper-threading enabled | 770 GB RAM | 4 x NVIDIA L40S GPUs per node :node interconnect: Ethernet (Speed: 25Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 2 PB. /u shared home filesystem; GPFS-based; user quotas (100 GB data, 1M files/) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. NO BACKUPS YET /otter/ptmp shared scratch filesystem; no quotas enforced. NO BACKUPS ! /nexus/posix0/bmm ONLY available on login nodes! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm ~~~~~~~~~~~~~~~~~~~~~~~~~~~ A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Raven home page `__. For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for OTTER cluster. Current Slurm configuration on OTTER: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 2 hours - current max. turnaround time (wallclock): 24 hours - p.otter partition include all batch nodes in exclusive usage and is default - s.otter partition can be used for serial jobs and can be shared Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 2 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 24 hours. Default memory per node in the shared partition is 94000 MB, maximum per allocated node per job is 770000 MB. To grant the job access to all of the memory on each node use :bolditalic:`\-\-mem=0` option for sbatch/srun. The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page). To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:l40s:X`, where :bolditalic:`X` is a number of resources from 1 up to 4 (:bolditalic:`l40s`). Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. ------------- Psychiatry ========== ---- Name of the clusters: .. toctree:: :maxdepth: 1 :glob: Psychiatry/Psychiatry-PIROL.rst.txt Institution: **Max Planck Institute of Psychiatry** .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Psychiatry .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Psychiatry PIROL ================ ---- Name of the cluster: **PIROL** Institution: **Max Planck Institute of Psychiatry** Login nodes: ~~~~~~~~~~~~ - **pirol01.hpccloud.mpcdf.mpg.de** The SHA256 ssh host key fingerprint is: **4VrucjZqrse36SLpQMRr26JiFffbj/rr2gGDzMXtpmk (ED25519)** Hardware-Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ Login node pirol01: - CPU Model: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz - 1 socket - 18 cores per socket - no hyper-threading (1 thread per core) - 120 GB RAM 6 cpu execution nodes pirolc[001-006] : - CPU Model: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz - 1 socket - 12 cores per socket - no hyper-threading (1 thread per core) - 80 GB RAM 6 gpu execution nodes pirolg[001-006] : - CPU Model: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz - 1 socket - 10 cores per socket - 400 GB RAM - 1 x Nvidia A40 node interconnect is based on 10 Gb/s ethernet Filesystems: ~~~~~~~~~~~~ /u shared home filesystem with user home directory in ``/u/``; user quotas (currently 200 GB, 250k files) enforced. User quotas can be checked using the ``quota`` command (e.g. ``quota --show-mntpoint --hide-device -f /pirol/u``). /nexus/posix0/MPI-psych shared scratch filesystem with user directory in ``/nexus/posix0/MPI-psych/`` Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ `Hierarchical environment modules `_ are used at MPCDF to provide software packages and enable switching between different software versions. There are no modules preloaded on PIROL. User have to specify the needed modules with explicit versions at login and during the startup of a batch job. Not all software modules are displayed immediately by the **module avail** command, for some user first needs to load a compiler and/or MPI module. You can search the full hierarchy of the installed software modules with the **find-module** command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ a brief introduction into the basic commands (srun, sbatch, squeue, scancel, sinfo, s\*...) can be found on the `Raven home page `__ or on the `Slurm handbook `__ Current Slurm configuration on PIROL: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * **two partitions:** c.pirol (default), g.pirol (for gpu jobs and high memory cpu jobs) * **current max. run time (wallclock):** (11 days, default runtime is 24 hours) * **default memory per node for jobs:** c.pirol (10000 MB), g.pirol (39600 MB) * **c.pirol, g.pirol:** resources on the nodes may be shared between jobs * **g.pirol partition:** to access GPU resources :bolditalic:`\-\-gres` parameter must be explicitly set for jobs * sample batch scripts can be found on `Raven home page `_ (must be modified for PIROL) Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 24 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 11 days Default memory per node is 10G & 38G. To grant the job access to all of the memory on each node use :bolditalic:`--mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script To run code with different memory limits than the defaults, choose appropriate partition and set the required memory(*c.pirol* with max 72000M per node and *g.pirol* with max 396000M per node) by using :bolditalic:`\-\-partition` option in a sbatch script: :bolditalic:`#SBATCH \-\-partition=c.pirol` or :bolditalic:`#SBATCH \-\-partition=g.pirol` | To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH \-\-gres=gpu:a40:1` | Valid gres options are: **gpu[[:type]:count]** | where | **type** is a type of gpu (*a40*) | **count** is a number of resources (*=1*) GPU cards are in default compute mode. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Quantum%20Optics .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Quantum Optics ============== ---- Name of the cluster: **TQO** Institution: **Max Planck Institute of Quantum Optics** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - **tqo401.bc.mpcdf.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ 1 login node tqo401 : 2 x Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz (Caskadelake); 48 cores per node; 385 Gb RAM; no hyper-threading 59 execution nodes tqo[402-460] : total amount of 2880 CPU cores; 2 x Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz (Caskadelake); 385 Gb RAM; no hyper-threading 66 execution nodes tqo[501-566] : total amount of 4752 CPU cores; Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (Icelake); 512 Gb RAM; no hyper-threading 1 execution nodes tqog02 for parallel GPU computing : total amount of 16 CPU cores; 2 x Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz (Skylake); 94 Gb RAM; no hyper-threading; 2 x Nvidia Tesla P100 GPUs per node 2 execution nodes tqog[03-04] for parallel GPU computing : total amount of 32 CPU cores; 2 x Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz (Skylake); 94 Gb RAM; no hyper-threading; 2 x Nvidia Tesla V100 GPUs per node .. - Node interconnect is based on 1Gb/s ethernet Filesystems: ~~~~~~~~~~~~ /u ($HOME) shared home filesystem; GPFS-based; user quotas (currently 500 GB, 0.5M files) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /ptmp shared home filesystem; GPFS-based; no quotas enforced. NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on TQO. Please use 'module available' to see all available modules. - Intel compilers (e.g. 'module load intel/19.1.3'): icc, icpc, ifort - Intel MKL ('module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (e.g. 'module load impi/2019.9'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - sbatch, srun, squeue, sinfo, scancel, scontrol, s\* - current max. turnaround time (wallclock) for partitions: 168 (partition s.168) & 672 (partition s.672) hours - s.gpu partition for GPU computing: 2xP100 & 4xV100 gpus; turnaround time is 672 hours - nodes are different in CPU architecture and memory capacity. Use \-\-constraint sbatch/srun option to run MPI jobs on homogeneous environment Useful tips: ~~~~~~~~~~~~ The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ \ page) | To use GPUs add in your slurm scripts :bolditalic:`\-\-gres` option and choose how many GPUs and/or which model of them to have: :bolditalic:`#SBATCH --gres=gpu:p100:1` or :bolditalic:`#SBATCH --gres=gpu:v100:2` | Valid gres options are: **gpu[[:type]:count]** | where | **type** is a type of gpu (*p100* or *v100*) | **count** is a number of resources (*1* or *2*) GPU cards are in default compute mode. To use a gpu interactively: login to the tqog01 node; load cuda module\ . GPU cards on tqog01 are in default compute mode. Default memory for jobs is 1600M per core. Use :bolditalic:`--mem` to set necessary amount of memory per job. To grant the job access to all of the memory on each node use :bolditalic:`--mem=0` option for sbatch/srun To run code on nodes with different memory capacity (*94G; 192G; 384G; 512G*) use :bolditalic:`\-\-constraint=` option in a sbatch script: :bolditalic:`\-\-constraint=94G` or :bolditalic:`\-\-constraint=192G` for instance. To run code on nodes with specific CPU architecture use :bolditalic:`\-\-constraint=` option in a sbatch script: :bolditalic:`\-\-constraint=cascadelake` or :bolditalic:`\-\-constraint=skylake` To check node features use :bolditalic:`sinfo -O nodelist,features` Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/radioastronomy .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Radioastronomy ============== ---- Name of the linux cluster: **HERCULES** Institution: **Max Planck Institute for Radio Astronomy** Login nodes: ~~~~~~~~~~~~ .. list-table:: * - **hercules11.bc.rzg.mpg.de** - **hercules12.bc.rzg.mpg.de** Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ :2 login nodes hercules[11-12]: | 2 x Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz | 24 cores per node | hyper-threading disabled - 1 threads per core | 188 GB RAM; :32 execution nodes hc[201-232] for parallel computing: | total amount of 1536 CPU cores | 2 x Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 48 cores per node | hyper-threading disabled - 1 threads per core | 377 GB RAM :54 execution nodes hcg[001-054] for parallel GPU computing: | total amount of 2592 CPU cores | 2 x Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 48 cores per node | hyper-threading disabled - 1 threads per core | 377 GB RAM | 3 x Quadro RTX 6000 GPUs per node :node interconnect: based on 25 Gb/s ethernet Filesystems: ~~~~~~~~~~~~ /u shared home filesystem; quoted to 1TB of data and 600K files; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota' /mkfs dedicated project area for selected users - NO BACKUPS! /hercules dedicated project area - NO BACKUPS! /scratch dedicated scratch area for all users - NO BACKUPS! /mandap incoming data from Bonn (only available on login nodes) - NO BACKUPS! Software Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on HERCULES cluster. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel/19.1.3'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL ('module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI 2019.9 ('module load impi/2019.9'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ...´ | Similar to the HPC systems, this module tree is `hierarchical `__. | To find a module and information about the available versions or what dependencies need to be loaded first one can use the ‘find-module’ command. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - sbatch, srun, squeue, sinfo, scancel, scontrol, s\* - five partitions: - **short.q** (default), **long.q**, **gpu.q** for serial jobs on shared nodes - **parallel.q** for multi-nodes prallel hybrind MPI/OpenMP jobs, nodes are allocated exclusively - **gpu42cores.q** for serial cpu jobs only with 42 cores and 50% of RAM per node - **gpu6cores.q** for serial gpu jobs only with 6 cores and 50% of RAM per node - **interactive.q** to debug serial/parallel jobs. Currently disabled. - sample batch scripts can be found on `Cobra home page `__ (must be modified for HERCULES) +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | Slurm partition | short.q | long.q | gpu.q | parallel.q | gpu42cores.q | gpu6cores.q | interactive.q | | | | | | | | | | | | (default) | | | | | | | +=========================+==============+=============+==============+=============+==============+=============+===============+ | number of nodes | 86 | 32 | 54 | 32 | 54 | 54 | 4 | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | hostnames | hc[201-232] | hc[201-232] | hcg[001-054] | hc[201-232] | hcg[001-054] | hcg[001-054]| hc*,hcg* | | | | | | | | | | | | hcg[001-054] | | | | | | | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | default run time limit | 4 hours | 24 hours | 24 hours | 48 hours | 24 hours | 24 hours | 2 hours | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | maximum run time limit | 4 hours | 240 hours | 240 hours | 240 hours | 240 hours | 240 hours | 12 hours | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | default memory per node | 8000 MB | 8000 MB | 120000 MB | 370000 MB | 4000 MB | 60000 MB | 8000 MB | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | maximum memory per node | 370000 MB | 370000 MB | 370000 MB | 370000 MB | 185000 MB | 185000 MB | 370000 MB | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | maximum nodes per job | 1 | 1 | 1 | 32 | 1 | 1 | 2 | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | maximum cpus per node | 48 | 48 | 48 | 48 | 42 | 6 | 48 | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | execute more than 1 job | Yes | Yes | Yes, | No | Yes | Yes, | Yes | | | | | | | | | | | at a time on each node | | | max. 3 jobs | | | max. 3 jobs | | +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ | gpus per node | \-\- | \-\- | 3 | \-\- | \-\- | 3 | 0(hc*),3(hcg*)| +-------------------------+--------------+-------------+--------------+-------------+--------------+-------------+---------------+ Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value and partition is 4 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 10 days on **long.q**, **gpu.q** and **parallel.q** partitions Default memory per job in serial partitions is 8000M. To grant the job access to all of the memory on each node use :bolditalic:`--mem=0` option for sbatch/srun The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ \ page). Exporting of **OMP\_PLACES=cores** also can be useful. | To run GPU codes add options :bolditalic:`\-p gpu.q` and :bolditalic:`\-\-gres=gpu:N`, where N is number of GPUs (min is 1, max is 3), into your batch scripts: | :bolditalic:`#SBATCH \-p gpu.q` | :bolditalic:`#SBATCH \-\-gres=gpu:1` Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk `_ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/Science_of_Light .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Science of Light ================ ---- Name of the cluster: **ZEROPOINT** Institution: **Max Planck Institute for the Science of Light** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - zp11.bc.rzg.mpg.de - zp12.bc.rzg.mpg.de - zp13.bc.rzg.mpg.de - zp14.bc.rzg.mpg.de Hardware-Configuration and Slurm partitions: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (phase 1): ^^^^^^^^^^ +------------------------------------+-------------------+----------------+-----------------+ | partition | HighMem | HighFreq | DGX | +====================================+===================+================+=================+ | # nodes | 4 | 8 | 1 | +------------------------------------+-------------------+----------------+-----------------+ | hostnames | zp[01-04] | zp[001-008] | zpx | +------------------------------------+-------------------+----------------+-----------------+ | *Slurm partition* | *highmem* | *highfreq* | *dgx* | +------------------------------------+-------------------+----------------+-----------------+ | .. | +-----+------------------------------+-------------------+----------------+-----------------+ | CPU | model | Xeon Gold 6130 | Xeon Gold 6144 | Xeon E5-2698 v4 | + +------------------------------+-------------------+----------------+-----------------+ | | architecture | x86_64 | x86_64 | x86_64 | + +------------------------------+-------------------+----------------+-----------------+ | | producer | Intel | Intel | Intel | + +------------------------------+-------------------+----------------+-----------------+ | | microarchitecture | Skylake-SP | Skylake-SP | Broadwell-EP | + +------------------------------+-------------------+----------------+-----------------+ | | CPUs per node | 2 | 2 | 2 | + +------------------------------+-------------------+----------------+-----------------+ | | cores per CPU | 16 | 8 | 20 | + +------------------------------+-------------------+----------------+-----------------+ | | threads per core | 1 | 1 | 1 | + +------------------------------+-------------------+----------------+-----------------+ | | clock rate (base/boost), GHz | 2.1\ /\ 3.7 | 3.5\ /\ 4.2 | 2.2\ /\ 3.6 | + +------------------------------+-------------------+----------------+-----------------+ | | cache size (L3) | 22 MB | 24.75 MB | 50 MB | + +------------------------------+-------------------+----------------+-----------------+ | | SIMD instruction set | AVX-512 | AVX-512 | AVX-2 | + +------------------------------+----------+--------+----------------+-----------------+ | | RAM size | zp[01-03]| 1 TiB | 96 GiB | 500 GiB | | | | | | | | | | | zp04 | 960 GiB| | | +-----+------------------------------+----------+--------+----------------+-----------------+ | .. | +-----+------------------------------+-------------------+----------------+-----------------+ | GPU | model | -- | -- | Tesla V100 | + +------------------------------+-------------------+----------------+-----------------+ | | producer | -- | -- | Nvida | + +------------------------------+-------------------+----------------+-----------------+ | | architecture | -- | -- | Volta | + +------------------------------+-------------------+----------------+-----------------+ | | GPUs per node | -- | -- | 8 | +-----+------------------------------+-------------------+----------------+-----------------+ | .. | +-----+------------------------------+-------------------+----------------+-----------------+ | Node interconnect | 1 Gb/s ethernet | +-----+------------------------------+-------------------+----------------+-----------------+ (phase 2): ^^^^^^^^^^ +------------------------------------+--------------------+----------------+ | partition | Standard (default) | GPU | +====================================+====================+================+ | # nodes | 68 | 32 | +------------------------------------+--------------------+----------------+ | hostnames | zp[101-168] | zpg[001-032] | +------------------------------------+--------------------+----------------+ | *Slurm partition* | *standard* | *gpu* | +------------------------------------+--------------------+----------------+ | .. | +-----+------------------------------+--------------------+----------------+ | CPU | model | Xeon Gold 6130 | Xeon Gold 6130 | + +------------------------------+--------------------+----------------+ | | architecture | x86_64 | x86_64 | + +------------------------------+--------------------+----------------+ | | producer | Intel | Intel | + +------------------------------+--------------------+----------------+ | | microarchitecture | Skylake-SP | Skylake-SP | + +------------------------------+--------------------+----------------+ | | CPUs per node | 2 | 2 | + +------------------------------+--------------------+----------------+ | | cores per CPU | 16 | 16 | + +------------------------------+--------------------+----------------+ | | threads per core | 1 | 1 | + +------------------------------+--------------------+----------------+ | | clock rate (base/boost), GHz | 2.1\ /\ 3.7 | 2.1\ /\ 3.7 | + +------------------------------+--------------------+----------------+ | | cache size (L3) | 22 MB | 22 MB | + +------------------------------+--------------------+----------------+ | | SIMD instruction set | AVX-512 | AVX-512 | + +------------------------------+--------------------+----------------+ | | RAM size | 187 GiB | 187 GiB | +-----+------------------------------+--------------------+----------------+ | .. | +-----+------------------------------+--------------------+----------------+ | GPU | model | -- | Quadro RTX 6000| + +------------------------------+--------------------+----------------+ | | producer | -- | Nvidia | + +------------------------------+--------------------+----------------+ | | architecture | -- | Turing | + +------------------------------+--------------------+----------------+ | | GPUs per node | -- | 2 | +-----+------------------------------+--------------------+----------------+ | | +-----+------------------------------+--------------------+----------------+ | Node interconnect | 1 Gb/s ethernet | +-----+------------------------------+--------------------+----------------+ Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 27 TB: /u shared home filesystem with user home directory in ``/u/``; GPFS-based; user quotas (currently 600 GB, 1M files) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /ptmp | shared scratch filesystem with user directory in ``/ptmp/``; | GPFS-based; no quotas enforced. | NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The hierarchical "module" subsystem is implemented on ZEROPOINT. Please use 'module available' to see all available modules. Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on ZEROPOINT is the Slurm Workload Manager. Current Slurm configuration on ZeroPoint: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: *3 days* - current max. turnaround time (wallclock): *7 days* - default partition: *standard* Useful tips: ~~~~~~~~~~~~ To run GPU codes use **gpu** partition add option :bolditalic:`\-\-gres=gpu:N`, where N is number of GPUs (max is 2) into your batch scripts: :bolditalic:`#SBATCH -p gpu \-\-gres=gpu:1` To run GPU codes on zpx add option :bolditalic:`\-\-gres=gpu:N`, where N is number of GPUs (max is 8): :bolditalic:`srun -p dgx \-\-gres=gpu:1 \-\-pty bash -l` How to use parallel COMSOL runs on cluster please look at `sample batch scripts `__ | To use large number of threads (subkernels) in Mathematica and circumvent timeout issue with loading the kernels locally from the software server one can load sequentially the subkernels before the parallel region. For instance, to use 32 subkernels, one would then have the following commands in a mathematica script file: | | LaunchKernels\[10\]; | LaunchKernels\[10\]; | LaunchKernels\[10\]; | LaunchKernels\[2\]; | | "Parallel region" Here 10 subkernels are launched at a time, but this value needs to be adapted depending on the networks performance and time out value. Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/iron-research .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic Sustainable Materials ===================== ---- Name of the cluster: **CMMC-CMTI-CMMG** Institution: **Max Planck Institute for Sustainable Materials** Login nodes: ~~~~~~~~~~~~ .. hlist:: :columns: 200 - cmti001.bc.mpcdf.mpg.de - cmti002.bc.mpcdf.mpg.de Hardware Configuration: ~~~~~~~~~~~~~~~~~~~~~~~ +--------------------+-------------------+---------------------------------+-------------------------------------------+ | | Login nodes | Compute nodes (14320 CPU cores) | Compute nodes (24.576 CPU cores) | + +-------------------+---------------------------------+-------------------------------------------+ | | cmti[001-002] | cmti[003-360] | cmmg[001-096] | +====================+===================+=================================+===========================================+ | CPU | Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz | AMD EPYC 9754 128-Core Processor | +--------------------+-------------------+---------------------------------+-------------------------------------------+ | CPU(s) | 40 | 40 | 512 | +--------------------+-------------------+---------------------------------+-------------------------------------------+ | Thread(s) per core | 1 | 1 | 2 | +--------------------+-------------------+---------------------------------+-------------------------------------------+ | Core(s) per socket | 20 | 20 | 128 | +--------------------+-------------------+---------------------------------+-------------------------------------------+ | Socket(s) | 2 | 2 | 2 | +--------------------+-------------------+---------------------------------+-------------------------------------------+ | RAM | 192 GB | 192 GB | 768 GB | +--------------------+-------------------+---------------------------------+-------------------------------------------+ - cmmg[001-096] nodes: interconnect is based on Mellanox Technologies InfiniBand fabric NDR200 (Speed: 200Gb) - cmti[001-360] nodes: interconnect is based on Intel Omni-Path Fabric Technologies (Speed: 100Gb) Filesystems: ~~~~~~~~~~~~ GPFS-based with total size of 1200 TB and independent inode space for the following filesets: /u shared home filesystem; GPFS-based; user quotas (currently default is 1TB) enforced; quota can be checked with '/usr/lpp/mmfs/bin/mmlsquota'. /cmmc/ptmp shared scratch filesystem ; GPFS-based; no quotas enforced; NO BACKUPS! Compilers and Libraries: ~~~~~~~~~~~~~~~~~~~~~~~~ The "module" subsystem is implemented on **CMMC**. Please use 'module available' to see all available modules. - Intel compilers (-> 'module load intel'): icc, icpc, ifort - GNU compilers (-> 'module load gcc'): gcc, g++, gfortran - Intel MKL (-> 'module load mkl'): $MKL\_HOME defined; libraries found in $MKL\_HOME/lib/intel64 - Intel MPI (-> 'module load impi'): mpicc, mpigcc, mpiicc, mpiifort, mpiexec, ... This module becomes visible and loadable only after a compiler module (Intel or GCC) has been loaded - Python (-> 'module load anaconda'): python Batch system based on Slurm: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The batch system on CMMC is the Slurm Workload Manager. A brief introduction into the basic commands (srun, sbatch, squeue, scancel, ...) can be found on the `Cobra home page `__ For more detailed information, see the `Slurm handbook `__. See also the `sample batch scripts `__ which must be modified for the CMMC-CMTI-CMMG cluster. Current Slurm configuration on CMMC-CMTI-CMMG: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - default turnaround time: 24 hours - current max. turnaround time (wallclock): 96 hours - 4 partitions s.cmfe (default), p.cmfe, s.cmmg, p.cmmg - s.cmmg partition: for jobs using less than 256 cores (MPI or openMP or hybrid). Nodes are shared. Jobs are limited to use CPUs only on one node. Default RAM per CPU is 2 GB - p.cmmg partition: for parallel MPI or hybrid MPI/OpenMP jobs using entire nodes. Resources are exclusively allocated on nodes. Default RAM is the entire node RAM. For optimizing CPU usage, please allocate (and use) 256 cores per node. - s.cmfe partition: for serial or OpenMP jobs. Nodes are shared. Jobs are limited to use CPUs only on one node. Default RAM per CPU is 2 GB - p.cmfe partition: for parallel MPI or hybrid MPI/OpenMP jobs. Resources are exclusively allocated on nodes. Max. nodes per job is 44 Please note that cmmg has hyperthreading enabled (2 logical cores per physical core). For best compute performance, do not use more threads/processes than physical cores unless you really know what you are doing. The number of allocated cores in the slurm reporting is automatically doubled once the jobs starts. (Example: you ask for 32 (physical) cores, after start 64 (logical) cores are shown). Useful tips: ~~~~~~~~~~~~ By default run time limit used for jobs that don't specify a value is 24 hours. Use :bolditalic:`\-\-time` option for sbatch/srun to set a limit on the total run time of the job allocation but not longer than 96 hours By default jobs use all memory on nodes in p.cmfe and p.cmmg partition. | In s.cmfe/s.cmmg partition default allocated memory per job is 2048 MB. To grant the job access to use more or less memory on each node use :bolditalic:`\-\-mem` or :bolditalic:`\-\-mem-per-cpu` options for sbatch/srun | Note that the maximum amount of memory available for slurm jobs is less than the hardware specification. Real memory is 188000 MB for cmti, and 730000 MB for cmmg. If you ask for more memory, your job cannot be scheduled. If you need to exceed the fair share of memory (4700 MB per core on cmti, 2850 MB per core on cmmg), please leave min. 2GB per unused core, to ensure that other jobs can use them. Occasional exceptions to this are acceptable if a job requires a higher RAM/CPU core ratio due to limitations in (efficient) parallelization - notably if using more cores would slow down the calculation, or increase the total memory demand The OpenMP codes require a variable **OMP\_NUM\_THREADS** to be set. This can be obtained from the Slurm environment variable **$SLURM\_CPUS\_PER\_TASK** which is set when :bolditalic:`\-\-cpus-per-task` is specified in a sbatch script (an example is on `help information `__ page) | To check node features use :bolditalic:`sinfo -O nodelist,features:30` Support: ~~~~~~~~ For support please create a trouble ticket at the `MPCDF helpdesk. `__ ------------------- Software ------------------- A wide range of software packages and libraries is provided on the supercomputers and HPC clusters. This section provides detailed information on how to access that software and build, optimize, and debug custom software using compilers and tools. .. toctree:: :maxdepth: 1 :glob: environment-modules.md.txt hpc-application-packages.md.txt data_analytics-machine_learning.md.txt compilers_languages.md.txt libraries.md.txt debugging-tools.md.txt performance-tools.md.txt mathematical-tools.md.txt bioinformatics.md.txt containers.md.txt vnc.md.txt # Environment Modules ## Introduction The MPCDF uses the Environment Modules system to manage the user environment for working with software installed at various locations in the file system or for switching between different software versions. Users are not required to explicitly specify paths for different executable versions, or keep track of PATH, MANPATH and related environment variables. With the modules approach, users simply 'load' and 'unload' modules to control their environment. System administrators provide modulefiles, typically named after the software package and an optional version number. All popular shells are supported, including bash, ksh, and tcsh. Besides handling different software versions, the modules approach allows system administrators to install software in non-standard locations and relocate software packages transparently for the user (by adapting the modulefile). It is therefore highly recommended for users to use the variables provided in the modules in their makefiles, scripts, etc. instead of relying on absolute paths (see below for examples). Since 2018, HPC systems as well as the increasing number of dedicated clusters all use hierarchical environment modules (see below for more details). ## Basic interactive usage Please find below a list of the most important commands (see for a complete reference): `module help` lists module subcommands and switches `module avail` lists available software packages and versions which can be enabled ("loaded") with the module command `module apropos ` searches available modulefiles for the specified keyword string and list all matching modules. `module help /` provides brief documentation for the specified module. `module load /` "loads" the module, i.e. modifies the user's environment ($PATH, $MANPATH, etc.) `module unload /` "unloads" the module `module list` lists all modules which are currently loaded in the user's environment ## Usage in scripts Instead of absolute paths to libraries, binaries etc. the environment variables set by the modulefile should be used in scripts, makefiles etc. By convention, an MPCDF modulefile sets an environment variable named `_HOME` (where PKG is the name of the package, for example: MKL\_HOME) which points to the root directory of the installation path (see below for example usage). Information about additional, package-specific environment variables can be obtained with the commands `module help /` and `module show /.` ## Examples 1\) Interactive session on the command line, using the Intel fortran compiler (version 19.1.3) and Intel MKL (version 2020.4 explicitly specified): ```bash module load intel/19.1.3 module load mkl/2020.4 ifort -I$MKL_HOME/include example.F -L$MKL_HOME/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core ``` 2\) Makefile (fragment): ```text FC=ifort example: example.F $(FC) -I$(MKL_HOME)/include test.F -L$(MKL_HOME)/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core ``` 3\) Handling long output from `module avail` which may not fit into a single terminal window: Piping the output to `less` using a bourne shell (e.g. bash): ```bash `( module avail ) 2>&1 | less` ``` Piping the output to `less`using a c shell (e.g. tcsh): ```sh `( module avail ) | & less` ``` ## Hierarchical environment modules To manage the wide variety of software packages resulting from all the relevant combinations of compilers and MPI libraries, we have decided to organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the top level, dependent libraries (e.g., MPI) on the second level, and further dependencies on a third level. This means that not all modules are visible initially: only after loading a compiler module will the modules dependent on it become available. Similarly, loading an MPI module in addition will make the modules dependent on that MPI library available. On current systems (e.g. Raven, Viper), no defaults are set for compilers and MPI libraries and no modules are pre-loaded. For older systems, after login, the Intel compiler, Intel MPI and Intel MKL module will be loaded by default. To start at the root of the environment modules hierarchy for those systems, issue `module purge`. For example, the FFTW library compiled with a certain Intel compiler and a certain Intel MPI library can be loaded as follows: First, load the Intel compiler module using the command ```bash module load intel/19.1.3 ``` second, the Intel MPI module with ```bash module load impi/2019.9 ``` and, finally, the FFTW module corresponding exactly to the compiler and MPI library via ```bash module load fftw-mpi ``` You may check using the command ```bash module available ``` that after the first and second steps the dependent environment modules become visible, in the present example `impi/2019.9` and `fftw-mpi`. Moreover, note that the environment modules can be loaded via a single `module load` statement as long as the order given by the hierarchy is correct, e.g., `module load intel/19.1.3 impi/2019.9 fftw-mpi`. Please always specify the exact version of the compiler and MPI library and please make sure to always use the same compiler and MPI modules for compiling your code as for running your code in a SLURM script. In case you know the name of the module you wish to load, but you are not sure about the available versions or what dependencies need to be loaded first, you can try to use the 'find-module' command. This tool searches for the MODULENAME string through a list of all installed modules ```bash find-module MODULENAME ``` You can then choose the desired module version, use the output of the command to determine the correct order to load dependencies, and finally load the module itself, e.g. ```bash $ find-module horovod horovod/cpu/0.13.11 (after loading anaconda/3/2019.03 tensorflow/cpu/1.14.0) horovod/cpu/0.15.2 (after loading anaconda/3/2019.03 tensorflow/cpu/1.14.0) horovod/gpu/0.13.11 (after loading anaconda/3/2019.03 tensorflow/gpu/1.14.0) horovod/gpu/0.15.2 (after loading anaconda/3/2019.03 tensorflow/gpu/1.14.0) $ module load anaconda/3/2019.03 tensorflow/cpu/1.14.0 horovod/cpu/0.13.11 ``` It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the top level of the hierarchy. Note that a hierarchy exists for Python modules with the 'anaconda' module files on the top level. ## Note on module dependencies Some projects require the loading of more than one compiler module and the use of dependent libraries. In other words, the projects must load at least two top-level modules and second level dependencies that exist in both module hierarchies. In this case, the second level module is chosen from the most recently loaded top-level module. This is also true for second/third level dependencies. In the following example, one would like to use the Intel compiler for a C++ code that relies on FFTW. As explained in our [compiler documentation](compilers_languages#c-header-files-and-standard-library), a GCC module will also need to be loaded in this case. Depending on loading order, different versions of the FFTW library are ultimately made available to the project: ```bash :~$ module purge :~$ module load gcc/11 intel/21.6 :~$ module load fftw-serial :~$ printenv | grep FFTW_HOME FFTW_HOME=/mpcdf/soft/SLE_12/packages/skylake/fftw/intel_21.6.0-2021.6.0/3.3.10 :~$ module purge :~$ module load intel/21.6 gcc/11 :~$ module load fftw-serial :~$ printenv | grep FFTW_HOME FFTW_HOME=/mpcdf/soft/SLE_12/packages/skylake/fftw/gcc_11-11.2.0/3.3.10 ``` # HPC Application Packages This page provides a list of scientific applications available on the HPC systems at the MPCDF. Note that different HPC systems may host only a subset of the packages listed below. Additional software can be made available on request. Software is generally made available via the [environment modules](environment-modules "Modules") system. The output of the command `module avail` provides an up-to-date list of available libraries and versions (see the section applications). Specific information about the application, links to the documentation and, if applicable, licensing restrictions are provided via the command `module help application_name`. ## Molecular dynamics and quantum chemistry packages * [ABINIT](https://www.abinit.org/) A DFT simulation package * [Amber](https://ambermd.org/) Classical molecular dynamics *(with the exception of [AmberTools](https://ambermd.org/AmberTools.php), access requires a valid Amber license to be presented to the MPCDF)* * [CP2K](https://www.cp2k.org/) _Ab-initio_ molecular dynamics * [CPMD](https://github.com/CPMD-code) _Ab-initio_ molecular dynamics * [DFTB+](https://dftbplus.org/) Quantum mechanical simulation software package * [ESPResSo++](https://espressopp.github.io/) Multiscale Simulation Package for Soft Matter Systems * [FHI-aims](https://fhi-aims.org/) _Ab-initio_ molecular simulations package * [GROMACS](https://www.gromacs.org/) Classical molecular dynamics * [LAMMPS](https://www.lammps.org/) Classical molecular dynamics * [NAMD](https://www.ks.uiuc.edu/Research/namd/) Classical molecular dynamics * [NECI](https://github.com/ghb24/NECI_STABLE) Full Configuration Interaction Quantum Monte Carlo code (FCIQMC) * [OCTOPUS](https://www.octopus-code.org/) Software package for performing _ab-initio_, time-dependent electronic structure calculations within the framework of (TD)DFT * [ORCA](https://orcaforum.kofo.mpg.de/) _Ab-initio_, DFT and semiempirical SCF-MO package *(for licensing reasons a valid EULA has to be presented to the MPCDF)* * [PLUMED](https://www.plumed.org/) Library for free energy calculations in molecular systems * [Quantum Espresso](https://www.quantum-espresso.org/) A DFT simulation package * [TURBOMOLE](https://www.turbomole.com/) Program package for _ab-initio_ electronic structure calculations *(operated under a license agreement for the MPG)* * [VASP](https://www.vasp.at/) The Vienna _Ab initio_ Simulation Package (VASP) for atomic scale materials modelling *(access is restricted to VASP licensees, contact [MPCDF helpdesk](../../../faq/help.md) in order to provide a license)* ## Other packages * [AlphaFold2](https://github.com/deepmind/alphafold) Neural-network-based model to predict the three-dimensional structure of a protein based solely on its amino acid sequence. Databases and the software are readily available on *Raven*, cf. `module help alphafold` for details. * [BioEM](https://gitlab.mpcdf.mpg.de/MPIBP-Hummer/BioEM) GPU-accelerated tool for Bayesian inference of electron microscopy images. * [COMSOL](https://www.comsol.com/) Multiphysics package *(operated under a license agreement for the MPG)* * [PRESTO](https://www.cv.nrao.edu/~sransom/presto/) Software for pulsar search and analysis * [RELION](https://www2.mrc-lmb.cam.ac.uk/relion/) Bayesian approach to refinement of (multiple) 3D reconstructions or 2D class averages in electron cryo-microscopy (cryo-EM) # AI software On this page you find a collection of information about software for data analytics and especially machine learning, which we support at the MPCDF. ```{contents} Table of Contents :local: :depth: 1 ``` ## Introduction The current best practice for using AI and data analytics software on our systems is to utilize [**containers**](#containers). Containers provide a consistent and reliable way to manage complex dependencies and ensure reproducibility across different environments. We recommend using [**Apptainer**](./containers#apptainer) (formerly Singularity) for this purpose, as it is well-suited for our HPC infrastructure. The software provided through environment modules is considered deprecated and should be used with caution. For more details on the recent changes in our Python infrastructure, please refer to the [Bits & Bytes article](https://docs.mpcdf.mpg.de/bnb/216.html#major-change-in-the-python-infrastructure-on-the-hpc-clusters). Please note that users are expected to install any required software themselves. The following sections provide guidance and resources to help you get started with containers, installing Python packages locally, and best practices for setting up your environment. ```{eval-rst} .. important:: Always verify the performance of your software setup, as it can vary depending on the installation method. You can monitor performance using our `monitoring system <../performance-monitoring>`_. ``` ## Containers AI frameworks often come with complex dependencies that can vary across systems. To manage this effectively on MPCDF systems, **containers** are the recommended solution. Containers allow you to encapsulate your software environment into a single, portable image. This greatly simplifies reproducibility and collaboration. ### Using containers on MPCDF Systems On MPCDF systems, the recommended way to work with containers is through **Apptainer** (formerly Singularity). It integrates well with batch systems, supports GPU usage, and does not require root access, making it ideal for HPC environments. **Apptainer** is available on our HPC systems via the **module system**. To see available versions, use the `find-module` command. For example: ``` $ find-module apptainer apptainer/1.3.2 apptainer/1.3.6 apptainer/1.4.1 ``` For more details on the general usage of Apptainer, please refer to: - [Our dedicated Apptainer documentation page](https://docs.mpcdf.mpg.de/doc/computing/software/containers.html) - [Apptainer's official documentation](https://apptainer.org/). ### Dedicated Apptainer examples for AI frameworks To help you get started with containers, we provide a curated AI Containers Repository on GitLab, featuring examples tailored for common AI frameworks like PyTorch and TensorFlow: - [AI Containers Repository on GitLab](https://gitlab.mpcdf.mpg.de/dataanalytics-public/ai_containers) The repository includes: - Python scripts for typical AI workflows (e.g., training) - Apptainer definition files - Slurm job submission scripts for running containers on HPC systems - Best practices and tips for using containers ### Use an apptainer image as a Jupyter kernel in RVS To add a Jupyter kernel in [RVS](../../visualization#remote-visualization-and-jupyter-notebook-services) running inside an apptainer image follow the instructions outlined in the [AI Containers Repository](https://gitlab.mpcdf.mpg.de/dataanalytics-public/ai_containers/-/blob/main/README.md#using-containers-with-rvs). ### Hardware compatibility To ensure optimal performance, it's crucial to match your containers with the appropriate hardware, especially when using GPUs. - **NVIDIA GPUs** (e.g., on the [Raven system](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html#system-overview)): - Use containers built with **CUDA** and install AI frameworks compiled with CUDA support. - Browse available images here: [NVIDIA NGC Catalog](https://catalog.ngc.nvidia.com/containers) - **AMD GPUs** (e.g. on the [Viper system](https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-user-guide.html#system-overview)): - Use containers built with **ROCm**, and ensure your AI frameworks are installed with ROCm support. - Browse available images here: [AMD ROCm Docker Hub](https://hub.docker.com/u/rocm) ## How to install Python packages locally For rapid experimentation, or if you want to leverage software already available on the HPC systems via environment modules, we recommend setting up a virtual environment to install any additional packages you may need. ### Setting up a venv First, load the Python interpreter via the Water Boa Python module: ```bash module load python-waterboa/2024.06 ``` Then load your required packages, if they are available on our module system. See our dedicated section for more information about how the [module system works](./environment-modules). Now, create your virtual environment via: ```bash python -m venv --system-site-packages ``` This command will create a directory at the given path where your software will be installed. The `--system-site-packages` flag gives the virtual environment access to the already loaded packages in the previous steps. ### Activate your venv To activate your newly created virtual environment, execute: ```bash source /bin/activate ``` ### Install packages Then you can simply install your required packages via `pip`. For example [to install PyTorch](https://pytorch.org/get-started/locally/): ```bash pip install torch ``` ```{eval-rst} .. important:: Take care of GPU support! If you require a particular build for CUDA or ROCm, consult the documentation of the software you want to install. For example, to install pyTorch: * With NVIDIA GPUs support, install a wheel built with CUDA 12.6: .. code-block:: bash pip install torch --index-url https://download.pytorch.org/whl/cu126 * With AMD GPUs support, install a wheel built with ROCm 6.3: .. code-block:: bash pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3 ``` ```{eval-rst} .. important:: Conda environments have very limited support (more details in `our documentation `_ and this `Bits and Bytes article `_). ``` ### Use a virtual environment as a Jupyter kernel in RVS You can add a Jupyter kernel in [RVS](../../visualization#remote-visualization-and-jupyter-notebook-services) running in a virtual environment. First install the `ipykernel` package inside the virtual environment: ```bash pip install ipykernel ``` Then install the kernel locally: ```bash python -m ipykernel install --user --name=my-env-name --display-name "Python (my-env-name)" ``` It will be automatically visible in the kernels list of a Jupyter Lab session in RVS. ## LLM Inference Service To support the interactive use of open-source language models, MPCDF provides an easy-to-use LLM Inference Service, available at [https://llm.mpcdf.mpg.de](https://llm.mpcdf.mpg.de). There are over 100,000 open language models hosted on [Hugging Face](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending). They cover a wide range of parameter sizes, from tiny models with a few million parameters to really gigantic models with up to one trillion parameters. While smaller models can outperform larger ones on specific tasks, especially when fine‑tuned, the general capabilities of the biggest open models rival those of closed models such as GPT or Gemini. However, running even the “smaller” models efficiently already requires notable compute resources, and the largest models require substantial AI hardware that is often out of reach for individual research groups. To enable researchers in running and evaluating open-source LLMs, MPCDF provides the necessary computational resources along with an easy-to-use inference service, available at [https://llm.mpcdf.mpg.de](https://llm.mpcdf.mpg.de) The LLM Inference Service is a flexible web application that allows you to create endpoints exposing a model via a REST API. For this, we rely on popular inference frameworks such as [vLLM](https://github.com/vllm-project/vllm) and [Ollama](https://ollama.com/). ```{eval-rst} .. important:: When a user creates an endpoint, the service submits the corresponding Slurm job under that user’s own account. This means that the job is queued, scheduled, and accounted for like any other regular Slurm job. ``` The service then exposes the model through a routed REST API endpoint, so it can be accessed conveniently from a local machine or existing tools. Via an intuitive UI, users can request the desired hardware and configure the framework. We provide sensible default configurations for the frameworks to help you get started quickly. At the same time, you remain free to tune the framework parameters to your needs and to run any model and modality supported by the respective framework, including your own fine‑tuned models, provided they are hosted on the Hugging Face Hub. Examples for starting and configuring the inference service, as well as for calling the API, are available on the [Recipes page](https://llm.mpcdf.mpg.de/recipes) in the LLM Inference UI. Currently, two of the most powerful GPU systems at MPCDF, [_DAIS_](https://docs.mpcdf.mpg.de/doc/computing/dais-user-guide.html) and [_Viper-GPU_](https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-user-guide.html), are connected to the service. ```{eval-rst} .. important:: To use the LLM Inference Service, you need an active MPCDF account with access to HPC systems enabled. With such an account, you should be able to run the service on the `Viper-GPU `_ system. Users with access to `DAIS `_ can also run the service on that system. To log in to the `LLM Inference Service `_, please use your normal Kerberos credentials. Please note that before creating your first endpoint on a given HPC system, you need to log in directly to that system at least once. ``` The LLM Inference Service is targeted at researchers who wish to interactively evaluate specific open models or conduct interactive user studies. For non-interactive workloads, such as extensive benchmarks or offline evaluations, we recommend using Slurm batch jobs to ensure efficient resource utilization. Example scripts for submitting such jobs are available in our [LLMs-meet-MPCDF](https://gitlab.mpcdf.mpg.de/dataanalytics-public/llms-meet-mpcdf) GitLab repository. Additionally, for users interested in testing "standard" open models, the [Chat AI](https://docs.hpc.gwdg.de/services/chat-ai/index.html) service by GWDG is an excellent alternative. It offers a user-friendly chat interface as well as access to an [inference API](https://docs.hpc.gwdg.de/services/saia/index.html). ## Cautions and best‑practice notes for AI workloads on HPC systems AI frameworks (e.g. PyTorch, TensorFlow, JAX) often automatically create a large number of OpenMP threads internally. On the HPC systems this can exhaust the available CPU resources, increase contention, and in the worst case cause node crashes or job termination. The following recommendations help to keep the nodes stable while preserving most of the performance: ### Set `OMP_WAIT_POLICY=PASSIVE` This tells the OpenMP runtime to put idle threads into a low‑power wait state instead of busy‑waiting. The impact on performance is typically negligible, but it can dramatically reduce the likelihood of node failures, especially on the **Viper** GPU nodes where aggressive thread spawning is a common cause of instability. ```{eval-rst} .. important:: When running AI workloads on the **Viper** GPU system we strongly recommend setting OMP_WAIT_POLICY to PASSIVE. ``` ### Limit the number of OpenMP threads `OMP_NUM_THREADS=1` is a safe starting point for many AI applications and guarantees that only a single thread per process is active. For workloads that benefit from multi‑threaded CPU kernels (e.g. data loading, BLAS operations) you may increase this value, but always benchmark the impact on your specific model and dataset. ```{eval-rst} .. important:: Use our internal monitoring system (see the `monitoring system <../performance-monitoring>`_ page) to watch for an unusually high number of threads, CPU usage, or memory pressure. If you observe such symptoms, try setting OMP_NUM_THREADS to a low number. ``` # Compilers and languages ```{contents} Contents :local: :depth: 1 ``` ## Intel Compilers ### Intel C/C++ Compiler for Linux #### Usage The name of the C compiler executable is `icx`, the name of the C++ compiler executable is `icpx`. Compilation and linking of a C program (source file `myprog.c`) is done as follows: `icx -o myprog myprog.c` To get an overview of the available command line options use the command `icx --help`. More information is provided by the manual page `man icx`. Extensive documentation and, e.g., information on code optimization strategies is provided by the official [Intel C/C++ Compiler Documentation](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compiler.html). To compile and link MPI codes with Intel MPI, use the wrappers `mpiicx` and `mpiicpx`, respectively. #### Compiling and linking against a more recent C++ standard library The C++ standard library headers and shared objects installed in the default system folders are relatively dated. This can cause errors such as * `<>` * `GLIBC_2.33 not found` at runtime when compiling and linking. The recommended procedure to avoid these errors is the following: 1. Clean the currently loaded environment modules with `module purge`. 2. Load the compiler module you want to use and all depending modules you need. 3. Export the environment variables `CC` and `CXX` with the compiler you want to use. 4. Lastly, load a recent gcc version, e.g. `module load gcc/`. Do not load any other modules afterwards. 5. Set `LDFLAGS` to point to the recent GCC version of the standard library, e.g. `export LDFLAGS="$LDFLAGS -L${GCC_HOME}/lib64 -Wl,-rpath,${GCC_HOME}/lib64"`. 6. Configure and build your application. Be aware that CMake checks `LDFLAGS` at the first invocation only. Make sure to create a new build with CMake, if applicable. ### Intel Fortran Compiler for Linux #### Usage The name of the Intel Fortran Compiler executable is `ifx`. Compilation and linking of a Fortran program (source file `myprog.f90`) is done as follows: `ifx -o myprog myprog.f90` To get an overview on the available command line options use the command `ifx --help`. More information is provided by the manual page `man ifx`. Extensive documentation and e.g. information on code optimization strategies is provided by the official [Intel Fortran Compiler Documentation](https://www.intel.com/content/www/us/en/developer/tools/oneapi/fortran-compiler.html). To compile and link MPI codes with Intel MPI, use the wrapper `mpiifx`. ### How to get access to the Intel Compilers On the Raven and Viper supercomputers, users need to load and specify a version of the Intel compiler explicitly, similarly for Intel MPI. No default versions exist for the Intel compiler and MPI modules, and no default versions are loaded at login. To get a list of all available Intel compilers, enter `module avail intel`. To get access to a specific Intel compiler, load the module by `module load intel/`. ### Intel Compiler for Linux Optimization Flags #### Compiler optimization flags Compiler optimization flags have strong influence on the performance of the executable. Some important flags are given below. First, different optimization levels are available: * `-O2`: Standard optimization. Default. * `-O3`: Aggressive optimization. Use it with care and check the results against a less optimized binary. * `-O0`: Disables all optimization. Useful for fast compilation and to check if unexpected behavior results from a higher compiler optimization level. * `-O1`: Very conservative optimization. In addition, [vectorization](https://software.intel.com/content/www/us/en/develop/articles/improve-performance-with-vectorization.html) is key to achieving good floating point performance on modern CPUs. For detailed information on how to specify the instruction set level during compilation please consult the [Intel Compiler Documentation](https://software.intel.com/content/www/us/en/develop/tools/compilers/c-compilers/documentation.html) In particular, the switches `-x`, `-ax`, `-m` are relevant, for example: * `-xCORE-AVX512 -qopt-zmm-usage=high`: Enable AVX512 vectorization for Intel Skylake, CascadeLake, IceLake processors. These flags are recommended on *Raven*. * `-march=znver4`, `-march=skylake-avx512`, or `–march=core-avx2`: Enable AVX512 or AVX2 vectorization compatible with AMD Zen4 CPUs. These flags are recommended on *Viper*. * `-xCORE-AVX2`: Enable AVX2 vectorization for Intel Haswell and Broadwell CPUs. * `-ipo`: Enable interprocedural optimizations beyond individual source files. **The meta switch `-fast` is not supported on MPCDF systems** because it forces the static linking of *all* libraries (i.e. it implies the switch `-static`) which is not possible with certain system libraries. \[\*\] To obtain information about the features of the Linux host CPU issue the command `cat /proc/cpuinfo | grep flags | head -1`. Instruction-set-related keywords are, among others, *avx, avx2, avx512*. The list of all supported switches and extensive information is covered by the official [Intel Compiler Documentation](https://software.intel.com/content/www/us/en/develop/tools/compilers/c-compilers/documentation.html). #### Floating point accuracy Intel compilers tend to adopt increasingly more aggressive defaults for the optimization of floating-point semantics. The default is `-fp-model fast=1`. We recommend to double check the accuracy of simulation results by using more conservative settings (which might come at the expense of computational performance) like `-fp-model precise` (recommended) or even `-fp-model strict`. See the compiler man pages for more details. ## GNU Compiler Collection The [GNU Compiler Collection](https://gcc.gnu.org/) provides -- among others -- front ends for C, C++, and Fortran. A default version of GCC comes with the operating system. More recent versions suitable for HPC can be accessed via [environment modules](./environment-modules). To compile and link MPI codes using the GNU compilers, use the commands `mpigcc`, `mpig++`, or `mpigfortran` in combination with Intel MPI. Find the full documentation at . ## GPU Programming The following packages are provided on the HPC clusters to enable users develop applications for NVIDIA GPUs. ### NVIDIA CUDA Toolkit [The NVIDIA CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) provides a development environment for the programming of NVIDIA GPUs. It includes the CUDA C++ compiler (`nvcc`), optimized libraries, debuggers and profilers, among others. Issue `module avail cuda` to get an up-to-date list of the CUDA versions available on a system. ### NVIDIA HPC SDK [The NVIDIA HPC SDK](https://developer.nvidia.com/hpc-sdk) provides a C, C++, and Fortran compiler for the programming of NVIDIA GPUs and multi-core CPUs. It is the successor product of the PGI compiler suite. In addition, the NVIDIA HPC SDK comprises a copy of the CUDA toolkit and various libraries for numerical computation, deep learning and AI, and communication. Issue `module avail nvhpcsdk` to get an up-to-date list of the versions available on a system. ### Kokkos C++ Performance Portability Library [The Kokkos performance portability framework](https://github.com/kokkos/kokkos) enables the development of applications that achieve consistent good performance across all relevant modern HPC platforms based on a single-source C++ implementation. It provides abstractions for parallel computation and data management, and supports several backends such as OpenMP and CUDA, among others. ### Python The high-level Python programming language can be extended with modules written in plain C++/CUDA to leverage GPU computing. In this case, the interfaces may be created with Cython or pybind11 in a comparably easy way. Alternatively, the PyCUDA module offers a straight forward way to embed CUDA code into Python modules. Codes that make heavy use of NumPy may compile such costly expressions to CPU or GPU machine code using the Numba package. Note that Numba is non-intrusive as it only uses decorators. It is part of the Anaconda Python distribution. ## NAG Fortran compiler Usually we have the latest NAG Fortran compiler installed. To see all available NAG compilers on UNIX, enter `module avail nagf95`. To use a specific NAG compiler, load the module by `module load nagf95/<$version>`. The compiler command is `nagfor`. To use the compiler on windows, follow the instructions given in ```text /afs/ipp-garching.mpg.de/common/soft/nag_f95/<$version>/windows/readme.txt ``` for versions equal or later *rel5.3*. For access a valid AFS-Token for the cell *ipp-garching.mpg.de* is necessary. More information about the NAG Fortran Compiler can be found in the documentation of NAG at [NAG Fortran Compiler Documentation](https://www.nag.com/content/nag-fortran-compiler). ## Python At the MPCDF, Python including a plethora of scientific packages for numerical computing and data science (NumPy, SciPy, matplotlib, Cython, Numba, Pandas, etc.) used to be provided in an up-to-date fashion via the Anaconda Python Distribution. Starting in 2024, a new Python basis is deployed, based on free software sources. Run the commands ```bash module avail python-waterboa module help python-waterboa ``` to get information on what's available, where the versioning is similar to the one of Anaconda. A list of the installed legacy Anaconda releases can be obtained via the following command: ```bash module avail anaconda ``` Please note that new versions of Anaconda Python cannot be provided any more due to licensing restrictions. ### Python for HPC Being an interpreted and dynamically-typed language, plain Python is not a language suitable per-se to achieve high performance. Nevertheless, with the appropriate packages, tools, and techniques the Python programming language can be used to perform numerical computation in a very efficient manner, covering both aspects, the program’s efficiency and the programmer’s efficiency. The aim of this article is to provide some advice and orientation to the reader in order to use Python correctly on the HPC systems and to take first steps towards basic Python code optimization. #### Performance The key to achieve good performance with Python is to move notably expensive computation from the interpreted code layer down to a compiled layer which may consist of compiled libraries, code written and compiled by the user, or just-in-time compiled code. Below, three packages are discussed for these use cases. ##### NumPy NumPy is the Python module that provides arrays of native datatypes (float32, float64, int64, etc.) and mathematical operations and functions on them. Typically, mathematical equations (in particular, vector and matrix arithmetic) can be written with NumPy expressions in a very readable and elegant way, which brings several advantages: NumPy expressions avoid explicit, slow loops in Python. In addition, NumPy uses compiled code and optimized mathematical libraries internally, e.g. Intel MKL on MPCDF systems, which enables vectorization and other optimizations. Parts of these libraries use thread-parallelization in a very efficient way by default, e.g. to perform matrix multiplications. In summary, NumPy provides the de-facto standard for numerical array-based computations and serves as the basis for a multitude of additional packages. ##### Cython Cython is a Python language extension that makes it relatively easy to create compiled Python modules written in Cython, C or C++. It integrates well with NumPy arrays and can be used to implement time-critical parts of an algorithm. Moreover, Cython is very useful to create interfaces to C or C++ code, such as legacy libraries or native CUDA code. Technically, the Cython source code is translated by the Cython compiler to intermediate C code which is then compiled to machine code by a regular C compiler like GCC or ICC. ##### Numba Numba is a just-in-time compiler based on the LLVM framework. It compiles Python functions at runtime for the datatypes these functions are being called with. Moreover, Numba implements a subset of NumPy’s functions, i.e. it is able to compile NumPy expressions. Functions are declared via a simple decorator-syntax to be suitable for jit-compilation, hence, Numba is only little intrusive on existing code bases. ### Parallelization While Python does implement threads as part of the standard library, these cannot be used to accelerate computation on more than one core in parallel due to cPython’s global interpreter lock. Nevertheless, Python is suitable for parallel computation. In the following, two important packages for intra-node and inter-node parallelism are addressed. #### multiprocessing The multiprocessing package is part of the Python standard library. It implements building blocks such as pools of workers and communication queues that can be used to parallelize data-parallel workloads. Technically, multiprocessing forks subprocesses from the main Python process that can run in parallel on multiple cores of a shared-memory machine. Note that some overhead is associated with the inter-process communication. It is, however, possible to access shared memory from several processes simultaneously. A typical use case would be large NumPy arrays. #### mpi4py Access to the Message Passing Interface (MPI) is available via the module mpi4py. It enables parallel computation on distributed-memory computers where the processes communicate via messages with each other. In particular, the mpi4py package supports the communication of NumPy arrays without additional overhead. On MPCDF systems, the environment module mpi4py provides an optimized build based on the default Intel MPI library. ### IO NumPy implements efficient binary IO for array data that is useful, e.g., for temporary files. A better choice with respect to portability and long-term compatibility are HDF5 files. HDF5 is accessible via the h5py Python package and offers an easy-to-use dictionary-style interface. For parallel codes, a special build of h5py with support for MPI-parallel IO is provided via the environment module h5py-mpi. ### The Python software ecosystem In addition to the packages discussed up to now, there is a plethora of solid and well-proven packages for scientific computation and data science available, covering, e.g., numerical libraries (SciPy), visualization (matplotlib, seaborn), data analysis (pandas), and machine learning (TensorFlow, pytorch), to name only a few. ### Software installation Often, users need to install special Python packages for their scientific domain. In most cases, the easiest and quickest way is to create an installation local to the user’s homedirectory. After loading the Anaconda environment module, the command `pip install --user PACKAGE_NAME` would download and install a package from the Python package index (PyPI), or similarly, the command `python setup.py install --user` would install a package from an unpacked source tarball. In both cases, the resulting installation is located below “~/.local” where Python will find it by default. ### Summary The software recommended in this article is available via the Anaconda Python Distribution (environment module “anaconda/3”) on MPCDF systems. note that for some packages (mpi4py, h5py-mpi), the hierarchical environment modules matter, i.e., it is necessary to load a compiler (gcc, intel) and an MPI module (impi) in addition to Anaconda in order to get access to these dependent environment modules. The application group at the MPCDF has developed an in-depth course on “Python for HPC” which covers all the topics touched in this article in more detail on two days. It is taught one to two times per year and announced via the MPCDF web page. Finally, it should be pointed out that Python 2 reaches its official end-of-life on January 1, 2020. Consequently, new Python modules and updates to existing ones will not take Python 2 compatibility into account in the future. Users still running legacy code are strongly encouraged to migrate to Python 3. # Libraries ## Intel Math Kernel Library The Intel Math Kernel Library (MKL) is installed on Linux systems and can be loaded using the modules environment. Just enter `module load mkl` to get access to the Math Kernel Library. The MKL library is composed of highly optimized mathematical routines. It contains, among others, LAPACK, the Basic Linear Algebra Subprograms (BLAS) and the extended BLAS. For parallel computing it provides ScaLAPACK, the Basic Linear Algebra Communications Subprograms (BLACS) and the Parallel Basic Linear Algebra Subprograms (PBLAS). Detailed information can be found on the [Intel Math Kernel Library Website](https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation). The link line for binding MKL to your application depends on the MKL functions used and if parallel or sequential use is required. You can find detailed information on the web at [https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor](https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor) or contact the application support team at MPCDF. ## ELPA [ELPA](https://elpa.mpcdf.mpg.de), a library of scalable, high-performance direct solvers for symmetric/Hermitian eigenvalue problems. On MPCDF computing platforms, ELPA is provided via the [environment module](environment-modules) system. ## ScaLAPACK ScaLAPACK includes routines for the solution of dense, band, and tridiagonal linear systems of equations, condition estimation and iterative refinement, for LU and Cholesky factorization, matrix inversion, full-rank linear least squares problems, orthogonal and generalized orthogonal factorizations, orthogonal transformation routines, reductions to upper Hessenberg, bidiagonal and tridiagonal form, reduction of a symmetric-definite/ Hermitian-definite generalized eigenproblem to standard form, the symmetric/Hermitian, generalized symmetric/Hermitian, the nonsymmetric eigenproblem, and the singular value decomposition. See for more information. On the MPCDF computing platforms, ScaLAPACK is provided as part of the [Intel MKL](https://software.intel.com/en-us/intel-mkl), a library of optimized mathematical routines. MKL has to be loaded via the [environment module](environment-modules) system. It is strongly recommended to use the [Intel MKL Link Line Advisor](https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor) for generating correct link lines. ## PETSc and SLEPc [PETSc](https://petsc.org), the Portable Extensible Toolkit for Scientific Computation from Argonne (ANL), is a suite of data structures and routines for the uni-processor and parallel-processor solution of large-scale scientific application problems modeled by partial differential equations. [SLEPc](https://slepc.upv.es/) is a software package for the solution of large sparse eigenproblems on parallel computers. It is built on top of PETSc. Both libraries are available on MPCDF machines via [environment modules](environment-modules). ## Intel Integrated Performance Primitives The Intel Integrated Performance Primitives API, containing highly optimized primitive operations used for **digital filtering, audio and image processing**, is installed on the Linux systems, and the current version can be made available by invoking `module load ipp`. In-depth information can be found in the [Intel IPP documentation](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/ipp.html). ## Intel Threading Building Blocks The Intel Threading Building Blocks (TBB) enables the C++ programmer to integrate (shared memory) parallel capability into the code. It is installed on Linux Systems and the current version can be made available by invoking `module load tbb` Then compiling is done in the following way: ```bash icpc -c -o main.o -I$TBB\_HOME/include/tbb main.cpp ``` Linking is only possible against shared libraries, for example: ```bash icpc -o main main.o -L$TBB\_HOME/intel64/gcc4.8 -ltbb ``` For detailed information please read [Intel Threading Building Blocks](https://www.intel.com/content/www/us/en/develop/documentation/onetbb-documentation/top.html). ## GPU-enabled numerical libraries Applications that rely on expensive library calls to standard routines from BLAS, LAPACK, FFTW, or similar may benefit from GPU-enabled libraries that can easily be used as drop-in replacements for the respective CPU libraries. Moreover, many GPU-specific libraries exist for specific purposes that may provide the functionality required for a user code. ### GPU-enabled libraries bundled with NVIDIA CUDA The NVIDIA CUDA Toolkit provides optimized numerical libraries that can be called and linked from user code in order to perform computations on the GPUs, for example - cuBLAS - cuFFT - cuSPARSE - cuSOLVER Please consult the [NVIDIA CUDA Toolkit documentation](https://docs.nvidia.com/cuda/index.html) for a comprehensive list and more detailed information. CUDA is available via the environment module 'cuda'. ### MAGMA The [MAGMA library](https://icl.cs.utk.edu/magma/) provides implementations of BLAS and LAPACK routines optimized for multi-core CPU and multi-GPU systems. MAGMA is available via the environment module 'magma'. ## NAG The [NAG library](https://nag.com) is a collection of mathematical and statistical routines developed by the [The **N**umerical **A**lgorithms **G**roup Ltd.](https://nag.com), Wilkinson House, Jordan Hill Road, Oxford, OX2 8DR, UK. ### Availability The NAG library is supported for Fortran and C compilers. The Fortran library is F77 style with additional modules or interfaces for usage from F90/95/2003. At the MPCDF a campus license (i.e. for arbitrary architectures and compilers and an arbitrary number of users) is available for the serial NAG Fortran and C libraries. The libraries are provided for Unix architectures (Linux, Solaris). ### Usage The NAG libraries are made available via the [environment module](environment-modules) system. The output of the command `module avail nag` provides an up-to-date list of available versions for the different compilers supported on that system. Use `module help ` for documentation and detailed usage instructions. For some NAG routines the license file is also required at run-time. Therefore, execute the command `module load ` before starting your program. ## How to link libraries on Linux systems ### Introduction Assume that there's a program you would like to build which depends on a (fictitious) library called "libfoo". As usual on MPCDF systems, the library is provided via an environment module: ```bash $ module load libfoo # LIBFOO_HOME is set to the full installation path $ ls $LIBFOO_HOME/lib # list the contents of the "lib" subdirectory libfoo.so libfoo.a ``` A file ending with ".so" (e.g. libfoo.so) is called a *shared library* or *shared object*, whereas a file ending with ".a" (e.g. libfoo.a) is called a *static library* or *archive*. Both types can be utilized (linked) from user code. In the following we assume that there is a C program named "main.c" which calls functions from the library "libfoo". ### Linking static libraries To compile the program "main.c" and link it statically with "libfoo.a", issue the following commands: ```bash $ icc -c -I$LIBFOO_HOME/include main.c # compile step $ icc -o main main.o $LIBFOO_HOME/lib/libfoo.a # link step ``` Note that the full path to "libfoo.a" is provided during the link step, without any additional command line flags (such as "-L" or "-l"). Finally, an executable "main" is created. In particular during the link step, all the functions in "libfoo" which are required by our program are actually copied from "libfoo.a" into the executable "main". Hence, the executable is independent of the file "libfoo.a" afterwards. ### Linking dynamic libraries, setting the RPATH To compile the program "main.c" and link it with the dynamic library "libfoo.so", issue the following commands: ```bash $ icc -c -I$LIBFOO_HOME/include main.c $ icc -o main main.o -L$LIBFOO_HOME/lib -lfoo -Wl,-rpath,$LIBFOO_HOME/lib ``` As a result, an executable "main" is created. Now, with dynamic linking, the functions are not copied from the library to the executable during the link step. They are only referenced from the executable. Hence, the executable depends on the file "libfoo.so" afterwards. It is smaller in size than the statically linked executable. Note that three command line arguments are involved in the link step: `-L`: path where the compiler looks for libraries at link time `-l`: name of the library to be linked (i.e. the file name without "lib" and ".so") `-Wl,-rpath,`: path where the executable needs to look for shared libraries at run time. Without the `-Wl,-rpath,$LIBFOO_HOME/lib` command line argument, the program would be linked correctly, but, at runtime, the executable would not be able to find the shared library because of lacking path information. This is the reason why that information must be added which is called the RPATH. Note that there's no whitespace in the argument. Note, in addition, that the environment variable LD\_LIBRARY\_PATH overrides the RPATH. Setting the LD\_LIBRARY\_PATH should be avoided, if possible. You can verify that the library "libfoo" is referenced correctly in the executable "main" by running `ldd main`. This will print a list of all dynamic libraries linked in your executable. Specifically, the output should contain a line for "libfoo.so": ```bash libfoo.so => /some/path/to/the/library/libfoo.so ``` Note that if "not found" is displayed for any library, this means that the RPATH is not set correctly and the executable will not be able to run. ### Compilers The information provided here is valid for Intel and GNU compilers (C/C++/Fortran) on Linux systems. ## Watson Sparse Matrix Package Licensing of WSMP was discontinued in 2020. # Debugging tools ```{contents} Contents :local: :depth: 1 ``` ## Using the compiler The easiest and always available tool for code validation is the compiler. Most modern compilers have many debugging flags, which can help to find different kinds of bugs in a code. This article should provide a guideline or checklist for validating arbitrary code. ### Checks during the compilation First step is clearly to compile the code and to remove all errors which are reported by the compiler. While this is an indispensable prerequisite for code development, the compiler is able of much more checking. After the code compiles with some standard flags needed for the code, one can add warning flags, which cause the compiler to issue warnings about suspicious code structures, unused variables or arguments. For the Intel Compiler, this is added by (the ... are the other compiler arguments) ```bash ifort -warn all ... ``` for the GNU compiler it is ```bash gfortran -Wall ... ``` In principle all of these warnings should be considered and removed, but sometimes, one does not want to have some special warnings (for example about unused variables), then one can disable some of the warnings by ```bash ifort -warn all -diag-disable [ID of the warning] ... ``` where the ID is given at the output of the warning. ### Run-time checks If all warnings are worked on, one can go a step further and run the code. The results have to be carefully checked by the user, as no tool is able to do this. But what the compiler can do are some run-time checks. The most important are the boundary checks (in Fortran programs) of arrays. But there are some other checks like usage of uninitialized variables or the creation of temporary arrays. These checks can be switched on at compile time with ```bash ifx -g -traceback -O0 -check all ... ``` It is usually a good idea to also add flags that cause the compiler to include information about the source code into the executable to better find the location of the reported errors. For the same reason, disabling optimization also helps in finding errors. For GNU, these run-time checks can be switched on by ```bash gfortran -g -fbacktrace -O0 -fcheck=all ... ``` These checks are important and often help to find hidden errors, which only come up in some situations (and are then difficult to debug). ### Floating point exceptions A further run-time check is the check for floating point exceptions, like division by zero and so on. This can be established by ```bash ifort -g -traceback -fpe0 ... ``` and ```bash gfortran -g -fbacktrace -ffpe-trap=invalid,zero,overflow ... ``` ## Linaro DDT DDT, the Distributed Debugging Tool from Linaro (formerly Allinea, then ARM), is a comprehensive graphical debugger for scalar, multi-threaded and large-scale parallel applications, written in C, C++ or Fortran. The MPCDF holds a license for **a total of 256 processes**. DDT is installed on the High-Performance Computing systems and on the Linux compute clusters. For MPI applications, it is possible to either debug a job running on the compute nodes, or to debug a small application (no more than 4 processes) on the interactive nodes. ### Option 1: Utilization of interactive nodes If the debugging session can be performed with **up to 4 processes** and memory requirements do not exceed the capacity of a single node, the utilization of an interactive node is possible as follows: Access to the interactive node from the cluster, enabling the X11 forwarding (graphics) and compression. ```bash ssh -YC user@raven-i.mpcdf.mpg.de ``` Load arm\_ddt and the desired version of compiler and mpi. ```bash module load intel impi arm_ddt ``` Launch the debugger. ```bash ddt ``` Configure your debugging session: Firstly, indicate your binary (compiled with debugging symbols), working directory and application parameters. Secondly, set up the ***number of MPI processes***, define the field '***implementation***' as '***SLURM (generic)***', and define the mandatory ***srun arguments*** for memory and estimated job time ('--mem=XX' and '--time=YY'). Finally, press the '***Run***' button and wait for the slurm response allowing the debugging session. ![](ddt-slurm-srun-config-screenshot.png "DDT Slurm srun config screenshot") ### Option 2: debugging a BATCH job The process of debugging a scheduled job is advised when a larger setup needs to be debugged. To perform a debugging session with a job running on the compute nodes, first it is necessary to start a ddt gui (graphical user interface) and afterwards to submit a job with a special command that is going to attach your job with your already running ddt session. **First**, access the cluster's login node enabling the X11 forwarding (graphics) and compression. ```bash ssh -YC user@raven-i.mpcdf.mpg.de ``` Load the arm ddt debugger. ```bash module load arm_ddt ``` Launch the debugger gui. You can use the '&' which will execute the program in the background. ```bash ddt & ``` **Second**, you need to modify your job submission script as follows: By adding a module load command for the arm\_ddt ```bash module load arm_ddt ``` And by updating your srun command line by inserting 'ddt --connect' before the call. ```bash ## srun ./binary ddt --connect srun ./binary ``` **Third**, launch your updated job from the same node where the GUI is running, and wait for the allocation on the compute nodes. Once the job scheduler (SLURM) has assigned the resources, the ddt gui will be attached and the debugging session will start. ```bash sbatch myjobscript.bash ``` Example of a job script for debugging a simple hybrid application (pincheck) running on 4 Raven nodes. ```bash ##!/bin/bash -l ##SBATCH -o ./job.out.%j ##SBATCH -e ./job.err.%j ##SBATCH -D ./ ##SBATCH -J DDT ##SBATCH --nodes=4 ##SBATCH --ntasks-per-node=2 ##SBATCH --cpus-per-task=36 ##SBATCH --mail-type=none ##SBATCH --time=01:00:00 export OMP_NUM_THREADS=36 module purge module load intel impi module load arm_ddt ddt --connect srun ./pincheck ``` Further information: - - [users guide](https://docs.linaroforge.com/24.1.3/html/forge/forge/introduction_to_forge/ddt.html#ddt) ## Forcheck Forcheck is a tool for static verification of the correctness of Fortran 77, 95 and 2003 programs. It also supports treatment of interface checking including module dependencies via a facility of creating Forcheck-specific library files. Forcheck is installed on x86 Linux cluster systems and can be initialized by invoking ```bash module load forcheck ``` Then, the following executables are available: - `forchk` for checking of Fortran syntax - `interf` for generation of explicit interfaces from Forcheck library files - `fcklib` for maintaining Forcheck library files _Installed versions are frozen at version 14 of the tool and provided as is._ ## Intel Trace Analyzer and Collector (ITAC) In addition to its comprehensive MPI profiling capabilities the [Intel Trace Analyzer and Collector](https://software.intel.com/en-us/intel-trace-analyzer) (ITAC) provides a tool for checking MPI correctness (detect deadlocks, ...). ITAC is installed on x86 Linux cluster systems as well as the HPC systems. Under Intel-MPI the MPI correctness check can be enabled with ```bash module load intel impi itac mpiexec -check-mpi ./a.out ``` In Slurm batch environments, where `srun` is the preferred MPI launcher, the following method with LD\_PRELOAD can be used: ```bash module load intel impi itac export LD_PRELOAD=$VT_SLIB_DIR/libVTmc.so:$I_MPI_ROOT/lib/debug/libmpi.so srun ./a.out ``` _Intel has announced the end-of-life for ITAC._ ## Intel Inspector / Thread Checker The Intel Inspector (formerly known as Intel Thread Checker) is a tool for analyzing and debugging a threaded application. The tool - helps you create threaded applications by identifying shared and private variable conflicts, - locates threading errors before they occur with an intuitive understanding of threaded application behavior, - isolates threading bugs to the source code line where the bug occurs, showing you exactly where in your program threading errors are likely to happen, - describes possible causes of threading errors and suggested solutions with one-click diagnostic help. The Thread Checker is installed on x86 Linux cluster systems as well as the HPC systems and can be initialized by invoking ```bash module load inspector ``` Then, the command `inspxe-cl` for the command line usage and the command `inspxe-gui` for a GUI are available. For more details please have a look at the following documents: - _Intel has announced the end-of-life for Inspector._ ## Heaptrack Heaptrack is a memory profiler originally developed for KDE. An overview of its features can be found [here](https://github.com/KDE/heaptrack). The command-line version is installed on Raven. Data collection for the executable `a.out` can be triggered in a job script in the following way: ```bash module load heaptrack srun hpcmd_suspend heaptrack ./a.out ``` A `.gz` file prefixed with `heaptrack` will be created in the job submission directory. To print the collected data to stdout, run `heaptrack --analyze` on that file. Note that heaptrack does not support MPI intrinsically, but it will create a separate data file for each rank. ## Valgrind Valgrind is a memory debugging and profiling tool. Documentation can be found at the [Valgrind](https://valgrind.org/) home page. # Performance tools ```{contents} Contents :local: :depth: 1 ``` ## Intel APS The Intel Application Performance Snapshot (APS) can give an overview of possible performance issues. We recommend using this tool first before looking into further details. You can invoke the analysis in your SLURM job script in the following way: ```bash module load vtune srun hpcmd_suspend aps --stat-level=4 -r aps_result -- ./my_application ``` This command will write the performance data into the subdirectory `aps_result`. After the job has finished, the report can be generated with `aps-report -a aps_result`. You will find two HTML files in the current directory, which you can view in any browser. ## Intel VTune Intel VTune is a statistical profiling and performance analysis tool for Intel processors. A command line interface (`vtune`) and a GUI (`vtune-gui`) are provided by the environment module `module load vtune`. Below, we give an example of how you can record the profiling data for your application in your SLURM job script: ```bash module load vtune srun hpcmd_suspend vtune -collect hpc-performance -r vtune_hpc_performance -- ./my_application srun hpcmd_suspend vtune -collect hotspots -trace-mpi -r vtune_hotspots -- ./my_application srun hpcmd_suspend vtune -collect uarch-exploration -r vtune_uarch -- ./my_application ``` This will create three subdirectories `vtune_*`, which contain the corresponding profiling data. You can inspect the data with `vtune-gui`, in which you can open the respective directory of interest. ```{eval-rst} .. important:: If you encounter error messages like `Failed to create data directory: Too many open files` or `Insufficient limit for open file descriptors in the system for driverless collection`, then add a line `ulimit -n 16384` after `module load vtune`. ``` For detailed documentation please have a look at the [Intel VTune documentation](https://www.intel.com/content/www/us/en/develop/documentation/vtune-help/top.html) and the [Vtune cookbook](https://software.intel.com/content/www/us/en/develop/documentation/vtune-cookbook/top.html). ## Intel Advisor The Intel Advisor is a threading design and prototyping tool for software developers. Since version 2016 comprehensive SIMD-vectorization analysis capabilities have been added. Features: - Analyze, design, tune and check your threading and SIMD-vectorization design before implementation - Explore and test threading options without disrupting normal development - Predict thread errors & performance scaling on systems with more cores It can be made available on Linux clusters and HPC systems by invoking `module load advisor`. Intel Advisor is particularly useful for analyzing the roofline data for your application. This can be achieved by inserting the following lines into your SLURM job script: ```bash module load advisor srun hpcmd_suspend advisor --collect survey --project-dir advisor_roofline -- ./my_application srun hpcmd_suspend advisor --collect tripcounts --project-dir advisor_roofline --flop --no-trip-counts -- ./my_application ``` For producing the roofline plot it's recommended to use the whole node, e.g. by specifying `--cpus-per-task=72` for serial jobs. The result can then be viewed either by invoking GUI with `advisor-gui advisor_roofline/` or by producing a roofline plot in html format by `advisor --report=roofline --report-output=advisor_roofline/roofline.html --project-dir=advisor_roofline`. For a more detailed overview and information please have a look at the [Intel Advisor Documentation.](https://software.intel.com/en-us/get-started-with-advisor-resources) and the [Advisor cookbook](https://software.intel.com/content/www/us/en/develop/documentation/advisor-cookbook/top.html). ## Intel Trace Collector and Analyzer (ITAC) The Intel® Trace Analyzer and Collector (ITAC) is a tool for understanding MPI behaviour of applications. In the execution phase it collects data and produces trace files that can subsequently be analyzed with the Intel® Trace Analyzer performance analysis tool. ITAC can be used to analyze and visualize MPI communication behaviour of a given program and possibly detect inefficiencies and load imbalances. To **collect** data do the following: 1. `module load itac` 2. Compile your code with an Intel compiler specifying "-g -tcollect", e.g. `mpiifort -g -tcollect *.f90` 3. Set the environment variable `VT_FLUSH_PREFIX` to some directory with plenty of space (`/ptmp/USERID`) 4. Run your program as usual By this a trace file (suffix `.stf`) should have been written. It can be **analyzed** with the trace analyzer tool as follows: 1. `module load itac` 2. traceanalyzer .stf ITAC also provides a mechanism to **check correctness** of MPI applications. The runtime checker will detect data type mismatches and deadlocks. Adapt your job script as follows to evaluate your MPI application: 1. `module load itac` 2. `srun --export=ALL,LD_PRELOAD=$ITAC_HOME/intel64/slib/libVTmc.so ` These recipes represent only the easiest and most obvious access. There are also more refined methods either by instrumenting the code or by specifying filters for selective analysis. Detailed documentation can be found in the [Intel® ITAC Documentation](https://software.intel.com/en-us/intel-trace-analyzer). ## Likwid The [Likwid Performance Tools](https://hpc.fau.de/research/tools/likwid/) are a lightweight command line performance tool suite for node-level profiling. Likwid is available as an environment module. ## Scalasca Scalable tool for performance analysis of MPI/OpenMP/hybrid programs Scalasca (**Sc**alable performance **a**nalysis of **la**rge**-sc**ale parallel **a**pplications) is an open-source project by the Jülich Supercomputing Centre ([JSC](https://www.fz-juelich.de/jsc)) which focuses on analyzing OpenMP, MPI and hybrid OpenMP/MPI parallel applications. The Scalasca tool can be used to identify bottlenecks and load imbalances in application codes by providing a number of helpful features, among others: profiling and tracing of highly parallel programs; automated trace analysis that localizes and quantifies communication and synchronization inefficiencies. The tool is designed to scale up to tens of thousands of cores and even more. Scalasca is able to automatically instrument code on subroutine level, or the user can instrument the code to investigate special regions. The performance measurements are carried out at runtime. The results are studied after program termination with a user-friendly interactive graphical interface which shows the considered event or performance metric together with the respective source code section and with regard to fluctuations over the used partition of the system. Scalasca can be used on all compute clusters by loading the module `module load scalasca`. Documentation on how to prepare the code and specify different modes of operation can be found on the Scalasca web page [www.scalasca.org/](https://www.scalasca.org/). ## Lightweight MPI profiling with mpitrace [MPItrace is an open-source library](https://github.com/IBM/mpitrace) that enables lightweight profiling of MPI programs by gathering statistics about the times spent in various MPI calls together with corresponding message sizes. The tool can be used on the HPC clusters by simply loading the module `module load mpitrace` at runtime. Functionality is based on the LD_PRELOAD mechanism, no instrumentation or recompilation of the program is required. By default the tool produces three files, corresponding to the MPI ranks with the minimum, maximum, and median of the total communication time. They are named according to the scheme `mpi_profile..`. Various knobs, including the option to record all MPI ranks or a user-defined subset thereof, can be set via environment variables, as documented in file `$MPITRACE_HOME/doc/env_variables.txt` An example output file (here, rank 0 was the one with minimum communication time) looks like this: ``` Data for MPI rank 0 of 16: Times from MPI_Init() to MPI_Finalize(). ----------------------------------------------------------------------- MPI Routine #calls avg. bytes time(sec) ----------------------------------------------------------------------- MPI_Comm_rank 1 0.0 0.000 MPI_Comm_size 2 0.0 0.000 MPI_Barrier 8 0.0 0.000 MPI_Reduce 94 1257.7 0.056 MPI_Allreduce 12 16.0 0.000 MPI_Gather 40 5.2 0.006 MPI_Gatherv 56 200.0 0.004 MPI_Alltoall 48 15000000.0 0.240 ----------------------------------------------------------------------- MPI task 0 of 16 had the minimum communication time. total communication time = 0.307 seconds. total elapsed time = 29.487 seconds. user cpu time = 458.794 seconds. system time = 9.018 seconds. max resident set size = 20027.258 MiB. ----------------------------------------------------------------- Message size distributions: MPI_Reduce #calls avg. bytes time(sec) 26 8.0 0.054 26 16.0 0.000 28 200.0 0.001 14 8000.0 0.002 MPI_Allreduce #calls avg. bytes time(sec) 12 16.0 0.000 MPI_Gather #calls avg. bytes time(sec) 28 4.0 0.006 12 8.0 0.000 MPI_Gatherv #calls avg. bytes time(sec) 56 200.0 0.004 MPI_Alltoall #calls avg. bytes time(sec) 48 15000000.0 0.240 ----------------------------------------------------------------- Summary for all tasks: Rank 12 reported the largest memory utilization : 20035.61 MiB Rank 9 reported the largest elapsed time : 29.54 sec minimum communication time = 0.307 sec for task 0 median communication time = 3.294 sec for task 7 maximum communication time = 4.040 sec for task 13 MPI timing summary for all ranks: taskid host cpu comm(s) elapsed(s) user(s) system(s) size(MiB) switches 0 vipc2001 0 0.31 29.49 458.79 9.02 20027.26 754 1 vipc2001 16 1.94 29.49 457.57 11.31 20033.55 843 2 vipc2001 32 3.01 29.49 455.71 12.72 20031.63 705 3 vipc2001 48 3.47 29.49 455.22 13.45 20029.03 748 4 vipc2001 64 1.59 29.49 457.42 10.64 20033.55 711 5 vipc2001 80 2.47 29.49 455.80 11.85 20030.67 693 6 vipc2001 96 3.16 29.49 454.81 13.03 20026.62 703 7 vipc2001 112 3.29 29.49 454.62 13.12 20030.29 699 8 vipc2002 0 2.83 29.49 455.53 12.42 20027.21 830 9 vipc2002 16 3.35 29.54 455.60 13.08 20025.33 694 10 vipc2002 32 3.42 29.49 455.02 13.16 20030.43 675 11 vipc2002 48 3.43 29.49 454.95 13.31 20024.61 726 12 vipc2002 64 3.92 29.49 454.54 13.96 20035.61 687 13 vipc2002 80 4.04 29.49 455.13 13.95 20027.00 677 14 vipc2002 96 3.54 29.49 455.92 13.22 20026.89 683 15 vipc2002 112 3.59 29.49 455.05 13.50 20035.12 666 ``` ## Simple Performance Library perflib This page describes the usage of the performance library libooperf.a. ### Description The perflib consists of an instrumentation library, which provides instrumented programs with a summary output containing performance information for each instrumented region in a program. If some regions are deep in the call tree in a loop, the library adds some overhead to the runtime. This overhead is not visible in the resulting times but in the runtime of the program. This library supports parallel (MPI and mixed mode) applications, written in Fortran. It only accounts for the master thread in a multithreaded program. There are the following libraries: > libooperf.a > perf library for MPI programs. > > libperfhpm.a > Use perf instrumentation to call hpmtoolkit. > > libperfdummy.a > Provides dummies for the perf instrumentation. You can link against > this library to avoid the perf overhead in production runs. > > ### Basic Interface The basic interface of the perflib consists of 4 fortran callable subroutines: > perfinit > must be called after MPI\_Init to do initialization > > perfon(name) > defines a starting point for performance measurement. Name is a > character string, which identifies this point in the output of > perfout. > > perfoff > defines an end point of performance measurement. > > perfout(name) > must be called before MPI\_Finalize and prints the results to standard > output. Name identifies a call of perfon, which is supposed to have > 100% of runtime. The percentage of runtime of all other perfons is > relative to this. You can call perfout from any number of MPI tasks. > If your MPI tasks all do the same work, it is enough to call perfout > from task 0 only. When you call perfout from several MPI tasks > concurrently the output in stdout is mixed. You can serialize the > calls to perfout as shown in the following example program. ### Advanced Interface In addition to the four basic subroutines, there are two more for context management. > perf\_context\_start(name) > starts a new context, in which all regions called from this context > are separated from the calls to the same regions from other contexts. > For example, if one calls some regions from within the initialization > and from the timeloop and one wants to discriminate these, one can > define a context for the init phase and one for the timeloop. > > perf\_context\_end > ends a previously started context and switches back to the parent > context There are also two more functions, which helps getting the results of the performance regions directly into the program. > perf\_get(name, double \*inctime, double \*inc\_MFlops) > gives back the inclusive time and inclusive MFlops from the > performance region "name". This is useful for loops, where one wants > to get the results for different loop iterations. > > perf\_reset(name) > resets the counters and the time of the performance region "name". > Only useful in conjunction with perf\_get, otherwise some performance > data is lost. ### Example Program ```fortran Program tperf Implicit none use mpi Integer:: ii, ierr, pe, npes, mype Real(8):: uu Integer:: omp_get_thread_num Integer, Parameter:: SZ=10000 Real(8):: a(SZ), b(SZ),pi(SZ) print *, 'Start of Program tperf' Call MPI_Init(ierr) if (ierr /= 0) Stop "MPI_Init failed" Call MPI_Comm_size(MPI_COMM_WORLD, npes, ierr); Call MPI_Comm_rank(MPI_COMM_WORLD, mype, ierr); !$omp parallel print "('This is MPI task',i5,' thread',i3)",& mype, omp_get_thread_num() !$omp end parallel Call perfinit Call perfon ('tperf') Call perfon ('calc') !$omp parallel do Do ii=1, 100 call calc (pi) Enddo !$omp end parallel do Call perfoff Call perfon ('random') Call random_number(a) Call random_number(b) Call perfoff Call perfon ('calc2') Do ii=1, SZ uu = calc2 (a, b) Enddo Call perfoff Call perfoff ! tperf do pe=0, npes-1 Call MPI_Barrier(MPI_COMM_WORLD, ierr) If (mype == pe) Call perfout('tperf') Enddo Call MPI_Finalize(ierr) if (ierr /= 0) Stop "MPI_Finalize failed" print *, 'End of Program tperf' CONTAINS Subroutine calc (p_pi) Integer:: ii Real(8):: p_pi(:) Do ii=1, size(p_pi) p_pi(ii) = sin(real(ii))*sqrt(real(ii)) Enddo End Subroutine calc Real Function calc2 (a, b) Real(8):: a(:), b(:), c c = Sum (a * b) calc2 = c End Function calc2 End Program tperf ``` ### Compiling and Linking The 'perf'' library is available as a module: ```sh module load perflib ``` To compile and link the program tperf.f use the following command line: ```sh mpif90 -o tperf tperf.f -L$PERFLIB_HOME/lib -looperf -lpfm -lstdc++ ``` Sometimes the C++ standard library (-lstdc++ in the linkline) is already set inside the compiler wrapper. When using OpenMP (-qsmp=omp) measurements are only made for thread 0. Be aware that performance of thread 0 might be different to other threads. ### Output By default the output from this program looks like this: ```sh Start of Program tperf Start of Program tperf This is MPI task 0 thread 0 This is MPI task 1 thread 0 Inclusive Exclusive Subroutine #calls Time(s) % MFlops Time(s) % MFlops -------------------------------------------------------------------------- tperf 1 1.878 100.0 142.763 0.000 0.0 0.000 calc 1 0.100 5.3 681.948 0.100 5.3 681.948 random 1 0.000 0.0 444.131 0.000 0.0 444.131 calc2 1 1.778 94.7 112.473 1.778 94.7 112.473 Size of data segment used by the program: 89.12 MB Inclusive Exclusive Subroutine #calls Time(s) % MFlops Time(s) % MFlops -------------------------------------------------------------------------- tperf 1 1.877 100.0 142.878 0.000 0.0 0.000 calc 1 0.100 5.3 682.227 0.100 5.3 682.227 random 1 0.000 0.0 465.735 0.000 0.0 465.735 calc2 1 1.777 94.7 112.565 1.777 94.7 112.565 Size of data segment used by the program: 89.12 MB End of Program tperf End of Program tperf ``` ### Remarks: - Column 1 is the name given in perfon. - Column 2 is the number of calls of perfon with this name. - Columns 3-5 are inclusive and the following 3 columns are exclusive. - Inclusive values measure all code between a call to perfon and its corresponding call of perfoff. - Exclusive values exclude those parts of the code, which are measured separately with calls to perfon and perfoff. - The subroutine calc has no subcalls of perfon. That's why inclusive and exclusive values are identical. By setting the environment variable ``` export PERFLIB_OUTPUT_FORMAT=xml ``` the output is not written to stdout but each MPI rank writes its results in a different XML file with a name perf.<PID>.xml. These xml files can then be further processed with own tools or with the python script comp\_perf.py. You can get help with ``` comp_perf.py --help ``` # Mathematical tools ```{contents} Contents :local: :depth: 1 ``` ## IDL The [**I**nteractive **D**ata **L**anguage](https://www.nv5geospatialsoftware.com/Products/IDL) is a complete package for the interactive analysis and visualization of scientific and engineering data, developed by [ITT](https://www.itt.com), Visual Information Solutions. ### Availability IDL is available on the compute platforms at the MPCDF. A pool of 131 floating development licenses and 60 floating run-time licenses sponsored by the institutes of the Campus Garching is available. Users are asked to use the much cheaper run-time licenses whenever possible. On MPCDF Linux systems IDL is made available via the [environment modules](./environment-modules "Modules") system. The output of the command `module avail idl` provides an up-to-date list of available versions. Use `module help idl` for documentation and detailed usage instructions. For use on **LinuxX86\_64, WindowsX86\_64** and **MacOSX** desktop systems of institutes of the Campus Garching install images for version 8.9 and successors together with the appropriate license file can be found via the [software download page](https://max.mpg.de/Service/Forschungsservice/Pages/MPCDF/Software-download-for-MPG-users.aspx). ## Maple Maple is an analytic computation system. It performs mathematical computations and manipulations for solving problems from various technical disciplines. Most significantly, Maple can compute both numerical as well as symbolic solutions to mathematical expressions. ### Availability and usage On MPCDF systems Maple can be accessed by invoking `module load maple`. Then the commands `maple` (command line version) and `xmaple` (graphical user interface) are available. ### Documentation Detailed information about Maple can be found on the [Maplesoft Website](https://www.maplesoft.com). ## Mathematica [Mathematica](https://www.wolfram.com/mathematica/) is an integrated software system and language intended for numeric, symbolic and graphical computation, developed by [Wolfram Research Inc. (WRI)](https://www.wolfram.com/), Champaign, Illinois 61280-7237, USA , ### Availability At the MPCDF a shared pool of 40 floating licenses for Mathematica sponsored by the institutes of the Campus Garching is available for use on arbitrary platforms. All licenses have Premier Service, i.e. [faculty home-use](https://software.additive-net.de/de/produkte/wolfram/lizenzierung/ps/mathematica-home-use?start=1#content) (needs approval by MPCDF administrator) and the rights to use [webMathematica Amateur](https://software.additive-net.de/de/produkte/wolfram/webmathematica), [Wolfram Workbench](https://software.additive-net.de/de/produkte/wolfram/produkte/workbench) and [Wolfram Lightweight Grid Manager](https://software.additive-net.de/de/produkte/wolfram/gridmathematica/lizgridmma?start=2#content). ### Usage Mathematica is invoked by using the command `math`, the notebook is started via the command `mathematica`. On MPCDF **Linux** systems Mathematica is made available for all users via the [environment modules](./environment-modules "Modules") system. The output of the command `module avail` provides an up-to-date list of available versions. Use `module help mathematica` for documentation and detailed usage instructions. For use on **Linux, Windows** and **MacOSX** desktop systems of institutes of the Campus Garching (e.g. PCs without the environment module system) images and license files can be found via the [software download page](https://max.mpg.de/Service/Forschungsservice/Pages/MPCDF/Software-download-for-MPG-users.aspx). ### Further Information In WRI's [Mathematica Information Center](https://library.wolfram.com/database/Books/Mathematica) a large collection of literature concerning Mathematica can be found. Additional information on Mathematica may be obtained from the company's [WRI](https://www.wolfram.com/) web-pages directly. Also extensive information and a mailing list are available from the [DMUG](https://www.mathematica.ch/) (**D**eutschsprachige **M**athematica **U**ser **G**roup) home page. ## MATLAB ### MATLAB and Simulink The MATLAB and Simulink products are developed by MathWorks. MATLAB is a high-level language and interactive environment for numerical computation, visualization, and programming. Using MATLAB, you can analyze data, develop algorithms, and create models and applications. The language, tools, and built-in math functions enable you to explore multiple approaches and reach a solution faster than with spreadsheets or traditional programming languages, such as C/C++ or Java. \*) Simulink is a block diagram environment for multidomain simulation and [Model-Based Design](https://www.mathworks.com/model-based-design/). It supports system-level design, simulation, automatic code generation, and continuous test and verification of embedded systems. \*) \*) cited from: and The German distributor of MATLAB & Simulink is: The MathWorks GmbH, Adalperostraße 45, 85737 Ismaning, Telefon 089-45235-6700, Fax 089-45235-6710 ### Availability The MPCDF presently provides MATLAB from a shared pool with base licenses plus various additional toolboxes (e.g. Simulink). ### Usage **MATLAB on MPCDF Linux systems** is available in multiple versions as loadable modules in all Linux systems at the MPCDF. See the output of `module avail matlab` for the default and a list of other available versions. After having loaded the MATLAB module the MATLAB graphical user interface providing a fully integrated development and run-time environment can be started from the command line just typing `matlab`. Using MATLAB with SLURM and running parallel matlab code is documented as part of the [FAQ](../../../faq/hpc_software.md). For use on **Linux, Windows** and **MacOSX** desktop systems of institutes of the Campus Garching (e.g. PCs without the environment module system) Matlab images and license files can be found via the [software download page](https://max.mpg.de/Service/Forschungsservice/Pages/MPCDF/Software-download-for-MPG-users.aspx). ### Documentation Documentation on MATLAB can be found [here](https://www.mathworks.de/products/matlab/). Documentation on Simulink can be found [here](https://www.mathworks.de/products/simulink/). Further information on MATLAB and Simulink can be obtained from the web-pages of the companies [MathWorks Inc.](https://www.mathworks.de) and [*scientific* COMPUTERS GmbH](http://www.scientific.de). # Bioinformatics The following biology-related web applications are hosted by MPCDF: - **Movebank** ([https://www.movebank.org):](https://www.movebank.org) a free, online database of animal tracking data (developed at the MPI of Animal Behavior with collaborators) hosted by MPCDF. - **HaloLex** ([https://www.halolex.mpg.de):](https://www.halolex.mpg.de) comprehensive annotation and information system for prokaryotic genome and proteome data (developed by the MPI of Biochemistry and MPCDF) *access restricted/on request* (contact: ) - **Galaxy** (decommissioned in June 2021): Galaxy is an open-source, web-based platform for data-intensive biomedical research. The MPCDF has operated a Galaxy instance for the MPG for a number of years. On request, MPCDF provides software, data sources and application support specific to computational biology applications. # Containers ## Apptainer Apptainer () is an open-source software developed to add containers and reproducibility to scientific high performance computing. Just like its predecessor Singularity, Apptainer is being developed to provide container technologies on HPC systems. It gives users an easy way to access different OSs on the HPC systems while still ensuring that containers run in an established user environment, without a pathway for privilege escalation on the host. Apptainer was born in 2021, when the Singularity open source project split into two separate projects: Apptainer and SingularityCE. The Apptainer branch has joined the Linux Foundation, while the Sylabs' fork of Singularity, dedicated to commercial use, was renamed SingularityCE. While, at least at the beginning, there has been continual alignment between Sylabs' SingularityCE and Apptainer, over time the paths of the projects will likely diverge as both projects continue to mature. As part of the transition, only open community standard interfaces will be supported in Apptainer. This includes removing the "Library" and "Remote Builder" support. In the event these become open community maintained standards (and not corporate controlled), these features may be left intact or re-added at a later date. For this reason, users of the old Singularity software are encouraged to adjust their scripts accordingly. On top of the `apptainer` command, Apptainer provides backwards compatibility offering `singularity` as a command line link. It is also committed to maintain as much of the CLI and environment functionality available in the old Singularity software as possible. From the user's perspective, very little, if anything, should change and the wrapper around the `singularity` command allows users to run commands like 'singularity pull', 'singularity run', etc. just as before. Please, visit for additional information on the Apptainer software and access to its documentation. ### Examples of Apptainer commands The following table summarizes some Apptainer commands (based on version 1.0.3). For more information see the Apptainer User Guide at https://www.apptainer.org/docs/.
General commands
help Help about any command
Usage commands
build Build an Apptainer image
cache Manage the local cache
capability Manage Linux capabilities for users and groups
exec Run a command within a container
inspect Show metadata for an image
instance Manage containers running as services
key Manage OpenPGP keys
oci Manage OCI containers
plugin Manage apptainer plugins
pull Pull an image from a URI
push Upload image to the provided URI
remote Manage apptainer remote endpoints
run Run the user-defined default command within a container
run-help Show the user-defined help for an image
search Search a Container Library for images
shell Run a shell within a container
sif siftool is a program for Singularity Image Format (SIF) file manipulation
sign Attach a cryptographic signature to an image
test Run the user-defined tests within a container
verify Verify cryptographic signatures attached to an image
version Show the version for Apptainer
Global options
-d --debug print debugging information (highest verbosity)
-h --help help for apptainer
-q --quiet suppress normal output
-s --silent only print errors
-v --verbose print additional information

The help command gives an overview of Apptainer options and subcommands. For example: ``` $ apptainer help [] $ apptainer help build $ apptainer help instance start ``` ### Apptainer on the MPCDF HPC systems On the HPC clusters at MPCDF, an environment module is provided in order to load the Apptainer software. For backwards compatibility, a Singularity module (singularity/link2apptainer) is also provided which will print a warning message and load the default Apptainer module. The old Singularity as well as the new SingularityCE software will not be supported on the HPC clusters. ### Minimal example To execute a PyTorch script in a containerized environment with CUDA support, the desired image should first be pulled from a Docker repository. From a login node: ``` # Load apptainer module module load apptainer/1.3.6 # Pull image from a docker repository apptainer pull nvidia-pytorch.sif docker://nvcr.io/nvidia/pytorch:25.04-py3 ``` The above command downloads an image from the [NGC catalog](https://catalog.ngc.nvidia.com/) and stores it in a SIF file named `nvidia-pytorch.sif`. To execute the script, use the `exec` command: ``` apptainer exec nvidia-pytorch.sif python your_script.py ``` For additional examples with both CUDA and ROCm support, as well as complete SLURM submission scripts, refer to the [AI Containers Repository](https://gitlab.mpcdf.mpg.de/dataanalytics-public/ai_containers). # VNC ## Introduction VNC (Virtual Network Computing) is a graphical remote desktop system that uses the RFB protocol (Remote Frame Buffer) to control a graphical desktop session on another computer over a network. It transmits keyboard and mouse events from one computer to another and sends the graphical screen updates back in the other direction. Open-source implementations exist for all relevant operating systems. ## VNC usage scenarios at the MPCDF VNC is useful to transmit the content of graphical user interfaces from one computer (the server) to another computer (the client). Technically, on a Linux platform, running a VNC session involves an X-VNC server that is used to draw X11 applications locally. Only bitmaps are then sent to the client. In most cases, the performance of VNC is superior compared to traditional X forwarding, especially when dealing with complex graphical interfaces on wide-area networks. At the MPCDF, a typical scenario would be to run a graphical tool (e.g. a debugger, a performance analysis tool, or data analysis software) interactively on a HPC cluster in a VNC session. A particularly useful feature of a VNC session is its persistence. Users may disconnect from the VNC session and reconnect later, potentially from different computers at different locations. Programs that run within the session continue to run as long as the VNC session is not shut down or killed. For plain text applications, the GNU screen tool offers similar persistent functionality at a much smaller resource footprint. Please consider using screen instead of VNC in case you don't need to run GUI applications. ## How to launch VNC sessions on HPC clusters at the MPCDF Important: Please read and follow the separate [instructions for remote visualization](../../visualization/index.html#remote-visualization-and-jupyter-notebook-services) in case you need GPU acceleration for OpenGL-enabled (3D) applications. Moreover, the remote visualization service allows users to conveniently launch a non-accelerated VNC session on dedicated resources, albeit with a limited run time. VNC sessions can be launched manually on most interactive Linux machines. As a general rule, the session should be run as close to the workload as possible in order to minimize the need for X forwarding to the X-VNC server. ### How to launch a VNC server manually on a login node On a login node please proceed as follows to start a VNC session. ```bash module load turbovnc module load vncsetup # Set up the VNC environment and password, vncsetup # is required only once. vncserver -geometry 1200x980 ``` At the launch of vncserver its X display is reported, e.g. 'toks01:10'. Expand the hostname to be fully qualified and note the information to be able to connect later, e.g. 'toks01.bc.rzg.mpg.de:10'. On **Raven** and **Viper**, please note that it is necessary to append an 'i' to the hostname reported for network reasons: E.g., when the vncserver reported 'raven02:10' at launch time, the fully qualified hostname and X display would read 'raven02**i**.mpcdf.mpg.de:10'. The VNC session uses the lightweight IceWM window manager by default. ### How to shut down a VNC server To terminate a VNC session, log in to the machine it is running on and enter the following command (where ":10" is the X display of our example which needs to be adapted to the actual X display of your session). ```bash module load turbovnc vncserver -kill :10 ``` ## How to connect to VNC servers running on Linux systems at the MPCDF For security reasons it is necessary to establish an SSH connection to gate1 (or gate2) and tunnel the VNC connection through that connection. On Linux, recent VNC viewers support a "-via" command line option that can be used to establish the SSH tunnel conveniently. On other platforms and with different VNC viewers, it is necessary to create the SSH tunnel manually. Users need to make sure to have a vncviewer locally installed (TigerVNC, TightVNC, or TurboVNC are recommended). Starting from the launch examples in the previous section, the steps necessary to connect to VNC sessions are described in the following. This example uses the server 'raven01i.mpcdf.mpg.de' on X display 10, which you would need to adapt to your actual session. ### How to connect from Linux clients using TurboVNC viewer (or compatible viewers) Proceed as follows to connect if you have a vncviewer that supports the "-via" option. (Make sure to adapt the hostname and the X display to your actual session.) ```bash vncviewer -via USER@gate2.mpcdf.mpg.de raven01i.mpcdf.mpg.de:10 ``` You will be prompted several times for a password: Enter your MPCDF password followed by an OTP to log in to gate2, and then enter your VNC password to connect to the session. ### How-to connect from Linux, Mac, or Windows clients using SSH and a generic VNC viewer For VNC clients that don't support the "-via" option a two-step process is necessary. 1. Open a new terminal window and establish an SSH connection to gate2: `ssh -L 5999:raven01i.mpcdf.mpg.de:5910 USER@gate2.mpcdf.mpg.de` Note that the "-L" option needs the target hostname and the TCP port of the VNC server. The TCP port is obtained by adding the X display number (here "10") to 5900, which gives 5910 in our example. On Windows, the ssh command is in general not available. The "plink.exe" binary from PuTTY provides comparable functionality and uses a similar syntax. 2. Once the SSH connection is established, open a second terminal window and connect vncviewer to the local port of the SSH tunnel. `vncviewer 127.0.0.1::5999` ## Security considerations In general, VNC (RFB) packets are not encrypted. It is the user's responsibility to establish transport encryption by tunneling VNC connections though SSH. Firewall rules at the MPCDF prevent plain VNC connections from external networks and enforce SSH connections, anyway. # Quickstart guide to HPC This document provides information relevant to users migrating from other HPC systems to supercomputers and clusters at the MPCDF. ## Software environment (modules) A [new module system with a hierarchical structure](software/environment-modules.md) has been introduced. Starting with Raven, no defaults are defined for the compiler and MPI library modules. Users need to explicitly specify the full version for compiler and MPI modules during compilation **and** in batch scripts to ensure compatibility of the MPI library. Due to the hierarchical module environment, many libraries and applications only appear after loading a compiler, and subsequently also an MPI module (*the order is important here: first, load compiler, then load the MPI*). These modules provide libraries and software consistently compiled with/for the user-selected combination of compiler and MPI library. To search the full hierarchy, the `find-module` command can be used. All fftw-mpi modules, e.g., can be found using `find-module fftw-mpi`. ## MPI parallel HPC applications As a default, the Intel compilers for Fortran/C/C++ and the Intel MPI library are recommended on Raven. The MPI wrapper executables are `mpiifort`, `mpiicc` and `mpiicpc`. These wrappers pass include and link information for MPI together with compiler-specific command line flags down to the Intel compiler. More information and links to the reference guides for the Intel compilers are provided on the following pages: * [Intel C/C++ compiler for Linux](software/compilers_languages#intel-c-c-compiler-for-linux) * [Intel Fortran compiler for Linux](software/compilers_languages#intel-fortran-compiler-for-linux) In addition, the [GNU Compiler Collection, GCC](https://gcc.gnu.org/) (C, C++, Fortran) is supported to be used with Intel MPI. ## Multithreaded (OpenMP) or hybrid (MPI/OpenMP) HPC applications To compile and link OpenMP applications pass the flag `-qopenmp` to the Intel compiler. Note that the recent Intel compilers do not support the legacy `-openmp` option anymore. In some cases it is necessary to increase the private stack size of the threads at runtime, e.g. when threaded applications exit with segmentation faults. On the systems at the MPCDF, a value of OMP\_STACKSIZE=256MB is set (in order to relax the very small default value of 4 megabytes). For example, to request a larger stack size of 512 megabytes, set the environment variable to OMP\_STACKSIZE=512m in the Slurm job script. For information on compiling applications which use pthreads please consult the [Intel C/C++ compiler documentation](https://software.intel.com/en-us/articles/intel-c-composer-xe-documentation/#lin). ### Intel Math Kernel Library (MKL) overview The Intel Math Kernel Library (MKL) is provided as the standard high-performance mathematical library. MKL provides highly optimized implementations of (among others) * LAPACK/BLAS routines, * direct and iterative solvers, * FFT routines, * ScaLAPACK. Parts of the library support thread or distributed-memory parallelism. Extensive information on the features and the usage of MKL is provided by the [official Intel MKL documentation](https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation/). ### Linking programs with MKL To use MKL, load the environment module `mkl` first. The module sets the environment variables `MKL_HOME` and `MKLROOT` to the installation directory of MKL. These variables can then be used in makefiles and scripts. The [Intel MKL Link Line Advisor](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl/link-line-advisor.html) is often useful to obtain information on how to link programs with MKL. For example, to link statically with the threaded version of MKL on Raven (Linux, Intel64) using standard 32 bit integers, pass the following command line arguments to the Intel compiler: ```text -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lm -ldl ``` ## Execution of (parallel) programs via Slurm Parallel programs on the HPC systems are started with `srun` (see also the man page of `srun`), not `mpirun` or `mpiexec`. For production runs it is necessary to run the MPI program as a batch job. Please refer to the sections on the Slurm batch system and to the sample batch job scripts for further information, which are contained in the user guides on [Raven](raven-user-guide.md) and [Viper](viper-user-guide.md). Be reminded that the compiler and MPI modules that were loaded for compilation also need to be loaded in the batch script in exactly the same version. # Performance Monitoring A comprehensive HPC performance monitoring system is deployed on the HPC systems, where a plethora of performance metrics is collected continuously per socket or per node, enabling MPCDF staff to monitor application performance on a system-wide scale in real time or retrospectively. Users get access to a performance report in PDF format for each of their jobs via a web service. ## Introduction The MPCDF operates HPC systems to provide compute services to scientists from the Max Planck Society. Having performance numbers available for the whole machine, but also down to each individual compute job, is essential for the stakeholders of the HPC systems (i.e. users, administrators, application support, and management). This helps stakeholders firstly, to become aware of potentially suboptimal usage of resources, and secondly, to take action to improve the way these resources are used. A comprehensive HPC performance monitoring system was developed at the MPCDF and is in operation on the HPC systems since fall 2018 to continuously monitor relevant performance metrics on all nodes and for each job. The HPC performance monitoring system is extremely lightweight and operates in the background, invisibly to the user. ## PDF Performance Reports for Users For each HPC job, we provide performance reports as PDF files for download via the following web service: After a login with the regular Kerberos credentials, the user first selects the machine of interest. Currently, the reports are available for finished jobs on the Raven HPC system with a runtime of at least 20 minutes. Once a machine was selected, a table of finished jobs is presented. To obtain the PDF performance report for a specific job, please click the 'Generate' button first. Depending on the size of the job and the load on the system, this may take from seconds to minutes. Once the PDF was created, the button label changes from 'Generate' to 'Download' at the next manual refresh of the page. ![PDF Performance Report](_static/hpc-report.png "Performance_Monitoring_PDF_Screenshot.png") The PDF file comprises multiple pages. On the first page, the most important parameters and environment variables of the job are presented in tabular form. For small jobs, a table with per-socket GFLOP/s and memory bandwidth data is shown. The following pages contain a plethora of plots showing performance data over time, e.g., GFLOP/s, memory bandwidth, retired instructions by SIMD set (scalar, SSE, AVX, AVX512), memory usage, GPU utilization, GPU memory usage, and various metrics from the HPC network and the parallel file systems. For small jobs, these plots are displayed per socket, node, or GPU device. For larger jobs, the data is instead presented in a more statistical way showing minimum, median, and maximum lines. The final page of each report contains extensive documentation and an explanation of each plot. As the system is evolving, the content and presentation of the PDF file may be changed and further improved in the future. ## Suspending the Performance Monitoring System for Specific Jobs The hpcmd software daemon uses the programmable hardware performance monitoring units (PMUs) of the CPUs to continuously measure performance data with negligible overhead. In case a user wants to use those units for the purpose of custom performance measurements, hpcmd needs to be suspended first. This is in particular relevant when one of the following software packages is used: Intel Amplifier XE (VTUNE), Intel Advisor, PAPI, LikWid, perf, and similar tools. To suspend the instances of hpcmd that monitor the compute nodes during the runtime of a batch job, we provide the wrapper 'hpcmd\_suspend'. Simply put it in between 'srun' and the executable you want to run as follows: ```bash srun hpcmd_suspend ./YOUR_EXECUTABLE ``` After the batch job has ended, hpcmd will re-enable itself automatically. Please do not suspend hpcmd unless you intend to perform your own measurements. ## Technical Background The HPC monitoring daemon (hpcmd) runs in the background on each compute node. Simple and lightweight by design, hpcmd is mostly written in Python. It queries standard Linux command line tools (e.g., perf, ps), virtual file systems (/proc, /sys), and some proprietary tools (e.g., opainfo, nvidia-smi, nvidia-dcgm). Thereby, metrics such as the GFLOP/s, the memory bandwidth, the mix of scalar and vector instructions, memory utilization, GPU utilization, network and disk I/O bandwidths, and many more, are captured on a per-socket or per-node resolution. In addition, hpcmd integrates with the Slurm batch system to gather information such as the jobid, the requested number of nodes, cores, GPUs, etc., to complement the actual performance data. hpcmd runs as a systemd service on each node of the HPC systems and performs measurements over regular 4 minute intervals synchronized between nodes. Measured and derived values are written to syslog messages, and are finally transferred to a central Splunk database and analytics platform, which enables MPCDF staff to inspect the performance data in real-time or retrospectively. As an entry point to Splunk, all the jobs are shown in a roofline-type of plot, representing a current picture of the system-wide performance. In addition, job-specific dashboards are available, enabling interactive graphical exploration of all the aforementioned performance metrics. Automated analysis based on machine learning technology is currently under development. Users are not allowed to work interactively on the MPCDF-internal Splunk system for licensing and data protection reasons, however they are provided with a static PDF Performance Report containing the full information for their specific job (please see above for details). Note that the HPC performance monitoring system was not designed as a replacement for profiling tools. It can provide, however, valuable information on performance issues, motivating in-depth profiling and code optimization work. ## Overhead hpcmd runs with a reduced scheduling priority (niceness) in the background. The Linux kernel therefore moves hpcmd and its child processes to cores that are not fully used by the application at a given time. Note that Linux perf measures mostly passively using programmable hardware counters, hence the overhead from perf is negligible. After extensive testing and several months in production on two HPC systems, we did not experience any measurable overhead or impact on the applications when running hpcmd in epochs of 10 minutes duration. ## Further information The hpcmd software is open source and available at . Online documentation on hpcmd is available at . Reference: Stanisic L., Reuter K., *MPCDF HPC Performance Monitoring System: Enabling Insight via Job-Specific Analysis*, Euro-Par 2019, [Lect. Notes Comput. Sci, 11997 (2020)](https://doi.org/10.1007/978-3-030-48340-1_47) ([arXiv](https://arxiv.org/abs/1909.11704)) # Training ## Courses and workshops arranged by or in collaboration with the MPCDF The MPCDF provides training events for the MPG and its partners. Please see the [MPCDF training web pages](https://www.mpcdf.mpg.de/14192/Training) for an overview on upcoming and past events. ## Training programmes of other institutions There are many **regularly offered courses**, both fundamental and advanced ones, in different topics of scientific computing. Consider e.g. the following: - Courses at the [HLRS](https://www.hlrs.de/training/hpc-training) Stuttgart (partly held at the LRZ Garching) - Courses at the [LRZ](https://www.lrz.de/services/compute/courses) Garching - [PRACE](https://training.prace-ri.eu) HPC training and tutorials ------------------- Data ------------------- .. image:: /_images/data01.png The MPCDF provides several services around the management of research data, covering storage, transfer and analysis of scientific data. Beside these generic services, the data group at the MPCDF provides consulting and dedicated high-level support for the development, optimization and analysis of data applications. This comprises the development and operation of database systems in collaboration with Max-Planck scientists, web hosting and repository services, and technical consulting, e.g. on web interfaces for data discovery and access. More detail about the different service areas can be found in the following sections: .. toctree:: :maxdepth: 1 :glob: share/index.rst.txt globusonline/index.rst.txt object-storage/index.rst.txt data-transfer/index.rst.txt gitlab/index.rst.txt publication/index.rst.txt backup-archive/index.rst.txt store/index.rst.txt --------------------------------- DataShare: Sync and Share Service --------------------------------- .. image:: /_images/datashare01.png :width: 800 DataShare is the Sync and Share service of the MPCDF. It offers a wide range of functionality around data sharing and synchronisation and can be used by every MPCDF user. .. toctree:: :maxdepth: 1 datashare.md.txt nextcloud-migration.md.txt switch-client.md.txt sync-share-clients.md.txt faq.md.txt # DataShare: An Introduction ## MPCDF DataShare Service ### Preparations Before you can use the MPCDF DataShare service, you have to opt-in for it at the MPCDF SelfService: Log in with your MPCDF account and go to "My account / Services" to opt-in for DataShare. DataShare requires Two-Factor-Authentication (2FA), so please make sure to set up a 2FA token in SelfService. Further information about the SelfService can be found on its [Help](https://selfservice.mpcdf.mpg.de/index.php?r=site%2Fhelp) page. At the Selfservice, you can also subscribe for other MPCDF services like GitLab. > **_NOTE_**: After opt in, it can take up to 20 minutes until the accounts are created in the services and you can log in to DataShare! ### First Login After you have opted in for the service, you can log in to DataShare: On the first login, a "getting started" page will be displayed with links to download the optional desktop- and mobile clients as well as a link to further documentation. You can use those clients to synchronize a folder on your local machine with data on the DataShare server. Alternatively, after closing the welcome message, you can create new documents directly in the web interface by clicking on the "Plus" sign, or upload files from your machine via drag&drop. ### Sharing Data At any time, you can share files with other DataShare users. If you want to share data with external, non-MPCDF users via DataShare, you can invite them via email from the [MPCDF SelfService](https://selfservice.mpcdf.mpg.de) where you have opted in for DataShare. External users will see the message "Your storage is full, files can not be updated or synced anymore" in their DataShare account. This is because external users have no storage quota. However, they will still be able to upload files into directories you or other DataShare users with fully qualified accounts have shared with them. In this case, the data will count towards your own quota. If you want to share a file only once with an external user, there is also the possibility to share a file or folder in DataShare just via a link. # DataShare: Migration to Nextcloud The MPCDF DataShare sync&share service based on the [ownCloud product](https://owncloud.com/product) has now been in operation for more than 10 years. It has been running reliably across multiple major software and hardware changes, hopefully providing value to our users. Already some time ago, ownCloud was forked into a new product called [Nextcloud](https://nextcloud.com/). For most of the time, ownCloud and Nextcloud existed in parallel, both occupying specific niches in the market. However during the last few years, development around ownCloud has slowed significantly, with the vendor finally announcing its end of life for the end of 2026. For this reason, the MPCDF **DataShare service was migrated to Nextcloud** on **September 13th and 14th, 2025.** Data, shares, calendars etc. were preserved during the migration, and the general functionality and look&feel remains very similar. Nevertheless, there are some important changes: ## New desktop and mobile clients If you used the *ownCloud* or branded *DataShare* client on your device before, you will need to **install the [Nextcloud client](https://nextcloud.com/install/#install-clients)** instead in order to continue being able to synchronize your data on that device. You may re-use the ownCloud data directory on your machine in order to avoid downloading all of the files again, if you **ensure that it is no longer accessed by the ownCloud client** by either uninstalling it, or removing the DataShare account configuration from it. See also [Switching from the ownCloud to the Nextcloud client](https://docs.mpcdf.mpg.de/doc/data/share/switch-client.html) for a more in depth guide. ## Shares by expired users no longer accessible Previously, data belonging to an expired user could still be accessed via public links (e.g. `https://datashare.mpcdf.mpg.de/s/`) or by other DataShare users it was shared with. **Since the migration to Nextcloud, this is no longer the case!** If you or your collaborators are still using data belonging to an expired user, **please transfer it to an active user** if possible and re-share it from there. In special cases where this is not feasible, for example if the shared link must remain the same, contact support@mpcdf.mpg.de for assistance. ## Expired accounts will be deleted after 6 months Expired accounts including all data, shares, calendars etc. associated with them will be deleted after 6 months. Please make sure to transfer data that should be preserved to another user before your account expires. This will be possible [via the Nextcloud web interface](https://docs.nextcloud.com/server/31/user_manual/en/files/transfer_ownership.html) after the migration. ## Old style "v1" chunking API no longer supported The old-style "v1" [chunking API](https://github.com/owncloud/core/wiki/spec:-big-file-chunking) for uploading large files is no longer supported. This should only impact a small number of users using old versions of the *pocli*, *pyocclient* or similar. ## Custom groups converted to Nextcloud Teams Any ownCloud "custom groups" that you may have created were converted to *Nextcloud Teams*, also called *Circles*. They generally work very similarly and can be managed through the new ["Contacts" app](https://docs.nextcloud.com/server/30/user_manual/en/groupware/contacts.html#circles). Since there needs to be exactly one *owner* of a *team*, this role will be assigned to the first *admin* of the ownCloud custom group. Any additional admins will be assigned the *admin* role, and non-privileged members the *member* role. Only the owner can delete the team or define additional admins, while admins may edit the settings and add/remove members. If the owner's account expires, another admin is automatically promoted to owner. ## Two-factor authentication mandatory Two-factor authentication (2FA) via MPCDF Login is now mandatory in accordance with general MPG security guidelines. This is the same Single sign-on (SSO) login that was already introduced for GitLab earlier this year. E.g. you should only have to sign in once per day for any of DataShare, GitLab, or other MPCDF services that will use SSO in the future. In case you have not [set up 2FA](https://docs.mpcdf.mpg.de/faq/2fa.html) in our [SelfService](https://selfservice.mpcdf.mpg.de) for other services such as GitLab already, you will need to do so before you can login to DataShare. If you use third-party clients that do not support MFA such as Thunderbird, Apple Calendar etc., you may create a device specific / [app password](https://docs.nextcloud.com/server/latest/user_manual/en/session_management.html#managing-devices) for them in the web interface under `Settings - Security` and use it in place of your normal MPCDF password: ![](_static/dsnext_add_third_party_device.png) # Switching from the ownCloud to the Nextcloud client The migration of our sync-and-share services from the ownCloud platform to the Nextcloud platform requires that users switch from the ownCloud desktop client to the Nextcloud desktop client. In the following, we show how to carry out this transition on a **Linux** machine, but most of the configuration steps can be applied to any other operating system. If you are running a recent Linux distribution, the transition is quite easy and we suggest to rely on the package provided by your distribution. If you want to run the very latest Nextcloud client or your Linux distribution is old and only obsolete clients are provided, then you may want to consider installing the client from the AppImage file, as described below. Regardless of your choice, please take care of reading the instructions below on how to delete your account from the old ownCloud client and configure the new Nextcloud client. ## Step 1: delete your account from the old ownCloud client. Before configuring your account on the Nextcloud client it is very important that you remove your account from the old ownCloud client. This will prevent the two clients from trying to access your data at the same time, possibly creating some inconsistencies in your files. ![](_static/switch-client1-old-client.png) In order to do so: 1. Open your ownCloud desktop client 2. Select the account you want to remove from the top row of accounts you have configured 3. Select the account button on the right and then "Remove" This will delete your account from the client, but all your data will still be available in the synchronized folder and on the online server. Note that at this point you should not edit the files in your synchronized folder, as any new change is not immediately committed to the online server. ## Step 2: download the Nextcloud desktop client. #### Nextcloud from repository If you are planning on installing the Nextcloud desktop client from your Linux distribution repository, look for the `nextcloud-desktop` package. For example, ``` sudo apt install nextcloud-desktop ``` This will install the basic client and its dependencies. You may also want to install some plugins that will allow you to see the synchronization status of your files and create shares from the context menu directly in the file manager. For example, `nautilus-nextcloud` provides Nextcloud integration for the Nautilus (GNOME) file manager. For OpenSUSE the packages are called `nextcloud-client-dolphin` (KDE) and `nautilus-extension-nextcloud` (GNOME). E.g. if you are using the default KDE desktop, you would run ``` sudo zypper install nextcloud-desktop nextcloud-client-dolphin ``` #### Nextcloud from AppImage On the other hand, if you want to install Nextcloud from the AppImage file, navigate to [this link](https://nextcloud.com/install/) and select "Download for desktop". Note that for Linux only a single AppImage file is provided for all the distributions. ![](_static/switch-client3-download-page.png) Change the permissions on the downloaded AppImage file to allow its execution. At this point you can simply run the downloaded file in order to configure your Nextcloud account. It is convenient to store the Nextcloud client AppImage file somewhere on your computer, as you will have to run this file every time you want to start your client (or when you set up the automatic startup for this application). ## Step 3: configure the Nextcloud desktop client. After starting the Nextcloud client (installed via the repository or via the AppImage file), you will be asked if you want to import the old ownCloud accounts detected on your machine. Please **skip** the import and configure your accounts from scratch, otherwise authentication may be configured incorrectly! ![](_static/switch-client4-import.png) You will be prompted with the wizard to add a new account. Select the "Log in" button ![](_static/switch-client5-add-account.png) and provide the URL of the sync-and-share server. ![](_static/switch-client6-add-server.png) Once you select the "Next" button, your web browser will redirect you to the login page of the server. Select the "Log in" button ![](_static/switch-client7a-access.png) and then the "MPCDF Login" button in the login page of the DataShare service. This will redirect your connection to the MPCDF Single Sign-On page where you can enter your credentials, including your One Time Password (OTP). ![](_static/switch-client8-sso.png) Finally, grant the Nextcloud desktop client access to your account. ![](_static/switch-client9-grant.png) At this point you can close the browser and continue the configuration of the desktop client. Select the folder that you want to synchronize with the server. If you select the same folder you were previously using for the ownCloud client, you will be asked if you want to keep the local data or if you want to erase the local data and start a clean sync. ![](_static/switch-client10-configure.png) Which option to choose depends on your internet connection and the available space on your local machine. If you have enough space available (or not much data on the server), you can make a clean synchronization of your data in a new folder. If you have a lot of files stored locally and you don't want to wait until they are downloaded again, you can select "Keep local data" (but be sure that files were not changed on the server while your local files are no longer up-to-date). Note that even if you select to erase the local data, the most recent Nextcloud clients will create a copy of the selected folder called "folder-name (backup)", so that your local files are not lost. This however is not ensured with older clients, so it is better to create a new empty synchronization folder if you want to be sure to have a local copy of your old data that you can safely remove at a later time. When the configuration is completed, select "Connect" to trigger the first synchronization of the new desktop client with the server. # Desktop and Mobile clients The Nextcloud desktop and mobile clients can be downloaded from . During the initial setup, just enter `datashare.mpcdf.mpg.de` as the server address. More information can be found in the [Nextcloud documentation](https://docs.nextcloud.com/server/latest/user_manual/en/desktop/index.html). ## Legacy ownCloud and MPCDF DataShare clients DataShare was migrated from ownCloud to Nextcloud on September 13th, 2025. The old ownCloud or branded MPCDF DataShare clients are no longer supported. In particular, they do not support the new 2FA authentication mechanism. This may be worked around by manually creating a [Device Password](https://docs.nextcloud.com/server/latest/user_manual/en/session_management.html#managing-devices) in the [web UI](https://datashare.mpcdf.mpg.de/settings/user/security), but should only be done as a last resort if you cannot install the Nextcloud client for some reason. There could be other issues when using the ownCloud client with Nextcloud, and it may stop working entirely with some future update. ## Third-party WebDAV clients Various third-party WebDAV clients are available and are often natively integrated into the operating system. Some work better than others, especially the integrated Windows WebDAV client has some known problems. For more information, refer to: Third party clients will generally not support the now mandatory two-factor authentication. To work around this, you may create a [Device Password](https://docs.nextcloud.com/server/latest/user_manual/en/session_management.html#managing-devices) (formerly known as "App Token") in the [web UI](https://datashare.mpcdf.mpg.de/settings/user/security). # FAQ Frequently asked questions about the MPCDF DataShare service. ## Authentication fails with third party clients Two-factor authentication (2FA) via MPCDF Login is now mandatory. If you use third-party clients (Thunderbird, Apple Calendar, rclone, cadaver, pocli etc.) that do not support MFA, you may create an [App Password](https://docs.nextcloud.com/server/latest/user_manual/en/session_management.html#managing-devices) for them and use it in place of your normal MPCDF password. ![](_static/dsnext_add_third_party_device.png) ## Can you increase my quota? Generally yes. Just open a ticket in our [helpdesk](../../../faq/help.html#how-can-i-get-help-and-support) shortly stating how much storage you need and what you intend to use it for. Please note though that the DataShare service is not intended as a replacement for network shares, but rather for data you want to share with other people. You should store a few thousand files in it at most, beyond that you will experience performance issues during sync. ## I deleted a file by mistake, what now? Deleted files are kept for some time in the [trashbin](https://datashare.mpcdf.mpg.de/apps/files/trashbin) - from where you can restore them yourself. Your data is also backed up to tape daily. So even if a file was deleted from the trashbin it may be possible to recover it if it was stored in the DataShare for at least a day. In this case, you may open a ticket in our [helpdesk](../../../faq/help.html#how-can-i-get-help-and-support), but please be aware since this takes manual intervention it may take a few days for your request to be processed. ## I can't upload big files The maximum size of files that can be uploaded through the web interface or thirdparty WebDAV clients is 20 GB. Some clients may only support 2GB or less. Bigger files must be chunked into smaller parts on upload. The Nextcloud [desktop clients](sync-share-clients.md) support this by default and include an [nextcloudcmd](https://docs.nextcloud.com/server/latest/admin_manual/desktop/commandline.html) interface for starting a sync from the command line. ## I can't open zip files downloaded through the web interface In the web interface, you can select multiple files or directories at once and then download them as a ZIP archive. Some older ZIP programs can't open these files. Note that this only applies to ZIP files generated on the fly by selecting multiple objects in the web interface, not objects that were already uploaded in ZIP format. To work around this problem, either download each file separately or install a third-party unzip program such as [7 Zip](https://www.7-zip.org) (Windows, Linux) or [The Unarchiver](https://itunes.apple.com/app/the-unarchiver/id425424353) (Mac OS X). ## I am an external user and can't upload files External accounts (usernames starting with g-) are meant for collaboration with MPCDF users only. As such, they don't have any quota of their own. Ask the person who invited you to share a folder with you and give you write permissions on it. You will then be able to upload data there which will be counted towards the quota of the inviter. ## Accessing shared folders via webdav / CURL You or your collaborators can upload files to a public link share like `https://datashare.mpcdf.mpg.de/s/` via WebDAV using the address ``` https://datashare.mpcdf.mpg.de/public.php/webdav/ ``` Where the user is the share token and password the share password. If there was no share password set, use the empty string "" as password. Using this you can upload data with curl like so: ```sh curl --user : https://datashare.mpcdf.mpg.de/public.php/webdav/ --upload-file ``` Of course, you can also use the [WebDAV client of your choice](https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html). ## How can I reset my password? You can request a password reset [here](https://selfservice.mpcdf.mpg.de/index.php?r=security%2Fforgot-password). ## Should I store Git Repositories in DataShare? Definitely - no. Both DataShare and Git(Lab) are versioning systems, but they don't know of each other. This can lead to confusing situations for one or the other system and in the worst case, you will lose data. ---------------------------------------------------- Globus Online: Large-Scale data Transfer and Sharing ---------------------------------------------------- .. toctree:: :maxdepth: 1 :glob: mpcdf-datahub-and-globus-online.md.txt datahub-file-staging.md.txt datahub-staging-flow.md.txt go-nexus.md.txt globus-demo-videos.md.txt # MPCDF DataHub and Globus Online To provide improved functionality for large scale data transfer and sharing the MPCDF has obtained a Globus Online Subscription. The subscription makes advanced functionality available on both the general Globus Online Server (DataHub) which is deployed at MPCDF and Globus Online Connect Clients which MPCDF users may deploy on laptops, desktops and login nodes on their Linux clusters. Globus Online is a third party transfer service which enables fire-and-forget data transfer at TB or multi-TB scale. Globus Online is well established and widely used with many computing centres and research institutes, as well as Universities, having Globus Services installed for their users. Here we will describe how you can gain access to Globus Online and make use of DataHub and Globus Connect Personal clients to transfer and share large data sets. This article focuses on: - [How to Create a Globus Online Account](mpcdf-datahub-and-globus-online.html#creating-a-globus-online-account) - [How to Transfer and Share Data](mpcdf-datahub-and-globus-online.html#data-transfer-and-sharing) - [How to Deploy Globus Personal Clients](mpcdf-datahub-and-globus-online.html#globus-connect-personal-plus) - [How to Find more information about Globus Online](mpcdf-datahub-and-globus-online.html#more-information) ## Creating a Globus Online Account To use MPCDF Globus endpoints, including DataHub, first opt-in to Globus via the [MPCDF SelfService](https://selfservice.mpcdf.mpg.de) Data Transfers and Sharing are managed in Globus Online via the Globus Online Web and to make use of Globus Online you will need to create an account within the Web App. Navigate to and click "Log in". ![Account-1](Account-1.png) Many MPG institutes can use the existing "Organizational login" by selecting "Max-Planck-Gesellschaft" as shown in the following screenshot. ![Account-1a](Account-1a.png) This will forward you to the MPG SSO site. ![Account-1b](Account-1b.png) If your institute is currently not supported, you will need to use a Google ID, ORCID ID or create a GlobusID. Note: This account is not linked to MPCDF and a different password should be chosen. ![Account-2](Account-2.png) Ideally we would suggest that you use a globus or ORCID ID. During the Signup process you will be asked for an email account, please use your MPG email account whenever possible. This will help us when accepting users in the Globus Connect Personal Plus Group (This is detailed later in this document). When selecting a Globus ID: ![Account-3](Account-3.png) ![Account-4](Account-4.png) When selecting an ORCID ID: ![Account-5](Account-5.png) ![Account-6](Account-6.png) ## Data Transfer and Sharing Now that you have a Globus Online account you will be able to access the Globus Online Connect Server instance at MPCDF (DataHub) and make use of Globus Connect Personal Clients. The DataHub can be used to transfer or share large data volumes, complementing the data services at MPCDF, such as [DataShare](../share/datashare.md). DataHub offers scratch based storage, with a quota of 50TB per user. Data is removed after 30 days. ### Transfer To start a transfer login to the Globus Online Web App and navigate to the File Manager section. ![Transfer-1](Transfer-1.png) The left and right panels allow you to access different storage resources (Data Collections). These may be either Globus Connect Servers or Globus Connect Personal clients. The MPCDF DataHub service offers a generic Data Collection called "MPCDF DataHub Stage-and-Share Area" - This provides the same scratch based /data area that was mounted on the previous DataHub Endpoint mpcdf#datahub. Note mpcdf#datahub is no longer available after the DataHub Upgrade (09.03.22). The collections can be found by using the search function in the File Manager or Bookmarks section of the Globus Online Web App. To access a Collection simply follow the usual login steps, entering your MPCDF username and password when prompted on the login.datahub.mpcdf.mpg.de site, then link an identity from "MPCDF DataHub OIDC Server (login.datahub.mpcdf.mpg.de)" and once this is linked use the identity (`username@login.datahub.mpcdf.mpg.de`) to access the Collection. Once you have accessed the collection you should navigate to your home area where you can store data. Note: You cannot copy data to the base directory /data of the collection. ![Transfer-2](Transfer-2.png) Globus Connect Personal Clients do not require password activation, once a client has been started you can simply search for it and open it in one of the panels. Now you are ready to Transfer or Share data. Note that clicking on the "Transfer and Timer Options" provides more options to configure transfers, such as adding encryption or tuning data synchronization options. ![Transfer-3](Transfer-3.png) To start a data Transfer simply select the files or folders you wish to transfer and click "Start". You can view the details of the transfers by clicking on the link provided in the green box (top right) or by selecting the "Activity" section from the left hand column. ![Transfer-4](Transfer-4.png) Note: the Activity Page can be used to view Transfer and Delete actions during the past 90 days. ![Activity-1](Activity-1.png) ### Sharing You can share a Folder by selecting the Folder and clicking the "Share" option in the central column. ![Share-1](Share-1.png) This will first ask you to consent to manage the shared/collection. After this you will need to click "Add Guest Collection". ![Share-2](Share-2.png) ![Share-3](Share-3.png) When creating a new Guest Collection you will be able to select the directory (Folder) and add basic metadata including keyword tags (which will later allow users to search for your data). At the very least you must enter a "Display Name" which will be the Guest Collection Name. ![Share-4](Share-4.png) After the Guest Collection has been created you will be able to choose which users (or groups) you wish to share with. Click "Add Permissions - Share With". ![Share-5](Share-5.png) Now you can choose if you wish to share with a Globus Online user, group, all globus users, or even make the data public. ![Share-6](Share-6.png) To share with a user simply enter their Globus username in the search box, as you enter you will see that a real-time search is performed to help you find the user. You can share with ANY Globus User, they do not need to have an account at the MPCDF. Once the permission has been added for the user they will receive an email and you can also manage their access to the shared collection. Change Read/Write options and even remove the users access. Additionally you can select further users that you wish to share this Guest Collection with. ![Share-7](Share-7.png) ## Globus Connect Personal (Plus) ### Installing Globus Connect Personal A Globus Connect Personal Client allows you to access data on a laptop,desktop or Linux Cluster Login node in the same way you would access data on a Globus Connect Server. You can have any number of Globus Connect Personal deployments, allowing access to data on several systems. To install a globus connect personal client simply follow this [link](https://docs.globus.org/globus-connect-personal/) or select the "File Manager" section in the Globus Online Web App, click to search for a collection, and then click on the link "Get Globus Connect Personal" at the bottom. ![GCP-1](GCP-1.png) You will then be able to select your operating system to download the correct client. ![GCP-3](GCP-3.png) By clicking on "Learn More about Globus Connect Personal" you can follow the links to more detailed documentation about the installation and configuration of Globus Connect Personal. ![GCP-4](GCP-4.png) Once Installed you can activate a client via the command line or GUI. The client will then be visible in your list of collections just like any standard Globus Connect Server. **Note:** By default Globus Connect Personal Clients only provide access to the users home area. To allow access to the large data filesystems (/ptmp on our HPC systems) you will need to edit the globus-connect-personal config file as follows. Basically you need to edit the config file ``` vim ~/.globusonline/lta/config-paths ``` to allow access to other filesystems For example to add access to /ptmp on a HPC cluster (edit the config file as follows) ``` @login02:~/.globusonline/lta> cat config-paths ~/,0,1 /ptmp/,0,1 ``` The config-paths file contains entries which have the following elements (see the Globus Online docs for more info). ``` ,, ``` A re-start of the client will be needed if it is running while the changes to the config file are made. ### Enhanced Functionality with Globus Connect Personal "Plus" The MPCDF Subscription allows users to gain access to enhanced Globus Connect Personal functionality, namely sharing from clients and client-to-client transfers, by joining the "Max Planck Computing and Data Facility" group. **Please only apply for access to the group if you are sure you need the advanced functionality. Basic Client-Server transfers and sharing from DataHub do not need the Plus Group** To join the group you should navigate to your "Groups" and type the name of the Group "Max Planck Computing and Data Facility" in the search box. Once you see the group you will be able to click the option to join the group. ![Groups-1](Groups-1.png) At this point MPCDF subscription Managers will receive a notification of your request and will grant access. Note: If you used your MPCDF email address when registering your account this process will be simplified for the subscription managers. Using a non MPCDF mail address may lead to some delay in you being admitted to the group. ## Next steps Once you have access to Globus and the MPCDF systems you can explore the functionality provided by Globus and consider the best ways for you to stage data to/from MPCDF. The links below provide the next steps for you to explore. A collection of demo videos, providing an overview of the functionality provided by Globus, can be found here: [Demo Videos](https://docs.mpcdf.mpg.de/doc/data/globusonline/globus-demo-videos.html) Information about possible solutions for staging data to/from MPCDF systems can be found here: [Staging data to/from MPCDF](https://docs.mpcdf.mpg.de/doc/data/globusonline/datahub-file-staging.html) ## More Information: More information on Globus Online and Globus Connect Personal can be found in the Globus Documentation: - [How To](https://docs.globus.org/how-to/) - [FAQ](https://docs.globus.org/faq/) - [Videos](https://www.youtube.com/@GlobusOnline) - Including Demos and use-cases For specific Questions about the MPCDF support for Globus Online please create a [helpdesk ticket](../../../faq/help.html#how-can-i-get-help-and-support). # Staging Files to HPC systems via Globus Online Large datasets can be staged to and from the HPC systems or Linux Clusters using Globus Online Globus Online is a third party data transfer service which can be used to transfer large data volumes in a fast and reliable manner (For more info see: [www.globus.org](https://www.globus.org)). General information about Globus Online registration and usage of MPCDF Globus Services and License can be found here: [MPCDF DataHub and Globus Online](mpcdf-datahub-and-globus-online.md) The best method for transferring data via Globus to/from MPCDF depends on the configuration of the external site. 1. If the external site has a Globus Online endpoint then deploying a Globus Connect Personal Client on an MPCDF cluster and directly moving data is often the simplest and best option. ``` Server (External Institute) <------> Client (MPCDF HPC). ``` 2. If the external site does not have a Globus Connect Server Endpoint then data can be transferred by deploying a Globus Connect Personal Client on the external site and on the MPCDF cluster and performing a client-to-client transfer. ``` Client (External Institute) <----> Client (MPCDF HPC). ``` Access to parts of the filesytem other than the user's home directory requires the client config file to be edited (see below). Note: client-to-client transfers require that the user is part of the ```Max Planck Computing and Data Facility Globus Connect Plus``` group. For more details see the following [link](mpcdf-datahub-and-globus-online.html#enhanced-functionality-with-globus-connect-personal-plus) 3. If the external site does not have a Globus Connect Server endpoint and client-to-client transfers are not possible then data can be staged via the MPCDF DataHub Server. To use the MPCDF DataHub server as a staging server install Globus Connect Personal clients on the source and target systems and perform the transfer in two steps. ``` Client (External Institute) <----> DataHub <------> Client (MPCDF HPC). ``` Access to parts of the filesytem other than the user's home directory requires the client config file to be edited (see below). Note: the storage on the DataHub server is scratch based and your data will be regularly cleaned, please do not use this service for permanent data storage. **Tip:** A Globus Flow has been created to automate this two stage transfer approach, making the process simpler and more reliable. If you need to use the staging approach we advise you to read the documentation and watch the Demo Video [Staging Data Via Flows](https://docs.mpcdf.mpg.de/doc/data/globusonline/datahub-staging-flow.html) Options 1 and 2 are the simplest while option 3 provides an alternative in the event of problems with 1 or 2 (Client-to-Client transfers work well in general but if the clients are deployed behind firewalls transfers between the two will not be possible). ### Configuring and using Globus Personal Clients: Setting up clients is easy and you can setup numerous clients (on HPC systems, desktops, laptops). For more info see the [Globus Online How To](https://docs.globus.org/how-to/) For large transfers (to/from our HPC systems) we recommend that you start a screen session on the HPC systems and run the globus client within it. By default Globus Connect Personal Clients only provide access to the user's home area. To allow access to the large data filesystems (/ptmp on our HPC systems) you will need to edit the globus-connect-personal config file as follows. Basically you need to edit the config file ``` vim ~/.globusonline/lta/config-paths ``` to allow access to other filesystems For example to add access to /ptmp on a HPC cluster (edit the config file as follows) ``` @login02:~/.globusonline/lta> cat config-paths ~/,0,1 /ptmp/,0,1 ``` The config-paths file contains entries which have the following elements (see the Globus Online docs for more info). ``` ,, ``` A re-start of the client will be needed if it is running while the changes to the config file are made. For more information [see the docs](https://docs.globus.org/faq/globus-connect-endpoints/#how_do_i_configure_accessible_directories_on_globus_connect_personal_for_linux) # Staging data via Flows A common use-case for data transfer is staging large volumes of data via the DataHub. To make staging data via DataHub easier for users we created a specific Flow that performs the two stage transfer in one logical workflow. Globus Flows is a service which allows users to define and execute data workflows using the Globus Subscription attached to servers at MPCDF, such as DataHub. These workflows consist of Actions which can be combined into a single logical operation. The Flow, depicted in the figure below, is formed from the following 3 actions. 1. Copy data to DataHub (e.g. from a client on a cluster) 2. Copy data from DataHub (e.g. to a client on a cluster) 3. Delete intermediate data on DataHub after staging has completed ![DataHub Staging Flow](GlobusFlow-via-DataHub.png) The Flow can be started via the Globus web portal via the [link](https://app.globus.org/flows/d9929fb9-35bc-468e-a22d-0a7274afea94/start) To use the Flow users need to be a member of the "MPCDF Flows Users" group, request access via the Globus web portal. The status and results from the flow can be viewed in the Globus web portal either for the whole flow or the individual actions. The staging Flow makes use of a common area within the DataHub storage which is not visible in the user's home area. The capacity of this shared area is limited to 150TB and transfers do not count against the user's quota. Due to the limited capacity of this shared area we urge any user that wishes to perform Flow based transfers above 50TB to contact MPCDF support via a helpdesk ticket. A demo video and slideset is available via the following [link](https://datashare.mpcdf.mpg.de/s/j7Cic1WKYulRfAX) # GO-Nexus GO-Nexus combines two of MPCDF's core data solutions, Nexus-Posix and Globus, to enable projects to benefit from permanent online storage which offers reliable transfer and sharing functionality as well as the ability to create data pipelines to automate data management. Nexus-Posix is an IBM Storage Scale filesystem which is commonly used by projects in MPCDF’s HPC-Cloud. Projects can rent a reservation on Nexus-Posix which can be scaled up as the project grows. The reservations are generally in the range of 10-100 TB and are accessed via mount points on the HPC systems _Raven_ and _Viper_ and/or HPC-Cloud VMs. Globus provides a fast, reliable and user-friendly way to transfer or share large amounts of data. Additionally, Globus can aid projects in publishing findable data for their communities. These qualities make Globus an ideal service to enable access to the large-scale data stored in Nexus-Posix. Combining these two services provides a solution for several core use cases and opens extra possibilities for projects which make use of Nexus-Posix. Two of the primary use cases are highlighted in Fig. 1, namely: 1. Transfer and Sharing service with mount points on Raven and possibly HPC-Cloud VMs. 2. Standalone Transfer and Sharing service, for data collection, publishing and sharing. The first use case highlights how projects can expose the Nexus-Posix filesystem mounted on the Raven HPC system and/or HPC-Cloud VMs, with users possibly performing large-scale simulations at MPCDF and then transferring results back to their home institute or even sharing them with colleagues in world-wide collaborations. ![GO-Nexus Example Use-Cases](Use-cases-Globus-Go-Nexus-2022.11.29.png) The second use case shows how standalone storage can be made globally available via GO-Nexus. This could be used when gathering data in the field for processing at a later date and/or for distributed collaborations where GO-Nexus would act as a central datastore, benefiting from the high-speed network connection at MPCDF. In both cases the reservations can be exposed either as findable or as private data collections via Globus, where users and community members can easily search for the data via the Globus web portal. In addition to the reliable transfer capabilities GO-Nexus benefits from all the advanced functionality which is available via the MPCDF’s globus subscription, enabling actions such as sharing and the use of Globus flows for automation. Several cloud projects have already adopted GO-Nexus for large-scale data transfers and to regularly sync data to and from Nexus-Posix by using “Globus timers”, a cron like service offered through the Globus web portal. # Globus Demo Videos - Demonstrating Globus Functionality for end users Globus offers a large range of functionality for data transfer and sharing and while many users have started to use basic Globus transfers the more advanced functionality often seems to be just out of reach. To help users gain a better insight into what is possible we created a set of short demos covering a range of topics, starting with the basics of client installation up to advanced topics such as creating groups and sharing. Each demo shows hands-on sessions where a specific topic is explored. The following demos exist and are available on datashare by following the links - Title (duration) - [Globus-Demos-Introduction](https://datashare.mpcdf.mpg.de/s/BBRBlQF8xHcqxeR) (1m37s) - [Globus-Connect-Personal-Raven-Install](https://datashare.mpcdf.mpg.de/s/GBxVchRy13ZW59G) (9m2s) - [Globus-Transfer-Example](https://datashare.mpcdf.mpg.de/s/5fPcg6sWjEzSOSM) (7m51s) - [Globus-Advanced-Transfer-Options](https://datashare.mpcdf.mpg.de/s/RxqEWfZbs5gF9S9) (7m32s) - [Globus-Groups](https://datashare.mpcdf.mpg.de/s/vsNW3rkr7rAitQx) (5m56s) - [Globus-Data-Sharing](https://datashare.mpcdf.mpg.de/s/peqhuy9Z2XTngrL) (7m38s) - [Globus-Advanced-Client-Functionality](https://datashare.mpcdf.mpg.de/s/awVu2a1pCQzrcwX) (4m58s) - [Globus-Timers](https://datashare.mpcdf.mpg.de/s/CtfjN2W8duUSCYR) (6m58s) The demos can be viewed online via a web browser or offline by downloading them. A slideset is also available with basic information about each demo : [Globus-Demos.pdf](https://datashare.mpcdf.mpg.de/s/Xx8DexCPdL7ZjaM) The folder with the complete set of demos is available on datashare by following the [link](https://datashare.mpcdf.mpg.de/s/UXgZ9vUSBiyypZO): ------------------------------------------------------ Nexus-S3: Object Storage for data Transfer and Sharing ------------------------------------------------------ .. toctree:: :maxdepth: 1 :glob: nexus-s3.md.txt publishing-data-via-s3.md.txt # Nexus-S3 Nexus-S3 is a scaleable object storage service compatible with the Amazon S3 protocol. MPCDF users can opt-in (see below) to Nexus-S3 which provide a free 1TB quota (up to 1 million objects). Data can be accessed using standard S3 clients and libraries such as [minio](https://min.io), [s3cmd](https://s3tools.org/s3cmd), [rclone](https://rclone.org/), [cyberduck](https://cyberduck.io/) and python-boto3 as well as via [Globus](https://www.globus.org/) (MPCDF GO Nexus S3 Collection) or via a web browser/curl in the case of public buckets. Nexus-S3 also supports object storage functionality such as versioning, life-cycle policies and temporary URL generation to allow users to download files with an expiry date. Together with the transfer and sharing functionality available via Globus this provides many solutions to use-cases such as large scale data sharing and publishing. ## Opt-in via SelfService Access to Object Storage is possible via the [MPCDF SelfService](https://selfservice.mpcdf.mpg.de). Log in with your MPCDF account and go to “My account / Services” to opt-in for Nexus-S3. Once the account has been created in the S3 service you can access your access/secret keys by clicking "View Access Token". These access/secret keys are used by S3 clients and Globus to access your S3 storage, please keep them safe and treat them as you would a password. If you feel these keys may have been exposed please create a helpdesk ticket and request that new keys be generated. NOTE: After opt-in, it can take up to 60 minutes until the accounts are created and for the access/secret keys to become available. ## Accessing Object Storage Nexus-S3 is globally available and can be accessed from MPCDF and external servers. To access using S3 clients such as minio, s3cmd or rclone simply copy the access and secret keys from selfservice and configure your client to use these with the service host/endpoint set to s3.nexus.mpcdf.mpg.de. #### Access via S3 _S3cmd:_ S3cmd is one of the most popular command line tools for accessing S3 based object-storage. To configure s3cmd run: ```sh s3mcd --configure ``` Many parameters can be left as default. However, the following need to be set to match the MPCDF systems. Your access and secret keys (please ensure these stay private). ``` Access Key: ***************** Secret Key: ***************** ``` Set the S3 endpoint to the endpoint name of the MPCDF storage you are using. ``` S3 Endpoint [s3.amazonaws.com]: s3.nexus.mpcdf.mpg.de ``` Once configuration is complete the config file is stored in ~/.s3cfg. This can be viewed and edited as any normal txt file. S3cmd can be used to manage data in the object storage. See ```man s3cmd``` or ```s3cmd -h``` for more info. A simple example set of commands follows, make a bucket, copy data to the bucket, list bucket contents and then query the bucket for info such as access policies : ``` s3cmd mb s3:// s3cmd put testfile s3:/// s3cmd ls s3:// s3cmd info s3:// ``` #### Access via Globus: 1. Log in to the [Globus Web UI](https://app.globus.org/) 2. Open the collections tab 3. Search for the collection: "MPCDF GO Nexus S3 Collection" 4. Click on the collection (see screenshot for the collection info) ![Nexus S3 Collection](Globus-GO-Nexus-S3-Collection.png) 5. Click on Credentials 6. Follow the steps to authenticate and provide consent 7. Copy and paste your access/secret keys into the relevant fields 8. Nexus-S3 Objects can then be accessed via the File Manager view as usual Note: When using Globus the keys will be encrypted on the Globus server at MPCDF and this encrypted form of the keys will be stored in the Globus Services in the Cloud (no decryption keys are stored in the Cloud). ## Bucket names and Public access Similar to Amazon’s simple storage service we use a global namespace for our buckets. Please think of a unique name for your bucket. If another bucket of the same name already exists you will see an error message stating: Error: Forbidden insufficient permissions on requests operation. A good practice for naming buckets is to prefix them with a project name or similar general prefix. This helps avoid possible contention with bucket names and also protects somewhat against public buckets being unexpectedly crawled by automated systems or malicious users on the internet. Additionally, care should be taken when creating and managing buckets to ensure that access rights are correctly set. When setting buckets as public some S3 clients will set read+write access by default. Please be careful. Publicly accessible buckets will quickly be discovered on the internet and may be abused if writeable. ## No backups of object storage Please be aware that the data stored in the Object Storage service is not backed up. # Publishing Data for public access via S3 Projects are free to set public access for download when needed. S3 command line clients such as s3cmd and minio-client support setting objects to be publicly available. For example, you can set public access by using s3cmd. Make the bucket listable: ``` s3cmd setacl --acl-public s3://public-bucket ``` Enable public download of objects in the bucket: ``` s3cmd setacl --acl-public --recursive s3://public-bucket ``` Individual objects can also be set public ``` s3cmd setacl --acl-public s3://public-bucket/my-public-object ``` Note: As new objects are added the access acl for these needs to be updated (this is not inherited from the bucket) Objects may be returned to private access by using: ``` s3cmd setacl --acl-private --recursive s3://public-bucket ``` And similarly for listing and/or individual objects. We advise against public access for upload since this would open the S3 storage to abuse. A few things to note: 1. Be aware that some clients set READ/WRITE access when you use the "public" option. e.g. for the minio-client: ```mc anonymous set public storage/public-bucket``` will actually allow anonymous writes as well as reads. 2. Any content you make public readable is likely to be crawled by Google etc ## Publishing data When publishing a data set it is advisable to provide a landing page with basic information about the data. This may be achieved by using the MPCDF [metastore](https://docs.mpcdf.mpg.de/doc/data/publication/metastore-documentation.html) or by creating a landing page within the public bucket. When using metastore, DataCite compatible metadata can be associated with the dataset which may be made available as links to the S3 objects. MetaStore makes the Findable as defined in FAIR data. When creating a stand alone landing page within the S3 Bucket it is advisable to: 1. Create an index.html page within the bucket 2. Describe the dataset within the index.html page (origin, owners, size etc) 3. Add a link to each object (including a checksum or a separate checksum file) 4. Provide basic information about how the objects can be downloaded (e.g. via curl, wget) ## Digital Object Identifiers (DOIs) for published data Digital Object Identifiers provide a persistent identifier for datasets which makes the data addressable and allows the underlying dataset to be moved in a transparent manner where the end users are simply re-directed to the new location. A DOI may be obtained via metastore or directly from the MPDL [MPDL-DOI](https://doi.mpdl.mpg.de/). ## Temporary Sharing: You can give temporary access to data via presigned URLs. These allow you to generate a short-lived URL that has an obscure form and a configurable lifetime. These URLs may safely be passed to data users to retrieve individual objects. More information about temporary file sharing can be found [here](https://docs.mpcdf.mpg.de/doc/cloud/technical/recipes/temporary-file-sharing.html) ------------------------------------ Small to Medium Scale Data Transfers ------------------------------------ .. image:: /_images/transfer01.png :width: 500px The MPCDF operates a bunch of different (storage) systems. These systems are different in both capabilities and size. Therefore, the question how to transfer data in and out the MPCDF has a lot of different answers. On this pages, you will find tools and tutorials about how to transfer data ín and out the MPCDF. Beside this documentation, we are happy to answer individual questions via our Helpdesk System (https://helpdesk.mpcdf.mpg.de). If you need individual support, please prepare the following questions in advance: * Do you want to transfer data into or out of the MPCDF? * Which MPCDF (storage) systems are involved? * What is the total size of the data to be transfered? * How many files you need to transfer in total? .. toctree:: :maxdepth: 1 :glob: data-transfer.md.txt mpcdf-datahub-and-globus-online.md.txt sharing-large-files-with-datashare.md.txt # Data Transfer: Tools & Tips Tools and tips for transferring data to and from MPCDF Several options exist to enable data transfer to and from MPCDF. Here we outline the main tools which are in general use for data transfers at MPCDF. Since each data transfer case is different we will break the tools down into three categories: * Large scale data transfers * Small/Medium scale data transfers * Sharing of small datasets (files) The purpose of the page is not to set in stone which tools should be used when, but rather to provide advice based on our experience. In some cases there is overlap in the use-cases which each tool is useful for. As you start to transfer data to/from MPCDF it is worth gaining some experience of each tool type before you settle on a specific tool. In addition to describing the data transfer tools we will also introduce the screen tool which can be used to help keep long running transfer processes going even when a users needs to log out of a session. ## Large Scale Data Transfers For large scale data transfers (often in the multi TB range) the MPCDF has made good experience using the bbcp and globus-online tools. ### Globus Online Globus Online (Globus.org) is a free service which allows users to move large volumes of data in a simple and reliable manner. In general Globus Online requires sites to set up a Globus Connect Server for data transfers, however, individual users can also install a Personal Client to enable them to move data to/from Globus servers. The Globus Online web portal provides a user friendly interface that enables users to transfer data between Globus Online servers. Many research centers have existing Globus Online servers which can be found via the web portal. Data transfers can be scheduled via the web portal and the Globus service will move the data reliably and transparently in the background. Below is a screenshot of the Globus Online portal enabling data transfers. The left hand pane is a session connected to a Globus Connect Server, the right is a session connected to a Globus Connect Personal Client on a laptop. Data can be move between these two endpoints by simply highlighting it and clicking on the transfer arrow. ![Globus Online Portal](4721b4d0-57b3-42d7-8a1a-ffd3378c4016.png) Globus enables, fast and reliable multi-stream data transfers, data syncing, checksum verification, encrypted transfers and more. The Globus Collection "MPCDF DataHub Stage-and-Share Area" can be used for staging data to and from MPCDF as well as Sharing data with other Globus Users. All MPCDF users have access to DataHub and in cases where a project does not have a dedicated globus endpoint the datahub should be used as a default. The MPCDF can aid with the deployment of Globus Online Servers for specific projects and Globus Online Personal Clients for individual users, including providing membership to Globus Connect Personal Plus. This means that a Globus Online solution can be found on a project and/or user level - enabling data to be transferred between external sites and MPCDF. For more information see: ### bbcp __As the bbcp tool is no longer under active development, it is only available for legacy reasons. We recommend to use Globus Online or another tool mentioned in this section.__ bbcp is a point-to-point network file copy application written by Andy Hanushevsky at SLAC as a tool for the BaBar collaboration. It is capable of transferring files at approaching line speeds in the WAN. bbcp is made available on MPCDF clusters via the modules environment. To enable bbcp: ```sh module load bbcp bbcp --help ``` bbcp is a peer-to-peer application. No server process is required - you just invoke bbcp on a source machine and in response a bbcp process is started on the target machine. You can also do this as a third party: the source and target machines do not need to be the same machine that you initiate the file transfer from. Note: this means that you need bbcp to be installed on both source and target machines and to have ssh login access on each machine. Among other features bbcp supports multiple streams, checksums, recursive copies, resumption of failed transfers, reverse connections and data compression. All of these features are explained in depth in the online docs listed below. A simple example follows: ```sh bbcp -P 10 -w 2M -s 10 test_10G_file user@remotehost.no.de:/userdata ``` This would cause bbcp to transfer a testfile (test\_10G\_file) to a remote location remotehost.no.de (where the remote username is user). The transfer would use 10 parallel streams (-S 10) with a TCP window size of 2MB (-w 2M) and report progress every 10 seconds (-P 10). In general some experimentation with the number of streams and window sizes may lead to better transfer rates, however in most cases the default behavior to use window auto-tuning is more than sufficient. More information about bbcp can be found here: ### Transferring small/medium data sets For transfers of smaller datasets, in the GBs range, tools such as rsync and scp and sftp are perfectly suitable. These tools can be used for transferring datasets to and from ssh enabled servers. The main benefits of these tools are that they are widely available, well known and relatively simple to use. The drawback is performance, they will simply not achieve the out-of-the-box data transfer rates that bbcp and globus online will. The rsync tool is a natural fit when syncing data and can improve data transfer speeds by simply avoiding transferring data that has not changed. Many systems at MPCDF allow outbound connections and thus command line tools can be used to start data transfers from these systems to external servers. In cases where a connection is required from an external system to an MPCDF system/linux-cluster users can make use of ssh tunnels via the gate1.mpcdf.mpg.de or gate2.mpcdf.mpg.de nodes. This is especially useful for SFTP. From an external system (e.g. laptop/desktop) create an ssh tunnel, in this example to a raven login node, but other MPCDF cluster login nodes will work similarly: ```sh ssh @gate1.mpcdf.mpg.de -L 2002:raven.mpcdf.mpg.de:22 -N ``` Once this tunnel has been established SFTP can be used to access the login node as it if were on your local system (in this case point your sftp client to port 2002 on localhost). This means that you can use file transfer tools such as FileZilla by just setting up the tunnel and configuring the FileZilla remote SFTP connection to use localhost and port 2002. When using FileZilla the 2FA may cause some problems (login requests can occur on each file transfer). To overcome this change the Login Type to interactive and set the Max number of connection to 1 in the Site Manager configuration. Note that for windows systems WinSCP is also capable of using the gate node as a proxy. Simply configure WinSCP to use an ssh tunnel in the Advanced Options section using gate1.mpcdf.mpg.de as the hostname and your usual MPCDF user name and password. To simplify direct access from Linux-based systems the ssh ProxyJump option can be used. To access the archive (or any cluster login node) ```bash sftp -o 'ProxyJump @gate1.mpcdf.mpg.de' @archive.mpcdf.mpg.de: ``` on newer OpenSSH versions (7.3 and above) you can use the -J option directly: ```bash sftp -J @gate1.mpcdf.mpg.de @archive.mpcdf.mpg.de: ``` Note: This will also work for ssh connections and rsync via ssh ```bash ssh -J @gate1.mpcdf.mpg.de @raven.mpcdf.mpg.de rsync -av -e 'ssh -J @gate1.mpcdf.mpg.de' source-dir @archive.mpcdf.mpg.de: ``` For more information see the scp, sftp and rsync man pages and/or search the internet (many tutorials and tips exist). Note: rsync, sftp and scp can also be used for transferring larger data volumes, you may need to monitor the transfers over a larger timeframe though (or write wrapper tools to parallelize their usage). ### Exposing data via MPCDF DataShare The MPCDF DataShare service provides MPCDF users with a web based sync and share service. This service allows users to upload and share data with external collaborators, or to simply upload it for later download from an external site (or home PC/laptop/tablet). This makes the Datashare service a perfect fit for exposing and sharing smaller datasets, or single files (documents). To avoid storing your central MPCDF username and password in scripts or apps, DataShare offers the possibility to create so called _device passwords_. These device passwords are additional credentials for your regular DataShare account. You can create as many as you need and use these credentials instead of your regular account within your apps or scripts to access DataShare. Once there is a security issue on one of your devices (lost smartphone, hacked account etc.), only the app password is affected and not your MPCDF account. Therefore, we strongly recommend using DataShare device passwords wherever possible. To create a new device password, log in to DataShare. In the top right corner, click on your user name and open "Settings". In the menu on the left, go to "Security" and scroll down to "Devices & Sessions". Here, you can create new app passwords for every device you are using DataShare on. In the text field, enter the name of the app for which you want to create an app password for. __DataShare will show you the automatically generated password only once, so make sure to copy and save it somewhere else!__ Now, your app can access DataShare with the combination of your username and the new created app password. ![DataShare App Password](datashareAppPasswords.png) #### Accessing DataShare via Pocli To enable this service for users of the HPC and linux clusters a command-line client called pocli (Python ownCloud/Nextcloud command line interface) was developed. The Pocli client supports basic operations such as upload or download of single or multiple files, directory creation, and file or directory removal. ```{eval-rst} .. admonition:: Uploads of files larger than a few GB using pocli can take a long time Please either split large files into smaller pieces or use other tools like rclone that can take care of this for you. ``` To get started with the pocli client use the commands: ```sh $ module load datashare $ ds --help ``` Help on individual commands is available as well, e.g.: ```sh $ ds put --help ``` Two basic usage examples are described in the following. Example 1: To upload a file to your DataShare space use the command ```sh $ ds put file.tar ``` As a second step, you can then log in to the DataShare web portal, share the file with another DataShare user, or create a download link for your external collaborator. Example 2: Let's assume that there's a file located at 'data/file.tar' in your DataShare space (owned by you, or shared by another DataShare user with you). To download the file to the current working directory, issue the command ```sh $ ds get data/file.tar ``` Technically, pocli is written in Python (tested with versions >=2.7) on top of the pyocclient library. At the first invocation of the 'ds' command a configuration file `~/.ocrc` is created. It is preconfigured for the MPCDF DataShare service, but can be edited and adapted to any ownCloud service. The 'ds' command asks for the password at each invocation. ### RClone [Rclone](https://rclone.org) is a command line program to manage files on remote/cloud storage. Rclone has a rich set of features and supports over 40 cloud storage systems including Nextcloud (Datashare), OpenStack Swift, as well as standard transfer protocols (HTTP, SFTP, FTP) and local filesystem. Rclone's ability to connect to many different storage services makes it a real swiss army knife when it comes to moving and managing data. It is a very valuable tool for modern day researchers whose data is often located in several different data silos. Within Rclone each storage resource is configured as a remote. Calling rclone config from the command line will open an interactive configuration session: ```sh rclone config e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> ``` Within this session, remotes can be added and/or altered. Alternatively you can call rclone config with a specific configuration option directly. (see the note below regarding secure configurations). Once remotes are configured they may be accessed to list content ```sh $ rclone ls remote:path ``` Data may be copied or moved between remote storage resources as follows ```sh $ rclone copy source:sourcepath dest:destpath $ rclone move source:sourcepath dest:destpath ``` The actual data transfer runs through the rclone client. Additionally Rclone allows for data syncing (similar to rsync) ```sh $ rclone sync source:path dest:path ``` This will sync the source to the destination, changing the destination only. Unchanged files will not be transfered and files at the destination may be deleted. Since this can cause data loss, always test first with the --dry-run flag to see exactly what would be copied and deleted. Note: Be advised that rclone sync acts differently to rsync w.r.t. the creation of target dirs, rclone will not auto-create dirs on the target For instance: _never_ do `rclone sync somedir datashare`: This will delete all the data in datashare - replacing it with the data in somedir. (using --dry-run will help avoid such problems) To use local storage simply ommit the remote prefix and use the data path as usual. Several remotes can be configured (using different protocols), allowing you to easily move data between services. The example below shows remotes configured to connect to aws-s3, the MPCDF DataHub and Datashare services and an openstack swift instance. ``` Current remotes: Name Type ==== ==== aws s3 datahub sftp datashare webdav openstack swift ``` Once this configuration is set up data can be easily moved between Datashare, Datahub and local storage as well as any cloud based storage (in this case swift and s3). Rclone is a Go program can be installed as a single binary file. For more information and to download Rclone please see the official Rclone website: Notes: Safe Configurations. When a remote is configured in Rclone the remote password is saved, in obscured mode, in the rclone configuration file. To secure the passwords you can create a password for the rclone configuration itself. When Rclone config is called from the command line, you will see several options. If you select "s" you can set a configuration password (see below). ```sh $ rclone config s) Set configuration password e/n/d/r/c/s/q> s Your configuration is not encrypted. If you add a password, you will protect your login information to cloud services. a) Add Password q) Quit to main menu ``` Once a secure configuation file has been created you will need to provide a password each time you start an rclone session. ALWAYS create a secure configuration file. #### RClone configuration for DataShare As stated above, you can create a new _remote_ for rclone via _rclone config_, after that choose "n": ``` rclone config e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> n ``` After rclone has asked you for a name for the new remote, you need to choose its type. For DataShare, choose "Webdav" (here, its number is 29, but numbers can change): ``` 29 / Webdav \ "webdav" ``` Next, enter the following DataShare URL and don't forget to enter your real user name: ``` 1 / Connect to example.com \ "https://example.com" https://datashare.mpcdf.mpg.de/remote.php/dav/files/YOUR_USERNAME/ ``` rclone supports several WebDAV based cloud solutions - DataShare's backend is Nextcloud: ``` 2 / Nextcloud \ (nextcloud) ``` Your username and password is required - __the password should be a new created Device Password and not your MPCDF password, see section above!__ ``` User name Enter a string value. Press Enter for the default (""). user> USERNAME Password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank Enter the password: password: Confirm the password: password: ``` The next questions for bearer tokens and advanced configuation can be skipped. Finally, rclone will show you a summary of the new configuration, confirm with "y": ``` url = https://datashare.mpcdf.mpg.de/remote.php/dav/files/YOUR_USERNAME/ vendor = owncloud user = USERNAME pass = *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y ``` Leaving the configuration interface, you can now access DataShare via rclone (replace "ds" with the name you gave the DataShare remote): ``` rclone lsf ds: ``` ### The screen command Although not a data transfer command, the screen command can be very useful when transferring large datasets. The screen command is a window manager that allows user sessions to continue even after a user has logged out. Long running data transfers can often take many hours or days, wrapping a session with screen means that a user can logout and re-connect to the session later, picking up from where they left off: * Type `screen` * Use the session as usual (starting a data transfer etc) * Detach from session using `CTRL-a d` * Then view existing sessions `screen -ls` * To re-connect to a session `screen -r ` Don't forget to kill the session when finished "CTRL-a k" for a single window, or "CTRL-a \\" for a multi-window session. See the screen man page for more information or search on the internet (many tutorials exist). ### Support: As always MPCDF support is available to answer questions - please submit a helpdesk ticket in cases where you feel support and/or advice is needed when transferring data. ### Some general notes: When moving large datasets tools like tar and zip are your friend. If you have a large number of small files to transfer across a network link then real performance gains can be found from forming compound archives from these files and compressing them. Transferring big chunks of compressed data is a good idea. Also using formats such as the BagIt format can help to ensure that the data payload is correctly described and that checksums of the data files exists. So when considering data transfers it is good to start by looking at your data and its format and asking if you can/should transform the data before transferring it. And finally a small note of caution: When transferring datasets over a wide area network (between data centers) a certain amount of fluctuation can be expected in the transfer rates. The networks are shared with many users and the storage system at both the source and sink are often shared. # MPCDF DataHub and Globus Online [This page has moved.](../globusonline/mpcdf-datahub-and-globus-online.md) # Sharing Large Files with DataShare To enable large file transfers via DataShare we advise using rclone chunker. This recipe will focus on sharing data via a public link, however, rclone can also be configured to use a standard user account in DataShare. ## Set up share folder in DataShare 1. Create a new folder for the data in DataShare 2. Via Sharing - Public Links, create a share with `read/write` permissions ![Create Public Link](datashare_create_link.png) 4. Copy link to clipboard and paste into the text editor of your choice 5. Extract the cryptic share token at the end of the url and save it for the rclone configuration ![Get Share Token](datashare_share_token.png) 6. Optionally repeat steps 2-5 to create another share with `readonly` permissions if recipient should only be able to download files ## Sender: Upload files using rclone 1. Configure rclone remote and chunking overlay. ``` > rclone config create testproject webdav url https://datashare.mpcdf.mpg.de/public.php/webdav/ user pass > rclone config create testproject-overlay chunker remote testproject: chunk_size 2G hash_type none ``` The default chunk_size of 2GB generally works fine. It can be increased up to 20GB if fewer chunks are desired. However using very big chunks might cause problems with slow clients or network connections (also relevant during download). Checksums can be enabled if desired (e.g. `hash_type md5`) but will of course take some additional time to calculate. 2. Upload individual files or a whole directory ``` > rclone copy 5g testproject-overlay: --progress --transfers 1 Transferred: 5G / 5 GBytes, 100%, 52.979 MBytes/s, ETA 0s Checks: 3 / 3, 100% Renamed: 3 Transferred: 1 / 1, 100% Elapsed time: 1m41.6s ``` The `--transfers 1` option ensures that only a single operation is running at a time. Please make sure to always use it when doing chunked uploads to DataShare; multiple concurrent transfers can actually slow things down due to synchronization overhead and generate unnecessary load on the server. ## Files on the server On the server, the folder will look like this (5g.rclone_chunk.001, 5g.rclone_chunk.002...): ![Chunked Files in DataShare Folder](datashare_chunked_files_in_folder.png) The file with the original name (`5g` in this example) just contains some metadata (number of chunks, checksums if enabled). Data is split into chunks of `-`. If desired, chunks can be downloaded via the web interface or curl and assembled manually e.g. with `cat -rclone_chunk-??? > `. ## Recipient: Download files again using rclone For larger data sets, setting up rclone on the recipient as well is recommended: 1. Configure rclone remote and chunking overlay ``` > rclone config create testproject-readonly webdav url https://datashare.mpcdf.mpg.de/public.php/webdav/ user pass > rclone config create testproject-readonly-overlay chunker remote testproject-readonly: ``` 2. Download individual files or a whole directory ``` > rclone copy testproject-readonly:5g 5g-from-remote --progress Transferred: 5G / 5 GBytes, 100%, 91.694 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1m2.1s ``` ---------------------------- GitLab: Software Development ---------------------------- .. image:: /_images/gitlab01.png :width: 800 GitLab covers the whole process of software development. Starting with the basic functionality of Git - a distributed version control system - GitLab offers today a wide range of tools and functionality for software developers (*dev ops*). The MPCDF GitLab instance is available to all MPCDF users and their external collaborators. *Continous Integration* can be done on central hosted shared GitLab Runners, results can be published via *GitLab Pages* and Docker images as well as software packages in various formats can be stored in GitLabs Image and Package Registry. .. toctree:: :maxdepth: 1 :glob: gitlab.md.txt devop-tutorial.md.txt gitlabrunners.md.txt # The MPCDF GitLab Instance: an introduction The MPCDF GitLab service offers git repository management, code reviews, issue tracking, activity feeds, wikis and many more. Before you can use the GitLab service, you have to opt-in for it at the MPCDF SelfService service: Login with your MPCDF account and go to "My account / Services" to opt-in for the GitLab service. Further information about the SelfService can be found on its [Help](https://selfservice.mpcdf.mpg.de/index.php?r=site%2Fhelp) page. At the Selfservice, you can also subscribe for other MPCDF services like DataShare. After you have opted in, you can log in to GitLab: You can find the official GitLab manual at: If you have any questions or you wish to request a new group / shared workspace for your project in GitLab, you can contact [support](../../../faq/help.md). # Poetry and GitLab: Devops for Python developers ## Introduction The MPCDF GitLab instance offers a wide variety of so called _Devops functionality_. This includes tools for manual project management as well as automation tools for code building, testing and deployment (_continuous integration_). Poetry is a tool for _Python packaging and dependency management_. While GitLab supports the software development life cycle on the level of collaboration and automation, Poetry supports the Python developer on a lower level: it helps the developer to set up and manage a Python project on their local computer. Together, Poetry and GitLab are building up an excellent tool ensemble to implement large Python projects. This tutorial will show how to use both GitLab and Poetry and how a combination of these two tools can lead to an efficient _Devop_ workflow. ## Overview ## The Poetry Project In [Poetry: Packaging and Dependency Management for Python](../../../bnb/207.html#poetry-packaging-and-dependency-management-for-python), the installation, configuration and usage of Poetry was already explained in detail: in this tutorial, only the basic and necessary Poetry commands will be shown again. Create a new Poetry based Python project (name "python-devop" which will create a new folder with that name): ``` poetry new python-devop ``` Change to the newly created directory "python-devop" and you will see a directory structure like this: ``` ├── pyproject.toml ├── python_devop │ └── __init__.py ├── README.rst └── tests ├── __init__.py └── test_python_devop.py ``` The file _pyproject.toml_ holds the metadata of your project. Beside dependencies to other packages, a project description etc., this file contains a versioning number for your project: ``` [tool.poetry] ... version = "0.1.0" ``` When later publishing your project via GitLab / CI, it is very important to increase the versioning number before every commit. If you do not change the version in your current commit, the package build itself will work, but due to version number conflicts, it can not be pushed to GitLab's package registry. ### Writing code ... Now it's time to write some code! For this tutorial, we just add a Python module "helloworld.py" to the sub folder "python_devop" with two easy functions: ``` #!/usr/bin/env/python def sayHello(): return "Hello world!" def add (a,b): return a + b ``` ### ... testing the code In this tutorial, only the first and basic steps of testing can be shown. For a more detailed overview of Pythons testing capabilities, see [Unit testing framework](https://docs.python.org/3/library/unittest.html). Poetry makes testing your code easy. In the "tests" sub folder, add a file "test_python_devop.py" with the following content: ``` import unittest import python_devop.helloworld class TestStringMethods(unittest.TestCase): def test_add(self): theSum = python_devop.helloworld.add(2,3) self.assertEqual(theSum, 5, "Sum should be five ...") def test_sayHello(self): greetings = python_devop.helloworld.sayHello() self.assertEqual(greetings, "Hello world!" , "Greetings should be: Hello world!") if __name__ == '__main__': unittest.main() ``` Now, activate the current virtual environment via ``` poetry shell ``` From the root folder of your project, install the package: ``` poetry install ``` After that, you can execute your tests from within the "tests" subfolder: ``` python3 -m unittest -v ``` If the tests were successful, you should see an output similar to this: ``` test_add (test_python_devop.TestStringMethods) ... ok test_sayHello (test_python_devop.TestStringMethods) ... ok ---------------------------------------------------------------------- Ran 2 tests in 0.000s OK ``` If you see a "FAILED" statement instead of "OK" in the last line, something went wrong in your code and / or tests. Check both again and correct them if necessary. Now, everything is set up on the local side. We can now go on and couple the Poetry project with GitLab. ## The GitLab Repository After your code works and the tests were running successfully on your local machine, it is time to create a GitLab project and upload ("push") your Poetry project into it. Log in to [GitLab](https://gitlab.mpcdf.mpg.de) and create a new project. You can use the "Blank project" template: just give the project a name ("python-devop") and if necessary, change the permissions ("Private / Internal / Public"). You will now be redirected to the start page of your new, still empty repository. Here, you can find some instructions how to fill the repository. As you have already your Poetry project on your local computer, follow the steps under "Push an existing folder" inside your Poetry project: ``` cd existing_folder git init --initial-branch=main git remote add origin git@gitlab.mpcdf.mpg.de:YOURUSERNAME/NAMEOFYOURREPO.git git add . git commit -m "Initial commit" git push -u origin main ``` If you are using an old version of Git, the option "--initial-branch" does not exist. In this case, just execute these two commands instead of the "git init" command from above: ``` git init git symbolic-ref HEAD refs/heads/main ``` In the GitLab web interface, you should see now the files and directories of your Poetry project (press "F5" (reload) in your browser). Congratulations - you have now a working Python project, ingested initially into a GitLab repository. ## The CI pipeline With GitLab's _Continous Integration_, it is possible to automate parts of a project's workflow. These automations are organized in _CI Pipelines_ which you can find in GitLab's menu on the left under the menu point "CI/CD". In this tutorial, we will * automate the tests you have already written in the chapter before * build a Python package out of the Poetry project * deploy this package to GitLab's internal package registry All of these steps will be executed automatically on the GitLab server if a new commit was done. All you need to do is to create a new file called _.gitlab-ci.yml_ (don't forget the dot!) in the root folder of your Poetry project. In the file _.gitlab-ci.yml_ we can now define so called _stages_ which are automatically executed on GitLab runners after every push to GitLab (in the [GitLab documentation](https://docs.gitlab.com/ee/ci/), you will find much more information about _Continous Integration_ & Co.) But let's add some content to the file _.gitlab-ci.yml_: ``` default: image: python:3.9 ``` At first, we need to define a Docker image which should be used as basis for a container to execute the CI Pipeline. GitLab will automatically clone the Git repository into the running Docker container, so all files of your Poetry project are available inside the container. As we created a Python project within Poetry, we will use the Python image in version 3.9 from DockerHub. As a next step, we need to prepare our Python/Poetry environment inside the Docker container. This has to be done _before_ any of the later defined stages are executed, so we put it into a section _before_script_. The three lines will install Poetry, then use Poetry itself to install our project and last but not least, activate our virtual environment. You will recognize the code: ``` before_script: - pip install poetry - poetry install - source `poetry env info --path`/bin/activate ``` Now, we have everything to execute Poetry functionality and we can start to define _stages_ which are doing something within our CI pipeline: ``` stages: - test - build ``` We are defining two stages - one for the _tests_ we defined already and a second one _build_ for building and deploying the project. This second one will also do the deployment into the GitLab package registry, because it makes no sense to define an extra stage here. Every job in a CI pipeline gets its own instance of a Docker container, executed one after the other. This means in a possible third stage for deployment, the build process needs to be done again, so to make life easier, we are combining these two steps already in the _build_ stage. After defining the stages, we can now define _jobs_ inside the stages. First, start with a job for executing the unit tests: ``` testing: stage: test script: - echo "This is the test stage" - cd tests - python3 -m unittest -v - pytest --junitxml=report.xml artifacts: when: always reports: junit: /builds/USERNAME/python-devop/tests/report.xml ``` The job _testing_ belongs to the stage _test_ and executes a _script_. The commands in here should be well known from the testing phase in the previous chapter. The parameter _--junitxml_ defines an XML file for the output of the tests. In the _artifacts_ section, this file is used by GitLab to produce a nice graphical report which can be found in the pipeline overview under _Tests_ (don't forget to change your GitLab username here): ![](testreport.png) Let's go on to the second job _building_ which belongs to the _build_ stage: ``` building: stage: build needs: [testing] script: - echo "This is the build stage" - poetry config repositories.gitlab https://gitlab.mpcdf.mpg.de/api/v4/projects/6373/packages/pypi - echo "Repository gitlab configured ..." - poetry build - echo "Build done ..." - poetry publish --repository gitlab -u YOURUSERNAME -p YOURTOKEN - echo "Publishing done!" ``` Here we have another option _needs_: it refers to the _testing_ job and means, that the _building_ job will only be executed if the _testing_ job was successful. Then we have again some Poetry commands, let's have a deeper look at them: ``` poetry config repositories.gitlab https://gitlab.mpcdf.mpg.de/api/v4/projects/XXX/packages/pypi ``` This Poetry command defines a new private package repository under the short name "gitlab". In the URL, you need to change the three XXXs with the project id of your GitLab project - you can find this id on the start page of your repository: ![](projectid.png) Next, we are building the whole project: ``` poetry build ``` And - last but not least - publish the outcome of the build process as PyPi package into GitLab: ``` poetry publish --repository gitlab -u YOURUSERNAME -p YOURTOKEN ``` Don't forget to change two things in this command: * YOURUSERNAME: your GitLab user name * YOURTOKEN: In GitLab, you can create so called _Access Tokens_ which can be used as kind of credentials for scripts and other tools. You can create them in your account under the menu "Access Tokens". The one which you need for publishing to the GitLab package registry needs GitLab API access. That's it - after you have saved the file _.gitlab-ci.yml_, you can now push your project again into GitLab (don't forget _add_ and _commit_ commands before pushing). The CI pipeline will run automatically, at first the _testing_ job and then the _building_ job will be executed. If everything worked well, you can take a look at the detailed overview page of the pipeline: ![](pipeline.png) Here you can also see if something went wrong, a click on the jobs opens their console output. Take a look at the _Package Registry_, you should see your package here now: ![](packageregistry.png) You can see that the package carries the version number you entered at the beginning into the Poetry config file. So if you did some changes to your project and the whole pipeline should run again, you need to change (increase) the version number. If you forget this, the CI process can't publish into the package registry and your _building_ job will fail. If you want to avoid giving your project permanently new version numbers, you can create another "developer" branch. Here, you can for example just execute the _testing_ but not the _building_ job, or you just get rid of Poetry's _publishing_ command. ## Using the published package If you want to use the new created package now on other computers or virtual environments, GitLab shows you how to do so. Just click on the package in the _Package Registry_: ![](usingpackage.png) The first thing a new user needs to do is to set up GitLab in their local environment as new package registry. After you have configured your _pip_ command to access GitLab's package registry instead of the "official" PyPi repository, you can use the "Pip command" shown in the page to install the package. Don't forget that you also need an access token here. # GitLab Runners for CI/CD ## Introduction If a GitLab repository contains a _continuous integration pipeline_, its jobs will be executed via a _GitLab Runner_. A GitLab Runner is a daemon running on another server, waiting to be contacted by the central GitLab server to execute CI pipelines. __You can find more detailed information about GitLab Runners in the [GitLab Documentation](https://docs.gitlab.com/runner/).__ There are two different ways of using a GitLab runner: * individual runners, installed on local machines, remote clusters, or cloud-based systems by individual users (without support from MPCDF) * shared runners, which are offered by MPCDF If you don't want to install and configure an individual GitLab runner, you can execute your CI pipelines on the _shared runners_ offered by MPCDF. If you don't add one or more _tags_ to your CI file _.gitlab-ci.yml_, your pipeline will automatically be executed by a shared runner. ## Runner Tags Sometimes, your pipeline needs to be executed on a runner with specific capabilities. Via _tags_ you can specify which runner should be used to execute your CI pipeline: ``` default: tags: - mpcdf-shared ``` __Valid tags:__ **mpcdf-shared:** All of our managed shared runners do have this tag. Use this tag if you want to make sure the job runs on one of the MPCDF shared GitLab runners instead of runners of other GitLab instances or user-started runners. **image-builder:** Use this tag when you need to build your own custom Docker image (read below "Build a docker image"). #### Hardware-specific tags They are organized hierarchically based on their level of specificity **gpu:** Use when a runner with a GPU is needed, but the vendor or architecture remains unspecified. ![Hierarchy of tags for GPU runner usage]( ../../../bnb/220/gpu.diagram.drawio.svg) **gpu-amd:** Use when an AMD GPU runner is needed, regardless of architecture or instruction set. **gpu-amd-gfx90a:** Use when an AMD runner with offload architecture gfx90a (e.g. for the MI200 GPU) is required. **gpu-nvidia:** Use when an Nvidia GPU runner is needed, regardless of architecture or instruction set. **gpu-nvidia-cc80:** Use when an Nvidia runner with Compute Capability 8.0 (e.g. for the A40 or A100 GPU) is required. If no tags are specified in the pipeline, the job is going to be picked up by any of the docker or podman runners. ## Docker images for CI with MPCDF environment modules To provide the developers of HPC applications with a familiar and comprehensive software environment also within GitLab-based continuous integration (CI) pipelines, the MPCDF is offering special Docker images. These images use environment modules to make software accessible, in a very similar way to how [software is managed on the HPC systems](../../computing/software/environment-modules.md). Hence, e.g. build scripts will work on both the HPC systems and the CI cloud runners in a consistent way. ### Images and tags The new software infrastructure is composed of various Docker images, each of which provides a software stack based on a _single_ combination of a compiler (and potentially MPI) variant. For the user the access to the software is enabled via environment modules. Currently, the images are based on *openSUSE Leap 15.5* which is largely compatible with the SLES 15 operating system used on many HPC clusters at MPCDF. Up-to-date lists of the images together with lists of the software contained are [documented in GitLab](https://mpcdf.pages.mpcdf.de/ci-module-image/). Please note that you need to [opt-in and login to GitLab](gitlab.md) before you can access this page. As indicated by its tag, each image only contains a single toolchain, namely a single compiler with optionally a single MPI library plus a selection of widely used additional libraries. The list of software may be extended upon request. Arbitrary further software from the official OpenSUSE repos may be installed by the users individually by deriving from the MPCDF images, if necessary. ### Image tagging-and-purging strategy #### Tagging using `latest` and the calendar year To limit the individual growth of these Docker images over time, we put the following tagging-and-purging strategy in place: Essentially, all images are tagged using `latest` and/or the calendar year. In the course of a year, say 2024, the images tagged with `latest` and the year (`2024`) are identical and receive regular updates and additions of software. With the beginning of the new year, all images tagged with the previous year stay unchanged (frozen). The newly created images for 2025, say, will start out in early January again in a slim state and will be tagged `latest`. Users can then choose to migrate to the more recent images (tagged `2025` and `latest` in our example) or stick with the older (but static!) images (tagged `2024`) for a while. In case a user opts for using the tag `latest`, please be warned that the software environment will likely change at the beginning of each year. #### Non versioned images tagged `latest` Moreover, we provide special images without an explicit version number pointing to the respective most recent compiler and MPI in the MPCDF software stack. For the compilers, for example, we offer the images `gcc:latest` and `intel:latest`, similarly for depending images containing Intel- or OpenMPI. See the page on the [`latest` image tag](https://mpcdf.pages.mpcdf.de/ci-module-image/latest.html) for an overview. A typical use case for these images is a user's code which should be built and tested with the newest available compiler. Only the tag `latest` exist for these non-versioned images. In order to seamlessly receive updates, the `module load` command should also omit the compiler's and MPI's version number in this case (for example, using `module load gcc` instead of `module load gcc/13`). ### Case Study: Set up a CI job using a recent Intel C++ compiler This section shows how to set up a CI job to compile and test some C++ software using the Intel compiler. The required steps are: * Identify the Docker image that provides the required software by checking the [CI Docker image website](https://mpcdf.pages.mpcdf.de/ci-module-image/). Copy the image tag, in our example we're using `gitlab-registry.mpcdf.mpg.de/mpcdf/ci-module-image/intel_2023_1_0_x:2024`. * Edit your `.gitlab-ci.yml` file and paste the image tag after `image:`. * Select proper tags for the shared runners you're intending to use. * The resulting `.gitlab-ci.yml` file might look as follows: ```yaml # .gitlab-ci.yml build_intel: image: gitlab-registry.mpcdf.mpg.de/mpcdf/ci-module-image/intel_2023_1_0_x:2024 tags: - mpcdf-shared script: - module load intel/2023.1.0.x - module load gcc/13 - module load cmake - icpx --version # compile C++ code and perform tests ... ``` ### Case Study: Interactive debugging of a CI job that uses an MPCDF CI module image Advanced users might be interested in having interactive access to the identical software environment that is used by their CI jobs, e.g. to speed up debugging. While it is not possible to get interactive access to the actual CI runners, users can create a local Apptainer-based copy of an MPCDF CI module image on a login node of an HPC cluster and do the debugging there. The necessary steps are outlined in the following. Before you can interact with our container registry at all, you will need to generate an [access token][access-token] with at least the scope `read_registry`. [access-token]: https://gitlab.mpcdf.mpg.de/-/user_settings/personal_access_tokens?scopes=read_registry Then you can use Apptainer to authenticate with the registry (needs to be done only once per cluster) and build a local copy of the ci-module-image of your choice, e.g. `gcc_15:2026`: ```console $ module load apptainer/1.4.3 $ apptainer registry login --username "$USER" docker://gitlab-registry.mpcdf.mpg.de Password / Token: $ apptainer build gcc_15.sif docker://gitlab-registry.mpcdf.mpg.de/mpcdf/ci-module-image/gcc_15:2026 INFO: Starting build... ``` Building the image will take a while. You can safely ignore the messages about "harmless EPERM on setxattr" that will be printed. After that you can get an interactive shell in the container using ```bash apptainer shell gcc_15.sif ``` By default this mounts devices and a number of host directories in the container. Importantly, environment variables are also forwarded into the container. This can lead to adverse interactions with the module system inside the container. For a clean environment use the `-e`/`--cleanenv` or `-c`/`--contain`. If you want a completely isolated environment, use the `-C`/`--containall` flag. See the Apptainer help screen for more details. ```bash # Increasing level of isolation apptainer shell --cleanenv gcc_15.sif apptainer shell --contain gcc_15.sif apptainer shell --containall gcc_15.sif ``` Before you can run the steps of the job interactively inside the container environment you need to make the environment modules available. For that you have to source the module initialization file which is normally stored in the environment variable "$BASH_ENV". ```console $ echo "$BASH_ENV" /mpcdf/soft/SLE_15/packages/x86_64/Modules/current/etc/profile.d/modules.sh $ . "$BASH_ENV" $ module load gcc ``` Note that the CI module images are not designed to be used in batch jobs, use the native environment modules instead. Moreover, as the CI module images are updated regularly, delete the Apptainer-based local copy after the debugging is finished to avoid using an outdated software environment. ### Migrating from the legacy `module-image` to the new CI module images Users of the previous `module-image` are encouraged to migrate to the new CI images now, report potential issues and request additional software modules via the helpdesk, if necessary. As shown in the previous section, the `module-image` can simply be replaced in the user's `.gitlab-ci.yml` file with one of the new images that provides the desired software stack for the respective CI job. The new CI module images were first introduced in the [December 2023 edition of the Bits and Bytes](../../../bnb/214.html#module-software-stacks-for-continuous-integration-pipelines-on-mpcdf-gitlab-shared-cloud-runners). ## Build a docker image With the addition of Podman runners, it is now possible to build your own docker image on a shared runner. Here's a CI pipeline template to build an image on a Podman runner: ```yaml build_image: tags: - image-builder variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG BUILDAH_FORMAT: docker BUILDAH_ISOLATION: chroot image: quay.io/buildah/stable before_script: - buildah login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY script: - buildah build -t $IMAGE_TAG . - buildah push $IMAGE_TAG ``` **Please, do not use "kaniko" as it has been archived and is no longer maintained.** ## Shared runners offered on MPCDF GitLab The following paragraphs describe the shared runners which are currently available on MPCDF GitLab. Some parameters are valid for all of them: * For security reasons, it is _not_ possible to execute shell scripts directly on the runners * The default Docker image is _"python:3.12"_ * Every runner has 4vCPUs and 32GB of RAM (except NVIDIA GPU runners -- see below) * All vCPUs have avx, avx2, avx512 flags * All runners have distributed cache configured ### Docker runners - MPCDF Cloud Runner 01 - MPCDF Cloud Runner 02 - MPCDF Cloud Runner 04 - MPCDF Cloud Runner 05 - MPCDF Cloud Runner 06 - MPCDF Cloud Runner 08 - MPCDF Cloud Runner 09 Tags: __docker, shared__ ### Podman Runners - MPCDF Podman Runner 01 - MPCDF Podman Runner 02 Tags: __podman, shared__ ### MPCDF GPU Runners #### Nvidia - MPCDF-GPU-01 - MPCDF-GPU-02 - MPCDF-GPU-03 - MPCDF-GPU-04 Resources: * 4 vCPUs, 16 GB RAM each Tags: __cloud-gpu, nvidia-cc80__ Each of these 4 GPU runners has access to a MIG partition which exposes about 50% of one Nvidia A30 GPU. Executing the command _nvidia-smi_ in a CI pipeline describes the available GPU resource: ``` (...) +-----------------------------------------------------------------------------+ | NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A30 On | 00000000:00:05.0 Off | On | | N/A 36C P0 60W / 165W | N/A | N/A Default | | | | Enabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | MIG devices: | +------------------+----------------------+-----------+-----------------------+ | GPU GI CI MIG | Memory-Usage | Vol| Shared | | ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG| | | | ECC| | |==================+======================+===========+=======================| | 0 2 0 0 | 13MiB / 11968MiB | 28 0 | 2 0 2 0 0 | | | 0MiB / 16383MiB | | | +------------------+----------------------+-----------+-----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` #### AMD - MPCDF-GPU-05 Resources: - 8 vCPUs 32GB RAM Tags: __amd-mi200__ Gitlab runner with an AMD Instinct MI210 GPU. This series doesn't allow a logical partition as Nvidia. The _rocm-smi_ command allows you to describe the GPU resources: ``` rocm-smi --showmeminfo vram ============================ ROCm System Management Interface ============================ ================================== Memory Usage (Bytes) ================================== GPU[0] : VRAM Total Memory (B): 68702699520 GPU[0] : VRAM Total Used Memory (B): 11001856 ========================================================================================== ================================== End of ROCm SMI Log =================================== ``` ---------------------------------------- Data Publication and Metadata Management ---------------------------------------- .. image:: /_images/publication01.png :width: 200px Due to the heterogenous landscape of storage systems, different sizes of datasets and permissions the publication of research data is a complex task. The MPCDF supports its users by providing tools for metadata management as well as the hosting of discipline specific data repositories. .. toctree:: :maxdepth: 0 :glob: datapublishing.md.txt mmd.rst.txt mmd-dev.rst.txt metastore-documentation.md.txt # Service: Data Repositories For data publishing, the MPCDF recommends a [CKAN](https://ckan.org/)-based data repository. CKAN is a software framework which allows to manage metadata as well as object data. Beside a web-based interface, CKAN offers a REST API for automation of common workflows. CKAN instances at the MPCDF are meant for Max Planck Institutes, groups or projects and __not__ for individual users. If you are interested in running a CKAN instance at the MPCDF, please contact us (). ## Support While establishing a CKAN instance, the MPCDF will support you in the following ways: * Generic CKAN support * First installation on an Ubuntu based VM * Support for the following plugins: * Hierarchical structures in CKAN organisations * Creation of individual metadata schemata * DOI support via the MPDL's DOI server * Maintenance and update of the basic installation ## Data Storage Integration While CKAN will store your metadata, the underlying object data does not need to be saved directly in CKAN. The graphic below gives an overview about how the heterogeneous storage landscape of MPCDF can be integrated or referenced via CKAN. ![Storage systmes](ckan-storage-systems.png) The MPCDF Metadata Tools: User Documentation ============================================ .. image:: /_images/mmd01.png The MMD Tools (short for *MPCDF Metadata Tools*) can be used to create and manage metadata in several common metadata schemata. Introduction ------------ The mmd tools consists of four callable scripts and one module which provides additional functionality. The tools should be available on MPCDF systems but could also installed easily on most other Linux systems. Installing ---------- The mmd tools should run on every system with a reasonably modern version of Python 3. We recommend using the `Anaconda Python Distribution `_ which is also available in MPCDF's module system on the HPC clusters. The following procedure installs the mmd tools into a new *virtual environment* utilizing Anaconda Python on one of the HPC clusters. At first, load the Anaconda Python Distribution via the module system: .. code-block:: sh module load anaconda/3/2021.11 You can check the availability of Anaconda via: .. code-block:: sh find-module anaconda Now create a new Python virtual environment for the mmd tools (you can give it another name of course): .. code-block:: sh conda create --name mmd Activate the new created environment: .. code-block:: sh conda activate mmd The prompt of your shell should now change the name of the new created environment. Next, clone the GitLab repository: .. code-block:: sh git clone git@gitlab.mpcdf.mpg.de:mmd/mmd-tools.git Change to the new created directory and install the mmd tools together with all necessary Python libraries: .. code-block:: sh pip install . The mmd tools are now ready to be used and should be accessible via shell completion. Try it via entering "mmd" and press the tab key - it should list all available mmd tools: .. code-block:: sh (mmdtest) thomz@cobra02:~> mmd mmd mmd2bagit mmdCreate mmdListBags mmdLoad mmdPublish mmdShow Let's take a look at the individual tools of the mmd suite. mmdCreate --------- The *mmdCreate* script can be used to create and edit metadata manually. As parameter, it needs an output file (parameter *o*) and a metadata format specification. These specifications can be found in the subfolder "formats" of the cloned repository. Please specify the path to the format definition as absolute or relative path: .. code-block:: sh mmdCreate.py -o ~/metadata.mmd --format /data/mmd/formats/dublinCore.json With the command above, you can create a metadata file in the well known DublinCore format [#]_. The script will guide you step by step through the necessary fields: .. code-block:: sh Outputfile: /tmp/metadata.mmd Metadata format: /data/mmd/formats/dublinCore.json Fill in each field. Type "?" for a description of the field. After the script guided you through the process of entering the metadata, you can find the result in the output file specified via the "-o" option. The file is in JSON format and can be displayed or further processed by the common JSON tools or libraries. mmdShow -------- Once you have created a metadata file via the *mmdCreate* script, you can display its content via the mmdShow script. The parameter "-i" takes the input file: .. code-block:: sh python3 mmdShow.py --i /tmp/metadata.mmd Optional, the parameter *--outputformat* can be set to "html" so that the printed output will be formated as simple HTML code. mmd2bagit --------- With the mmd2bagit script, you can combine a folder with your data files and its metatada description in mmd format into a BagIt container [#]_: .. code-block:: sh python3 mmd2bagit.py --folder ~/testdata/ --metadata /tmp/metadata.mmd .. important:: Please be aware that the script changes the structure of the input folder! All content of the folder will move to the "data" subfolder while on the top level, you can find some additional files which were created by the script! mmdPublish ---------- The mmdPublish script can be used to publish a metadata file into a CKAN instance. Before you can use the script, you need to create an access token in CKAN and store it into an environment variable: .. code-block:: sh export CKAN_API_KEY=YOUR_CKAN_ACCESS_TOKEN .. important:: Without this access token, the script can not write into the CKAN instance. Please make sure that the environment variable can not be read by unauthorized people! The script itself needs several parameters: - i: the input file in mmd metadata format - c: the URL of the CKAN instance, followed by the path to its API. For example: https://ckanexample.com/api/3/action/ - t: the field in the metadata corresponding to the "Title" field in CKAN (*not* the title itself!) - o: the CKAN organisation under which the dataset should be stored .. note:: TODO: Screenshots of the whole workflow! The Metadata Formats -------------------- The basic idea of the MMD tools is to be as flexible as possible when it comes to the creation and management of metadata. Therefore, the MMD Tools are working *schemaless* with plain pairs of keys and values. Additional, some common metadata formats and schemata from our users are supported. .. important:: If you need support of further metadata schemata, please contant the developers via support@mpcdf.mpg.de The integrated metadata formats are stored in a JSON based format and can be found in the subfolder "formats" of the GitLab repository. So far, the following schemata are included: * DublinCore simple * DataCite Metadata Format * MPCDF default metadata schema .. [#] https://www.dublincore.org/ .. [#] https://en.wikipedia.org/wiki/BagIt The MPCDF Metadata Tools: Developer Documentation ================================================= The MMD tools are hosted in a GitLab repository: https://gitlab.mpcdf.mpg.de/mmd/mmd-tools If you want to participate in the further development of the tool chain, please contact the MPCDF via support@mpcdf.mpg.de # MetaStore User Documentation ## Introduction MetaStore is the catch-all data publishing platform of the MPCDF. It is meant as a place to create and publish metadata, describing datasets stored in the various storage systems at MPCDF. Beside this use case, small datasets (<1GB) can be uploaded directly into MetaStore. Digital Object Identifiers (DOI) can be added to the describing metadata. You can attribute to each dataset a DOI via DataCite. The page of your dataset on MetaStore will be then used as a landing page for the DOI. You can also create a DOI for your dataset later in time, and not necessarily directly at the upload of the dataset. You must just be careful that, once a DOI is created, you cannot delete it anymore. Each dataset can contain at least one or more resources. A resource can be either a URL or an uploaded file (smaller than 1GB). ## Who can use it If your institute has an account on MetaStore, you can use it with this account. Know that in doing so, you are bound to the agreement with MPCDF about data quality. MetaStore is not meant as a platform for the individual researcher. Instead, access and permissions are granted to Max Planck institutes, groups, departments or big research projects. As MetaStore supports nested organisations, it is possible to have groups or departments below a Max Planck Institute, but the top level always has to be an MPI. If you are interested in using MetaStore, please name one or two administrators from your institute who will be in charge of managing your data publications. You can contact us via [support@mpcdf.mpg.de](mailto:support@mpcdf.mpg.de) ## Creating a dataset There are several ways to create, update and delete datasets on MetaStore. - [Web UI](./metastore/docs/interacting_with_ckan_webui.md) - [API, via scripts or CLI Tools](./metastore/docs/interacting_with_ckan_api.md) ### Supported dataset schemas As the main use case of MetaStore is the assignment of DOIs, the default metadata set is DataCite's metadata schema (in version 4.4). To keep the creation of metadata as easy and simple as possible, the default method of creating a metadata set in MetaStore offers the necessary key value fields as simple full text fields. In contrast, the extended metadata schema supports the full functionality of DataCite's metadata schema, including the controlled vocabulary. There are two metadata schemas allowed on MetaStore, all from DataCite. - [The default metadata schema](./metastore/docs/datacite-standard-format.md) - [The extended metadata schema](./metastore/docs/datacite-extended-format.md) ### Linking Data versus Uploading Data The main use case for MetaStore is to create metadata and assign DOIs to _existing_ datasets at the MPCDF. Without moving (big) datasets around, MetaStore provides a _landing page_ and a DOI for this kind of data. It doesn't matter where exactly the data is stored or how the data is accessible - MetaStore fulfills the "F" in FAIR which stands for "findable". In addition, supplementary data like papers or visualizations can be uploaded to MetaStore directly. This goes also for small datasets: MetaStore allows the upload and publishing of files up to 1 GB directly via its web interface. For bigger datasets, it is recommended to store them on other storage at MPCDF like the Nexus S3 Storage Solution. ### Digital Object Identifier (DOI) Technically spoken, a Digital Object Identifier (DOI) is a link, pointing to a landing page or directly to a piece of data which is available via the World Wide Web. It is a so called _persistent identifier_ (PID), which still should be available when the data behind it is no longer available. This property makes a DOI the perfect identifier for big datasets stored at MPCDF: when the data moves to another place, for example from Nexus S3 to the HPSS tape library, the DOI will still stay the same and references in research articles or other kinds of documents will still point to the data's landing page, which is the record in MetaStore. .. image:: /_images/backup01.png :width: 250px ------------------- Backup and Archive ------------------- A **backup** is a copy of data that you have, intended for disaster recovery: if you lose your data because of a hardware or software failure or a user mistake, you can recover the lost data from the backup. The backup is typically done automatically and periodically so that you always have a current copy of your data. In addition, a good backup system will let you recover not only the last version of your files, but also older versions. An **archive** is a collection of data that you want to store somewhere other than your local disk because you currently don’t need the data anymore, but you might need it again in the future. Or you might have a legal or contractual requirement to keep certain data for very long periods of time, even if the data belongs to projects which are already finished. An archive is usually done by hand by the user, who must decide what data to archive. For safety, a good archive system will automatically store at least two copies of the archived data. .. toctree:: :maxdepth: 2 :glob: backup-archive-system.md.txt archives.md.txt * BA_HPC/index.rst.txt BA_Linux_clusters/index.rst.txt BA_AFS/index.rst.txt BA_desktops/index.rst.txt ![banner](Images/banner7.jpg) # Backup & Archive Systems The MPCDF uses [Spectrum Protect](https://www.ibm.com/products/data-protection-and-recovery) (formerly known as **TSM**) for backups and [HPSS](https://hpss-collaboration.org) for archives. ### Backup & Archive for users of local MPCDF systems If you are a user of a local MPCDF system, you can find information about backups and archives below, depending on the system you use: - [All systems: general information about archiving at MPCDF](archives.md) - [High-Performance Computer ('viper' & 'raven')](BA_HPC/BA_HPC.md) - [Linux clusters](BA_Linux_clusters/BA_Linux_clusters.md) hosted by MPCDF - [AFS](BA_AFS/BA_AFS_backups.md) - [Desktop PCs and laptops](BA_desktops/index) (Windows, Linux, Mac) at IPP and at MPCDF, including: [checking the backups of a Windows PC](https://selfservice.mpcdf.mpg.de/index.php?r=site%2Ftsm) (requires login) ### Backup & Archive for all Max Planck Institutes The MPCDF offers backup and archive services to **all Max Planck Institutes**. The IT administrators can [find here further information](BA_MPG/backup-archive-for-any-max-planck-institute.md). # How to archive data ## Overview The MPCDF has installed a migrating filesystem on the archive-server called "archive". Data written to this filesystem will automatically be moved from disk to tape to free space on disk when necessary or back from tape to disk when needed again by the user. This service is open to all users of MPCDF. ## Accessing the archive-server You may log onto the server with your <userid> and kerberos-password using ssh: ```sh ssh @archive.mpcdf.mpg.de ``` The ssh key fingerprints are: *0M7PnQy8+R9baOQM3zrpykQJrby0eqIKGkfbm2XBXj8 (RSA)* *m4SvenGFe4J45oOfCDPjfBXtxpgpO8GEkyVnuXGVs5Q (ED25519)* Your HOME directory is located under **/ghi/r/<initial>/<userid>** There is also a symbolic link **/r** pointing to **/ghi/r**, so in practice a user with ID **smith** would work with **/r/s/smith** (or **/ghi/r/s/smith**) All data within users' HOME directories will automatically be archived to tape. For further information see the section "[Information about the operation and usage of this filesystem](#Informations)" below. ## Archiving project directories In addition to the users' HOME directories described above, there are also project-specific directories which automatically archive files on tape. The project main directories are available under **/r2ghi/proj** and there is also a symbolic link **/p** pointing to it. In case you have a new project which wants to use the archive server, please contact [MPCDF Support](mailto:support@mpcdf.mpg.de?subject=New%20project%20for%20archive%20service). ## Information about the operation and usage of this filesystem ### Additional basic information for any user - **Automatic archival to tape**: The system regularly (usually every hour) copies all new files to tape. The copy on disk remains as long as there is enough space. When the filesystem gets full above a certain value, some files which have already been copied to tape will be wiped from disk, beginning with the largest files which have been unused the longest time. - **Automatic retrieval from tape**: If (by using some program or command) you access a file which has been migrated to tape, the file will automatically be transferred back from tape to disk. This of course implies a certain **delay**. The command will **appear to hang**, but it will just wait until the data is online and then continue. - **Redundancy**: Every file being migrated gets simultaneously written to **two** **different tapes**. In this way, in case of a tape failure while reading back the data from the first tape, the file can probably still be read from the second tape. - **Optimal file size for efficiency**: The system can only migrate files which are bigger than the disk block size, which for this filesystem is 1 MB (one megabyte). Files smaller than 1MB stay resident on disk, permanently occupying disk space and, what's worse, making the total number of files grow so large that operations like scanning the filesystem for making backups become increasingly slow. In addition: while files larger than 1 MB can be migrated, **the** **system works efficiently only for file sizes larger than about 1 GB** **(one gigabyte)**. The reason is that reading or writing data to/from tape implies waiting for a tape drive to become available, then waiting for a tape to get mounted in the drive and then waiting for the tape to get rewinded/positioned. This can typically take several minutes. Once a tape is available and in position, the system can read or write data very fast. A 1 GB file can be read in under 10 seconds. Contrast this with reading 1 GB of data spread across 1000 files, each 1 MB in size, which would need at the very least 1000 tape-positioning operations, maybe also mounting several tapes (possibly hundreds!). For these reasons, **all users are kindly asked to keep the size of** **files stored on '/r' and '/p' filesystems within a range of about** **1 GB (one gigabyte) to about 1 TB (one terabyte)**. - **Maximum file size**: The above recommendation (1 GB to 1 TB per file) is not a strict limit. A small quantity of files not within that range is still ok. **BUT**: files larger than 20 terabytes will not be migrated to tape. They will stay on disk and, in the event of a disk crash, they will be **lost**. Do not store files larger than 20 terabytes. - **Disk quotas** limiting the number of files stored are enabled on '/r' and '/p' filesystems. On the '/r' filesystem, a user quota is enabled. Usually, it is 100.000 files. You can check your user quota with /usr/lpp/mmfs/bin/mmlsquota command: ```sh archive:~ $ /usr/lpp/mmfs/bin/mmlsquota -u hpss_ghi_r1ghi Block Limits | File Limits Filesystem Fileset type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks hpss_ghi root USR 384 0 0 0 none | 21 100000 120000 0 none r1ghi.rzg.mpg.de hpss_ghi MPIN USR no limits r1ghi.rzg.mpg.de where "hpss_ghi_r1ghi" is the name of '/r' filesystem. It can be obtained with the command 'df -h'. On the '/p' filesystem, the quota for the project is enabled. You can check the project quota with **/usr/lpp/mmfs/bin/mmlsquota** command: ```sh archive:~ $ /usr/lpp/mmfs/bin/mmlsquota -j hpss_ghi_r2ghi Block Limits | File Limits Filesystem type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks hpss_r2ghi FILESET 0 0 0 0 none | 1 100000 101000 0 none r2ghi.rzg.mpg.de ``` where "hpss_ghi_r2ghi" is the name of /p filesystem. It can be obtained with the command 'df -h'. is the name of your project. For example, if your project directory is /p/NAME, then your project name is "NAME". But if your project directory is /p/NAME/SUB, then your project name is "NAME_SUB". When your quota is exceeded, you get this message when trying to create a file or a directory: ```sh mkdir: cannot create directory ‘mydir’: Disk quota exceeded""" ``` If you have a lot of small files, please, pack them in a **tar**, **zip** or similar archive. The size of this archive should be bigger than 1 GB and smaller than 1 TB. If all your files are bigger than 1 GB and you still exceed the quota, please send us an email to [MPCDF Support](mailto:support@mpcdf.mpg.de?subject=Archive%20disk%20quota%20exceeded). We can then increase your quota. ### Additional information for users directly logging onto the server using ssh - You can manually force the recall of a migrated file by using any command which opens the file. You can recall in advance all files needed by some job with a command like ```sh file myfiles/* ``` or you can use **ghi_stage** command for that: ```sh ghi_stage myfiles/* ``` - You can see which files are resident on disk and which ones have been migrated to tape with the command **ghi\_ls** (located in /usr/local/bin), optionally with the option **-l**. Here is a sample output: ```sh archive% ghi_ls -l G -rw-r--r-- 1 ifw rzs 22 Nov 21 15:12 a1 H -rw------- 1 ifw rzs 138958551040 Sep 18 22:22 abc.tar H -rw-r--r-- 1 ifw rzs 1073741312 May 06 2009 core G -rw-r--r-- 1 ifw rzs 0 Jun 20 2008 dsmerror.log B -rw-r--r-- 1 ifw rzs 1079040000 Aug 03 2010 dummyz3 ``` The first column states where the file resides: a 'G' means the file is *resident on the **G**PFS disk*; a 'H' means the file has been transferred to the underlying **H**PSS archiving system, probably on tape; a 'B' means '**b**oth': the file has already been copied to HPSS but is still present on disk and can be removed immediately if the system needs to free disk space. - If you have many small files, please pack them first together to a large file with a suitable tool like **tar**, **cpio**, **ar**, **zip** or whatever. Please try to keep the size of files stored on the '/r' and '/p' filesystem within a **range of about 1 GB** **(one gigabyte) to about 1 TB (one terabyte)**. Here is a simple example of how to use **tar** to pack some small files **small000**, **small001**, etc to a big file **big.tar**: ```sh tar cvf big.tar small* ``` Additionally, if you want, you can write a "Table of Contents" file with a command like ```sh tar tvf big.tar > big.tar.toc ``` Files with a '.toc' extension will stay on-line, provided they are smaller than 1 MB, so you can read them any time without having to wait for a tape to be mounted. Likewise, files with a '.md5' extension also stay on-line if they are smaller than 1 MB. - Amount of stored data. If you need to compute the total amount of data that you have stored in the archive, use a command like this: ``` find . -type f -printf "%s %p\n" | awk '{sum+=$1} END {print sum}' ``` You can easily adapt this to compute the amount of data under a certain subdirectory only, or 'grep' for files of a certain type only, etc. - Please pay attention when working with **sparse files**. Sparse files are files which contain stripes of zeros and these zeros are not stored on disk. Therefore the disk usage (obtained with **du -sh** command) for such files is smaller than their actual size (obtained with **ls -l** command). When packing these files in a **tar** archive, the disk usage of the resulting tar file will be bigger than disk usage of source files. **tar** by default writes all zeros explicitly on disk. You can use **-S (--sparse)** option for **tar**. In this case, **tar** handles sparse files efficiently. Disk usage of this tar archive is as small as disk usage of the source files. To extract the files, **-S** option is not needed. The extracted files will be sparse again. This is in contrast to using **zip** or another archiving tool with compression. When compressing sparse files with **zip**, the resulting archive is small but extracted files will not be sparse anymore. A **problem** can arise when using **tar -S** to create a tar archive from the **source files stored on '/r' or '/p' filesystems**. If your source files are migrated to tape and purged from disk (**ghi_ls -l** outputs **'H'**) then, **tar without -S** option will recall files from tape before creating a tar archive whereas **tar with -S** option will not. In this case, an empty tar archive is created and no error message is shown! To avoid data loss, always check that source files are on disk (**ghi_ls -l** outputs **'G'** or **'B'**). If source files are on tape only (**ghi_ls -l** outputs **'H'**), then recall these files first for example, using **ghi_stage** command. ------------------- Backup HPC ------------------- .. toctree:: :maxdepth: 1 :glob: BA_HPC.md.txt # For HPC Viper & Raven ## Backups No backups of user files are done on the High Performance Computers '**viper**' and '**raven**'. To protect your data, keep an additional copy of all important data on another system, e.g. in your AFS volume. ## Archives The recommended method for archiving data for long-term storage from the High Performance Computers '**viper**' and '**raven**' is to use the migrating filesystem **/r**. (For information about archiving data with TSM, please see the next section.) Data written to **/r** will automatically be moved from disk to tape to free space on disk when necessary or back from tape to disk when needed again by the users. Please try to keep the size of files stored on the **/r** filesystem within a range of **about 1 GB to about 1 TB** (one gigabyte to one terabyte). [More details about archiving data and reading data back.](../archives.md) ## TSM archives Archiving data using TSM will not be supported in the near future. Please use the migrating filesystem **/r** as described above. If you have already archived data with TSM in the past, it will be automatically transferred to **/r**. You will be informed by e-mail when the transfer of your archives gets started and when it finishes. --------------------- Backup Linux Clusters --------------------- .. toctree:: :maxdepth: 1 :glob: BA_Linux_clusters.md.txt # For Linux clusters ## Backups Some Max Planck Institutes operate a Linux cluster hosted by MPCDF. Automatic backups of such clusters are possible, but are set up **only on request**. Members of the IT staff of a Max Planck Institute wishing to have backups of their cluster should contact MPCDF to discuss the details. ## Archives To archive data from a Linux cluster, use **scp**, **sftp** or **rsync** to connect to the machine **archive.mpcdf.mpg.de** and log in with your Kerberos user id and password. Your HOME directory will be **/ghi/r/<initial>/<userid>** (for example: **/ghi/r/s/smith**). Everything you store there will be automatically copied to tape. And everything you read from there will be automatically retrieved from tape, which will imply some delay of typically between 1 and 10 minutes. [More details about archiving data and reading data back.](../archives.md) ------------------- Backup AFS ------------------- .. toctree:: :maxdepth: 1 :glob: BA_AFS_backups.md.txt restore-afs-files-from-the-tsm-backup.md.txt ```{eval-rst} .. warning:: The AFS-Cell 'ipp-garching.mpg.de' will turn **read-only** in November 2025, and will be finally **decommissioned** in November 2026. ``` # For AFS ## TSM backups of AFS volumes ### Backups of AFS volumes - **Which volumes**: all AFS volumes are examined every night (with 'vos examine') and, if they have been modified, a backup is done. - **Permissions**: for the backup to succeed, the data must be readable by the special user **afsbackup** (ACL: 'afsbackup rl'), so make sure not to delete that ACL if you care about backups. - **Retention period**: for files which are still present on the AFS volume, all versions of the last 60 days are kept in the backup. In addition, for a file which has been deleted from the AFS volume, the last version is kept in the backup for 2 years. - **Access**: in order to access the backups of your AFS volumes, you must log in to a Linux machine with AFS. Most IPP users can login to machine **toks01.bc.rzg.mpg.de**. - **Check last backup**: once you are logged in to a Linux machine with AFS and you have an AFS token, use this command to check when the last backup of your AFS Home Volume was done: **/afs/ipp/@sys/bin/adsm showlast** Or use this command to see the logfile of the last backup: **/afs/ipp/@sys/bin/adsm showlog** - **Restore**: to recover files from the backup, follow [these steps](restore-afs-files-from-the-tsm-backup.md). - **Foreign volumes**: if you need to recover files from a volume which is not your AFS Home volume, please contact the [helpdesk](../../../../faq/help.html#how-can-i-get-help-and-support). ```{eval-rst} .. warning:: The AFS-Cell 'ipp-garching.mpg.de' will turn **read-only** in November 2025, and will be finally **decommissioned** in November 2026. ``` # Restore AFS files from the TSM backup How to recover AFS files from the TSM backup ## With a graphical user interface 1. Log in to a Linux machine with AFS and get an AFS token. If you are an IPP user, you can log in to machine **toks01.bc.rzg.mpg.de**. Other Linux machines with AFS should also work. 2. Start **/afs/ipp/@sys/bin/adsm**. You have to make sure to have your X11 display correctly set. 3. Click on ***Restore*** on the upper right of the window. 4. If you want to recover one file or a few files which you just lost, proceed to **step 5**. 5. If you want to recover one file or a few files which you lost *before the last backup* (typically, before last night), then the file is probably still in the backup, but it is marked *inactive*. In this case, first you need to select from the top menu: ***View -> Display active/inactive files***. Then proceed to **step 5**. 6. Click on the **+** sign next to ***File level*** on the left hand side of the window, then open the folder and subfolder where the file(s) you lost were located. 7. Mark the files you want to recover by clicking on the grey square to the left of the file name. You can also mark a whole folder. 8. Click on the ***Restore*** button (on the top left). You **must** then specify the location where you want to restore the files to. (**Do not** just choose original location: it will not work because the original location refers to the temporary mount point of the volume at backup time, which you cannot access.) After that, click on ***OK*** and wait until the program finishes getting the files. ## On the command line 1. Log in to a Linux machine with AFS and get an AFS token. 2. Start **/afs/ipp/@sys/bin/adsmc** 3. Use TSM client commands to query and restore files, for example: ``` query filespace # shows your backup mountpoint query backup # path = / query backup -subdir=yes # list your files recursively restore # restore backup to specified location restore -pick # pick from previous file versions ``` 4. For more details, options and examples, see the IBM documentation - [for performing incremental backups](https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.8/client/c_bac_cmndline.html), - [for querying backups](https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.6/client/r_cmd_querybkup.html), - [for restoring backups](https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.8/client/c_res_cmdlineunx.html). ------------------- Backup Desktops ------------------- .. toctree:: :maxdepth: 1 :glob: BA_Windows_restore.md.txt BA_check_backup_status.md.txt BA_desktops_Garching.md.txt BA_desktops_Greifswald.md.txt BA_windows_exclude.md.txt linux_ba_client.md.txt # Recover Windows files from the TSM backup ### How to recover files from the TSM backup 1. Click on ***Start*** -> ***IBM Spectrum Protect*** -> ***Backup-Archive GUI*** and wait for a new window to appear. (for older TSM-clients the path applies: ***Start*** -> ***All programs*** -> ***Tivoli Storage Manager*** -> ***Backup-Archive Client***) 2. Click on ***Restore*** on the upper right of the window. 3. If you want to recover one file or a few files which you just lost, proceed to **step 5**. 4. If you want to recover one file or a few files which you lost *before the last backup* (typically, before last night), then the file is probably still in the backup, but it is marked *inactive*. In this case, first you need to select from the top menu: ***View -> Display active/inactive files***. Then proceed to **step 5**. 5. Click on the **+** sign next to ***File level*** on the left hand side of the window, then open the drive, folder and subfolder where the file(s) you lost were located. 6. Mark the files you want to recover by clicking on the grey square to the left of the file name. You can also mark a whole folder. 7. Click on the ***Restore*** button (on the top left). You can then choose whether to recover the files to the original location or to a different location. After that, click on ***OK*** and wait until the program finishes getting the files. # Check Backup Information ## Selfservice 1. Visit the [MPCDF SelfService](https://selfservice.mpcdf.mpg.de/) and log in to your account. 2. Select *TSM Backup* in the *My Account* dropdown menu. 3. Enter the name of your system or a matching string in the *PC Name* field. ![TSM Backup SelfService header](./tsm_selfservice1.png) Be aware that the website lists **only Windows** systems. ## Check Locally On a Linux or MacOS system you can simply run `dsmc query filespace` (or `dsmc q fi` for short) in the terminal to see the last backup date. Keep in mind you might need to be root to run `dsmc` depending on your client configuration. # For Desktops in Garching ## Backups and Archives for Desktops and Laptops in Garching ## Windows ### Backups - You can **[check the backups of a Windows PC](BA_check_backup_status.md)**. If the last backup is old or there is no backup at all, please inform the IT staff in your department. - Backups of Windows PCs at IPP Garching and MPCDF are done automatically **every day**. By default, **all local drives** get backed up, including any **USB drives** attached to the PC at backup time. You might notice that your PC reacts more slowly while the backup is running, but for most PCs it usually takes under 15 minutes to complete. - For **desktop PCs**, the backup is done in the **night**, so please leave the computer running every night or at least one night per week. - For **laptops**, the backup is done around **noon**. If your laptop is not connected often to the network, it might miss the backup window. You can start a backup by hand any time by starting the TSM program and choosing "Actions -> Backup domain" from the menu bar. - If you need to **recover files** from the backup, follow [these steps](BA_Windows_restore.md). - Some files are automatically **excluded** from the backup. Here is a [list of them](BA_windows_exclude.md). - For files which still exist on your PC, all versions of the **last 60 days** are kept in the backup, so you can recover any of them. In addition, for a file which has been deleted from your PC, the newest version is kept in the backup for 2 years. ### Archives To archive data from your Windows PC, use an **scp** or **sftp** client (for example [WinSCP](https://winscp.net), [FileZilla](https://filezilla-project.org/), [pscp](https://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html), [psftp](https://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html)) to connect to the machine **archive.rzg.mpg.de** and log in with your Kerberos user id and password. Your HOME folder will be **/ghi/r/<initial>/<userid>** (for example: **/ghi/r/s/smith**). Everything you store there will be automatically copied to tape. And everything you read from there will be automatically retrieved from tape, which will imply some delay of typically between 1 and 10 minutes. [More details about archiving data and reading it back.](../archives.md) ## Linux ### Backups Backups of Linux desktops and laptops at IPP Garching and MPCDF are configured only on request. If you have a Linux machine you want to include in the backup, please contact the [helpdesk](../../../../faq/help.html#how-can-i-get-help-and-support). ### Archives To archive data from your Linux machine, use **scp**, **sftp** or **rsync** to connect to the machine **archive.rzg.mpg.de** and log in with your Kerberos user id and password. Your HOME directory will be **/ghi/r/<initial>/<userid>** (for example: **/ghi/r/s/smith**). Everything you store there will be automatically copied to tape. And everything you read from there will be automatically retrieved from tape, which will imply some delay of typically between 1 and 10 minutes. [More details about archiving data and reading it back.](../archives.md) ## Mac ### Backups Backups of Macintosh machines at IPP Garching and MPCDF are configured only on request. If you have a Mac that you want to include in the backup, please contact the [helpdesk](../../../../faq/help.html#how-can-i-get-help-and-support). ### Archives If you have an **scp** or **sftp** client on your Mac, you can archive files by connecting to the machine **archive.rzg.mpg.de** and logging in with your Kerberos user id and password. Your HOME directory will be **/ghi/r/<initial>/<userid>** (for example: **/ghi/r/s/smith**). Everything you store there will be automatically copied to tape. And everything you read from there will be automatically retrieved from tape, which will imply some delay of typically between 1 and 10 minutes. [More details about archiving data and reading it back.](../archives.md) # For desktops in Greifswald Backups of desktop PCs and laptops (Windows, Linux, Mac) at IPP Greifswald are usually done with local infrastructure in Greifswald. Please contact the IT staff at IPP Greifswald for details. # Windows files excluded from backup All files and folders on any drive matching the following names are automatically excluded from the backup. Here **"..."** means "at any depth" in the folder tree: - \\Temp - \\Windows\\Temp - \\Recycled - \\Recycler - \\Winnt\\Temp - \\...\\nobackup - \\...Netscape\\...\\cache - \\...\\system32\\config # IBM Spectrum Protect for Linux Desktops Backups of Linux desktops and notebooks at IPP Garching and MPCDF are configured only on request. If you have a Linux machine you want to include in the backup, please contact the [helpdesk](../../../../faq/help.html#how-can-i-get-help-and-support). ## Client Setup To use our service to backup your system once a day please follow the steps below. ### Registration Please provide the helpdesk with the following information about your system: - Hostname - Contact (Name, Phone, Room) - Notebook or desktop PC - Operating System The 'nodename' will be your FQDN and a password will be set by the administrator. ### Installation IBM offers packages for `rpm` and `dpkg` based distributions. You will receive a download link from us. 1. Extract the files from the tar archive, the "8.1.x.x" represents the client version. ``` tar xfv 8.1.x.x-TIV-TSMBAC-LinuxX86.tar or tar xfv 8.1.x.x-TIV-TSMBAC-LinuxX86_DEB.tar ``` 2. Install the GSKit packages. ``` rpm -U gskcrypt64-8.x.x.x.linux.x86_64.rpm gskssl64-8.x.x.x.linux.x86_64 or dpkg -i gskcrypt64_8.x.x.x.linux.x86_64.deb gskssl64_8.x.x.x.linux.x86_64.deb ``` 3. Install the IBM Spectrum Protect API. ``` rpm -i TIVsm-API64.x86_64.rpm or dpkg -i tivsm-api64.amd64.deb ``` 4. Install the backup-archive client. ``` rpm -i TIVsm-BA.x86_64.rpm or dpkg -i tivsm-ba.amd64.deb ``` For further explanation please have a look at the [IBM documentation.](https://www.ibm.com/docs/en/spectrum-protect/8.1.11?topic=clients-installing-linux-x86-64-client) ### Configuration The BA client will be installed in `/opt/tivoli/tsm/client/ba/bin`. You can also find the configuration files in this directory. Important files are `dsm.opt` and `dsm.sys`. Those will be provided by the helpdesk. Once you have the right configuration please run `dsmc` in your terminal and login: - confirm the username (=nodename) - enter the password provided by the helpdesk After your client is configured start the scheduler service: ``` systemctl daemon-reload systemctl enable --now dsmcad.service ``` ## Basic Operations How to view and restore your backups. ### With a graphical user interface 1. Make sure you have `OpenJDK 7` installed. 2. Open your terminal and run `dsmj`. 3. Click on ***Restore*** on the upper right of the window. 4. If you want to recover one file or a few files which you just lost, proceed to **step 5**. 5. If you want to recover one file or a few files which you lost *before the last backup* (typically, before last night), then the file is probably still in the backup, but it is marked *inactive*. In this case, first you need to select from the top menu: ***View -> Display active/inactive files***. Then proceed to **step 5**. 6. Click on the **+** sign next to ***File level*** on the left hand side of the window, then open the folder and subfolder where the file(s) you lost were located. 7. Mark the files you want to recover by clicking on the grey square to the left of the file name. You can also mark a whole folder. 8. Click on the ***Restore*** button (on the top left). You **must** then specify the location where you want to restore the files to. After that, click on ***OK*** and wait until the program finishes getting the files. ### On the command line 1. Open your terminal 2. Start `dsmc` 3. Use TSM client commands to query and restore files, for example: ``` query filespace # shows your backup mountpoints query backup # path = / query backup -subdir=yes # list your files recursively restore # restore backup to specified location restore -pick # pick from previous file versions ``` 4. For more details, options and examples, see the IBM documentation [for performing incremental backups](https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.8/client/c_bac_cmndline.html) [for querying backups](https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.6/client/r_cmd_querybkup.html) [for restoring backups.](https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.8/client/c_res_cmdlineunx.html) .. warning:: The AFS-Cell 'ipp-garching.mpg.de' will turn **read-only** in November 2025, and will be finally **decommissioned** in November 2026. ---------------------------------------- Deprecated: The Andrew File System (AFS) ---------------------------------------- .. toctree:: :maxdepth: 1 :glob: introduction-to-afs.md.txt introduction/index.rst.txt specific-technical/index.rst.txt glossary/index.rst.txt troubleshooting/index.rst.txt ```{eval-rst} .. warning:: The AFS-Cell 'ipp-garching.mpg.de' will turn **read-only** in November 2025, and will be finally **decommissioned** in November 2026. ``` # Store (AFS) Main storage systems - AFS ## Introduction to AFS AFS is one of the main networked file-systems at MPCDF. AFS is a distributed, global file-system. Given proper internet access you are able to use it wherever you are: at Garching, at home or somewhere in the world, e.g. at a conference. Its most prominent feature for the user is the global namespace. That means, independent of the computer you are using, you can access your files on UNIX under /afs/ipp-garching.mpg.de/. In Windows, UNC-paths are used: \\\\AFS\\ipp-garching.mpg.de\\, some paths may be mapped to a network-drive on your desktop, such as Y: \\\\afs\\ipp-garching.mpg.de\\ and H: to your home-directory. Your data is being protected by Kerberos 5. That way you can access only data you are authorized to and also authorize other people to your data. Further details regarding AFS are available in the following subsections - [AFS Basics](introduction/index) - [Technical details](specific-technical/index) - [AFS at the IPP](specific-technical/afs-at-ipp.md) - [Glossary](glossary/index) - [Troubleshooting](troubleshooting/index) Read more on the technical details of AFS, or the IPP-specific points about it. .. warning:: **The AFS-Cell 'ipp-garching.mpg.de' will turn read-only in November 2025, and will be finally decommissioned in November 2026.** ------------------- Introduction ------------------- .. toctree:: :maxdepth: 1 :glob: understanding-afs.md.txt # AFS Basics ## Understanding AFS Some general information regarding AFS. ### Separation of data and metadata In AFS, the information where the data are stored is stored on different servers than the data itself. The servers storing the data are called fileservers. The servers storing the information are called volume-location-server (database-server). ### A client connects to AFS for the first time after a reboot When a client wants to find a file in an afs-cell it has not connected to before, it first asks the database-server which fileserver serves the volume it is asking for. Then it goes to that server and tries to retrieve the data. ### Required Firewall rules In order for AFS to work correctly, you must allow connections from outside on the UDP port 7001 coming from the UDP ports 7000-7012. ### Structure of the /afs - filesystem On the highest level, AFS is split into Cells. An AFS-cell is an administrative unit. The MPCDF administers the cell "ipp-garching.mpg.de" and e.g. CERN's cell is reachable under /afs/cern.ch/. Within one AFS-Cell, the filesystem itself is built out of volumes, which are connected by mountpoints. Volumes are comparable to disc-partitions with a few extensions: - A volume may be moved from one disc to another or even from one server to another. - A volume may have a read-only (RO) snapshot spread on different servers. Thus, /afs/ipp-garching.mpg.de is a mountpoint to the volume "root.cell" within the AFS-cell "ipp-garching.mpg.de" See a simple diagram (taken from a talk) about the structure ![afs structure](afs-structure.png) ### Security within AFS **Authentication** (who are you ?) To prove to AFS-Servers who you are you need to present an AFS-**token**, which can be derived from a Kerberos-**Ticket**. Read more about this [here.](../specific-technical/authentication-within-afs.md) **Authorisation** (what are you allowed to do ?) The access rights to a directory in AFS are controlled by so-called Access Control Lists (ACLs). Read more about them [here.](../specific-technical/authorisation-within-afs.md) ### Further documentation: Manpages are installed under /afs/ipp-garching.mpg.de/common/man/. When your MANPATH environment variable is containing that path (like it should, e.g. on the MPCDF login node), then you can use those for most AFS-Commands. Modern linux distributions also ship the man-pages with the client packages. Also consider the official documentation at [openafs.org](https://docs.openafs.org/index.html). .. warning:: **The AFS-Cell 'ipp-garching.mpg.de' will turn read-only in November 2025, and will be finally decommissioned in November 2026.** ------------------- Specific-Technical ------------------- .. toctree:: :maxdepth: 1 :glob: afs-at-ipp.md.txt authentication-within-afs.md.txt authorisation-within-afs.md.txt installing-an-afs-client-on-windows.md.txt specific-technical-information.md.txt statistic.md.txt # Specific & Technical ## AFS at IPP/RZG Some information about IPP specific details of the AFS. ## Home directories Every user registered at RZG has a home-volume, which is mounted as her/his home-directory. There are two places, where your home-volume is mounted : /afs/ipp-garching.mpg.de/u/*<username>* or /afs/ipp-garching.mpg.de/home/*<first letter of username>*/*<username>* Working in either of these directories is completely equivalent. ## Recovering just deleted/corrupted files (RO-Volumes) ### Home Volumes As briefly described in [understanding AFS](../introduction/understanding-afs.md), all Volumes at IPP have a READ-ONLY copy. This RO (read-only) copy of the Volumes is refreshed ("The RW-Volume is released" in AFS-speak) every day by a cron-job. The RO of your home-Volume (home-directory) is mounted /afs/ipp-garching.mpg.de/**.u**/*<username>*. For windows: \\\\AFS\\ipp-garching.mpg.de\\.u\\*<username>* Please note the dot before the "u". This gives you the possibility to browse through a snapshot of your home-directory which is up to 24hours old and recover data which just got missing. More information about recovering older files than this is given under Data -> Backup in the navigation column on the left. ### Project Volumes For the RO-Replics of project-Volumes, we have (currently) no fixed mountpoints. Thus, you need to create your own mountpoint directly to the RO-Volume in order to access it. For this, you need the "a" right on the volume. Example: Say, there are some files missing in /afs/ipp/common/soft/openafs. If you have this, you can do the following : 1. get the volume name, where the files have been : *fs listquota /afs/ipp/common/soft/openafs* gives you the volume name "**openafs**" 2. Mount the readonly Volume somewhere in AFS: cd /afs/ipp-garching.mpg.de/u/USERNAME fs mkmount openafs.RO openafs.readonly 3. **Done**! now in /afs/ipp-garching.mpg.de/u/USERNAME/openafs.RO you have access to the files of yesterday night. 4. After copying over the files, you can remove this mountpoint again : fs rmmount openafs.RO # Authentication within AFS How to get the proof of who you are. ## Unix In AFS you have a so-called "token" which proves to the fileserver who you are, which then grants access to restricted areas within AFS. To obtain a token, you first need to get a kerberos ticket via "kinit" and then use "aklog" to get an AFS-token. The kerberos ticket then may also be useful for other services (web, ssh) at RZG. Useful commands for dealing with Kerberos tickets are: - **kinit** : get a kerberos ticket *(man kinit)* - **klist** : shows present tickets in the credential-cache *(man klist)* - **kdestroy** : destroys an existing credential-cache (and all tickets in there) *(man kdestroy)* For converting a kerberos ticket into an AFS token use the command : - **aklog** : convert a Kerberos Ticket into an AFS-Token (man aklog) The preferred way of doing things is : kinit \# to get a Kerberos5 ticket aklog \# to create an AFS token out of it **NOTE:** Both Kerberos and AFS have "Authentication Containers", through which credentials are made available. This is necessary for multi-user machines, but also if you work alone on your machine it helps you to work with different identities at the same time. In Kerberos it is called "Credential Cache", AFS it is a "PAG" (process authentication group). For AFS, you might find following commands useful : - **unlog** : destroys AFS-Token (man unlog) - **pagsh** : open a shell in a new PAG. (man pagsh) ## Windows For Windows, you should click on the lock-symbol in your system-tray and then type your password there: The lock shows a red x if you have no AFS-token (you are not authorised against AFS at all) : ![](schloss-ohne-token.png "Schloss ohne Token") clicking on the lock leads you to a dialog where you should enter your password. After a successful authentication, it should look like this : ![](copy_of_Schlossmittoken.png "Schloss mit Token") # Authorisation within AFS How to authorize access to directories and files. Access control in AFS is done via ACLs (Access Control List). You can give different rights to multiple AFS-users and AFS-groups. In this respect the access control is more fine-grained than normal Unix-rights. However, access rights in AFS are based on directories. That means if you have access to one file in a directory, you have access to all of them. ACLs for new directories are inherited from the parent directory. The rights one can give/have are explained in the [AFS-Glossary](../glossary/afs-glossary.html#LetterA "AFS-Glossary"). ## UNIX ### Using access rights You can **list the access rights** for a directory in unix with the command *"fs listacl <path>"*. e.g. ``` # fs listacl /afs/ipp-garching.mpg.de/ Access list for /afs/ipp-garching.mpg.de/ is Normal rights: system:administrators rlidwka system:anyuser rl afsbackup rl ``` **!** You can get more detailed information on the man-page on the login-node rzgate *"man fs\_listacl"*. Just make sure the path *"/afs/ipp-garching.mpg.de/common/man"* is in your MANPATH-environment variable. Similarly, you can **set the access rights** for a directory in unix with the command *"fs setacl <path> (user|group) rights"*. The details are given again on the man-page : "man *fs\_listacl"*. Unfortunately, there is no recursive version of the fs setacl command. However, you can set ACLs recursively by using the (GNU!) *"find"* command : ```sh find -noleaf -type d -exec fs setacl "{}" (user|group) \; ``` **! It should be noted that access rights are inherited when creating a new directory.** **! Best Practices :** When setting up a shared directory for a project, you should really create a group. With groups it is much easier to give rights to new members of the project. Usually, 2 groups are better: *<Project>-readers and <Project>-Writers.* * * ### AFS-groups Each AFS-user can create up to 20 groups for her personal use, which is encouraged when sharing directories with more than 2 persons. The relevant commands are : - **pts creategroup** **:** create a new group *(man pts\_creategroup)* - **pts delete** **:** delete a group *(use with care!) (man pts\_delete)* - **pts listowned :** show your groups *(man pts\_listowned)* - **pts membership :** show the members of a group -- or -- the groups a user is member of. *(man pts\_membership)* - **pts removeuser :** remove a user from a group *(man pts\_removeuser)* In case you wonder, there are some global groups: - system:administrators : admins of this cell - system:authuser : people with a valid token for this AFS-cell. - system:anyuser : Anyone in the world with an AFS-client You should be careful when giving rights to any of those groups. ## Windows In windows you may see AFS-information and manipulate access rights through the Explorer-extension. For this, you just right-click on a directory in the explorer: ![](explorerextension.png "Explorer-Extension") ### You can then see AFS-information either through the AFS-submenu or the properties (here: " Eigenschaften") menu. In these submenus you can also modify the ACLs on an AFS-directory ### Powershell You can also use the powershell to change the ACLs via a script. The relevant commands are called "pts.exe" and "fs.exe". The usage is as described above. If you want to change a directory-hierarchy recursively, in the powershell do a : ```sh gci -Recurse -Directory | Foreach { fs.exe setacl "$_.fullname" } ``` Here you need to substitute: - <PATH> is a UNC path like \\\\AFS\\ipp-garching.mpg.de\\... - <user|group> the AFS-user or group - <rights> the AFS-rights as explained in the [AFS-Glossary](../glossary/afs-glossary.html#LetterA "AFS-Glossary"). ### NOTE When adding/removing a user to a group, it may take up to 2 hours until it takes effect. The user added/removed to the group can speed up this process by discarding the old token and obtaining a new one. # Installing an AFS client on windows Installation directions about installing everything required to access the AFS at IPP/RZG First you need to know if your PC is running 32 Bit or 64Bit Windows. This, you can check at .... openAFS requires [Kerberos](https://en.wikipedia.org/wiki/Kerberos_(protocol)) for Authentication. Following packages are required: - heimdal (the kerberos utilities) from [Secure Endpoints](https://www.secure-endpoints.com/heimdal/). - NetIDManager (Kerberos GUI) from [Secure Endpoints](https://www.secure-endpoints.com/#Network%20Identity%20Manager). - openAFSClient from [openafs.org](https://www.openafs.org/windows.html) For using AFS in the cell ipp-garching.mpg.de download the file krb5.conf. Download the correct version for your windows, but do not install them yet. We are going to install Kerberos first, test it and then install the openAFS client. 1. Start the installation process of the heimdal-package. 2. move the file krb5.conf to C:\\ProgramData\\Kerberos\\ Install NetIdmanager Install openafs-client kinit.exe klist.exe kdestroy.exe Install kerberos-GUI : set default identity get ticket destroy it Install openafs-client # AFS: Specific & Technical Information ## [AFS at IPP/MPCDF](afs-at-ipp.md) Some information about IPP specific details of the AFS. ## [Authentication within AFS](authentication-within-afs.md) How to get the proof of who you are. ## [Authorisation within AFS](authorisation-within-afs.md) How to authorize access to directories and files. # Statistics Here is some statistics about the AFS used at RZG: Latest info from : Number of fileservers : Number of clients : Space provided in GB : Number of files : Histogram : .. warning:: **The AFS-Cell 'ipp-garching.mpg.de' will turn read-only in November 2025, and will be finally decommissioned in November 2026.** ------------------- Glossary ------------------- .. toctree:: :maxdepth: 1 :glob: afs-glossary.md.txt # Glossary # AFS-Glossary ### A ### B ### C ### D-E ### F ### G-J ### K ### L ### M ### N-O ### P ### Q ### R ### S ### T ### U ### V ### W-Z ### A - AccessControl ACLs are enforced on the directory-level. Within one directory all files have the same protection. Available Rights : - a - administer : change ACL in this directory - d - delete : delete a file in a directory - i - insert : insert a file in this directory - k - lock : lock files in this directory - l - lookup : change into this directory, see all filenames - r - read : read files - w - write : write to files Notes : - To create new files both i+w rights are necessary. - In order to see files in the windows explorer or to do a ls -l on unix, both r+l rights are required. ### B - bosserver : server-program which takes care that the other server-programs are running and restarts those in case of a failure - bos <unix-command> : client program to query and manipulate the bosserver. ### C - Cell a Cell is an administrative unit within AFS. All servers and clients belong to one cell. The name of the cell is usually the second directory of the namespace /afs/<cellname>/..., e.g. here it is /afs/ipp-garching.mpg.de/ - CellAlias abbreviation of the cellname in the namespace. Here it is e.g. /afs/ipp/ for /afs/ipp-garching.mpg.de/ - Client sometimes also referred to as cache-manager. Program which provides access to AFS. Listens on port 7001/udp. - Client-cache filespace in memory or disc of the client to store data locally. Thus, data do not have to be retrieved from the server every time the client accesses them. - Callback Mechanism of the fileserver to tell the client, that some files in its cache have been changed by a different client and must be discarded from the client-cache. ### D - ### E - ### F - fs <unix-command> : client program to query the fileserver. Gives various information about the files and directory a client is using. - fileserver : server-program which actually serves the file-data. Controlled by the bosserver. Listens on port 7000/udp - firewall settings client : following rules should be applied : outgoing connections to afs-servers, ports 7000-7011/udp incoming connections from afs-fileservers on port 7000/udp ### G - ### H - ### I - ### J - ### K - Kerberos5 network security framework. On Unix there are (at least) two free implementations : [heimdal](https://www.heimdal.software "external-link") and [MIT](https://web.mit.edu/Kerberos/ "external-link"). ### L - ### M - Mountpoint point in the namespace to a particular volume. It is implemented as a symbolic link. A mountpoint may point to a RW or a RO Volume. Unix-commands: fs listmount <path>, fs mkmount <path>, fs rmmount <path> ### N - ### O - ### P - Protection Server : Server-program which deals with users and groups within AFS. Listens on port 7002/udp. - PRDB : PRotectionDataBase, used by the Protection server - pts <unix-command> : used to query the protection server in order to create/manipulate groups, users and other things connected to authorization. ### Q - ### R - RX <network-Protocol> : used by AFS for the data connections between client and server, and server and server - rxdebug <unix-command> : client program to query the afs-servers on the network level, prints various information about the peer queried. ### S - ### T - (Kerberos-) Ticket : Ticket (as for a ski-lift) identifying you as yourself in a network structure. Used for creating AFS-tokens. Generally known as credential. - (AFS-) Token : Credential which identifies you as a user within AFS. Without a token you will not have write access to AFS. The rights you have within AFS depend on your token and the ACL. ### U - ubik : network protocol to synchronize databases on different servers. Used by AFS for the VLDB, PRDB. - udebug <unix-command> : client-program to query servers running the ubik protocol. ### V - Volume Segment of the file-space on a fileserver. Comparable to a partition on a harddisk or a directory tree. Volumes can be moved between servers with almost no interruption of the accessibility. Types of Volumes: Readwrite (RW),Readonly(RO),Backup(BK) - Volumegroup group of corresponding RW, RO and BK Volumes. There are only 7 members in a single volumegroup allowed. - Volserver (Volume-Server): Server-program which deals with volume operations (like release). Controlled by bosserver. Listens on port 7005/udp - VLServer (VolumeLocation-Server) : Server-program which handles the volumelocation database. Controlled by bosserver. Listens on port 7003/udp - VLDB : Volume Location DataBase. Contains information which volume is on what server. - vos <unix-command> : client-program to query the volserver and the vlserver. ### W - ### X - ### Y - ### Z - .. warning:: **The AFS-Cell 'ipp-garching.mpg.de' will turn read-only in November 2025, and will be finally decommissioned in November 2026.** ------------------- Troubleshooting ------------------- .. toctree:: :maxdepth: 1 :glob: troubleshooting.md.txt ```{eval-rst} .. warning:: The AFS-Cell 'ipp-garching.mpg.de' will turn **read-only** in November 2025, and will be finally **decommissioned** in November 2026. ``` # Troubleshooting AFS is a complex thing, so when you're experiencing problems with it, this site might help you and us track down the problem. ## FAQ - **How can I get tokens from another cell when logged in a UNIX-machine at MPCDF ?** There are two things to consider here. The Kerberos REALM and the AFS-Cell. If the Kerberos REALM is version 5 only (like e.g. MPA), then you first need to get a Kerberos ticket and then use it to create an AFS-token. So, in order to get a token from another Cell, you can do the following : ```sh kinit USERNAME@KRB-REALM aklog -c AFSCell ``` **NOTE**: this will overwrite your present KRB5-ticket, unless you specify another ticket cache (env KRB5CCNAME) for this shell - As an example use following for the *MPA* : ```sh kinit USERNAME@MPA-GARCHING.MPG.DE aklog -c mpa-garching.mpg.de ``` - **What are these .\_\_afsNNN files ? **Sometimes you happen to see files starting with .\_\_afsNNN and wonder where they come from. Those files are created by the client when one application has opened a file, but on the same client this very file was deleted. On closing the (original) file, this .\_\_afsNNN file will be deleted. It may happen due to some kind of error condition that the client does not remove this. Then, it will be removed by the next file-system check ("salvage") or you just do it your self. - **What does "Connection timed out" mean ?** When a client cannot reach a server for a certain period of time, it does not try to talk to it anymore, but simply returns "Connection timed out". You can make the client re-establish the server connection by issuing "fs checkservers &" on the command line. It will give you a list of servers, this client cannot reach. If your server is reachable again, the access should be working normally again. - **What does "Unable to authenticate to AFS because Authentication Server was unavailable." mean ? **Very likely you are issuing the command "klog" on a UNIX-based machine, which uses Kerberos4. Kerberos4 is deprecated and will be shut down completely soon. For some parts of the institutes it is already disabled. Thus, use the following commands : ```sh kinit # to get a Kerberos5 Ticket aklog -noprdb # to turn this ticket into an AFS-Token ``` ## General problems known to cause errors - Are you coming from outside the campus network ? Then, a simple "aklog" will take a long time to login. You should use ```sh kinit # get a Kerberos ticket aklog -noprdb # turn it into an AFS-token, do not query the PT-Server ``` to avoid this timeout. - Do you use Kerberos 5 ? Make sure not to use "klog" anymore on UNIX-Systems. See FAQ above. - Computer time correct ? Please check that the time on your client is correct. - Do I have a token ? Check that you have a valid token. Tokens expire after some amount of time, thus a sudden loss of access rights can occur. - Which version do I have ? Please use the recommended version 1.6 for unix and 1.7 for windows (or any later version) when accessing AFS. ## Windows - Even a reboot does not help: remove the cache-file - When using a client prior to version 1.7, make sure you are using the AFS-Loopback-Adapter. or even better: upgrade to 1.7.x ## Unix - X-connection does not work: When logging into a remote machine via ssh, something like : ``` xauth: timeout in locking authority file ``` appears and then no X-application can be started. The problem here is, that when logging in, you do not get an AFS-token when logging in remotely. The solution is to either use an authority file which is local on the remote machine or to make sure that you get a token when logging in. HPC-Cloud ========= The HPC-Cloud provides MPG projects with a flexible self-service cloud solution based on OpenStack, Ceph, and IBM Spectrum Scale. Conceptually, the HPC Cloud enables scientists to combine batch and cloud-based computing within the same pipeline, taking advantage of both massive HPC cluster resources and highly flexible software environments. This enables projects to realize novel hybrid solutions while also benefiting from simple scaling within the cloud and the ability to rapidly prototype new solutions. Practically, the system offers standard cloud computing “building blocks”, including virtual machines based on common Linux operating systems, software-defined networks, routers, firewalls, and load balancers, as well as integrated block and S3-compatible object storage services. All resources can be provisioned and managed via a web browser or industry-standard RESTful APIs. Each project will receive a quota for each resource type, within which institute-based admins have the freedom to allocate their resources as necessary to realize their projects. The MPCDF Cloud Team is available to provide consulting and advice during both the project planning and realization phases. Moreover, to ensure the seamless flow of data between the HPC Cloud and Raven, an IBM Spectrum Scale filesystem has been deployed. Each project may request space within the filesystem which can then be mounted on both systems. This provides one data repository for the users and enables simple data flows between, e.g., high-performance simulations and post processing or data analytics and machine learning on cloud servers. Note: The HPC Cloud is not intended for the hosting of general IT services from institutes, such as email and institute web pages, but rather for the support of IT infrastructure for research projects. Resources in the HPC-Cloud can be rented by MPG projects as detailed in the Renting section below. .. toctree:: :maxdepth: 1 :glob: technical/index.rst.txt technical/faq/index.md.txt renting/index.rst.txt terms_of_use/index.rst.txt * Technical and User Documentation ================================ .. toctree:: :maxdepth: 2 :glob: quickstart.rst.txt compute.rst.txt storage.rst.txt network.rst.txt clients.rst.txt recipes.rst.txt faq/index.md.txt Quick Start =========== This document describes how to create and login to your first virtual machine (also called an *instance* or simply *server*) on the HPC Cloud. To proceed you will need a user account with the *OpenStack* service enabled and access to a *project*. .. important:: Projects are typically organized at the institute level, not for individual users. For questions about project creation and access to OpenStack please use the `helpdesk `_ Preparation ----------- 1. Login to the `dashboard `_ using your username/password and domain *mpcdf*. Note that the dashboard is only accessible from MPG networks. When accessing the dashboard from external networks we advise you to use a socks proxy. A socks proxy can be started on both Linux- and Windows-based systems, with your browser subsequently being configured to use the proxy. For Linux a socks proxy can be created using ssh as follows: .. code-block:: sh ssh -D 1337 -q -C -N @gate.mpcdf.mpg.de For Windows a socks proxy can be created using PuTTY as follows: - Launch PuTTY - enter the hostname (gate.mpcdf.mpg.de) and port (22). - On the left side, in the Category window, go to Connection -> SSH -> Tunnels. - For 'Source Port' enter '1337'. - Under 'Destination' select the 'Dynamic' radio button and leave the 'Auto' button selected. - Press the 'Add' button. You should see 'D1337' in the 'Forwarded ports:' box. - Click the 'Open' button. - A terminal window will open allowing you to login as usual with your username/password and OTP. Once the socks proxy has been created you can configure your browser to use it. For Firefox Preferences-> Network Settings and create a Manual Proxy using localhost and port (1337) For Chrome .. code-block:: sh google-chrome --proxy-server="socks5://localhost:1337" --host-resolver-rules="MAP * 0.0.0.0, EXCLUDE localhost" For Windows Edge Settings->Open Proxy Settings and create a Manual proxy with host Address (socks=127.0.0.1) and port (1337) To enable command line client access (once the ssh tunnel has been enabled) .. code-block:: sh export ALL_PROXY=socks5h://localhost:1337 openstack Note: The PySocks python module may need to be installed by hand as a dependency. Many MPG Institutes will be able to directly access the OpenStack dashboard (and the MPCDF VPN can be used as a fallback). However, if you are not able to load the page from your local institute network (or institute-provided VPN), please `let us know `_. 2. Create or import an RSA SSH key pair via `Project / Compute / Key Pairs `_. This key pair belongs to your *user* and will be used for SSH authentication on the servers you personally create. Please be aware that the name of the key will be visible to other users of the same project. Save the private key to your personal computer. Example of adding a new RSA key pair locally and importing it to the OpenStack dashboard: a. First, create your RSA key pair via your local terminal and save it to the desired location (`$HOME/.ssh/id_rsa` by default): .. code-block:: sh ssh-keygen -t rsa b. Now, display the content of the public key and copy the printed content: .. code-block:: sh cat $HOME/.ssh/id_rsa.pub c. Go to the `Project / Compute / Key Pairs `_ -> "Import Public Key" and paste the copied content of the public key into the "Public Key" field. 3. Modify your default security group to allow ssh-connections: a. Go to `Project / Network / Security Groups `_ -> "Manage Rules" for the default security group. b. Click on "+Add Rule", choose "SSH" in the "Rule" dropdown list and click "Add". .. important:: For security reasons the private key is not stored in the system. Thus, if you don't save it at the time it is created, there will be no way to recover it! Create a server --------------- 1. Set the active project via `Identity / Projects `_ or the dropdown menu near the upper left as in the example shown below. .. image:: /_images/active_project.png :alt: Active Project Dropdown Menu 2. Create a new server via the |launch_button| button on `Project / Compute / Instances `_. A wizard will guide you through the process, as shown in the composite image below. In this example we want a single-core server running a standard Linux distribution on a shared network, so after specifying the instance name, select *Ubuntu 20.04* as the source, *mpcdf.small* as the flavor, and *cloud-local-1* as the network. Assuming your user has only a single key pair, it should have been pre-selected automatically. .. |launch_button| image:: images/launch_button.png :alt: Launch Instance .. image:: /_images/launch_instance.png :alt: Launch Instance Wizard 3. Once the server is up and running, you can login via SSH using the key pair defined above. Its randomly-assigned IP address can be found on the `list of instances `_. Servers on *cloud-local* networks are already accessible from internal MPCDF networks without requiring any additional configuration. Here are several common login scenarios: a. Direct login from your personal computer via the `gateway machines <../../computing/gateways.html>`_: .. code-block:: sh ssh -i KEYFILE -J USERNAME@gate.mpcdf.mpg.de root@10.186.XX.XX b. Login to the gateway machines with SSH agent forwarding... .. code-block:: sh ssh-add KEYFILE ssh -A USERNAME@gate.mpcdf.mpg.de ...and then to the server itself: .. code-block:: sh ssh root@10.186.XX.XX .. tip:: Once an instance has been launched additional users can be provided with ssh access to the root account by adding their ssh key pairs to the root user's authorized_keys file. There is no way to inject keys via the dashboard or openstack cli. Next steps ---------- At this point the server may be administered like any other virtual machine. Outgoing internet access is allowed by default, e.g. for software installation and updates. Common next steps are to attach additional `block storage `_ or mount a `shared filesystem `_ .. tip:: The dashboard is not the only way to manage HPC Cloud resources. Throughout the documentation there are boxes (like the one below!) containing equivalent procedures using the `command line interface `_. Note that *UPPERCASE* placeholder values should be modified before running the commands and refer to names (e.g. *demosrv*) and not uuids (e.g. *123e4567-e89b-12d3-a456-426655440000*), unless otherwise indicated. .. admonition:: CLI Example .. code-block:: sh openstack keypair create KEYNAME > KEYFILE openstack server create SERVER --image "Ubuntu 20.04" --flavor mpcdf.small --network cloud-local-1 --key-name KEYNAME After a short delay, the server's IP address can be found with: ``openstack server show SERVER -f value -c addresses`` .. [#splittunnel-note] Note that *SplitTunnel* profiles are supported, but only if the *10.186.XX.XX* address range does not conflict with your institute's local network environment. Compute ======= This page expands upon `Quick Start `_ by describing the various options in more detail, including image/flavor selection and config scripts. Also covered are two mid-lifecycle operations -- rebuild and resize -- which change a running server's image and flavor, respectively. Finally, possible locations for the system disk are detailed, as this is the mechanism by which local SSD storage is provisioned in the HPC Cloud. Images ------ A newly-created server's operating system type and version are determined by the template or *image* used to create the system (root) disk. The following cloud-ready images are maintained by the admins: +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | OS | Firmware | Security Model | Software Mirrors | Availability | End of Support | +====================+==========+================+================================================================+===================================+==================+ | AlmaLinux 8 | BIOS | SELinux | local, including `EPEL `_ | all projects | 2029-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | AlmaLinux 9 | UEFI | SELinux | local, including `EPEL `_ | all projects | 2032-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | AlmaLinux 10 | UEFI | SELinux | local, including `EPEL `_ | all projects | 2035-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | RHEL 8 | BIOS | SELinux | Red Hat via subscription | on request, subscription required | 2029-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | RHEL 9 | UEFI | SELinux | Red Hat via subscription | on request, subscription required | 2032-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | RHEL 10 | UEFI | SELinux | Red Hat via subscription | on request, subscription required | 2035-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | Debian 12 | UEFI | AppArmor | `debian.org `_ | all projects | 2028-06-30 (LTS) | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | Debian 13 | UEFI | AppArmor | `debian.org `_ | all projects | 2030-06-30 (LTS) | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | Ubuntu 22.04 | UEFI | AppArmor | local | all projects | 2027-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | Ubuntu 24.04 | UEFI | AppArmor | local | all projects | 2029-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | Ubuntu 26.04 | UEFI | AppArmor | local | all projects | 2031-05-31 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | openSUSE Leap 16.0 | UEFI | SELinux | `opensuse.org `_ | all projects | 2027-11-30 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ | SLES 16.0 | UEFI | SELinux | SUSE via subscription | on request, subscription required | 2027-11-30 | +--------------------+----------+----------------+----------------------------------------------------------------+-----------------------------------+------------------+ The QEMU virtual machine type of all images is now the modern Q35. Note that maintaining support until the specified date may require minor release upgrades. The default login of the public images is ``root`` with *publickey* being the only enabled authentication method. Further configuration details can be found in `GitLab `_. Support for images is limited to the initial deployment of the server. On-going maintenance, including security updates, and application/service configuration remain the responsibility of the project admin(s). .. tip:: You may also build and upload your own images in RAW or QCOW2 format. Note that `cloud-init `_ or an equivalent service is required to retrieve the hostname, SSH keys, etc. via the `metadata `_ service. Snapshots ~~~~~~~~~ A *snapshot* is an image containing a copy of a server's system disk at a particular point in time. Snapshots are useful for duplicating a pre-configured server or creating a "restore point" before performing maintenance. Note that there is no single action to "revert" to a previous snapshot, but the same result can be achieved with the `rebuild <#rebuild-a-server>`_ function. To create a snapshot, click the |snapshot_button| button next to the desired source server on `Project / Compute / Instances `_. If the server is running, it will be paused briefly while the data is copied. [#snapshot-consistency-note]_ .. |snapshot_button| image:: images/snapshot_button.png :alt: Create Snapshot .. admonition:: CLI example .. code-block:: sh openstack server image create SERVER --name SNAPSHOT See `Project / Compute / Images `_ for a list of images and snapshots available on the active project. To launch a new server from a snapshot, simply choose *Instance Snapshot* as the boot source and select the desired snapshot from the list. The resulting instance will be a clone of the source at the time the snapshot was taken, but with different MAC and IP addresses. [#clone-note]_ .. image:: /_images/launch_snapshot.png :alt: Launch Snapshot .. admonition:: CLI example .. code-block:: sh openstack server create NEWSERVER --image SNAPSHOT ... Rebuild a server ~~~~~~~~~~~~~~~~ A server can be *rebuilt* from an image, thereby keeping the same configuration including MAC and IP addresses. .. warning:: All data written to the system disk since the initial deployment will be lost during the rebuild! To rebuild a server, select it from `Project / Compute / Instances `_ and then choose the |rebuild_button| action. You can either start over with the same OS by choosing the running image (listed in the "image name" column) or change the OS by picking a different image. Even if your server is based on a now out-of-date image (identified by the presence of a datestamp *YYYYMMDD* in the name), you still have the option of rebuilding from the original image. Note that for technical reasons changing the machine type (pc-i440fx or pc-q35, see table above) by rebuilding is not supported. If you need to do this, please contact MPCDF for help. .. |rebuild_button| image:: images/rebuild_button.png :alt: Rebuild Instance .. hint:: To revert to a previous snapshot, simply rebuild the server *from its own snapshot*. .. admonition:: CLI example .. code-block:: sh openstack server rebuild SERVER --image IMAGE Provide a *SNAPSHOT* in place of *IMAGE* to revert to that snapshot, or omit the ``--image`` option entirely to use the original image. Flavors ------- The *flavor* assigned to an instance determines its hardware resources including number of CPU cores, amount of memory, and size/location of the **system** disk. [#ephemeral-disk-note]_ Here is a partial list of pre-defined flavors: +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | Name | vCPUs | Memory | Disk Size | Disk Location | Compute Node | Availability | +==================================+=======+========+===========+===============+======================+==========================================================+ | mpcdf.small | 1 | 2 GB | 25 GB | Ceph | shared | all projects | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.medium | 2 | 4 GB | 25 GB | Ceph | shared | all projects | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.large | 4 | 8 GB | 25 GB | Ceph | shared | all projects | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.xlarge | 8 | 16 GB | 25 GB | Ceph | shared | all projects | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.\ *N*\ c\ *M*\ g | *N* | *M* GB | 25 GB | Ceph | shared | all projects up to 24 vCPUs and 64 GB, larger on request | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.\ *N*\ c\ *M*\ g.ssd | *N* | *M* GB | 250 GB | local SSD | shared | on request | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.\ *N*\ c\ *M*\ g.nvme | *N* | *M* GB | 25 GB | Ceph | shared, with NVMe | on request | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.\ *N*\ c\ *M*\ g.gpu.\ *T* | *N* | *M* GB | 25 GB | Ceph | shared, with GPU *T* | on request | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ | mpcdf.dedicated.\ *N*\ c\ *M*\ g | *N* | *M* GB | 25 GB | Ceph | dedicated | on request | +----------------------------------+-------+--------+-----------+---------------+----------------------+----------------------------------------------------------+ Shared compute nodes allow a limited level of virtual CPU oversubscription. This configuration benefits applications with intermittent load profiles (very common in a cloud environment), by allocating more virtual cores than the total number of physical cores on the host. In this way load peaks can be processed more quickly and idle cores can be utilized by other applications. On the other hand, continuously-busy applications should run on dedicated nodes to have predictable performance and avoid negatively affecting other projects. In such cases a *.dedicated* flavor should be employed to ensure that each vCPU is pinned to a physical core. Users are not allowed to create their own flavors, but additional variations, e.g. more memory-per-core or a specific disk size, are available upon request. A complete list of flavors available to the active project is shown in the "Flavor" section of the `launch wizard `_. Attached devices ~~~~~~~~~~~~~~~~ Instances using *.gpu.* flavors see the usual virtual hardware, plus an Nvidia A30, A40, or A100 GPU attached in PCI-passthrough mode. The standard "baremetal" driver and software stack may be employed, including ``nvidia-smi`` command to query the state of the GPU. Similarly, instances using *.nvme* flavors see the usual virtual hardware (including Ceph-based system disk), plus a 1.6 TB NVMe SSD attached in PCI-passthrough mode. This device will appear as block device ``/dev/nvme0n1`` to be formatted with a filesystem or written directly by an application. Before deleting an NVMe-enabled instance, please erase the device as this does not happen automatically: .. code-block:: sh apt install nvme-cli # or equivalent on other systems nvme format -s1 /dev/nvme0n1 .. caution:: The contents of an NVMe SSD are not automatically cleared between uses. Before deleting an instance with attached NVMe, it is **strongly recommended** to erase the device as described above, otherwise the data could potentially be read from a different project. Please be aware that neither attached GPUs nor NVMe SSDs support live migration, meaning that occasional scheduled downtimes will be required for maintenance purposes. System disk storage location ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Flavors **not** ending in *.ssd* store the system disk on the Ceph backend, which has certain advantages: - Data is replicated across the Ceph cluster for a high degree of availability. - Instances can be quickly restarted on a different host in the event of a hardware failure. In contrast, *.ssd* flavors will place the system disk directly on the local SSD-based storage of the compute host, which may offer higher I/O performance. [#boot-from-volume-note]_ .. caution:: Locally-stored system disks may become unavailable or even permanently lost in the event of a hardware failure. Resize a server ~~~~~~~~~~~~~~~ The resources of a running server may be *resized* up or down by changing its flavor -- a semi-automated process during which the server is stopped, a temporary snapshot is taken, and then provisionally restarted. This provides the opportunity to revert the change in case the operating system proves to be incompatible with the new flavor. You may pick any new flavor provided the system disk location remains the same. .. danger:: Changing the system disk storage location is **not supported** and will result in data loss! 1. Select the server from `Project / Compute / Instances `_ and then choose the |resize_button| action. .. |resize_button| image:: images/resize_button.png :alt: Resize Instance 2. Wait while the server gracefully shuts down, a snapshot is made of its system disk, and it restarts using the new flavor. [#resize-snapshot-note]_ 3. Login to the server and check if the operating system and services are running normally. If so, click the |confirm_resize_button| button, otherwise choose |revert_resize_button| to undo the operation. .. |confirm_resize_button| image:: images/confirm_resize_button.png :alt: Confirm Resize .. |revert_resize_button| image:: images/revert_resize_button.png :alt: Revert Resize .. admonition:: CLI example .. code-block:: sh openstack server resize SERVER --flavor NEWFLAVOR ssh -i KEYFILE root@10.186.XX.XX systemctl status openstack server resize confirm SERVER If SSH login fails or systemd reports problems, use ``revert`` in place of ``confirm`` on the last line. Customization script -------------------- Server deployment can be further automated via *customization scripts* (also called *user data*). The two most common `formats `_ are scripts, which are called by `cloud-init `_ during startup, and `cloud config `_ files, which modify the behavior of cloud-init itself. The formats are distinguished by the presence of ``#!`` or ``#cloud-config``, respectively, on the first line of the script. To use a customization script simply paste its contents into the text box in the "Configuration" section of the launch wizard. .. image:: /_images/launch_configuration.png :alt: Launch Configuration .. hint:: Adapt the following example block to automatically deploy a *second* SSH key in addition to your personal `key pair `_. .. code-block:: text #cloud-config ssh_authorized_keys: - ssh-rsa ... .. admonition:: CLI example .. code-block:: sh openstack server create SERVER --user-data SCRIPTFILE ... .. [#snapshot-consistency-note] Consistency of snapshots is only guaranteed at the *block* level, i.e. pending operations at the *file* level may not be captured. Normally this is not a problem for modern filesystems, databases, etc. .. [#clone-note] Note that unique identifiers such as ``/etc/machine-id`` will be carried over from the original. In some operating systems this includes SSH host keys as well. If this is not the desired outcome, regenerate them with, e.g.: ``ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -N ""`` .. [#ephemeral-disk-note] Note that the HPC Cloud does not offer ephemeral disks. To add additional disks see `block storage `_. .. [#boot-from-volume-note] This option has **no effect** on boot-from-volume instances, for which the storage location is determined by the volume type. .. [#resize-snapshot-note] Additional `volumes `_ are not included in the automatic snapshot, so any data written to these disks during the provisional restart will remain in place, even if the resize is reverted. .. important:: Interface change We will rename the volume types as follows: Ceph to standard, CephBulk to data, CephIntensive to intensive. Volume type fast, will keep its name. Storage ======= The HPC Cloud offers both *block* storage, in the form of disk volumes which can be directly attached to a server, and *file* storage, in the form of shared filesystems which can be NFS-mounted by the operating system. In addition, there is an *object store* providing *containers* (also called *buckets*) which allow data to be accessed from multiple clients though standard REST APIs. [#container-note]_ The block and object storage services are based on Ceph, while file storage offers a choice between CephFS and IBM Storage Scale (GPFS). The latter is also mounted on `Raven <../../computing/raven-user-guide.html#gpfs>`_ (by default) as well as `Robin <../../visualization/index.html#robin>`_ (on request) for efficient exchange of data between the various systems. Block ----- Storage *volumes* are logically independent block devices of arbitrary size. They can be attached to a running server and later detached or even reattached to a different server. [#no-multiattach-note]_ Typically, volumes are used to store data which should persist beyond the lifetime of a single instance, although they are also useful as "scratch" space. The list below details the types of volumes available on the HPC Cloud. Only the *standard* volume type is enabled for all projects. If you feel you need another volume type, please contact the cloud enabling team on our helpdesk. +-----------+----------------------------------------------------------------+ | Name | Description | +===========+================================================================+ | standard | Reasonable performance for most data types and access patterns | +-----------+----------------------------------------------------------------+ | data | Best for large sequential access patterns | +-----------+----------------------------------------------------------------+ | fast | High bandwidth, and more iops | +-----------+----------------------------------------------------------------+ | intensive | Lowest available IO latency | +-----------+----------------------------------------------------------------+ 1. Create an empty volume via the |volume_button| button on `Project / Volumes / Volumes `_. The maximum allowable size depends on your project's quota. .. |volume_button| image:: images/volume_button.png :alt: Create Volume .. image:: /_images/volume_size.png :alt: Volume Size 2. Select a server from `Project / Compute / Instances `_, perform the |attach_volume_button| action, and pick the newly created volume. .. |attach_volume_button| image:: images/attach_volume_button.png :alt: Attache Volume 3. The device names such as */dev/sdb* or */dev/vdb* are not guaranteed to be the same each time they are attached to a server. To definitively identify a volume, look into */dev/disk/by-id*. Volumes attached will appear as `scsi-0QEMU_QEMU_HARDDISK_ID` where ID is the OpenStack volume ID. On servers launched before 08.2026, the name will be `virtio-ID` where ID is truncated at 20 characters. For example: .. code-block:: sh openstack volume list --name VOLUME +--------------------------------------+--------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+--------+-----------+------+-------------+ | 708e61cf-b80a-4488-9a32-eb532715ce38 | VOLUME | available | 10 | | +--------------------------------------+--------+-----------+------+-------------+ Will appear as */dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_708e61cf-b80a-4488-9a32-eb532715ce38*. It is also possible to look up the device name from `here `_ in the "attached to" column). .. important:: We strongly advise using the disk ID to positively identify the block device on the server with an OpenStack volume Create and mount a new filesystem using the following example commands: .. code-block:: sh mkfs.xfs /dev/disk/by-id/virtio-708e61cf-b80a-4488-9 mkdir /demovol echo "/dev/disk/by-id/virtio-708e61cf-b80a-4488-9 /demovol xfs defaults 0 0" >> /etc/fstab mount /demovol .. admonition:: CLI Example .. code-block:: sh openstack volume create VOLUME --size SIZE [--type TYPE] openstack server add volume SERVER VOLUME .. tip:: It is possible to migrate an existing volume to a new type online, even while still attached to the server. Simply click the "Change Volume Type" action, choose the target type, and set the migration policy to "On Demand". The data will then be transparently copied to the new backend storage pool. .. code-block:: sh openstack volume set VOLUME --type TYPE --retype-policy on-demand Shared Filesystems (Shares) --------------------------- Projects can deploy various parallel filesystems within the HPC Cloud. The table below provides some rough guidelines to decide which type of shared filesystem suits your use-case best. ========== ========== =============== ================= ==================== =================== Service FileSystem User management Additional mounts Strengths Limitations ========== ========== =============== ================= ==================== =================== NexusPOSIX NFSv4 MPCDF LDAP HPCs, Clusters Performance & Backup Avg File > 16MB Manila NFSv4 project-managed no Simplicity Limited performance Manila CephFS project-managed no Performance Client maintenance Own NFS NFSv3/4 project-managed project-managed Flexibility Single server ========== ========== =============== ================= ==================== =================== NexusPOSIX ^^^^^^^^^^ NexusPOSIX is based on IBM Spectrum Scale, a high performance parallel filesystem. It can be mounted on both HPC Cloud VMs and the HPC systems (Dais, Raven & Viper). This cross mounting allows projects to easily access data from both cloud and HPC systems, allowing for hybrid, HPC/Cloud, solutions. Highlights of NexusPOSIX include: - Mounted on Dais, Raven and Viper and on HPC Cloud VMs - Data Security - Automatic backup included - Possible high performance data transfers and sharing via Globus (GO-Nexus) - Reservations can be increased as projects grow NexusPOSIX reservations can be requested by projects starting at 5TB and ranging into 100s of TBs. For more information, please make a request via the MPCDF helpdesk. Because NexusPOSIX is natively mounted on our HPC systems you must use MPCDF UIDs and GIDs in your user management. The filesystem is designed to house large files, meaning your files must be *larger than 16MB on average*. Manila ^^^^^^ Manila is the HPC Cloud service providing *Shares*, i.e. shared filesystems. The service is backed by our cloud Ceph cluster. Access to and from this cluster is limited to the HPC Cloud, therefore no additional mounts can be provided. As project administrator you have root access to the filesystems provided by the HPC Cloud, and are free to set up user management as suits your project. These flexible shared filesystems can be created in two configurations: * NFS: This type of share is a single-threaded export that may be mounted as an NFS share, limiting the number of requests that can be handled. Best suited for shared configuration files. * CephFS: This type of share is a direct connection to the HPC Cloud Ceph cluster providing increased performance. However the client instances must use a ceph client version compatible with our Ceph servers (v17.2.8, v18.2.4, v19, or greater). Manila NFS """""""""" The Manila client is not included with the standard openstack clients. To get started ensure you have the manila client installed, for example: .. code-block:: pip install --user python-manilaclient Now you are ready to create a share .. code-block:: openstack share create NFS $SIZE --share-type CephNFS --name $SHARE openstack share show $SHARE Note the `$SUBDIR` from export_locations, i.e. the part coming after `/volumes/_nogroup/`. You will need it to mount the file system later. The SUBDIR value can be retrieved using the following command: .. code-block:: export SUBDIR=$(openstack share show $SHARE -c export_locations -f value | grep path | sed -E 's@.*:/volumes/_nogroup/([^ ]*)@\1@') Make sure your client server(s) are connected to the `manila-nfs` network. If not add the required network connection: .. code-block:: openstack server add network $SERVER manila-nfs Get the IP of your server on the manila-nfs network. Note for the below command to work you must have the tool `jq` installed. .. code-block:: CLIENT_IP=$(openstack server show perf-tests -c addresses -f json | jq -r '."addresses" | ."manila-nfs" | .[]') Access is managed through a whitelist of IPs, so add the client IP of your VM to the share: .. code-block:: openstack share access create $SHARE ip $CLIENT_IP Now ssh into your client server and mount the share: .. code-block:: apt install -y nfs-common echo "manila-nfs.hpccloud.mpcdf.mpg.de:/volumes/_nogroup/$SUBDIR /share nfs _netdev 0 0" >> /etc/fstab systemctl daemon-reload mkdir /share && mount /share Manila CephFS """"""""""""" The Manila client is not included with the standard openstack clients. To get started ensure you have the manila client installed, for example: .. code-block:: pip install --user python-manilaclient Now create the new share: .. code-block:: openstack share create CEPHFS $SIZE --share-type CephNative --name $SHARE openstack share show $SHARE Note the `$SUBDIR` from export_locations, i.e. the part coming after `/volumes/_nogroup/`. You will need it to mount the file system later. The SUBDIR value can be retrieved using the following command: .. code-block:: export SUBDIR=$(openstack share show $SHARE -c export_locations -f value | grep path | sed -E 's@.*:/volumes/_nogroup/([^ ]*)@\1@') Make sure your client server(s) are connected to the `manila-cephfs` network. If not add the required network connection: .. code-block:: openstack server add network $SERVER manila-cephfs With CephFS access is managed through access keys. Create Key for the client: .. code-block:: openstack share access create $SHARE cephx $CEPHUSER Wait a moment while the credentials are generated and made available, then .. code-block:: openstack share access list $SHARE Note the access key, which you will need to mount the file system on the client machine. Now ssh into your client machine, there do: .. code-block:: apt install -y ceph-common cat >> /etc/ceph/keyring <> /etc/fstab systemctl daemon-reload mkdir /share && mount /share Should the mount command produce error messages about missing configuration or failing lookups of the ceph monitors, these can be addressed by creating a file `/etc/ceph/ceph.conf` file with content: .. code-block:: toml [client] keyring = /etc/ceph/keyring [global] mon_host = manila-cephfs.hpccloud.mpcdf.mpg.de:3300 Object ------ The HPC Cloud includes an *object store* for saving and retrieving data through a publicly-accessible REST API. Both OpenStack Swift- and S3-style APIs are supported. Objects are stored in *containers* (or *buckets*) which in turn belong to the *project* (or *tenant*). Folders within buckets are supported but typically handled as a part of the object name. .. important:: Similar to Amazon's simple storage service we use a global namespace for our buckets. Please think of a unique name for your bucket. If another bucket of the same name already exists you will see an error message stating: *Error: Forbidden insufficient permissions on requests operation*. A good practice for naming buckets is to prefix them with a project name or similar general prefix. This helps avoid possible contention with bucket names and also protects somewhat against public buckets being unexpectedly crawled by automated systems or malicious users on the internet. .. important:: Please be aware that the contents of **public** buckets are not only visible but can be modified by anyone on the internet. 1. Create a new bucket via the |container_button| button on `Project / Object Store / Containers `_. Note that buckets, both public and private, share a global namespace. If a bucket name is already taken by another project you will receive an error message. Please use the `S3 bucket naming rules `_. This will help avoid possible problems when accessing objects via the S3 API. .. |container_button| image:: images/container_button.png :alt: Container Button .. image:: /_images/container_name.png :alt: Container Name 2. Select the bucket and click the |upload_button| button to upload a file. The object name (confusingly labeled "File Name") defaults to the filename, but may be an arbitrary string. You may create a folder for the object at the same time by prepending one or more names, separated by "/". .. |upload_button| image:: images/upload_button.png :alt: Upload Button .. image:: /_images/object_filename.png :alt: Object Filename 3. If you created a public bucket, the contents of the file would then be available at, e.g.: `https://objectstore.hpccloud.mpcdf.mpg.de/swift/v1/demobucket/demofolder/demoobject` or `https://objectstore.hpccloud.mpcdf.mpg.de/demobucket/demofolder/demoobject` .. admonition:: CLI Example .. code-block:: sh openstack container create BUCKET openstack object create BUCKET FILE --name FOLDER/OBJECT Object store quotas are separate from those of other storage types, and are not displayed on the dashboard. For questions about your quota, please contact the `helpdesk `_. Note that while the Ceph backend itself supports very large objects, uploads through the dashboard and Swift API are limited to **5GB**. [#swift-segments-note]_ Use the S3 API to avoid this limitation. .. tip:: The object store supports a subset of the Amazon S3 API. See `CLI and Scripting `_ to get started with `s3cmd `_ and/or `Boto3 `_. .. [#container-note] Note that the term *container* is unrelated to Docker containers. Where possible we use *bucket* to avoid confusion. .. [#no-multiattach-note] Attaching a volume to multiple servers simultaneously is not supported. .. [#swift-segments-note] The ``openstack`` command also uses the Swift API and is therefore subject to the same limitation. Note that the ``--segment-size`` feature of the older ``swift`` command splits the file into a collection of smaller objects which cannot easily be downloaded by other clients. Network ======= The HPC Cloud supports virtual network resources such as *routers*, *floating ips*, and *security groups*, which, together with the physical and virtual networks themselves form the communication infrastructure needed by compute instances. .. hint:: Networking topics can become quite complex, but *many* use cases are already covered by `local networks <#local-networks>`_ combined with the appropriate `security group <#security-groups>`_ rules. Thus, you may want to skip directly to those sections. Shared Networks --------------- The HPC Cloud offers multiple *shared* networks, summarized in the table below. These networks allow communication between projects (only if explicitly allowed by `security group <#security-groups>`_ rules) as well as traffic to and from the HPC Cloud (subject to some restrictions explained below). IP addresses are assigned randomly from the pool but typically won't change over the lifetime of the port, i.e. as long as the instance or floating ip is not deleted or released, respectively. +--------------------+----------------+------------------+--------------+--------------------------------------------------------+ | Network | Address Range | Routing | Availability | Supported Port Types | +====================+================+==================+==============+========================================================+ | cloud-local-1 | 10.186.1.XX | mpcdf campus/vpn | all projects | instances | +--------------------+----------------+------------------+--------------+--------------------------------------------------------+ | cloud-local-2 | 10.186.2.XX | mpcdf campus/vpn | all projects | instances | +--------------------+----------------+------------------+--------------+--------------------------------------------------------+ | cloud-local-float | 10.186.7.XX | mpcdf campus/vpn | all projects | floating ips, routers | +--------------------+----------------+------------------+--------------+--------------------------------------------------------+ | cloud-public | 130.183.217.XX | global | on request | floating ips, routers, instances [#fixed-public-note]_ | +--------------------+----------------+------------------+--------------+--------------------------------------------------------+ Local Networks ~~~~~~~~~~~~~~ The *cloud-local* networks provide each server with an IP address in the 10.186.XX.XX range and support connections **from** internal MPCDF networks without the need for a project-specific gateway or jumphost. Note that the reverse is not the case; in general servers on these networks can connect **to** publicly-available services such as `GitLab `_ and `DataShare `_, as well as the public internet [#internet-nat-note]_, but not Raven, local clusters, etc. The *cloud-local-float* network has similar properties but is normally used in combination with `private networks <#private-networks-and-routers>`_. Public Network ~~~~~~~~~~~~~~ The HPC Cloud includes a single globally-routed network *cloud-public* for the purpose of hosting internet-accessible services. Access is granted per-project, based on requirements and a discussion between the responsible local admins and MPCDF. In practice this network is normally used via `floating ips <#floating-ips>`_ which are attached to an instance running on a local (or private) network. [#fixed-public-note]_ Automated domain name service (DNS) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Hostnames are now automatically generated for most devices attached to the public or local cloud networks, including virtual machines and floating IP addresses. The system works like this: 1. Each virtual machine is assigned a hostname of the following form: ``VM_NAME.PROJECT_NAME.hpccloud.mpg.de`` If the name of the virtual machine is invalid according to the requirements of DNS, then a unique hostname based on the fixed IP address will be substituted automatically. 2. Each floating IP is assigned a hostname of the following form: ``FIP_DESCRIPTION.PROJECT_NAME.hpccloud.mpg.de`` If the description field is empty or invalid, then a unique hostname based on the floating IP address will be substituted automatically. 3. Hostnames are synchronized with the MPCDF DNS servers every five minutes. For devices on the public cloud network, both forward and reverse entries are propagated to the global DNS, whereas on local networks only the forward (i.e. hostname->IP address) entries are published. Thus, within the framework described above it is possible to deploy and configure many applications on the HPC-Cloud without tracking individual IP addresses. Private Networks and Routers ---------------------------- Private networking is available to meet more complex requirements, as well conserve IP addresses on the shared networks. The following steps correspond to a typical use case, but by no means cover all the possibilities. 1. Create a new network and associated subnet via |network_button| on `Project / Network / Networks `_, specifying *192.168.0.0/24* as the network address and *130.183.9.32*, *130.183.1.21* (one per line) as the DNS name servers. .. |network_button| image:: images/network_button.png :alt: Create Network 2. Create a new router via |router_button| on `Project / Network / Routers `_, choosing either *cloud-local-float* or *cloud-public* (if available) as the external network. .. |router_button| image:: images/router_button.png :alt: Create Router 3. Select the router, switch to the "Interfaces" tab, and click |interface_button| to add a port in the newly-created subnet. .. |interface_button| image:: images/interface_button.png :alt: Add Interface .. admonition:: CLI Example .. code-block:: text openstack network create NETWORK openstack subnet create SUBNET --network NETWORK --subnet-range 192.168.0.0/24 --dns-nameserver 130.183.9.32 --dns-nameserver 130.183.1.21 openstack router create ROUTER openstack router set ROUTER --external-gateway cloud-local-float openstack router add subnet ROUTER SUBNET You can now launch instances on the private network, but there is no way to reach the network from outside, and therefore no way to login! [#no-port-forwarding-note]_ To get around this limitation, attach a floating ip to at least one of the servers in the private network. Floating IPs ------------ *Floating ips* give a server an alternate ip address without reconfiguring the server itself. This works because the software-defined networking layer implements a one-to-one translation between the *fixed* address that the operating system "sees" and the *floating* address. Note that the fixed address will continue to be usable inside the local (or private) subnet, while all other hosts must use the floating ip. [#own-floating-ip-note]_ 1. Reserve a floating ip via |allocate_fip_button| on `Project / Network / Floating IPs `_, selecting either *cloud-local-float* or *cloud-public* (if available) as the pool. .. |allocate_fip_button| image:: images/allocate_fip_button.png :alt: Allocate IP to Project 2. Select the floating ip, click |associate_fip_button|, and then choose the primary port of the target instance. .. |associate_fip_button| image:: images/associate_fip_button.png :alt: Associate Floating IP .. attention:: The floating ip's pool must match the router gateway to successfully associate with a server: - For servers on local networks, only floating ips from *cloud-public* can be used. - For servers on private networks, it depends on which external network was chosen for the private router -- see step 2 `above <#private-networks-and-routers>`_. .. admonition:: CLI Example .. code-block:: sh openstack floating ip create cloud-local-float openstack server add floating ip SERVER 10.186.7.XX Security Groups --------------- The HPC Cloud provides a flexible per-instance, network-based access control mechanism in the form of *security groups*. The default security group allows connections between instances in the same project (and network), plus SSH from internal MPCDF networks and ping from anywhere. .. attention:: The default security group allows SSH ingress from all other hosts in the cloud networks and from other machines at MPCDF. While this is convenient you may want to limit access by using your own security groups. The recommended way to "open" additional ports is as follows: 1. Create a new group via |secgroup_button| on `Project / Network / Security Groups `_. .. |secgroup_button| image:: images/secgroup_button.png :alt: Create Security Group 2. Select the group, choose |rules_button|, followed by |add_rule_button|, and then add one or more rules, e.g.: .. |rules_button| image:: images/rules_button.png :alt: Manage Rules .. |add_rule_button| image:: images/add_rule_button.png :alt: Add Rule - Rule: Custom TCP Rule - Direction: Ingress - Port: 80 - CIDR: 0.0.0.0/0 3. Select the desired instance on `Project / Compute / Instances `_, choose |edit_secgroup_button|, and then add the newly created security group. In most cases you should keep, rather than replace, the default security group, unless your new security group includes rules for all necessary ingress and egress traffic. .. |edit_secgroup_button| image:: images/edit_secgroup_button.png :alt: Edit Security Groups .. admonition:: CLI Example .. code-block:: sh openstack security group create SECGROUP openstack security group rule create SECGROUP --protocol tcp --dst-port 80 openstack server add security group SERVER SECGROUP One thing to keep in mind is that what traffic ultimately reaches the server is determined by the properties of the network **and** the security group rules. For example, since *cloud-local* networks are not globally routed, opening a port to all sources only affects traffic from other MPCDF networks. Adding a public floating ip, however, changes the situation dramatically. Thus, it is good practice to choose the CIDR carefully, rather than relying exclusively on the network topology for security. .. [#fixed-public-note] Certain applications do not support floating ips, since the operating system detects only the fixed address. On request, it is technically possible to place an instance directly on the public network. .. [#internet-nat-note] Internet access is provided via a NAT gateway. The external address of the gateway (currently 130.183.254.22) may change and should not be used for access control. .. [#no-port-forwarding-note] Unlike many routers used for home internet service, the software-defined routers in the HPC Cloud do not support *port forwarding*, i.e. directing incoming traffic on a particular port to a particular local host "behind" the router. .. [#own-floating-ip-note] One might wonder if local peers can communicate with the floating ip as well. For private networks the answer is yes; for local networks the answer is **no**. Command Line Interface and Scripting ==================================== The HPC Cloud includes a comprehensive `REST API `_ to enable command line usage and scripting. Internal MPCDF network access is not required to use the API; like the dashboard, it is accessible from MPG networks. If you can load the `dashboard `_ then you should be able to reach the API endpoints as well. Utilizing the API typically requires a specific set of credentials along with locally-installed software. [#curl-note]_ Preparation ----------- 1. Login to the `dashboard `_ and set the active project as described in `Quick Start `_. 2. Use the left sidebar to navigate to `Identity -> Application Credentials `_ to generate a project specific credential. .. image:: /_images/application_credential.png :alt: Identity -> Application Credentials Sidebar Menu 3. Hit the aptly named `Create Application Credential `_ button .. image:: /_images/application_credential_create.png :alt: Create Application Credential Button 4. Fill out the form specifying the application credential. You should follow the Principle of Least Privilege when deciding the authority to give to an application credential. .. image:: /_images/application_credential_creation_form.png :alt: Application Credential Creation Form 5. Download the credential either as a shell script to source or a yaml configuration to place in ``$HOME/.config/openstack/clouds.yaml`` use from the clients. .. image:: /_images/application_credential_download.png :alt: Application Credential Creation Form If you have multiple projects or application credentials, append to the ``clouds.yaml`` content and edit the yaml to identify your projects. 6. Most resources can be managed via the unified command line `client `_. This Python-based software is available for many Linux distributions, most commonly as "python3-openstackclient". If it is not available on your machine, or you would prefer to use the latest stable version, simply install it with `pip `_, e.g.: ``pip install --user openstackclient`` Depending on your local environment and preferences, you may want to add pip's directory to the search path with ``export PATH=~/.local/bin:$PATH``. To make the additional search path persistent, add the same command to ``~/.profile`` or ``~/.bash_profile``. Command line usage ------------------ To use the command line client you must either * activate the RC file for the current terminal session: .. code-block:: sh source PROJECT-openrc.sh * use the `clouds.yaml` you downloaded: .. code-block:: sh alias o='openstack --os-cloud=openstack' then use the `o` alias afterwards. At this point the ``openstack`` command is usable without any additional parameters. [#parameters-note]_ For example, the following prints a list of virtual machines in a human-readable format: .. code-block:: sh openstack server list Scripting --------- The most basic scripting option is to call ``openstack`` from a shell script, e.g. to automate tasks, generate reports, etc. The ``-f``/``--format`` and ``-c``/``--column`` options are handy when feedback is required, as they make the output easier to parse than the default table format. The following example uses the plaintext output to run a ping-check on all running servers. .. code-block:: sh #!/bin/bash for nets in $(openstack server list --status active -f value -c Networks) ; do for addr in $(echo $nets | grep -Po "\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}") ; do echo -n "Ping $addr ... " ping -w1 $addr > /dev/null && echo OK || echo FAIL done done Also of interest are the ``json`` and ``yaml`` formats, which enable further processing of the output with `jq `_ and `yq `_, respectively, or any other program that accepts these standards as input. Language bindings ~~~~~~~~~~~~~~~~~ More complicated automation is perhaps better implemented using the `bindings `_ for your favorite programming language, such as the `Python OpenStack SDK `_. Here is the Python version of the previous example: .. code-block:: python #!/usr/bin/env python3 import openstack import subprocess conn = openstack.connect(cloud='envvars') for nets in [srv.addresses for srv in conn.compute.servers(status='active')]: for net in nets.values(): for ip in net: print('Ping ' + ip['addr'] + ' ... ', end='') ping = subprocess.run('ping -w1 ' + ip['addr'], shell=True, stdout=subprocess.DEVNULL) print('OK' if ping.returncode == 0 else 'FAIL') Naturally the command-line-based procedures listed in "CLI example" boxes are also achievable via language bindings. Here is another Python-based example, based on `Quick Start `_. .. code-block:: python #!/usr/bin/env python3 import openstack conn = openstack.connect(cloud='envvars') keypair = conn.compute.create_keypair(name='KEYNAME') with open('KEYFILE', 'w') as f: f.write(keypair.private_key) conn.compute.create_server(name='SERVERNAME', image_id=conn.compute.find_image('Ubuntu 20.04').id, flavor_id=conn.compute.find_flavor('mpcdf.small').id, networks=[{"uuid": conn.network.find_network('cloud-local-1').id}], key_name=keypair.name) Alternatives ~~~~~~~~~~~~ Depending on your automation goals, it may also be worth considering the Ansible `cloud modules `_ for OpenStack or even the built-in `orchestration service `_. S3 API ------ The `object storage `_ service provides an S3-compatible API to support many common clients. Note that the `RC file <#preparation>`_ used above is **not** sufficient by itself to use the S3 API. Instead, you must generate an access/secret pair from your domain/user/project credentials with: .. code-block:: sh openstack ec2 credentials create You may create more than one pair per project, for example to avoid sharing credentials between applications, but each will grant the same level of access to the project's buckets. From here on the endpoint ``objectstore.hpccloud.mpcdf.mpg.de`` plus the keys generated above are all that is needed for many S3 clients to use the object store. Unlike the dashboard and OpenStack APIs, the S3 (and Swift) endpoints are not limited to MPG networks. For example, the `s3cmd `_ config file ``~/.s3cfg`` should contain (at least): .. code-block:: text [default] host_base = objectstore.hpccloud.mpcdf.mpg.de host_bucket = objectstore.hpccloud.mpcdf.mpg.de access_key = ACCESS secret_key = SECRET You can then create a bucket and upload a file: .. code-block:: sh s3cmd mb s3://BUCKET s3cmd put FILE s3://BUCKET/FOLDER/OBJECT Upload performance may benefit from a larger chunk size: .. code-block:: sh s3cmd --multipart-chunk-size-mb=1000 put ... There are also separate language bindings for the S3 API, such as `Boto3 `_. Here is a Python version of the above operations. .. code-block:: python #!/usr/bin/env python3 import boto3 s3 = boto3.client( 's3', endpoint_url='https://objectstore.hpccloud.mpcdf.mpg.de', aws_access_key_id='ACCESS', aws_secret_access_key='SECRET' ) s3.create_bucket(Bucket='BUCKET') s3.upload_file('FILE', 'BUCKET', 'FOLDER/OBJECT') .. note:: Not all features of `AWS S3 `_ are supported. See the Ceph `documentation `_ for details. .. [#curl-note] Technically, it is also possible to access the APIs directly via `curl `_, but this is primarily used for development and testing. .. [#parameters-note] If you prefer not to use environment variables, the credentials can also be passed explicitly, e.g.: .. code-block:: sh openstack \ --os-auth-url https://hpccloud.mpcdf.mpg.de:13000/v3 \ --os-domain-name mpcdf \ --os-project-name PROJECT \ --os-username USER ... Recipes ======= This page contains a selection of recipes to address common tasks and use-cases which are encountered in HPC Cloud projects. Although each recipe is defined in a stand-alone manner, they can be combined to achieve more complex deployments. .. toctree:: :maxdepth: 1 :glob: recipes/Get_Part_of_Tar_Archive_from_S3.md.txt recipes/S3PublicSharing.md.txt recipes/S3UsageHowto.md.txt recipes/custom-images.md.txt recipes/kubernetes.md.txt recipes/nexus-posix-mount.md.txt recipes/remoteDesktop.md.txt recipes/reverse_proxy.md.txt recipes/securing-webservers.md.txt recipes/shelving-instances.md.txt recipes/temporary-file-sharing.md.txt recipes/vnc.rst.txt # Accessing a single file in a Tar archive on S3 If you have an uncompressed (!) Tar archive stored in S3 and you want to retrieve only one or some of its files, you don't need to download and extract the whole Tar archive. As S3 supports downloading only a range of bytes of a stored file, you just need to know where in the Tar archive the wanted file is stored. ## Creating a table of contents of a Tar archive Before you upload your Tar archive to a S3 storage, you need to create a table of its content with start points and length of the files: ``` tar -tvf AA.tar -R > toctemp.txt ``` Next, you need a little awk script which calculates the length of the files and print the result together with the starting point. Save the following code as "calculateFiles.sh": ``` awk ' BEGIN{ getline; f=$8; s=$5; } { offset = int($2) * 512 - and((s+511), -512) print offset,s,f; f=$8; s=$5; }' ``` Now, pipe the content of the previous file you created into that script and save the result as "toc.txt": ``` cat toctemp.txt | /root/calculateFiles.sh > toc.txt ``` The _toc.txt_ contains now lines like this: ``` 316492288 3501474 AA/wiki_85 ``` The first number indicates the starting byte of the file, the second one its length and the third column contains the filename. Once you have uploaded the Tar archive to an S3 storage, you can now download an individual file with the following script. Don't forget to adjust the S3 configuration, name of the Tar archive and the start position and length of the file you want to retrieve: ``` import boto3 import botocore import s3credentials import sys # Initialize a S3 client: session = boto3.session.Session() s3_client = session.client( service_name='s3', aws_access_key_id=s3credentials.rdo["key"], aws_secret_access_key=s3credentials.rdo["secret"], endpoint_url=s3credentials.rdo["url"], config=botocore.client.Config(signature_version='s3'), ) # Start position and length of the file we want to get from the Tar archive: startPos = 316492288 length = 3501474 stopPos = startPos + length # Lets get the file and print it: resp = s3_client.get_object(Bucket='parttest', Key='AA.tar', Range='bytes={}-{}'.format(startPos, stopPos)) text=resp['Body'].read().decode("utf-8") print(text, end='') ``` # S3 Policies You can manage access to your buckets and objects using [policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html). Policies are JSON files. In a policy you specify a set of targets (buckets or objects), users, and operations the users are allowed to perform on the targets. A list of supported bucket and object operations are listed in the Ceph [docs](https://docs.ceph.com/en/latest/radosgw/bucketpolicy/). ## Open/public access S3 buckets can be opened to the public, allowing read and/or write functionality to any user. In this article we will explore how this can be achieved using the s3cmd client. Note: before we start it would be appropriate to warn that allowing public upload to an S3 bucket is something which should be used with utmost caution. Please be careful if you enable this. Equally, buckets which allow public downloads can quickly be discovered and scanned; choosing bucket names prefixed with a project name can help avoid them being discovered easily. ### Using S3cmd The s3cmd can be used to set bucket policies but requires that the policy be provided as a json document (no canned policies are available with s3cmd). Assuming that the json policy is stored in public-policy.json (which can be obtained via the minio client get-json command). Firstly, to ensure that no policy is set, we can use the ```s3cmd info``` command. ```bash s3cmd info s3://publictest ``` Create the policy file, `public-policy.json`, here for public access: ```json { "Statement": [ { "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Principal": { "AWS": [ "*" ] }, "Resource": [ "arn:aws:s3:::publictest" ], "Sid": "" }, { "Action": [ "s3:GetObject" ], "Effect": "Allow", "Principal": { "AWS": [ "*" ] }, "Resource": [ "arn:aws:s3:::publictest/*" ], "Sid": "" } ], "Version": "2012-10-17" } ``` Now set the policy for the bucket ``` s3cmd setpolicy public-policy.json s3://publictest ``` Check the policy ``` s3cmd info s3://publictest ``` For public (unauthenticated) users simple curl commands can be used to access the bucket. ``` curl https://objectstore.hpccloud.mpcdf.mpg.de/publictest/test.1mb \ -o download.file ``` Information about the bucket itself, including the contents, can be found by accessing the bucket URL via a web browser or curl ```bash curl https://objectstore.hpccloud.mpcdf.mpg.de/publictest/ | xmllint --format - ``` And finally to delete the policy ``` s3cmd delpolicy s3://publictest ``` ## Share with specific projects You can manage access of other HPC Cloud projects to your buckets using policies. Projects are identified by the project UUID; you can look it up with the openstack client: ```bash openstack project show -c id -f value ``` For example address a policy to the project `2a949b0cb461482ea7671b5bfdadc4f1` use the following `principal` in the policy: ```json "Principal": { "AWS": [ "arn:aws:iam::2a949b0cb461482ea7671b5bfdadc4f1:root" ] } ``` A policy allowing read access to this principal in the `test` bucket would then look like this: ```json { "Statement": [ { "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::2a949b0cb461482ea7671b5bfdadc4f1:root" ] }, "Resource": [ "arn:aws:s3:::test" ], "Sid": "" }, { "Action": [ "s3:GetObject" ], "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::2a949b0cb461482ea7671b5bfdadc4f1:root" ] }, "Resource": [ "arn:aws:s3:::test/*" ], "Sid": "" } ], "Version": "2012-10-17" } ``` # S3 Usage Howto There are several ways to view your storage utilization in S3. Here we'll summarize how to view usage on a global level and also per container/bucket using standard S3 tools. In addition a recipe is provided to allow access to S3 usage statistics following the installation of an extension to the standard aws-cli. The standard tools tend to be slower than the extension but allow for querying of individual buckets rather than the total account quota. _See the note at the end of this article about storage quotas._ ## Using standard S3 tools Firstly using the ```openstack``` command: To view global storage statistics ``` $ openstack object store account show +------------+---------+ | Field | Value | +------------+---------+ | Account | v1 | | Bytes | 1048604 | | Containers | 2 | | Objects | 2 | +------------+---------+ ``` To view container level statistics: ``` $ openstack container show mycontainer +--------------+-------------+ | Field | Value | +--------------+-------------+ | account | v1 | | bytes_used | 28 | | container | mycontainer | | object_count | 1 | +--------------+-------------+ $ openstack container show mycontainer2 +--------------+--------------+ | Field | Value | +--------------+--------------+ | account | v1 | | bytes_used | 1048576 | | container | mycontainer2 | | object_count | 1 | +--------------+--------------+ ``` Using the ```s3cmd``` command: To gain a global view ``` $ s3cmd du 28 1 objects s3://mycontainer/ 1048576 1 objects s3://mycontainer2/ ------------ 1048604 Total ``` ``` $ s3cmd du s3://mycontainer/ 28 1 objects s3://mycontainer/ ``` To get human readable numbers ``` $ s3cmd du -H 28 1 objects s3://mycontainer/ 1024K 1 objects s3://mycontainer2/ ------------ 1024K Total ``` Note: working with sub-containers/buckets Although S3 containers/buckets are the real units of storage many clients allow you to address data within these buckets in a similar manner to a filesystem (with / being used as a separator for sub-containers/buckets similar to a directory). To view the storage usage of sub-containers/buckets you can use the s3cmd command. ``` $ s3cmd ls s3://mycontainer2/phase2/ 2021-08-31 09:37 0 s3://mycontainer2/phase2/ 2021-08-31 09:40 1048576 s3://mycontainer2/phase2/1mb_testfile_1 2021-08-31 09:40 1048576 s3://mycontainer2/phase2/1mb_testfile_2 ``` Using the s3cmd: ``` $ s3cmd du s3://mycontainer2/ 3145728 4 objects s3://mycontainer2/ ``` Viewing the sub-container/bucket ``` $ s3cmd du s3://mycontainer2/phase2/ 2097152 3 objects s3://mycontainer2/phase2/ ``` For human readable output ``` $ s3cmd -c s3cfg.dev2 du s3://mycontainer2/phase2/ -H 2048K 3 objects s3://mycontainer2/phase2/ ``` ## Using the usage-stats extension Ceph's RadosGW provides a custom S3 API endpoint that can be used to efficiently retrieve that information (similar to df instead of du): [https://docs.safespring.com/storage/usage-statistics/](https://docs.safespring.com/storage/usage-statistics/) Once installed the extension can be used from the aws client as follows: ``` $ aws --profile --endpoint-url https://objectstore.hpccloud.mpcdf.mpg.de s3api get-usage-stats { "Summary": { "QuotaMaxBytes": 214748364800, "QuotaMaxBuckets": 1000, "QuotaMaxObjCount": 51200, "QuotaMaxBytesPerBucket": -1, "QuotaMaxObjCountPerBucket": -1, "TotalBytes": 1168113870, "TotalBytesRounded": 1168130048, "TotalEntries": 16 } } ``` The extension can be enabled by placing the ceph-specific json api definition file from [1] in ```~/.aws/models/s3/2006-03-01/``` as described in [2]. After the extension has been installed the following python code can also be used to gather the usage-stats: ```python #!/usr/bin/python3 import boto3 import json # S3 connection details session = boto3.session.Session(profile_name='') s3 = session.client( service_name='s3', endpoint_url='https://objectstore.hpccloud.mpcdf.mpg.de' ) print(json.dumps(s3.get_usage_stats(), indent=2)) ``` To enable the usage of profiles, as per the python code above, please set your profile in ```~/.aws/credentials```. ## Storage Quotas. It is important to remember that the S3 quotas apply to a whole openstack project and not individual users which are granted access to that project. Also note that Quotas are applied to both Total Storage Volume and Number of Objects [1] https://raw.githubusercontent.com/ceph/ceph/master/examples/rgw/boto3/service-2.sdk-extras.json [2] https://github.com/ceph/ceph/tree/main/examples/rgw/boto3 # Custom Image Creation This recipe shows how a custom image can be created using the Packer tool. ## Packer Packer is a cloud agnostic tool to automate image builds. For more details see [Packer](https://www.packer.io/) Packer allows you to create a coded (documented) image creation template. You can use it to reproducibly create images, evolve their configuration and commit this all into a version control system. In the mid-term it will make your image creation faster and more reliable. Here we're just going to cover a basic example by following the steps below. - Install Packer - Configure the openstack environment (command line) - Create a Packer template file - Run the build command ## Install Packer Packer is available for numerous operating system releases out of the box. For more info check: [Packer Download](https://www.packer.io/downloads) ## Configure Openstack environment To configure your environment simply source the rc file from your openstack account ``` . openrc.sh ``` Packer can take advantage of the environment variables which this creates. Note: For the mpcdf release you will need to remember that packer uses ```OS_TENANT_NAME``` or ```OS_TENANT_ID``` rather than ```OS_PROJECT_NAME``` and ```OS_PROJECT_ID```. This means you will need to explicitly set the tenant as we do in the config file below or modify the openrc.sh file to export both ```*_PROJECT_*``` and ```*_TENANT_*```. Once you have the environment setup you can use the openstack command line tool to interact with openstack and determine configuration variables etc. for the template. ## Create a Template The template file is a simple text config file in json or HCL. In the example below we will use the HCL format. The definition of the template will use information that can be gained from the OS_* environment vars or via queries to openstack itself. We will look at the two main blocks needed to define a template file. The source block and the build block. ### Source Block The source block contains openstack related information which may be static (related to the openstack instance and project) or more dynamic (related to the base image on which to build etc). For instance: ``` SOURCE_ID=`openstack image list -f json | jq -r '.[] | select(.Name == "Ubuntu 20.04") | .ID'` FLAVOR_ID=`openstack flavor list -f json | jq -r '.[] | select(.Name == "mpcdf.small") | .ID'` NETWORK_ID=`openstack network list -f json | jq -r '.[] | select(.Name == "cloud-local-1") | .ID'` ``` And the openstack environmental variables: ``` OS_REGION_NAME OS_PROJECT_DOMAIN_ID OS_INTERFACE OS_AUTH_URL OS_USERNAME OS_PROJECT_ID OS_USER_DOMAIN_NAME OS_TENANT_NAME OS_PASSWORD OS_IDENTITY_API_VERSION ``` Some of these are read from the environment variables by packer (many are explicitly defined in the config below but omit the OS_PASSWORD to ensure this isn't recorded plain text in a file). For more info see: [Packer Openstack Builder](https://www.packer.io/docs/builders/openstack) Example (with some redaction): ``` source "openstack" "autogenerated_1" { flavor = "1011" identity_endpoint = "https://rdocloud.mpcdf.mpg.de:13000" image_name = "CustomImage" networks = ["***-***-***-***"] region = "regionOne" source_image = "c42847a8-3456-43d4-82bb-5ad05f402d7a" ssh_ip_version = "4" ssh_username = "root" tenant_id = "**************" username = "**********" } ``` The entries in the above template are relatively self explaining and some can actually be omitted but are included here to help you get a better insight into the process. See the packer docs for more info about which variables can read from the OS_ environment: [Packer Openstack Builder](https://www.packer.io/docs/builders/openstack) ### Build Block The build block defines which source should be used for the build process and then how to provision the image. Provisioning is the step where the virtual machine is configured. Tasks such as software installation and adaption of config files are undertaken during the image provisioning step. There are several different types of provisioners, here we will show an ansible and shell provisioner Ansible Example: ``` build { sources = ["source.openstack.autogenerated_1"] provisioner "ansible" { playbook_file = "provision-template.yml" } } ``` ```provision-template.yml``` is a standard ansible playbook. Shell Example: ``` build { sources = ["source.openstack.autogenerated_1"] provisioner "shell" { script = "script.sh" } } ``` ```script.sh``` is a simple shell script which will be uploaded and executed on the machine being provisioned. You can also supply an array or several scripts to be executed or define explicit commands to be executed via the inline option. ### Run the Build To run the build process simply call packer build with the template file ``` packer build mytemplate.pkr.hcl ``` This will create a temporary instance based on the base image and provision it using the provisioner you defined. In a final step the new image will be created using the image name you defined in the template, in this case "CustomImage". You can check the image after creation using: ``` openstack image list ``` # Kubernetes in the HPC Cloud You can deploy Kubernetes on the HPC Cloud. We provide a recipe using OpenStack HEAT as well as step-by-step instructions to set up a cluster using the command line interface or the cloud dashboard. The instructions are on GitLab: | setup | step-by-step | (deprecated) HEAT template | | --- | --- | --- | | Production | [production](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/kubernetes/-/tree/production/step-by-step) | (deprecated) [production](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/kubernetes/) | | Testing | [dev](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/kubernetes/-/tree/dev/step-by-step) | (deprecated) [dev](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/kubernetes/-/tree/dev) | The HEAT template has been deprecated. # The NexusPOSIX Filesystem ## Mount on a cloud server This guide assumes you have already created a VM running Ubuntu 20.04 and noted its IP address (e.g. *10.186.xx.xx*). For details about how to launch a new VM and connect as *root*, see the general [documentation](../quickstart/). Additionally you will need to know the project directory assigned in Nexus-Posix and the Linux group the project was created with. These values will be used in the following configuration to ensure the NFS mount points are correctly configured and that the VM users have access to the mounted Nexus-Posix filesystem. ### Procedure 1. Initial setup to be performed by an admin of the VM: a. Install the necessary nfs client software packages: ```sh apt update apt upgrade -y apt install -y nfs-client ``` b. Configure uid and gid mapping: ```sh sed -i.bak 's/# Domain = localdomain/Domain = mpcdf.mpg.de/' /etc/idmapd.conf echo "options nfs nfs4_disable_idmapping=N" >> /etc/modprobe.d/nfs.conf ``` Now `reboot` the VM. c. [Request](../../../../faq/help.html#how-can-i-get-help-and-support) a Nexus-Posix project directory and dedicated network. Use the [dashboard](https://hpccloud.mpcdf.mpg.de/dashboard/project/instances/) to attach a second interface (not floating ip) in the *nexus-private-...* network to the VM. ```sh echo "nexus-posix0.hpccloud.mpcdf.mpg.de:/nexus/posix0/PROJECT_DIR /nexus/posix0/PROJECT_DIR nfs _netdev" >> /etc/fstab mkdir -p /nexus/posix0/PROJECT_DIR mount /nexus/posix0/PROJECT_DIR ``` where PROJECT_DIR is the name of the project in Nexus-Posix. d. Install sssd for user identities: ```sh apt install -y sssd ``` e. Create the sssd configuration (/etc/sssd/sssd.conf) ```sh [sssd] config_file_version = 2 domains = CUSTOM [nss] filter_users = root filter_groups = root [pam] offline_credentials_expiration = 3 [domain/CUSTOM] id_provider = ldap access_provider = ldap ldap_search_base = ou=general-u,ou=ua,o=rzg,c=de ldap_access_filter = (gidNumber=GROUP_ID) ldap_uri = ldap://directory1.hpccloud.mpcdf.mpg.de/,ldap://directory2.hpccloud.mpcdf.mpg.de/ auth_provider = krb5 krb5_realm = IPP-GARCHING.MPG.DE krb5_server = kerberos.mpcdf.mpg.de,kerberos1.mpcdf.mpg.de,kerberos2.mpcdf.mpg.de,kerberos3.mpcdf.mpg.de cache_credentials = true enumerate = true min_id = 1000 override_homedir = /home/%u ``` where GROUP_ID is the ID of the Linux group the Nexus-Posix project was created with. Note: for some Operating systems and variants (e.g. CentOS, AlmaLinux) the sssd pam and nss responder services need to be explicitly started by sssd, by adding them to the config file, or socket activation needs to be enabled. ```sh [sssd] services = nss, pam ... ``` or ```sh systemctl enable sssd-nss.socket systemctl start sssd-nss.socket systemctl enable sssd-pam.socket systemctl start sssd-pam.socket ``` f. Complete the sssd configuration ```sh chmod -R 0600 /etc/sssd/* systemctl restart sssd pam-auth-update --enable mkhomedir ``` Note: Overriding the home dir to /home/ is useful for login nodes/workstations. g. Re-boot the VM ```sh shutdown -r now ``` A re-boot of a VM, once initially deployed, helps ensure that the configuration is correct and that the VM will correctly re-start in case of any outages. 2. Users may now read and write to the project directory from both the VM and *Raven*. ## Quota and usage From our parallel systems (Raven, etc) where the NexusPOSIX file system is mounted, you can check how much space and how many inodes your project is entitled to and using. To do so use the standard linux tools: ```sh df -h /nexus/posix0/ df -i /nexus/posix0/ ``` # Remote desktop connection This guide is specific for users of Windows images. Connecting to a Linux desktop remotely will function analogously, just that the server-side setup will of course be different. You can upload your own Windows images to the HPC cloud or request access to the Windows 10 Professional image we have set up. When creating your own image be aware that Windows does not come with drivers that can handle the devices provided by the cloud infrastructure. Working Windows drivers for these devices are provided by the Fedora Project's [virtIO driver disk](https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md). ## Connecting After booting a Windows Instance you can interact with it through the console interface provided by OpenStack. You can get the URL of the console with: ```sh INSTANCE_NAME="myWin" openstack server create "$INSTANCE_NAME" --image "Windows 10 Pro" --flavor mpcdf.medium --network cloud-local-2 openstack console url show "$INSTANCE_NAME" --novnc -c url -f value ``` and connecting to the resulting URL with your browser or by clicking on the instance in the dashboard and selecting the console tab. Use this console for the initial configuration of the Windows instance. ## Remote Desktop You can configure Windows to provide a remote desktop separate from the OpenStack console. Microsoft provides detailed [documentation](https://support.microsoft.com/en-us/windows/how-to-use-remote-desktop-5fe128d5-8fb1-7a23-3b8a-41e636865e8c); here is just a summary: - go to Start > Settings > System > Remote Desktop - enable the slider ### Virtual Network Computing (VNC) You can of course also install a VNC server on the machine. Then the ports will work slightly differently. Authentication will slightly differ a bit too. ### Permitting External Access By default the remote desktop is available at port 3389, create a security group allowing traffic to this port from the gateway machines: gate1 has address 130.183.12.24 and gate2.mpcdf.mpg.de has address 130.183.12.25. Here is how this could be done with the command line: ```sh INSTANCE_NAME="myWin" SECGROUP_NAME="Allow RDP from the gateways" openstack security group create "$SECGROUP_NAME" openstack security group rule create "$SECGROUP_NAME" --dst-port 3389 --protocol tcp --remote-ip 130.183.12.16/28 --description "Allow RDP from the gateways" openstack server add security group "$INSTANCE_NAME" "$SECGROUP_NAME" ``` Look up your instance IP address from the dashboard or with: ```sh o server show $INSTANCE_NAME -c addresses -f value ``` Then establish an ssh connection through the gateways: ```sh ssh -f -N -L 4321:IP:3389 gate2 ``` You can now connect to the Windows machine by pointing your remote desktop client to `localhost:4321`. # Reverse Proxy This recipe aims to explain what a reverse proxy is, why you may want to use one, and your options for setting one up in the HPC Cloud. ## Introduction A reverse proxy acts as an intermediary that forwards the client's requests to one or more different internal services, such as a web server or WordPress blog. A reverse proxy is placed near the server serving client requests. Reverse proxies are used to improve service security and stability. A reverse proxy may balance load across, cache content from, or simply redirect traffic to a number of servers. In addition we encourage projects to use reverse proxies to conserve the limited number of IPv4 addresses we have available. ## Setup Depending on your requirements, we provide two recipes: - Define a proxy using the OpenStack LoadBalancer service: [cloud-native](#Cloud-Native) - Manage your own instance running NGINX configured as reverse proxy: [classic](#Classic) Going the cloud native way saves you another VM to manage. The classic route allows you to do TLS termination on the proxy [^1]. ## Cloud Native The following steps outline how to test and deploy a reverse proxy using the OpenStack CLI. This setup demonstrates the use of a reverse proxy to expose two hypothetical web servers running in the same private network. Only a single public IP address is required, which can be allocated through the OpenStack dashboard (GUI). ### Create private network Set the following variables to your preferences: ```bash FLOATING_IP="" DOMAIN_1="www.mysite01.com" DOMAIN_2="www.mysite02.com" SUBNET_RANGE="192.168.0.0/28" ``` Assume that the domain names for the services are registered in DNS. This is meant to ease the use of the instructions below. You can, of course, enter values in place of using the variables below. ```bash openstack network create priv-net00 openstack subnet create sub-net00 --network priv-net00 \ --subnet-range "${SUBNET_RANGE}" \ --dns-nameserver 130.183.9.32 --dns-nameserver 130.183.1.21 ``` Create the following route to allow access to public internet for instance updates and installation ```bash openstack router create rout00 openstack router set rout00 --external-gateway cloud-public openstack router add subnet rout00 sub-net00 ``` ### Create servers Launch two virtual machines (VMs) within the previously created private network. Each VM should have Apache HTTP Server installed and running to serve as a basic web server for testing the reverse proxy setup. Using the following user data script ensures that Apache is installed and running as soon as the VM boots up. ```bash cat < web-init.sh #!/bin/bash DEBIAN_FRONTEND=noninteractive apt-get update DEBIAN_FRONTEND=noninteractive apt-get upgrade -y DEBIAN_FRONTEND=noninteractive apt-get install -y apache2 systemctl start apache2 systemctl enable apache2 IP=\$(hostname -I | awk '{print \$1}') echo "Hallo from my web server: \$IP" | tee /var/www/html/index.html EOF ``` Using the following commands to create the VMs ```bash openstack server create server01 --image "Ubuntu 24.04" --flavor mpcdf.small \ --network priv-net00 --security-group web\ --security-group default --user-data web-init.sh openstack server create server02 --image "Ubuntu 24.04" --flavor mpcdf.small \ --network priv-net00 --security-group web\ --security-group default --user-data web-init.sh ``` ### Create Loadbalancer The reverse proxy is implemented using the OpenStack Load Balancer service (Octavia). This allows incoming requests to a single public IP address to be intelligently routed to the appropriate backend Apache web servers. The load balancer acts as a reverse proxy by distributing traffic based on defined listener rules and pool configurations. ```bash pip install python-octaviaclient openstack loadbalancer create --name load-bal00\ --vip-subnet-id sub-net00\ --vip-address "${FLOATING_IP}" openstack loadbalancer show load-bal00 ``` ### Create Listener Create a listener on the load balancer to handle incoming HTTP requests. The listener defines the protocol and port on which the load balancer will accept traffic—typically HTTP on port 80. This can be changed according to the incoming traffic. ```bash openstack loadbalancer listener create --name http-listener01\ --protocol HTTP --protocol-port 80 load-bal01 ``` ### Create a pool Normally, pools are used to group multiple backend servers for redundancy and load distribution. However, in this setup, each pool will contain only a single server to simulate different services behind the reverse proxy. ```bash openstack loadbalancer pool create --name web-pool01\ --lb-algorithm ROUND_ROBIN --loadbalancer load-bal01 openstack loadbalancer pool create --name web-pool02\ --lb-algorithm ROUND_ROBIN --loadbalancer load-bal01 ``` ### L7 policy To route traffic to the correct backend server based on the requested URL path, create **L7 policies** and **rules**. These policies inspect incoming HTTP requests and redirect them to the appropriate pool based on path patterns. ```bash openstack loadbalancer l7policy create --name l7-mysite01\ --action REDIRECT_TO_POOL --redirect-pool web-pool01 http-listener01 openstack loadbalancer l7policy create --name l7-mysite02\ -action REDIRECT_TO_POOL --redirect-pool web-pool02 http-listener01 openstack loadbalancer l7rule create --type HOST_NAME\ --compare-type EQUAL_TO --value "${DOMAIN_1}" l7-mysite01 openstack loadbalancer l7rule create --type HOST_NAME\ --compare-type EQUAL_TO --value "${DOMAIN_2}" l7-mysite02 ``` ### Populate the pool Now that the pools are created, you need to add your web servers (VMs) as members of each pool. Each member represents one of your backend Apache servers that will serve requests. ```bash VM_IP1=$(openstack server show server01 -f value -c addresses | grep -oP "(?<=\[')[^']+(?='\])") VM_IP2=$(openstack server show server02 -f value -c addresses | grep -oP "(?<=\[')[^']+(?='\])") openstack loadbalancer member create --subnet-id sub-net00\ --address "${VM_IP1}" --protocol-port 80 web-pool01 openstack loadbalancer member create --subnet-id sub-net00\ --address "${VM_IP1}" --protocol-port 80 web-pool02 ``` ### Testing To test the reverse proxy setup and verify that the internal IP addresses of the backend web servers are different while the public IP remains the same, you can open a web browser and go to the corresponding web url. The private IP shows up for both and they are different. This means they all point to different internal servers. You can also verify that both domains point to the same public IP by using https://www.nslookup.io/website-to-ip-lookup/. ## Classic ## Network First set up a private network: https://docs.mpcdf.mpg.de/doc/cloud/technical/network.html#private-networks-and-routers For the sake of this example we assume the following: - `$NET_NAME` resolves to the network name - `SUBNET_NAME` resolves to the subnet name - `192.168.0.0/24` is the subnet ip range The network section in the cloud documentation has instructions on how to setup a network. ### Create security groups and rules We create two security groups: one to add to the proxy the other to add to the nodes hosting the applications. Then add a rule allowing all `tcp` traffic from the proxy to the machines hosting the applications. Finally add rules to the proxy to restrict traffic as is appropriate for the application, in this case: - Allow `HTTP` (port `80`) access from the internet (`0.0.0.0/0`) to support certbot certificate verification - Allow `HTTPS` (port `443`) from the internet (`0.0.0.0/0`) to access our web services from anywhere - Allow `SSH` (port `22`) from local MPCDF networks (`10.0.0.0/8` and `130.183.0.0/16`) ```bash # create security groups openstack security group create apps openstack security group create proxy # allow all traffic from proxy to apps openstack security group rule create apps --remote-group proxy --protocol tcp # allow access to proxy openstack security group rule create proxy --remote-ip "0.0.0.0/0" --protocol=tcp --dst-port=80 openstack security group rule create proxy --remote-ip "0.0.0.0/0" --protocol=tcp --dst-port=443 openstack security group rule create proxy --remote-ip "10.0.0.0/8" --protocol=tcp --dst-port=22 openstack security group rule create proxy --remote-ip "130.183.0.0/16" --protocol=tcp --dst-port=22 ``` ### Network configuration Now create network ports for your application servers and the proxy and associate the appropriate security groups ```bash openstack port create proxy \ --network $NET_NAME \ --fixed-ip "subnet=$SUBNET_NAME,ip-address=192.168.0.3" \ --security-group=proxy openstack port create app1.0 \ --network $NET_NAME \ --fixed-ip "subnet=$SUBNET_NAME,ip-address=192.168.0.10" \ --security-group=apps openstack port create app2.0 \ --network $NET_NAME \ --fixed-ip "subnet=$SUBNET_NAME,ip-address=192.168.0.20" \ --security-group=apps ``` #### External access Get a floating IP from the `cloud-public` network and associate it with the port of the proxy port: ```bash openstack floating ip create cloud-public --port proxy ``` #### Server Names You can use the DNS names we provide for your HPC Cloud project [^2]. If you control your own domain add an `A` record pointing to the floating IP you assigned to the proxy. You can look up the value of the floating ip from the dashboard or using the openstack CLI: ```bash openstack floating ip list --fixed-ip-address 192.168.0.3 ``` ### Create the servers Create the proxy server: ```bash openstack server create proxy --flavor=mpcdf.small --image="Ubuntu 24.04" --key-name=fberg --port=proxy openstack server create app1.0 --flavor=mpcdf.small --image="Debian 12" --key-name=fberg --port=app1.0 openstack server create app2.0 --flavor=mpcdf.small --image="Debian 12" --key-name=fberg --port=app2.0 ``` ## Configure the Reverse proxy SSH into your proxy using the floating IP: ```shell PROXY_IP="$(o floating ip list --fixed-ip-address 192.168.0.3 -f value -c 'Floating IP Address')" ssh "$PROXY_IP" -l root ``` Install nginx [^3] and certbot [^4]. The instructions on certbot also show you how to run certbot to get a certificate for the domain you setup. It should configure an entry for the domain in the nginx configuration. Look for an entry with `server_name` set to the domain name(s) you set up above. From the proxy you can also SSH into the instances running your applications. The following example assumes that your applications will be available at port 80 on their servers. ### Single domain Suppose you want your applications to be available at `https://examnple.com/app1` and `https://examnple.com/app2`. When you asked for the certificate, certbot should have generated a virtual host entry in the NGINX configuration for `server_name example.com`; look for it. It should look something like this: ```nginx server { server_name example.com; # managed by Certbot root /var/www/html; ... location / { ... } listen 443 ssl; # managed by Certbot listen [::]:443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/app1.example.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/app1.example.com/privkey.pem; # managed by Certbot ... } ``` Now add two new location blocks, so the server entry looks like this: ```nginx server { server_name example.com; # managed by Certbot root /var/www/html; index index.html index.htm index.nginx-debian.html; .... location / { ... } location /app1 { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://192.168.0.10/; } location /app2 { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://192.168.0.20/; } listen 443 ssl; # managed by Certbot listen [::]:443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/app1.example.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/app1.example.com/privkey.pem; # managed by Certbot ... } ``` You may also want to disable buffering on the proxy with `proxy_buffering off;`. NGINX provides documentation for the many reverse proxy settings[^5][^6]. Don't touch the lines `# managed by Certbot`. Restart nginx: ```bash systemctl restart nginx ``` Now going to `https://example.com/app1` with your browser will get to the nginx proxy and be redirected to the application running on the server `app1.0`. Similarly, `https://example.com/app2` will be directed to the application running on the server `app2.0`. Note that this configuration assumed your applications will be available at port 80 on their servers. ### Multiple domains Suppose you have setup A records for your applications called `app1.example.com` and `app2.example.com`. Each domain will need a virtual host in the NGINX configuration. You should find an entry in your NGINX configuration looking like this: ```nginx server { server_name app1.example.com; # managed by Certbot root /var/www/html; index index.html index.htm index.nginx-debian.html; location / { ... } listen 443 ssl; # managed by Certbot listen [::]:443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/app1.example.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/app1.example.com/privkey.pem; # managed by Certbot ... } ``` Edit the location entry for `/`: ```nginx location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://192.168.0.10/; } ``` Where `192.168.0.10` is the IP of the instance running app1. For app2, it would then look like this: ```nginx server { server_name app2.example.com; # managed by Certbot root /var/www/html; index index.html index.htm index.nginx-debian.html; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://192.168.0.20/; } listen 443 ssl; # managed by Certbot listen [::]:443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/app2.example.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/app2.example.com/privkey.pem; # managed by Certbot ... } ``` In this setup I assume you grabbed separate certificates for your domains. `certbot` lets you get one certificate for multiple domains as well by repeating the `-d` option. In that case the `ssl_certificate` and `ssl_certificate_key` entries will look slightly different. Restart nginx: ```bash systemctl restart nginx ``` Now going to `https://app1.example.com` with your browser will get to the nginx proxy and be redirected to the application running on the server `app1.0`. Similarly, `https://app2.example.com` will be directed to the application running on the server `app2.0`. #### Notes You may also want to disable buffering on the proxy with `proxy_buffering off;`. NGINX provides documentation for the many reverse proxy settings[^4][^5]. Don't touch the lines `# managed by Certbot`. [^1]: [wikipedia.org](https://en.wikipedia.org/wiki/TLS_termination_proxy): TLS termination proxy [^2]: [docs.mpcdf.mpg.de](https://docs.mpcdf.mpg.de/doc/cloud/technical/network.html#automated-domain-name-service-dns) Automated domain name service [^3]: [nginx.org](https://nginx.org/en/linux_packages.html) installation instructions [^4]: [certbot.eff.org](https://certbot.eff.org) instructions [^5]: [docs.nginx.com](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) setting up an NGINX reverse proxy [^6]: [nginx.com](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) NGINX proxy module parameter reference # Securing Webservers The HPC Cloud operates under a shared responsibility model w.r.t server and service security. This means that MPI Project admins are responsible for ensuring that the services they open to the internet are as secure as possible and are also regularly checked and upgraded. Online services exist to both generate secure example configuration and to test any service which is exposed to the public internet. ## Configuration generator To generate secure configuration the following tool from Mozilla is often useful: Mozilla SSL config generator: (note this also provides example configs for databases such as MySQL and PostgreSQL) The highest level of protection "Modern" is obviously advised if possible. However, in many real life cases the "Intermediate" Level protection is a good compromise between security and access (ensuring most clients can access the service). ## Scanning Services When scanning webservers the following webpage is simple to use and provides detailed information about the security level of the scanned server. ssl labs page: Moreover several command line scanning tools exist including: - - _Note:_ The command line tools can be especially useful when services are exposed on non-standard ports and/or when you wish to scan a service which has yet to be open to the public internet (in general we would advise scanning locally before opening the service to the public internet). # Shelving Instances (by hand) When an instance of a VM is stopped the resources it requires are not released to the project quotas. This is a design feature that ensures a stopped instance can always be restarted, since the resources it needs are still available. In some cases it is useful to stop an instance and release the resources, thus allowing projects to deploy other instances. In openstack this is referred to as "shelving". The current deployment of openstack at MPCDF does not fully support shelving and although it can be used to stop an instance the quotas are not updated. This recipe details how an instance can be shelved by hand as a work-around. This process consists of 3 parts. 1. Creating a snapshot of a VM 2. Deleting the VM 3. Re-creating the VM once it is again required ## Creating a Snapshot Stop the VM (ensuring no disk I/O is active) and then create a snapshot with a meaningful name. ``` $ openstack server stop $ openstack server image create --name -snapshot $ openstack image list ``` Wait until the image is in the "active" state before continuing to delete the VM instance. ## Deleting the VM Before deleting the VM check the current configuration; instance flavor, floating IP, security groups, networks etc. This information will be needed if the VM is to be re-created with the same configuration as the existing instance. Then delete the VM. ``` $ openstack server delete ``` Now the resources will have been returned to the project quotas. ## Re-creating the VM The VM instance can be re-created by creating an instance using the snapshot as the image source. ``` $ openstack server create --flavor --image -snapshot --network ``` Take care to assign all the needed infrastructure to the newly created VM instance; networks, security groups, floating IPs etc. Ensure that the flavor of the new instance is equal to, or larger, than the original instance w.r.t the boot disk size. # Temporary File Sharing A common use-case for web applications is the need to asynchronously run a long running process for a user and then provide the user with a download link where they may collect their data. This recipe will combine two of the more advanced s3 functionalities to address this use-case. Namely: - Temporary urls for short lived shares - Bucket lifecycles to enable automatic object deletion after a set lifetime The s3 object storage service allows users to generate a temporary url which can be shared with colleagues and collaborators, enabling anonymous access to a data object for a restricted amount of time. This can be thought of as a short-lived-share. Bucket lifecycles can be implemented to provide a lifetime for uploaded objects. Any uploaded object will be deleted n days after upload (where n is configurable). In this recipe we will step through the creation of a bucket with a lifecycle, uploading of objects and generation of temporary URLs. Although this can all be achieved using command line clients we will explore using a python script for the object upload and temporary URL generation. ## Create a bucket with a lifetime policy The lifecycle policy documented here marks any object added to the tempstore bucket for deletion after one day. The lifecycle application is run in the background and asynchronously. That means the time of deletion is fuzzy, but always happens some time after the expiry date is reached. ### Using s3cmd Make a bucket ```bash s3cmd mb s3://tempstore ``` Create a lifecycle policy called `delete-1day.xml`. Here is an example for a policy where objects will be deleted after one day: ```xml Delete-After-1-Day Enabled 1 ``` And apply the policy to your bucket: ```bash s3cmd setlifecycle delete-1day.xml s3://tempstore ``` Check that the policy has been applied: ```bash s3cmd getlifecycle s3://tempstore s3cmd info s3://tempstore ``` ### Using MinIO client Similarly with the minio-clients (where the alias is mys3) Create a bucket with a lifecycle ```bash mc mb mys3/tempstore ``` Set a lifecycle policy where objects are marked to be deleted after one day: ```bash mc ilm add mys3/tempstore --expiry-days "1" ``` Check the lifecycle ``` mc ilm ls mys3/tempstore mc ilm export mys3/tempstore ``` ## Create temporary URLs The commands below generate a URL which may be used to fetch the file via a browser or command line tool such as `curl`. The URL is only valid for a limited amount of time. Downloads attempted after the expiry date of the URL will result in an `Access Denied` message. The `s3cmd` or MinIO client commands could easily be wrapped in a script to create temporary URLs. It is also possible for us to make use of the python API via `boto3`. Below is a small example script which will both upload an object and create a temporary URL. ### Using s3cmd: To generate a temporary URL which is valid for the next 10 mins ``` s3cmd signurl s3://tempstore/temporary_file.txt +600 ``` Note: It is advisable to set https support in the .s3cfg file ```python use_https = True signurl_use_https = True ``` ### Using the MinIO clients: For info see: ``` mc share download -h ``` To create a temporary URL allowed to read the object: ``` mc share download mys3/mctempstore/temporary_file.txt ``` ### Using Boto ```python #!/usr/bin/env python3 import boto3 import click # temp-share - example - upload object and generate temp url # John Alan Kennedy 2022 def put_object(s3, bucket, key): s3.put_object(Bucket=bucket, Key=key, Body=open(key, "rb")) def generate_presigned_url(s3, bucket, key, expires): """Generate pre-signed URL for a key in bucket. :param s3: s3_client connection :param key: Name of the S3 object :param bucket: Name of the S3 bucket :param expires: Time to expiration of temp url """ url = s3.generate_presigned_url( ClientMethod="get_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=expires, ) print("{0!s}{1!s}".format(key, url)) @click.command() @click.argument("bucket", required=True) @click.argument("key", required=True) @click.option("--expires", "-e", default=600, help="Time to expire in seconds.") def main(bucket, key, expires): # S3 connection details session = boto3.session.Session(profile_name="rdoopenstack") s3 = session.client( service_name="s3", endpoint_url="https://objectstore.hpccloud.mpcdf.mpg.de" ) put_object(s3, bucket, key) generate_presigned_url(s3, bucket, key, expires) if __name__ == "__main__": main() ``` The script can be called as follows: ```bash temp-share tempstore temporary_file.txt ``` The `--expires` option can be added to define the URL validity lifetime. The script is very basic. However, it is easy to see how this could form the basis of a more complex application. VNC Deployment Recipe ===================== This recipe documents the deployment of a VNC server on a cloud VM This guide assumes you have already created a VM running Ubuntu 20.04, noted its IP address (e.g. *10.186.xx.xx*), and plan for both admin(s) and users to connect via the `gateway machines <../../../computing/gateways.html>`_. For details about how to launch a new VM and connect as *root*, see the general `documentation <../quickstart.html>`_. Procedure --------- 0. Initial setup for all VMs: Create a `security group <../network.html#security-groups>`_ called *vnc-gates*. Its purpose is to allow VNC connections from the `gateway machines <../../../computing/gateways.html>`_ to the VNC server(s). The IP range of the gateway machines is: 130.183.12.24/31. VNC usually requires TCP ports 5900 to 5900+N, where N is the number of separate displays. .. code-block:: sh openstack security group create vnc-gates openstack security group rule create --protocol tcp --dst-port 5900:6000 --remote-ip 130.183.12.24/31 vnc-gates 1. Initial setup to be performed by an admin of the VM: Install the necessary software packages: .. code-block:: sh apt update apt upgrade -y apt install -y xorg icewm nautilus eog evince firefox libturbojpeg update-alternatives --set x-terminal-emulator /usr/bin/xterm wget -O /tmp/turbovnc_2.2.6_amd64.deb https://sourceforge.net/projects/turbovnc/files/2.2.6/turbovnc_2.2.6_amd64.deb/download apt install -y /tmp/turbovnc_2.2.6_amd64.deb At this point it is a good idea to `reboot` the VM. Add the pre-made *vnc-gates* security group to the VM. 2. Initial setup performed by the each user: .. code-block:: sh mkdir ~/.vnc cat << EOF > $HOME/.vnc/xstartup #!/bin/sh unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS exec icewm-session EOF chmod 755 ~/.vnc/xstartup cp ~/.vnc/xstartup ~/.vnc/xstartup.turbovnc /opt/TurboVNC/bin/vncpasswd 3. Users can now launch their own VNC sessions by running `/opt/TurboVNC/bin/vncserver -autokill` on the VM and then `vncviewer -via MPCDF_USER@gate.mpcdf.mpg.de 10.186.XX.XX::PORT` from their personal computer. The port number is defined as the display number added to 5900, e.g. *5901* for display *:1*, *5902* for display *:2*, and so on. Alternative (tunneled connection) --------------------------------- A slightly more secure solution is to bind the vnc session to the localhost on the VM and use an ssh tunnel directly to the VM to gain access. This way no VNC connection is required from the gate.mpcdf.mpg.de node to the VM serving the VNC. 1. Launch the VNC session on the VM as follows `/opt/TurboVNC/bin/vncserver -localhost -autokill`. 2. Identify the VNC port `/opt/TurboVNC/bin/vncserver -list` on the VM 3. Create an ssh tunnel from your local server to the VM `ssh -L 2345:localhost:PORT -J @gate.mpcdf.mpg.de ` 4. Connect to the vnc server from your local server `vncviewer localhost::2345` # Frequently Asked Questions ```{contents} Contents :local: :depth: 2 ``` ### How to boot a server from a volume? 1. Create a bootable volume from an image ```sh openstack volume create --image="IMAGE" --size="SIZE" "DEMO_ROOT" ``` 2. Create a new instance ```sh openstack server create "DEMO" --flavor="FLAVOR" --volume="DEMO_ROOT" --network="NETWORK" --key-name "KEY" ``` ### How to get a larger root disk for my server? Yes. By default instances get a 25G root disk. If you need more space on your root drive boot from a volume. Below are the necessary steps for two cases: if you need more space on an existing instance or know that the server you are about to boot will need more room. This will guide you through the process of creating a volume from your running instance, then recreating the instance based on that volume. Suppose I have a server named `DEMO`. It was booted using the standard 25G root disk provided for all instances. Steps: 1. Stop the server ```sh openstack server stop "DEMO" ``` 1. Once the state has reached `SHUTOFF`, make a snapshot ```sh openstack server image create "DEMO" --name="DEMO_SNAP" --wait ``` 2. Create a volume from the snapshot image: ```sh openstack volume create --image="DEMO_SNAP" --size="SIZE" "DEMO_ROOT" ``` 3. Delete the old instance ```sh openstack server delete "DEMO" ``` 4. Create a new instance ```sh openstack server create "DEMO" --flavor=FLAVOR --volume="DEMO_ROOT" --network=NETWORK ``` ### How to live resize a volume? In the future you can grow the volume like this: ```sh openstack --os-volume-api-version 3.42 volume set "DEMO_ROOT" --size ``` Then you need to grow the partition and filesystem on the server, for example if this was the root disk of your server: ```sh growpart /dev/vda 1 resize2fs /dev/vda1 ``` Rental Model ============ Introduction ------------ The MPCDF HPC Cloud provides access to computing and storage resources. The service is not offered to individual users but is intended for projects or working groups of Max Planck Institutes. Cost components --------------- The fees for the requested resources consist of three components: investment, infrastructure and support. The fees for the investment are based on the cost of purchasing the respective hardware, the fees for infrastructure are mainly based on the averaged costs for the consumption of electricity and cooling of the systems. Support costs cover the personnel required to deploy and operate the systems and are calculated in a degressive way based on the size of the requested resources. For each specific resource, a total monthly price is calculated considering the three cost components. 2025 prices for cloud resources can be found in the `HPC Cloud Price Lists `_ in the MPG internal MAX web-site. Prices from the past years can be found here: `2025 `_, `2024 `_, and `2023 `_. Setup procedure and billing procedure ------------------------------------- When a project is production ready, the requested resources are to be ordered and their availability confirmed. Once an agreement is reached the resources will be allocated to the project via standard technical means such as quotas, host aggregates, disk pools, etc, and mapped to a set of physical resources by MPCDF. Projects are required to order a minimum set of physical resources and are required to run for a minimum duration, usually six months. Adjustments of the size of the resources are possible, however, such adjustments should be requested at least 3 months in advance. In general, projects run until an explicit cancellation request is made or a pre-defined cancellation date is met. Cancellations or extensions of projects must be communicated at least six weeks in advance. Billing will take place shortly after the end of a project or after a configuration change. Additionally, for projects which span several years, billing will take place close to the end of each calendar year, planned for November. Although quotes are provided based on a monthly average the actual billing will be performed on the exact number of days during the billing period. In rare cases where the above described mechanisms are not sufficient, a special solution can be discussed and realized in collaboration with MPCDF. Special pricing may apply. Available Resources ------------------- A brief description of the resources available for renting follows. This list provides an indication of the available resource types, however, the exact technical solution for a project is to be discussed on a project by project basis to allow for an optimal solution. Compute resources ----------------- As of 2024, there are three types of computing hardware resources: * Standard memory compute node (<= 4GB/vCPU) * High memory compute node ( 4GB+ - 8GB/vCPU) * Extreme memory compute node ( 8GB+ - 16GB/vCPU) The compute resource which is best suited for the request will be evaluated together with MPCDF. In some special cases directly attached storage is available either as SATA SSD or as NVMe SSD. Furthermore, GPUs, either of type A30, A40, A100, or soon H100 are available on request. Note: The use of GPUs requires negotiation and also implies a mandatory amount of provisioned vCPUs. Storage resources ----------------- The following types of storage resources are available: * Block storage based on HDD or SSD * Nexus mass storage, either as RAID6-POSIX or S3 object storage. The latter is implemented either with erasure coding or replication. Terms of Use ============ General ------- The MPCDF HPC-cloud is a shared responsibility Infrastructure as a Service (IaaS) offering which the MPCDF provides for MPI projects. The HPC-Cloud opens up opportunities for rapid development and innovation but also requires a strong commitment from both MPI and MPCDF teams w.r.t the maintenance and operation of the cloud and the projects housed within it. Three roles cover the management and usage of cloud based projects. - MPCDF Cloud team - MPI Cloud Project admins - Research Users **MPCDF Cloud team**: Administrators of the HPC-Cloud and MPCDF project enablers who consult with MPI Cloud Project Admins to provide HPC-Cloud based solutions. **MPI Cloud Project admins**: Administrators appointed by the MPIs to manage their cloud project; deploy services in cloud, VMs, storage etc and act as first level support for the Research End Users **Research End Users**: Generic users of the services put in place by the MPI Cloud Project admins. The researcher users may be from within the associated MPI or external users from collaborating partners (universities etc). For each project the responsible MPI names one or more Cloud Project admins who take over the operational responsibilities for this project. The MPCDF Cloud team provides operational support for the cloud infrastructure and enabling support for cloud projects. This leads to a shared responsibility model, the cornerstones of which are outlined below. The usage of the HPC-Cloud requires that MPI project admins agree to the standard MPCDF/MPG terms of usage [`1 `__] and ensure that users of the cloud based services will equally comply with these standard terms. The MPI project admins are responsible for several aspects of the cloud, these are outlined here and described in more depth below as well as in the general MPCDF Service Maintenance Agreement `2 `__. - Security and Service maintenance - Backup and Recovery - Data Privacy and management of Sensitive Data - Obtaining Licenses for proprietary software - Managing the Cloud project including decommissioning the cloud resource and any associated migration at the end of the project - Providing first level support for Researcher End Users Each MPI Cloud Project admin is required to agree to the terms and responsibilities which are detailed in this document. Security and Service Maintenance The MPCDF cloud team is responsible for the security and maintenance of the cloud infrastructure. The MPI Cloud Project admins are responsible for servers (Virtual Machines) and services which are deployed within the cloud. This includes tasks such as performing prompt and regular security patches as well as ensuring that services are scanned for possible security issues. In addition to the general maintenance of the VMs and services MPI cloud project admins are advised to follow cloud management best practices e.g. follow the principle of least access privileges, and also to delete / decommission resources that are no longer needed. Each MPI Cloud Project Admin will have full access to the cloud resources and the ability to create, modify and destroy them. Moreover, each researcher that is granted root access to a VM deployed within the cloud has elevated rights on the VM and is capable of installing and modifying software on the VM as well as accessing any attached data (filesystem). The MPI Cloud Project admins are responsible for providing and managing access to VMs. Backup and Recovery ------------------- The HPC Cloud does not provide any automated backup of VMs, their associated data, or the object storage (buckets). Data Privacy and Sensitive Data ------------------------------- The MPI cloud project admins are responsible for ensuring that the project complies with all applicable legal requirements, especially regarding data protection and copyright laws, as well as the stipulations of the general MPCDF terms of use. The MPCDF HPC-Cloud does not provide services to manage sensitive data. Projects which aim to manage sensitive data will need to provide a project specific solution. Proprietary Software -------------------- The MPI Cloud Projects admins are responsible for obtaining and managing licenses for any proprietary software, including operating systems, which is deployed within the cloud. In some cases licenses are available from the MPCDF or via the central software licensing service of the MPG (SOLI) `3 `__. Service Interventions and Scheduled Down times ---------------------------------------------- The HPC-Cloud is provided in a manner which is analogous to the Linux clusters hosted at MPCDF. General maintenance activities will be announced in advance. However, short term interventions may occur, for instance when critical security related patching is required and when security incidents occur or preventative action is needed. Performance ----------- The HPC Cloud has been designed to support flexible and scalable computing and data solutions on a large scale. For services which depend on shared resources, including network, storage and “shared” compute, there are no performance commitments. However, while shared resources are the standard offered in the HPC Cloud a “dedicated” compute model is available for compute sensitive applications. Support ------- Support from the MPCDF cloud team is available during working hours via the helpdesk (https://helpdesk.mpcdf.mpg.de) or by email: support@mpcdf.mpg.de. To ensure timely responses can be made to support requests clear communication channels are to be set up for each project between the MPCDF cloud team and the MPI Cloud Project admins. The MPI Cloud admins are to act as first level support for their Research End Users. Potential Sanctions ------------------- Failure to observe any of the listed points or other actions that endanger the security and/or proper operations of the MPCDF infrastructure or other third-parties can result in a temporary or permanent suspension of the service. - [1] https://www.mpcdf.mpg.de/userspace/terms-of-use - [2] https://www.mpcdf.mpg.de/userspace/service-agreement - [3] https://www.soli.mpdl.mpg.de/de/ # Visualization ## Support for the Visualization of Scientific Data The [application support group](../computing/application-support.md) at the MPCDF supports Max Planck scientists in producing high quality scientific visualizations. ## Remote Visualization and Jupyter Notebook Services Web-based services are available on RAVEN and ROBIN for all the users with an account on the clusters that want to access CPU and GPU resources for visualization, development, computation and analysis of results on HPC systems. The Remote Visualization Service (RVS) and Jupyter Notebook as a Service (JNAAS) provide the following sessions: - **Remote desktop**: suitable for efficient use of GUI tools (e.g. VTUNE, Matlab) and for activities that do not require intensive calculations. Resources on RAVEN: 8 CPU cores, up to 256 GB of (shared) main memory. - **Remote visualization**: suitable for GPU-accelerated calculations and rendering using tools like VisIt or Paraview. Resources on RAVEN: a single A100 GPU, 18 cores and up to 512 GB of (shared) main memory. - **Jupyter**: suitable for standard data analysis and computation using Jupyter notebooks. Resources on RAVEN: 8 CPU cores, up to 256 GB of (shared) main memory. - **Jupyter for machine learning**: suitable for GPU-accelerated machine-learning applications using Jupyter notebooks. Resources on RAVEN: a single A100 GPU, 18 cores and up to 512 GB of (shared) main memory. - **RStudio**: launch RStudio, an IDE for R. Only available on ROBIN, with 12 CPU cores and 64 GB of main memory. For technical reasons, an Ubuntu-based container ("rocker") is used to run RStudio. Make sure to enter your actual MPCDF username in combination with your RVS password (set up at initialization, see below) when prompted at login. Each remote graphical session (desktop and visualization) has a maximum run time of 24 hours, while Jupyter sessions are allowed to run for up to 8 hours. All sessions have access to the software provided via the module environment, including visualization software (e.g., VisIt, ParaView, Blender, ImageMagick, ffmpeg) and [software for Data Analytics](../computing/software/data_analytics-machine_learning), and the data stored on the HPC systems. ### Web Interface To start using the Remote Visualization and Jupyter Notebook services, users can login to with their Kerberos user name and password, using a web browser that supports HTML5 (we recommend to use Mozilla Firefox or a Chrome-based browser). Please, also note that the service doesn't currently support zsh shell on the compute clusters (i.e. if your account on Raven is using zsh shell, you will likely have problems initializing and submitting new RVS sessions, see the Troubleshooting section below). Three options are available: 1. **Initialize Remote Visualization**: ![](_static/rvs-jnaas_initialize.png) This step is only required the first time a user wants to access the RVS and JNAAS on a specific HPC system. Using the web form, the user can set the VNC and Jupyter password which is required to connect to running sessions (both remote visualization and Jupyter) later. We encourage users to select a password different from their Kerberos' one and with a minimum of 8 characters, following common password policies. Note that the VNC and Jupyter password is used only for connections to the VNC and Jupyter servers and does not replace in any way your Kerberos password. For example, you still need your Kerberos credentials to authenticate to before being able to connect to your running sessions with your VNC and Jupyter password. **Important**: users that initialized the RVS prior to April 2020 should re-initialize their password in order to also set a Jupyter password. In the section *Default modules for JN session* the user can specify the name of any module that should be loaded before starting all the Jupyter notebook sessions. This list is stored in a file located in the home folder of the user ($HOME/.jupyter/modules.conf). For example, in order to have some machine learning modules pre-loaded to each Jupyter notebook session, a user can write: ``` gcc/6 cuda/10.1 cudnn/7.6.2 nccl/2.4.8 impi/2019.4 tensorflow/gpu/2.1.0 ``` in the text box to have the respective modules immediately available at the start of each session. Note that the "rvs" and "Anaconda/3" modules are loaded by default and additional modules can be loaded from a running Jupyter notebook, if needed (see the Troubleshooting section). Use this option only if instructed by MPCDF staff. The initialization step also creates a symbolic link called "ptmp\_link" in the home of the user to easily access the /ptmp volume and a folder called "rvs" in the home of the user, where logs of the remote visualization and Jupyter sessions will be stored. 2. **Submit new session**: ![](_static/rvs-jnaas_submit.png) This page provides the form to request a remote desktop, remote visualization or Jupyter notebook session on the HPC system. Note that some types of sessions are not available on all clusters (for example, the *Remote desktop* is currently available only on the RAVEN cluster). Depending on the *Machine* and on the selected *Session Type*, different resources are available for each session (see above). The information icon next to the session type reminds of the exact resources allocated to each session. In this page, users can specify parameters of the session such as the length (default 4 hours) and, depending on the *Session Type*, the screen *Resolution* of the remote session (for *Remote desktop* and *Remote visualization* sessions) or the *Interface* and *Software* (for *Jupyter* and *Jupyter for machine learning*). Two values for the *Interface* are available: a *Classic* interface for standard Jupyter sessions and a *Lab* interface for users that prefer the Jupyter Lab layout. In the *Software* menu, it is possible to specify if the Anaconda3 package used for the Jupyter sessions should be based on Python only or should include the R language. Once the form has been confirmed, the job is automatically submitted to the batch queue on the HPC system, and a notification e-mail is sent to the user as soon as the session is ready. 3. **Connect to session**: ![](_static/rvs-jnaas_manage.png) This page provides information about the status of the submitted sessions, including the possibility to cancel a session or connect to a running one. All sessions are opened in a new browser tab. The VNC and Jupyter password that has been specified in step one has to be entered to finally give the user access to the remote session directly from the web browser. ### Command line interface Users that prefer to use a command line interface can manually initialize and request remote visualization and Jupyter sessions by submitting jobs to the batch system on the HPC clusters. The initial setup needs to be performed by the user only once on each cluster as follows: ```sh $ module load rvs $ module load anaconda/3/2020.02 $ setup_rvs $ setup_jn ``` This will create a password for the VNC and Jupyter connections, the folders "$HOME/rvs" (where logs will be stored) and "$HOME/.jupyter" (for the Jupyter configuration files) in the home directory of the user, with an automatic backup of previously existing folders with the same name. In addition, a symbolic link to the /ptmp volume named "ptmp\_link" is created in the home of the user. For example, to start a 4-hour long interactive visualization session, the user can then simply run the following commands: ```sh $ module load rvs $ cd $HOME/rvs $ sbatch --time=04:00:00 $RVS_HOME/bin/rvs.cmd ``` Similarly, to start a 4-hour long interactive Jupyter session, the user can run the following commands: ```sh $ module load rvs $ cd $HOME/rvs $ sbatch --time=04:00:00 $RVS_HOME/bin/jn.cmd ``` As soon as the session starts, the user will receive a notification e-mail with instructions on how to connect via a URL, via the new web interface, or via a conventional VNC software client (for remote desktop and remote visualization sessions). Please, note that only the MPCDF authorized scripts can run on the RVS and JNAAS partitions: scripts modified by the users will result in an immediate failure of the job. ### Technical details The web-based Remote Visualization Service at MPCDF is based on a combination of Virtual Network Computing (VNC) and WebSocket technologies. VNC provides a method to access a virtual graphical desktop on the HPC systems via the Remote Frame Buffer protocol, thus efficiently transmitting a stream of images between the server and the client. Access to the GPU hardware accelerated graphics is provided via VirtualGL. On the other hand, the [noVNC](https://github.com/novnc/noVNC) software provides a TCP proxy between the VNC server running on the host and a browser that supports the HTML5 standard (browser capabilities can be tested at [this link](https://websocketstest.com/)). Combining these two technologies, the RVS and JNAAS start a VNC or a Jupyter server for the user on the host and redirects the connection to the user web browser using the TCP protocol. ![](_static/session.png) An example of a running session can be seen in the image above. Please note that the noVNC toolbar on the left-hand side of the browser window provides additional settings to personalize the connection, like re-scaling, full screen and a clipboard function that should be used when copying and pasting content between the user's current desktop and the virtual desktop running in the browser. ### Troubleshooting In the following we present a list of the most common questions and problems users may encounter, with a suggestion on how to solve them. 1. *The cluster where I want to submit my job does not appear in the list of available machines*. The remote visualization and Jupyter services are currently supported only on ADA, DAIS, LEO, RAVEN, and ROBIN. If a cluster name does not appear in the drop-down menus, you may not be registered as an official user of that cluster. Please check that you can login directly via "ssh" to the cluster. If this is the case, but you still don't see the cluster entry, please contact the [MPCDF support](../../faq/help.md). 2. *I don't remember my password for connecting to a session*. If you forgot your password or if you want to change the list of default modules available in your Jupyter session, you can re-initialize your remote visualization and Jupyter sessions as described in step 1 of the Usage section. The new initialization will create a backup of your "$HOME/rvs" (containing the session logs), "$HOME/.vnc" and "$HOME/.jupyter" folders before setting the new password for you. 3. *I have a 'Xlib: extension "GLX" missing on display "127.0.0.1"' error when I start software in a remote session*. Software that uses the OpenGL libraries requires a remote visualization session on a GPU (not a 'remote desktop' session). In order to access the OpenGL libraries, you should run your commands with the prefix "vglrun" (e.g. vglrun blender, vglrun visit, vglrun paraview, etc.). 4. *I want to load a module from my Jupyter session*. You can list, load and unload modules available on the cluster directly from your Jupyter notebook using: ```python module('list') module('load','module_name') module('unload','module_name') ``` Note however that inside a Jupyter session the command ```python module('purge') ``` will not unload all modules, but will restore the Jupyter notebook to the initial state instead (with all the modules that were loaded when the notebook was first launched). Please note that for technical reasons it is not possible to module-load some Python modules that set the LD\_LIBRARY\_PATH variable from within a Jupyter session. Please put such environment modules into the list of *Default modules for JN session* to module-load them before the session starts. 5. *The State of my session is PENDING for a long time*. Usually, your session should start within a few seconds from the submission. If your session shows the PENDING status for a long time, all the available hardware resources may already be in use. You can try submitting a new session to a different cluster, or select a session that requires less resources. For example, selecting a *Remote desktop* instead of a *Remote visualization* session or a *Jupyter* session instead of a *Jupyter for machine learning* one increases the chances that a slot for your session is more readily available. 6. *I have a '503 Service Unavailable' error when I try to connect to the Jupyter session*. When a Jupyter session is started on a cluster, it may take a few seconds for the Jupyter server to be ready and accept your connection. In case you see this error message, please wait a few seconds and then try again connecting to your session. If the problem persists minutes after the start of the session, please contact the [MPCDF support](../../faq/help.md). 7. *The Jupyter kernel in my session is not behaving properly*. If you installed conda environment in your home directory on the cluster or if you have other commands that are executed automatically in your bash configuration, the Jupyter session may not work as intended. In these cases, please deactivate your conda environment on the cluster and check that your bash configuration does not contain commands that are automatically executed at login time before trying again. 8. *My session is killed a few seconds after starting*. There are 3 possible reasons for this. a) One possibility is that a password was not set for the cluster where the session was submitted. In this case, please follow the instructions above and initialize your sessions. b) Another possibility (usually when requesting a Jupyter session) is that some python packages installed in the user's home are creating conflicts with the software (Anaconda in particular) run by the remote visualization job. In such cases, please inspect the log files for the failed job in the folder $HOME/rvs on the cluster and check which package is creating the error or the conflict. c) Finally, the remote visualization service does not currently support zsh shell. If you are using this shell on the cluster where you are submitting a session, then all your requests (including the initialization of a new password) to the RVS server will be cancelled after a few seconds. ## Robin The Remote Visualization Service at MPCDF has recently been expanded with a new cluster called **Robin**, available only via the RVS web interface. Robin is our first compute cluster in the MPCDF HPC Cloud and can currently host up to 20 CPU sessions and 24 GPU sessions. RVS resources at Robin are available to all users with an account on our HPC systems (i.e. Raven and Viper). Each session on Robin provides 12 virtual CPUs and 64GB of RAM, with GPU sessions having access to a shared NVIDIA A40 GPU (up to 2 sessions can share a single GPU). Robin mounts the Raven's file system, providing access to all the software and data available on the Raven cluster, including the user's home directory. A runtime of up to 7 days is currently allowed (with a plan to increase to up to 28 days of maximum runtime), but users are encouraged to stop their sessions once their calculations are completed and should be aware that long running jobs can be killed in case of maintenance of the cluster. Users requesting GPU sessions are encouraged to limit the memory used by their code to roughly 1/2 of the available GPU memory (~24GB out of the 48GB available), in order to avoid disrupting the calculations of other users sharing the same GPU. This is particularly important for Machine Learning software (e.g. Tensorflow, Pytorch) that can allocate the entire available GPU memory for a single process. Robin is designed to provide a single solution for the remote visualization needs of future HPC clusters at MPCDF: the file system of new clusters (like the upcoming cluster Viper) can be made available on Robin, providing easy access to software and data without the need of a dedicated installation of the Remote Visualization Service on each cluster. Users interested in using the Remote Visualization Service on Robin are reminded to initialize their sessions on the cluster once (before submitting their first session) at [this link](https://rvs.mpcdf.mpg.de/rv/initialize), as described above. ------------------- Campus ------------------- The MPCDF offers a range of general services (as of 2023, this is mostly restricted to MPCDF employees). .. toctree:: :maxdepth: 1 software/index.md.txt wifi/index.md.txt ## Software The MPCDF provides a number of scientific software packages with MPG-wide or Garching-campus-wide licensing agreements for download and local installation via the [MPG MaxNet](https://max.mpg.de/Service/Forschungsservice/Pages/MPCDF/Software-download-for-MPG-users.aspx) (for MPG-internal use only!) ## Wi-Fi ### Guest networks In some meeting rooms, unencrypted guest networks are available. ### Eduroam Eduroam (education roaming) is a secure, world-wide Wi-Fi roaming service developed for the international research and education community. Eduroam allows students, researchers and staff from participating institutions to obtain internet connectivity on campus and when visiting other participating institutions. Eduroam users will be assigned guest addresses and may not be able to access internal services. Internal networks and services can be accessed using VPN. ### Installation (*MPCDF/MPQ staff only*) The step-by-step guide for configuring Easyroam is provided below: 1. Ensure that an internet connection exists, LAN or WLAN (not Eduroam). 2. If applicable, delete the existing Eduroam profile and the Eduroam CAT tool. *After removing the profile, access to Eduroam will be temporarily unavailable.* 3. Log in to the Easyroam platform ["Easyroam WAYF"](https://get.eduroam.de) and select your institution. * MPCDF staff, please select "Max Planck Computing and Data Facility (MPCDF)" * MPQ staff, please select "Max Planck Institute for Quantum Optics (MPQ)" 4. On the following page, you will find links to install the Easyroam app for your device. The client for Windows or Linux (Debian-based) can be downloaded by selecting "Download for desktop" in the drop-down menu. *Should you encounter problems during installation on Windows, please request help from the [IT-Servicedesk](mailto:it-support@ipp.mpg.de).* There is also an `easyroam-desktop` RPM package available in the [OBS system repository](https://ginster.mpcdf.mpg.de/obs/system/). *This is only tested with openSUSE Leap 15.5 and might not work as intended on other RPM-based distributions.* 5. After installation the devices can be configured for the use of Eduroam via the app. * Mobile devices can use "connect my phone" on the Easyroam website and scan the provided QR code. * The apps for desktops will forward to the Easyroam website for further configuration. A detailed documentation in German language for setting up specific clients can be found [**here**](https://doku.tid.dfn.de/de:eduroam:easyroam-anleitungen). ##### Troubleshooting: - **Issue**: On Android, the Eduroam Wi-Fi suddenly stops working, while the certificate is still valid. **Solution**: In the easyroam app, select "*Manage*" under the currently used profile and press "*Reinstall*" Bits and Bytes -------------- .. The edition list below is generated by conf.py from the numbered files in this directory (docs/bnb/.md), newest first. Do not edit the list by hand and keep the placeholder token intact. To add an edition, drop in the file; to publish it, bump BNB_PUBLISHED_UP_TO in conf.py. .. toctree:: :maxdepth: 2 222.md.txt 221.md.txt 220.md.txt 219.md.txt 218.md.txt 217.md.txt 216.md.txt 215.md.txt 214.md.txt 213.md.txt 212.md.txt 211.md.txt 210.md.txt 209.md.txt 208.md.txt 207.md.txt 206.md.txt previous.md.txt Bits and Bytes Logo # No.222, August 2026 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_222.pdf) ## High-performance Computing ### Procurement of next-gen HPC and AI systems of the MPG Earlier this year, MPCDF in collaboration with administrative headquarters of the Max-Planck Society (MPG) launched a Europe-wide call for tenders for procuring a successor for the HPC system _Raven_ of the MPG. With additional budget allocated by the President the procurement included a second, GPU-only system dedicated to scientific AI applications. Bids from different vendors and with different technical characteristics were ranked primarily by the compute performance the offered system will deliver (based on a representative set of 10 HPC applications of the MPG for the HPC system, and the aggregated GPU-performance for the AI system, respectively). In summary, the main characteristics of the new systems are as follows: HPC system: * 320 CPU-only compute nodes based on AMD EPYC Turin 9655 (61440 CPU cores, 240 TiB RAM total), * 59 GPU-accelerated compute nodes based on Nvidia GraceBlackwell GB200 NVL4 (236 GPUs, 43 TiB HBM total), * Nvidia Infiniband NDR (nonblocking fat tree, 400 Gb/s per CPU node, 800 Gb/s per GPU node), * IBM StorageScale with 24 PB HDD, 360 TB NVMe. AI system: * 106 4-way GPU-compute nodes based on Nvidia GraceBlackwell GB200 NVL4 (424 GPUs, 77 TiB HBM total), * Nvidia Infiniband XDR (nonblocking fat tree, 800 Gb/s per node) * IBM StorageScale with 1 PB NVMe. The new systems will both be delivered by the well known German company pro-com Datensysteme GmbH, in partnership with AMD, Nvidia, IBM and Lenovo. Despite the currently very difficult global market situation, the new systems will provide MPG researchers with increased computational power compared to _Raven_, and in addition a new line of MPG resources for AI research. Installation of the systems will commence in autumn this year with the aim to have the systems in production in the course of 2027. _Erwin Laure, Markus Rampp_ ### _Viper-GPU_: extension to final configuration A number of 27 additional compute nodes (with 54 AMD MI300A APUs) are currently being added to _Viper-GPU_ as a compensation for the delayed deployment of the entire _Viper_ machine. With this extension _Viper-GPU_ will reach its final configuration comprising 327 compute nodes (654 MI300A APUs) within the next few weeks. _Markus Rampp_ ## Software News ### Compilers The latest GNU compiler 16.1.0 (module `gcc/16`) and the Intel oneAPI version 2026.0 (modules `intel/2026.0`, `impi/2021.18`, and `mkl/2026.0`) have been installed together with the corresponding software stacks on the HPC machines and on many institute clusters. The Intel compiler provides new features, including more fine-grained loop optimization control and the ability to filter optimization reports. The new version 23.2.0 of the LLVM-based AMD compiler (module `amd-llvm/23.2`) has been made available on _Viper-GPU_. _Tobias Melson_ ### CUDA and Nvidia HPC SDK CUDA 13.2 (module `cuda/13.2`) has been installed on _Raven_, together with the matching NCCL module `nccl/2.30.4`. The Nvidia HPC SDK version provided by the module `nvhpcsdk/26` has been upgraded to version 26.5 and will further be updated to the most recent version once available. The current version ships with CUDA 13.2 supplied by the module `cuda/13.2-nvhpcsdk_26`. See our [previous announcement](https://docs.mpcdf.mpg.de/bnb/215.html#cuda-modules-on-raven) for an explanation about CUDA modules. The update has also been applied to the CI module images in our GitLab instance. _Tobias Melson, Tilman Dannert_ ### Amber molecular dynamics package The new version 26 of the popular classical molecular dynamics (MD) package [Amber](https://ambermd.org/) is available on the MPCDF HPC systems. Notably, with HIP support added, the new version is provided on _Viper-GPU_ for the first time. With the new release we also update the naming of the corresponding environment modules, as the licensing distinction that motivated the earlier split no longer applies. Starting with Amber26, environment modules previously named `amberMD` (providing the license-restricted pmemd) and `amber` (providing the free AmberTools) are now provided on our HPC systems as a single unified module named `amber` containing both pmemd and tools. Earlier module versions stay untouched, but note that there is no `amberMD` module for Amber26 and job scripts loading `amberMD` need to be adapted accordingly. Since pmemd releases happen only every other year, an AmberTools-only package (i.e. without pmemd) will continue to be provided as module `ambertools`. The new version can be loaded using `module load amber/26`. Using tools that depend on Python requires additionally loading _Water Boa Python 2024.06_ (environment module `python-waterboa/2024.06`). Note that pmemd is free of charge for non-commercial use; users affiliated with for-profit organizations require a separate license agreement with UCSF (see the [Amber26 license terms](https://ambermd.org/GetAmber.php)). Amber's protein force fields are widely used beyond Amber itself: MPCDF, in close collaboration with the Max Planck Institute for Multidisciplinary Sciences and Johnson & Johnson and together with co-authors representing Amber, has ported the protein force fields ff14SB and ff19SB for use with the [GROMACS](https://www.gromacs.org/) MD package, and cross-validated their consistency in both packages. The results are detailed in the recent preprint [_Force Fields and Simulation Engines: Lessons Learned from Porting Amber ff14SB and ff19SB to GROMACS_](https://doi.org/10.26434/chemrxiv.15006112/v1). _Vedran Miletic, Markus Rampp_ ## Using AI Coding Agents on MPCDF Systems Command-line AI coding agents such as Claude Code, OpenAI Codex, Google Antigravity, OpenCode, or Cline have quickly become popular tools for software development. They read and modify source files, run commands, execute tests, and iterate largely on their own. Increasingly, MPCDF users are using them to develop their scientific software, refactor simulation codes, write analysis scripts, or explore an unfamiliar codebase. To help users do this more safely, MPCDF now provides a containerized environment that runs these agents with reduced access to the surrounding file system — a meaningful improvement over running them directly on the machine, though it offers no guarantee of security. #### Why a sandbox is needed An AI coding agent does more than suggest text: it runs with your user account and executes real actions. It opens files, runs shell commands, and edits code, often with only minimal confirmation. By default it inherits all your permissions, so it can read any file you can read and write any file you can write. Started in your home directory, it could therefore access SSH keys, API tokens, and unrelated project data and send them to an external service. A single misunderstood instruction can also overwrite or delete files. Running an agent with unrestricted access to your account is therefore strongly discouraged. A practical mitigation is to confine the agent as far as possible to just the project it is supposed to work on. This is what the MPCDF AI agents container is designed to do: it reduces what an agent can reach, without claiming to make its use completely safe. #### The MPCDF AI agents container The environment is built on [Apptainer](https://apptainer.org/), the user-space container runtime already used across MPCDF HPC systems. When you launch it, only two things are visible to the agent inside: - the **current working directory** (your project), and - a dedicated **fake home directory**, which holds the agents' own configuration, credentials, and caches, kept separate from your real `$HOME`. Everything else — the rest of your home directory, other users' data, and other file systems — remains invisible. The container image itself is read-only, so the agents cannot modify their own installation. Launching directly from `$HOME` is also blocked, to avoid accidentally exposing all of your files. It is worth being clear about what this does and does not achieve. A container adds a layer of isolation on top of the ordinary Linux user account, but it is not an impenetrable security boundary. Container runtimes are themselves software, and known or as-yet-undiscovered vulnerabilities may in principle allow an agent to break out of the sandbox. Running an agent inside the container is therefore considerably safer than running it directly on the machine, but it does not make it entirely safe, and it does not remove your responsibility for what the agent does. The container ships with a set of pre-installed command-line agents, so you can pick the one you prefer without installing anything yourself: | Command | Provider | |------------|-----------------------| | `agy` | Google Antigravity | | `claude` | Anthropic Claude Code | | `cline` | Cline | | `codex` | OpenAI Codex | | `copilot` | GitHub Copilot CLI | | `gemini` | Google Gemini CLI | | `opencode` | OpenCode | | `vibe` | Mistral Vibe | The container comes in three flavours — based on the Linux distributions SLES, RHEL, and Ubuntu — and the matching one is detected automatically from the host system. On the SLES- and RHEL-based HPC systems (e.g. _Raven_ and _Viper_) the MPCDF software tree is bind-mounted read-only into the container, so the familiar `module load` command works exactly as on the login node, and agents can build and test software against the usual HPC software stack. #### Getting started Clone the [project](https://gitlab.mpcdf.mpg.de/mpcdf/ai-cli-agents-container), build the container image, and install the `agents` launcher once (make sure `~/bin` is in your `PATH`): ```bash git clone https://gitlab.mpcdf.mpg.de/mpcdf/ai-cli-agents-container.git cd ai-cli-agents-container ./build-containers.sh # builds the container image (auto-detects the OS flavour) ./install-agents-launcher.sh # creates the ~/bin/agents launcher ``` After that, the typical workflow is simply: ```bash cd /path/to/your/project agents ``` This drops you into a shell inside the container, indicated by an `[Agents]` prompt. From there you start the agent of your choice, for example `claude` or `codex`, and work as usual. When you are done, exit the shell to leave the sandbox. For all details (prerequisites, configuration options, and the two-stage build) please refer to the `README.md` in the AI agents container [project repository](https://gitlab.mpcdf.mpg.de/mpcdf/ai-cli-agents-container). #### Best practices Because the sandbox reduces but does not eliminate the risks, a few habits are essential to using AI coding agents on MPCDF systems safely and effectively: - **Start in the project directory, never in `$HOME`.** The isolation is only as good as the directory you open. Change into the specific project you want to work on before launching the container. - **Mind confidentiality and data protection.** Prompts and file contents are sent to external cloud services operated by the respective providers. Do not expose personal data, sensitive research data, credentials, or otherwise confidential material to an agent. Check that using a given service is compatible with the terms under which your data was obtained. - **Keep credentials in the fake home.** API keys and login tokens accumulate in the fake home directory. Treat it like any other secret store, and wipe it if you want to remove all agent state. - **Review everything the agent produces.** AI agents make mistakes, introduce subtle bugs, and can "hallucinate" plausible-looking but incorrect code or results. Read the diffs, run your tests, and keep your work under version control so that unwanted changes can be reverted. - **Use the batch system from outside the container.** For safety, Slurm client tools are not available inside the container. Submit and manage jobs (`sbatch`, `srun`, `squeue`) from a regular login shell; the agent can then wait for the results and analyse the output files. - **Keep an eye on cost and quotas.** Agents can issue many API calls in a short time. Be aware of the usage limits and billing associated with your chosen provider and account. #### Availability and support The AI agents container is available now on MPCDF systems. It adds a valuable layer of protection, but — as noted above — it is not a guarantee of security: you remain fully responsible for the agents and workloads you run, and for assessing the associated security and compliance risks. Questions, feedback, and problem reports are welcome via the MPCDF helpdesk. _Klaus Reuter, Andreas Marek_ ## HPC-Cloud ### Standard cloud images The MPCDF HPC-Cloud provides flexible compute and storage resources to Max Planck Institutes. Compute typically takes the form of a virtual server running the Linux operating system (OS). Rather than installing the OS onto a blank disk as one might with a personal computer, the lifecyle of a virtual server begins with a disk _image_ containing a pre-installed Linux distribution. Data written to the server's disk is stored separately from the image, which is itself immutable. This technical approach allows new servers to be launched quickly, and also conserves space in the backend storage system. The cloud team provides a set of standard images based on feedback from project admins, while taking into account the support timelines of the respective distributions. Each one is based on the upstream "cloud-ready" image, to which a limited set of site-specific customizations is applied. The full details can be found in the image-builder [project](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/image-builder). **Ubuntu 26.04** is the latest addition, joining supported releases of **Debian**, **AlmaLinux**, and **openSUSE Leap**. A complete list can be found in the [documentation](https://docs.mpcdf.mpg.de/doc/cloud/technical/compute.html#images). Images contain not only primary data, but also metadata which determines certain aspects of the servers's virtual hardware. On this subject we would like to highlight an important recent change: The virtual storage controller is now set to type _virtio-scsi_, which supports up to 255 block volumes. A side effect of this change is that volumes appear as `/dev/sdX`, with persistent path `/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_UUID` where `UUID` is the OpenStack ID of the volume. Thus, it may be necessary to adjust automation scripts to use the correct paths on newly-launched servers. No changes are required on existing servers. _Brian Standley_ ### S3-IAM - Self Service user management for object storage The CEPH Squid release introduces Self Service, multitenant, functionality via Identity and Access Management (IAM) accounts. IAM accounts allow projects to independently manage their resources including users, groups, roles, policies, and buckets; using an API interface modeled after AWS IAM. IAM provides a hierarchy of accounts within a project; a root account for the project admins and end user accounts for individual users. - Root account: The admin responsible for the Account. Manages resources within a specific project/IAM account, using clients such as the AWS CLI and the IAM API endpoint. - S3 end users: Operate within the confines of permissions granted by the root account. The root account can create and manage users, groups, roles, and permissions within the IAM account. This allows multiple users to gain access to a project or specific buckets within a project. However, all buckets and data are owned by the root account. Operations that the IAM root account can perform without the intervention of a CEPH admin include: - Create, modify, and delete users - Manage account users' access and secret keys - Manage IAM policies - Manage IAM user policies - Manage IAM groups - Create, modify, and delete OIDC providers - Create, modify, and delete notification topics - Create, modify and delete quotas on buckets This allows the root account to fully manage the resources and easily provide solutions for numerous use-cases. Common use-cases: 1. Providing a bucket per user: Multiple users can be created with policies set to allow dedicated read/write access to a single or multiple buckets. Additionally quotas can be set on each bucket for fine grained resource usage. 2. Read/write and read-only users: Within a project, or even for specific buckets, read/write and read-only users can be created allowing control over who generates and who consumes the data. Projects supporting IAM can now be created upon request. Existing projects may be migrated to IAM. However, this process is irreversible and requires additional planning. Ideally we suggest starting with a new project. _Robert Hish, Florian Kaiser, John Alan Kennedy_ ## News & Events ### Securing MPCDF account application form against bots Lately, it turned out that the [form to apply for an account at MPCDF](https://selfservice.mpcdf.mpg.de/index.php?r=registration) was not sufficently protected against being filled by bots. In order to prevent such fake applications triggering a huge amount of E-mails to the account approvers we implemented several means to protect the form against misuse by bots. In addition to some purely technical hurdles for bots, the E-mail address of the applicant now requires a confirmation, before the application is finally sent to the approver, which establishes a two-step mechanism already in the application process. _Andreas Schott_ ### New CoE projects EuroHPC has just finished the evaluation for a new round of HPC Centres of Excellence (CoEs). This time, two flavours of CoEs are envisaged, so-called “community CoEs” focussing on fostering the uptake and efficient usage of European HPC applications, and “lighthouse” CoEs focussing on improving and further developing important HPC applications. MPCDF has been part of four proposals (two community and two lighthouse ones) in the areas of biomolecular modeling and fusion plasma simulations, and all four of them have been proposed for funding and are now in contract preparations with a planned start early 2027. Specifically, the long-running CoE [“BioExcel”](https://bioexcel.eu/) will enter a new phase as “community” CoE and MPCDF will work alongside the MPI for Multidisciplinary Sciences on supporting the GROMACS community. The [GROMACS code](https://www.gromacs.org/) itself will be the focus of a new “lighthouse” CoE, where MPCDF will work together with KTH, BSC and FZJ on improving and extending GROMACS. In the area of fusion plasma simulations, the existing [Plasma-PEPSC CoE](https://plasma-pepsc.eu/) will continue as a “community” CoE and MPCDF will work together with the Max Planck Institute for Plasma Physics (IPP) on their [GENE code](https://www.genecode.org/) family. Another IPP code, [JOREK](https://www.jorek.eu/), will be the focus of a new lighthouse CoE. _Erwin Laure_ ### MPCDF at Garching Campus Open Doors (October 3) MPCDF will open its doors for the general public at the [_Open Day Campus Garching_](https://forschungscampus-garching.de/) at October 3, 10:00-17:00. We will provide short talks, posters about scientific high-performance computing, data science, and artificial intelligence and offer the opportunity to have a peek into the machine hall. [The program](https://www.mpcdf.mpg.de/opendoors2026) is targeted at the general public, but we, at MPCDF, always appreciate exchange with our friends and expert users who might take the opportunity of the campus event to meet in person with MPCDF staff. _Friederike Neu_ ### MPCDF Hands-On Cloud Computing Workshop On 13-14 October 2026, MPCDF organizes a Cloud Hands-on Workshop. This workshop is designed for system administrators, cloud practitioners, and DevOps engineers and there is no prior experience required. We will guide you from the basics to a solid, working understanding of cloud infrastructure. If you already use an HPC-Cloud project from the MPCDF, you'll also come along with new insights and ideas for your service planning. The workshop will be conducted online and we are going to provide a limited number of cloud projects for the time of the workshop. For more details and registration, please visit the [event homepage](https://plan.events.mpg.de/event/806/) _Fabio Baruffa_ ### Agentic AI for Science: An on-site workshop at MPCDF, 7-8 October 2026 AI agents are reshaping the way scientific research is conducted. Unlike traditional AI tools, agents can plan, reason across multiple sources, and carry complex tasks through many steps autonomously, from literature review and data analysis to code development and simulation workflows. To explore the current state and future potential of this technology, MPCDF is organising a 2-day in-person workshop bringing together researchers, research software engineers, and industry partners from across the Max Planck Society. The first day will combine hands-on tutorials with real-world experiences from MPG researchers, offering a practical introduction to AI agents alongside talks from groups already using them in their workflows. The second day will feature contributions from leading industry partners, showcasing their latest solutions and discussing how agentic AI is being adopted across the research landscape. The workshop is open to all researchers across the Max Planck Society and aims to build a shared understanding of where agentic AI can genuinely add value, what the current limitations are, and how the community can approach adoption responsibly. Further details and registration are available on the [event homepage](https://plan.events.mpg.de/e/ai-agents-for-science). _Piero Coronica_ ### Workshop on OpenMP Offloading with AMD GPUs, 27-28 October 2026 This online workshop, lead-organized by [HLRS](https://www.hlrs.de/) and AMD in collaboration with MPCDF, teaches the usage of OpenMP for GPU programming with a focus on the AMD MI300A APU which, for example, is employed in the HPC system _Viper-GPU_ at MPCDF. Target audience are beginners in GPU programming having already some basic knowledge of parallelization with OpenMP on CPUs. After this course participants will have learned the basics to confidently start porting applications from a CPU-only system to systems with discrete GPU accelerators or APUs like the MI300A. Further details and registration can be found on the corresponding [MPCDF event webpage](https://www.mpcdf.mpg.de/events/46269/2825). _Tilman Dannert_ ### Introduction to MPCDF Services The next introductory course will take place online on October 15, 2026, from 14:00 to 16:30. It is designed to familiarize new users with the MPCDF compute and data services. No registration is required; interested users can simply join via the [Zoom link](https://mpcdf-mpg-de.zoom-x.de/j/69500447868?pwd=5fQ3xerZMaMDh77215EaPyPVm4MGnM.1), which is also published on our website. Please note that the link only becomes active at the start of the course. _Klaus Reuter_ ### Seminar on Kokkos by Christian Trott On September 9th, MPCDF will host a seminar talk on _Kokkos: The Why, the Who and the How_ by Christian Trott, who is a co-leader of the Kokkos core team at Sandia National Laboratories and a co-Chair of the ISO C++ committees library working group (LWG). The [Kokkos C++ Performance Portability Ecosystem](https://kokkos.org/about/overview/) is a popular, production level solution for writing modern C++ applications in a hardware agnostic way, and has been widely adopted for writing performance-portable high-performance-computing applications, especially across GPU platforms. The seminar will be given on September 9th, 10:00 in the main lecture hall of building D2, with the option of online participation via zoom (see the [MPCDF event webpage](https://www.mpcdf.mpg.de/events/46503/2825)). The speaker will be available for extended discussions after the talk. _Erwin Laure, Markus Rampp_ ### IT4Science Days The next _MPG DV-Treffen_ will take place during the _IT4Science Days 2026_ together with colleagues from other German research organizations performing organizations, specifically from the Helmholtz Association and the Fraunhofer Society. This year, we will meet in Göttingen from Monday, September 28th, through Thursday, October 1st. The main meeting starts on Tuesday around noon. The focus of this year's meeting will be on _Artificial Intelligence_ in all its facets, but expect a wide spectrum of further contributions. More information, including the program as well as the registration form is available from the [website of the meeting](https://plan.events.mpg.de/event/670/). _Raphael Ritz_ ### MPG Research Data Management (RDM) Conference 2026 The next _MPG Research Data Management (RDM) conference_ will be held November 3-5, 2026, in Potsdam Golm preceded by an RDM introduction session on Monday, November 2, 2026. As with the RDM workshops of recent years, a special focus will be on the research support services of the hosting institutes (MPI of Colloids and Interfaces supported by the MPI of Molecular Plant Physiology and the Albert Einstein Institute). For the first time, the language of this event will be English. Participation in the conference is free of charge but limited to MPG members. More information, including the program as well as the registration form is available from the [website of the meeting](https://rdm.mpdl.mpg.de/mpdl-services/events/rdm-conference-2026/). _Raphael Ritz_ Bits and Bytes Logo # No.221, April 2026 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_221.pdf) ## High-performance Computing ### Resource limits at login nodes of the HPC-Systems In order to maintain the responsiveness of the login nodes on the HPC machines, per-user resource limits were introduced [on _Raven_](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html#resource-limits) and also [on _Viper_](https://docs.mpcdf.mpg.de/doc/computing/viper-user-guide.html#resource-limits) in late 2024. Over time, even stricter limits had to be enforced, actually. The following table summarizes the current limits. | |raven[01-02]i|raven[03-04]i|viper[01-02]i|viper[03-06]i|viper[11-12]i|viper13i| |-------|-------------|-------------|-------------|-------------|-------------|--------| |cores | 2 | 6 | 2 | 6 | 2 | 6 | |memory | 50 GB | 100 GB | 50 GB | 100 GB | 50 GB | 100 GB | |tasks | 768 | 1536 | 768 | 1536 | 768 | 1536 | As a consequence, especially because of the limitation of tasks (which actually is the sum of Unix processes and threads per user), some programmes might fail with `fork: retry: Resource temporarily unavailable` In such cases, please double check your process list for stale sessions that might have not terminated properly (`ps auxH`), and clean those up using standard tools like `kill`. Furthermore, we'd encourage users to explicitly choose a login node based on the above table, and no longer rely on our legacy DNS aliases `raven` or `raven-i` (likewise for _viper_). These legacy aliases will be withdrawn in late 2026. In case you require more powerful interactive sessions to analyze data on _Raven_ and _Viper_, we strongly recommend to use our RVS service at _Raven_ and _Robin_. The latter meanwhile has access to the filesystems `/raven/ptmp`, `/viper/ptmp1` (_Viper-CPU_), `/viper/ptmp2` (_Viper-GPU_) and `/nexus/posix0`. For further insights on potential impacts on parallel builds and GitLab CI jobs on the login nodes, please also refer to [our initial article on this topic in issue 217 of Bits&Bytes (December 2024)](https://docs.mpcdf.mpg.de/bnb/217.html#resource-limits-on-the-hpc-machines). _Christian Guggenberger_ ## Software News ### AMD software Version 7.2.1 of the ROCm software stack has been installed on _Viper-GPU_. The corresponding module is called `rocm/7.2`. On _Viper-CPU_, a new OpenMPI module has been installed for the AOCC compiler. Users can now compile Fortran codes with MPI using AOCC Flang by loading the modules `aocc/5.1 openmpi/5.0`. The MPI wrapper scripts follow the standard naming scheme, that is, `mpif90` for Fortran. Also modules providing HDF5 and NetCDF are available for this software stack: `hdf5-mpi/1.14.1` and `netcdf-mpi/4.9.2`. _Tobias Melson_ ### Nvidia HPC SDK A new module `nvhpcsdk/26` has been started with the Nvidia HPC SDK 26.1. Minor version updates will follow during the year without changing the module name. Update to 26.3 has already been done. The previous `nvhpcsdk/25` is now frozen on version 25.11. Be aware that the only CUDA version installed with `nvhpcsdk/26` is CUDA 13.1. It can be explicitly loaded with the module `cuda/13.1-nvhpcsdk_26` to set the CUDA relevant paths. With the change to CUDA 13+, some deprecations and API changes have been done. For further details, refer to the [CUDA 13 release notes](https://docs.nvidia.com/cuda/archive/13.0.0/cuda-toolkit-release-notes/index.html). _Tilman Dannert_ ### New ELPA version 2026.02 The latest [ELPA eigensolver library](https://elpa.mpcdf.mpg.de) release (version 2026.02.001) brings further performance improvements for GPU-based computations. In particular, ELPA 1-stage GPU tridiagonalization and backtransformation are now both ~10% faster, which leads to overall ~10% speedup for the standard eigenproblems. As [announced in the previous issue 220 of Bits&Bytes](https://docs.mpcdf.mpg.de/bnb/220.html#new-elpa-module-version-scheme), the bugfix number is now dropped from the ELPA module name. For example, the new release is available as `module load elpa/mpi/standard/gpu/2026.02`. In addition, the 2025.06.001 module has received a bugfix update, adding support for multi-architecture Nvidia GPU builds and fixing a bug in the tridiagonal solver. This and all previous versions are still available with their full version numbers, e.g., `module load elpa/mpi/standard/gpu/2025.06.001`. All available ELPA modules can be queried with `find-module elpa`. _Petr Karpov, Tobias Melson, Andreas Marek_ ### Fortran support added to structured diff and merge tools via incremental parsing ![A screenshot of Difftastic displaying differences between two Fortran files.](221/difftastic.svg) [Difftastic][difftastic] and [Mergiraf][mergiraf] are syntax-aware diff and merge tools that operate on concrete syntax trees instead of raw text. Difftastic computes structural diffs by parsing source code with [Tree-sitter][tree-sitter], an incremental parsing library that generates concrete syntax trees for many programming languages. Mergiraf builds on the same Tree-sitter infrastructure to perform structured, language-aware merges with improved conflict resolution. Tree-sitter provides fast, error-tolerant parsers and a uniform AST interface, enabling both tools to reason about code structure rather than line-based changes. Therefore the tools can align changes based on actual language structure (functions, expressions, blocks) rather than lines chosen by complicated algorithms, making diffs far more accurate and readable as shown in the screenshot in Figure 1 of Difftastic displaying differences between two Fortran files. Changes are displayed side-by-side by default, because they are no longer line-based. Note how changes are resolved almost to character level and embellished with basic syntax highlighting in bold and italic. This structure awareness also enables smarter merges that reduce spurious conflicts and preserve intent, unlike traditional line-based tools that often misinterpret reformatting or code movement as substantive changes. Difftastic and Mergiraf can be used directly in Git as diff and merge drivers, respectively. Recently, Fortran language support has been added by MPCDF to both Difftastic[^1] and Mergiraf[^2] on their respective development branches via a [Tree-sitter Fortran grammar][tree-sitter-fortran]. [^1]: https://github.com/Wilfred/difftastic/pull/951 [^2]: https://codeberg.org/mergiraf/mergiraf/pulls/717 [difftastic]: https://difftastic.wilfred.me.uk/ [mergiraf]: https://mergiraf.org/ [tree-sitter]: https://tree-sitter.github.io/tree-sitter/ [tree-sitter-fortran]: https://github.com/stadelmanma/tree-sitter-fortran _Henri Menke_ ## DataShare: Public link passwords Since the migration of the DataShare service to Nextcloud last year, creating password protected public link has been a two step process: Initially the share is created without a password, which must then be set if desired in the _Customize link_ dialog (see Figure 2). If there is an issue with saving the password - for example because it doesn't meet the password strength requirements - the displayed error can easily be overlooked and may lead to the share remaining without a password set at all. For this reason, new link shares will now always be created with a secure random password by default. The password can still be changed or removed if needed. ![A screenshot of the new DataShare link share dialog](221/datashare_link_share_password.png) _Florian Kaiser_ ## News & Events ### AMD workshop Coming up soon, MPCDF will run another AMD GPU workshop with a focus on the MI300A technology in _Viper-GPU_. This time, the workshop is split into two parts: A first part in collaboration with [HLRS](https://www.hlrs.de) (they also have AMD MI300A APUs in their Hunter system) with four half-day sessions of lectures and exercises in the afternoons of April 21st to 24th, 2026, and a second part with a hackathon on _Viper-GPU_ from April 27th to 29th. The online lectures of the first part will be given by AMD, and the accompanying exercises will be done on AMD cloud resources. The detailed program and the registration link can be found on the HLRS website [AMD Instinct GPU Training](https://www.hlrs.de/training/2026/gpu-amd). For the online hackathon on the MPCDF _Viper-GPU_ system the week after, users are invited to apply with their code to bring in and to work on profiling and optimization aspects or specific porting issues, supported by experts from AMD and MPCDF. Participants are expected to have a good understanding of their code and they should also be familiar with the relevant parts of the lectures of the previous week, as there will be no introductory lectures for the week of the hackathon. Registration for this event has to be done separately at [MPCDF AMD GPU Hackathon](https://plan.events.mpg.de/e/amd-gpu-hackathon-2026). _Tilman Dannert_ ### Introduction to MPCDF Services The next session of our introductory online course, which is designed to familiarize new users with the MPCDF compute and data services, will be held on April 30th, 2026, 14:00-16:30, online. No registration is necessary, you can just join with the [link](https://mpcdf-mpg-de.zoom-x.de/j/69500447868?pwd=5fQ3xerZMaMDh77215EaPyPVm4MGnM.1) published on our website. The link is only active at the time of the workshop. _Tilman Dannert_ ### IT4Science Days 2026 Save-the-date: This year's IT4Science Days - including the "MPG DV-Treffen" - will take place from September 29th to October 1st at the MPG Faßberg campus in Göttingen. Further information as well as registration will become available through the [website of the conference](https://plan.events.mpg.de/event/670/). As usual, the "IT-Verantwortlichen Treffen" will start the day before (Monday, September 28). For reference, see also [last year's meeting site](https://plan.events.mpg.de/event/474/). _Raphael Ritz_ ### RDA Deutschland Tagung 2026 The Research Data Alliance Germany had its yearly conference again at the "Geoforschungszentrum (GFZ)" in Potsdam February 24th-25th, 2026. Focus topics were the forthcoming "Forschungsdatengesetz" as well as the future of the National Research Data Infrastructure (NFDI). Further information including slides from most of the contributions are available from the [conference website](https://indico.desy.de/event/50156/overview). As in previous years, MPCDF helped to organize the event and contributed to the programm. _Raphael Ritz_ Bits and Bytes Logo # No.220, December 2025 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_220.pdf) ## High-performance Computing ### Viper News In early December also the second part of the Viper deployment, _Viper-GPU_, was formally accepted resulting in the removal of the formal warning of "user-risk" mode operation. The compute nodes are still awaiting a BIOS and firmware update to fix an issue with nodes occasionally crashing when Inifiniband traffic is routed across the two sockets of a node. As a workaround, the default MPI runtime configuration on _Viper-GPU_ accounts for each of the two sockets communicating only via their local network interface into the (dual-rail) Infiniband network and disables the automatic splitting of large MPI messages across the two interfaces. Similar mitigations are necessary for applications built on top of other communication libraries such as rccl. Specifically for machine-learning frameworks like Pytorch, users are advised to follow the [recommendations for running on Viper-GPU](https://docs.mpcdf.mpg.de/doc/computing/software/data_analytics-machine_learning.html#cautions-and-bestpractice-notes-for-ai-workloads-on-hpc-systems). On _Viper-GPU_ an additional storage system based on "NVMe over Fabrics" was deployed which allows users to optionally attach fast node-local scratch storage to all nodes of a job using Slurm commands. Details can be found in the [technical documentation](https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-flash-accelerators.html). Early next year the machine will be further expanded by 27 compute nodes (_Viper-GPU_, with 2 MI300A APUs each) and 5 additional login nodes (_Viper-CPU_ and _Viper-GPU_) as a compensation for various delays in the delivery and deployment process. MPCDF continuously updates the software stack on both _Viper_ machines and provides training and application support for users, in collaboration with software engineers and application experts from AMD and Eviden. _Markus Rampp_ ## Software News ### AMD software on *Viper-GPU* The two most recent versions of the AMD ROCm software suite have been installed on *Viper-GPU*. The modules `rocm/7.0` and `rocm/7.1` provide the ROCm versions 7.0.1 and 7.1.0, respectively. A new version of the LLVM-based AMD compiler `amd-llvm` has also been made available. Its version number 22.2 follows a new versioning scheme aligned to the LLVM release numbers. Hence, `amd-llvm/22.2` is the successor of `amd-llvm/7.1`. Fortran users, in particular, are advised to use the `amd-flang` compiler provided by this module, as AMD continues to release updates (so called "drops") for `amd-flang` with a higher cadence than for ROCm. _Tobias Melson_ ### New version of JAX on *Viper-GPU* and *Raven* [JAX](https://docs.jax.dev/en/latest/) is a Python library offering high-performance numerical computing and large-scale machine learning functionality on various backends comprising CPUs and accelerator devices from various vendors. On *Viper-GPU* (AMD/ROCm), JAX 0.7.1 has been installed in the module hierarchy for `python-waterboa/2025.06` and `rocm/7.0`. As usual, it can be found via `find-module jax`. On _Raven_ (Nvidia/CUDA), JAX 0.7.1 is also provided as a module compatible with `cuda/12.8` for convenience. But for Nvidia GPUs, users who prefer different releases can also generally install JAX following the [offical documentation](https://docs.jax.dev/en/latest/installation.html#nvidia-gpu). _Sebastian Kehl_ ### Major NumPy version update introduced with `python-waterboa/2025.06` Since summer 2025 a newer version of our own Python distribution (`python-waterboa/2025.06`) is available on the HPC systems and clusters. This comes with a few notable changes from the previous version, in particular with respect to the scientific package NumPy which is now available in version 2.1.3. Most importantly, with the major version 2.0.0 the developers of NumPy have decided to *remove* many things that have been marked as deprecated in previous versions. The most obvious breaking change is probably the removal of the type aliases `np.int` and `np.float`. Users will either have to adapt their code or explicitly load the old version of the module to fix these issues. For a detailed list of all changes, refer to the [NumPy 2.0.0 release notes][numpy-release-notes] and for guidance on how to port old code to the [NumPy 2.0 migration guide][numpy-migration-guide]. [numpy-release-notes]: https://numpy.org/devdocs/release/2.0.0-notes.html [numpy-migration-guide]: https://numpy.org/devdocs/numpy_2_0_migration_guide.html _Henri Menke_ ### Intel software stack The new Intel oneAPI 2025.3 has been made available on *Raven*, *Viper* and other clusters. It provides the compiler module `intel/2025.3`, the MPI module `impi/2021.17`, the MKL module `mkl/2025.3`, and modules for the Intel profiling tools. As usual, the scientific software stack has been compiled with this toolchain. The new `ifx` compiler also contains an enhancement for Fortran codes using the `COMPLEX` datatype. The compiler is now in many cases able to apply vector load-store (VLS) operations that combine load and store instructions. Codes with strided access of complex arrays in loops can benefit from this improvent, which is enabled by default. On *Raven*, the default MKL version being loaded by invoking `module load mkl` will change. Currently, it points to version 2021.1. Since then, substantial improvements have been made to MKL. Starting from January 1st, the `module load mkl` command will always refer to the most recent version, which is 2025.3 at the moment. Users can still explicitly load older releases by specifying a version (e.g., `module load mkl/2024.0`). _Tobias Melson_ ### Provisioning of AI Software Both AMD and Nvidia provide highly optimized software stacks for machine-learning (ML) and artificial intelligence (AI) applications via containers. It is thus recommended to run ML & AI workflows by using the latest containers provided by AMD (for _Viper_) or Nvidia (for _Raven_). Specific hints and further references can be found in our [documentaion](https://docs.mpcdf.mpg.de/doc/computing/software/data_analytics-machine_learning.html). _Andreas Marek_ ### New ELPA module version scheme The [ELPA library](https://elpa.mpcdf.mpg.de) provides highly optimized solvers for dense symmetric (Hermitian) eigenproblems. It delivers great performance on CPUs and GPUs and is available on *Viper*, *Raven*, and other clusters. Users can invoke `find-module elpa` to see all available ELPA modules. With the upcoming ELPA release next year, we will change the versioning scheme of the corresponding environment modules. Instead of using the full version number (e.g., `elpa/mpi/standard/2025.01.001`), the bugfix number will be dropped (e.g., `elpa/mpi/standard/2026.01`). Bugfix releases can thus be provided seamlessly without the need for users to adapt their build scripts. _Tobias Melson_ ## Introducing the MPCDF LLM Inference Service ![A screenshot of the LLM Inference Service app](220/llmis.png) We are proud to welcome a new member to our service portfolio: the MPCDF LLM Inference Service (LLMIS), available at [https://llm.mpcdf.mpg.de](https://llm.mpcdf.mpg.de). There are over 100,000 open language models hosted on [Hugging Face](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending). They cover a wide range of parameter sizes, from tiny models with a few million parameters to really gigantic models with up to one trillion parameters. While smaller models can outperform larger ones on specific tasks, especially when fine‑tuned, the general capabilities of the biggest open models rival those of closed models such as GPT or Gemini. However, running even the “smaller” models efficiently already requires notable compute resources, and the largest models require substantial AI hardware that is often out of reach for individual research groups. MPCDF has both the resources and the expertise to operate these models. With our new *LLM Inference Service*, it becomes straightforward for you to interactively test even the largest open models available. ![Screenshot of the "Add endpoint" dialog in the LLMIS](220/llmis_add_endpoint.png) The LLM Inference Service is a flexible, yet easy‑to‑use web application that allows you to create endpoints exposing a model via a REST API. For this we rely on popular inference frameworks such as [vLLM](https://github.com/vllm-project/vllm) and [Ollama](https://ollama.com/). Via an intuitive UI, users can request the desired hardware and configure the framework. The service then takes care of submitting the Slurm job and routing the endpoint, so that you can conveniently access the REST API from your local machine or your existing tools. Currently, two of the most powerful GPU systems at MPCDF, [_dais_](https://docs.mpcdf.mpg.de/doc/computing/dais-user-guide.html) and [_Viper-GPU_](https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-user-guide.html), are connected to the service. We provide sensible default configurations for the frameworks to help you get started quickly. At the same time, you remain free to tune the framework parameters to your needs and to run any model and modality supported by the respective framework, including your own fine‑tuned models, provided they are hosted on the Hugging Face Hub. ![Screenshot of the Recipes page in the LLMIS](220/llmis_recipes.png) The LLM Inference Service is targeted at researchers who wish to interactively evaluate specific open models or conduct interactive user studies. For non-interactive workloads, such as extensive benchmarks or offline evaluations, we recommend using Slurm batch jobs to ensure efficient resource utilization. Example scripts for submitting such jobs are available in our [LLMs-meet-MPCDF](https://gitlab.mpcdf.mpg.de/dataanalytics-public/llms-meet-mpcdf) GitLab repository. Additionally, for users interested in testing "standard" open models, the [Chat AI](https://docs.hpc.gwdg.de/services/chat-ai/index.html) service by GWDG is an excellent alternative. It offers a user-friendly chat interface as well as access to an [inference API](https://docs.hpc.gwdg.de/services/saia/index.html). We hope the LLM Inference Service will facilitate your scientific work and open up new research opportunities. You can expect the service to evolve over time as we add more hardware options and additional inference frameworks. Any kind of feedback is much appreciated; we are looking forward to hearing from you in the [AI@MPCDF discourse channel](https://discourse.hpc.gwdg.de/c/ai-at-mpcdf/), or via our [helpdesk](mailto:support@mpcdf.mpg.de). _David Carreto Fidalgo, Nastassya Horlava, Andreas Marek_ ## GitLab CI ### Building Docker images with Podman-Runners Some users of the _Continuous Integration_ in GitLab need to create their own custom container images. As the Docker-based runners in our GitLab infrastructure are running in unprivileged mode, using _Kaniko_ was one of the possible solutions. With Kaniko, a user can create container images from Dockerfiles inside Docker containers or Kubernetes pods. Due to some recurring issues, Kaniko's repository was archived earlier this summer, and is no longer maintained. After taking alternatives into consideration, we decided to introduce an alternative, more stable solution: two new GitLab runners specifically designed for building custom Docker images. The new runners are: * podman-runner-01 * podman-runner-02 As the names suggest, these runners are Podman-based rather than Docker-based. [Podman](https://www.redhat.com/en/blog/podman-inside-container) offers a smoother, more secure way to create new container images inside a CI pipeline. Below is an example of how to build an image that you can integrate into your CI pipeline: ``` .gitlab-ci.yaml build_image: tags: - image-builder variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG BUILDAH_FORMAT: dockerNew GitLab Runner Tags BUILDAH_ISOLATION: chroot image: quay.io/buildah/stable before_script: - buildah login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY script: - buildah build -t $IMAGE_TAG . - buildah push $IMAGE_TAG ``` #### Required Values: **tags** Your job must be tagged with "image-builder" or it won't run on the correct GitLab runner. **BUILDAH_FORMAT** Use docker instead of oci, especially when building derivative images, for better compatibility. **BUILDAH_ISOLATION** Set this to chroot because the default runc runtime doesn't work in a rootless environment. **image** We recommend using either quay.io/buildah/stable or quay.io/podman/stable (Buildah is included in the Podman binaries). **before_script and script** These sections handle authentication to Gitlab's container registry and the image build/push process. Use [buildah](https://docs.gitlab.com/ci/docker/buildah_rootless_tutorial/#configure-the-job) instead of docker commands to avoid confusion and ensure compatibility. As the names suggest, these runners are Podman-based rather than Docker-based. Building container images inside a rootless, unprivileged environment can be tricky, and Podman offers a smoother, more secure solution for this use case. Keep in mind that different stages of your CI pipeline may run on separate GitLab runners. If your build stage produces files or binaries that need to be included in your final container image, you have to make those artifacts available across jobs and runners. To do this, add an artifact definition at the end of your build stage: ```.gitlab-ci.yaml [...] artifacts: paths: - path/to/artifact expire_in: 1h [...] ``` This ensures that the output from one job can be reused in a later job that builds the container image. If you are still using Kaniko or need to build custom Docker images within a containerized environment, this setup provides a secure, modern, and fully supported alternative! _Francesco Turcinovich_ ### New tags for MPCDF GitLab runners For executing Continuous Integration Pipelines, the MPCDF operates shared GitLab Runners, which can be used by any GitLab user. To better reflect the hardware landscape of the HPC clusters, the runner infrastructure has been continuously extended and equipped with new hardware, including AMD and Nvidia-based GPUs. In parallel, the basic capabilities of the shared runners have been streamlined so that all of them now support the same features (e.g. distributed cache). To choose a runner with specific capabilities, users can specify tags in their CI pipelines. To make this tagging system more explicit and future-proof, we decided to re-implement it from scratch. The new tags will be added in addition to the existing ones. Until March 1st, 2026, both tagging systems can be used in parallel. After that date, the old tags will be removed, and only the new ones will remain. Please ensure that you have adapted your CI pipelines by then. #### New Tags **mpcdf-shared:** All of our managed shared runners do have this tag. Use this tag if you want to make sure the job runs on one of the MPCDF shared GitLab runners instead of runners of other GitLab instances or user-started runners. **image-builder:** Use this tag when you need to build your own custom Docker image (read article "Building Docker images with Podman-Runners" above). #### Hardware-specific tags They are organized hierarchically based on their level of specificity **gpu:** Use when a runner with a GPU is needed, but the vendor or architecture remains unspecified. ![Hierarchy of tags for GPU runner usage](220/gpu.diagram.drawio.svg) **gpu-amd:** Use when an AMD GPU runner is needed, regardless of architecture or instruction set. **gpu-amd-gfx90a:** Use when an AMD runner with offload architecture gfx90a (e.g. for the MI200 GPU) is required. **gpu-nvidia:** Use when an Nvidia GPU runner is needed, regardless of architecture or instruction set. **gpu-nvidia-cc80:** Use when an Nvidia runner with Compute Capability 8.0 (e.g. for the A40 or A100 GPU) is required. **cpu:** Use when a runner with a CPU is needed, but the vendor or architecture remains unspecified. ![Hierarchy of tags for CPU runner usage. Currently no arm64 runner is available.](220/cpu.diagram.drawio.svg) **cpu-x64:** Use when a runner with x64 architecture is needed. **cpu-arm64:** Use when a runner with ARM64 architecture is needed in the future (dashed line in Fig. 5). #### Old Tags As our pool of runners is homogeneous in configuration, most of the old tags have become redundant or superfluous. Here is what we removed and why: **docker:** No longer needed, as all runners are Docker-ready. **distributedcache:** All runners are now configured to enable the distributed cache for job artifacts. **avx / avx2 / avx512:** All runners are compatible with these CPU flags. **shared:** Renamed to mpcdf-shared for clarity. **cloud:** Previously used to differentiate underlying hardware; no longer necessary. **modules:** Previously used before modules were made available as Docker images. _Francesco Turcinovich_ ### New naming scheme for CI module images containing CUDA and ROCm MPCDF offers a variety of Docker images for running jobs in GitLab CI pipelines. These images provide environment modules that are very similar to those on our clusters, enabling users to run their codes on the GitLab CI runners with the same software stack as in production on the clusters. Detailed documentation can be found [here](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html#docker-images-for-ci-with-mpcdf-environment-modules). Following our standard update policy, all images labelled 2025 will be frozen on January 1st, 2026. New images for 2026 will contain updated versions of the included packages. Currently, we refrain from removing software from the images during the year. However, the rapid development and size of the ROCm package forces us to adapt the image naming scheme to further maintain this strategy. The following changes will apply with the turn of the year. Based on the existing images for certain compiler and MPI combinations, we will create additional images containing `cuda` or `rocm` modules. For example, the image `gcc_15-openmpi_5_0-rocm_7_1` will be based on `gcc_15-openmpi_5_0` with an additional `rocm/7.1` module installed. Images that are not labelled accordingly will not contain installations of `cuda` or `rocm` modules. All new images will be added to the existing list on [this website](https://mpcdf.pages.mpcdf.de/ci-module-image/). Unversioned images will also be provided, for example `gcc-openmpi-cuda` or `gcc-openmpi-rocm`, pointing to the latest versioned images. This will be useful for users who are not concerned with the specific version of the compiler, MPI library, or GPU software, but who want to run their codes with the latest releases. Be aware that only the latest LLVM-based new AMD compiler `amd-llvm` will be included in the `rocm` images. Due to a size limitation, older versions of this particular module will be deleted when the module gets updated. To use the latest version of the `amd-llvm` module in your pipeline automatically, load it with the command `module load amd-llvm` without specifying any version number. _Tobias Melson, Klaus Reuter_ ## HPC-Cloud Software Updates The MPCDF HPC-Cloud provides on-demand computing and storage resources to research projects of the Max Planck Institutes. As an infrastructure-as-a-service cloud, it offers self-provisioned servers, networking, and storage resources through a high-level API, CLI, and GUI. ![](220/tripleo_logo_1p5in.png) For the past four years, the HPC-Cloud infrastructure has been deployed and managed by the TripleO tool. It has served us well during this time for tasks such as adding compute resources and managing software upgrades. However, in February 2023, it was [announced](https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032083.html) that the project would be discontinued, leaving us without an upgrade path for the OpenStack "Wallaby" release currently in production. Therefore, we started the process of switching to a different tool. Since the HPC-Cloud is a productive platform on which many projects and users depend, this posed the following technical challenge: How do we migrate away from TripleO with as little disruption as possible for our users? #### Looking for alternatives After an initial evaluation round covering many of the [deployment tools](https://www.openstack.org/software/project-navigator/deployment-tools) that can be used to provision a full-featured OpenStack cloud, we made the decision to migrate to Kolla-ansible. Some of the factors that tipped the scales in favor of this tool were its similarity in terms of architecture of the control, network and compute planes to TripleO, and an extensive community that has been keeping the project healthy, supporting the newest releases of the OpenStack components. ![](220/kolla_logo_2in.png) Kolla-ansible has several interesting characteristics that make it a good fit: 1. Control plane and compute architecture similar to TripleO, i.e. containerized services running on baremetal servers. 2. Companion project Kolla provided us with a workflow to build our own versions of the containerized OpenStack services, allowing us to patch them much more easily and freely than with TripleO. 3. Written in Ansible, which is already used extensively throughout the Virtualization and Storage Teams at the MPCDF, and is also highly customizable. #### Preparing for the migration Having decided on a new tool, the next challenge was to design a migration plan that allowed us to replace the TripleO-deployed services with Kolla-ansible-deployed equivalents, while still keeping the system up and running. In order to achieve this, we had to make sure that the Kolla-ansible services had the same configuration and features as before. This meant first changing the software versions of the deployed services as little as possible to maintain compatibility to the rest of the system, second, making sure the Kolla-ansible-generated configuration would be as similar as possible to the TripleO counterpart, and finally, testing that the system was still stable after deploying each replacement. None of this could be fully automated, since it is not a feature that either of the two tools provide, and it is highly dependent on the initial configuration. Thus, it required significant manual work to adjust all details. For each service, a rough outline of the migration procedure would be: 1. Analyze the TripleO configuration and try to match relevant settings to Kolla-ansible counterparts. 2. If not possible to map an option, customize the Ansible code that deploys the service or any relevant configuration file templates. 3. Move TripleO components out of the way, but leave containers and configuration files on any affected servers to permit changes to be reverted if needed. 4. Deploy with Kolla-ansible. 5. Compare the generated configuration to TripleO. 6. Test service integration. 7. Repeat until satisfied with the end result. Several core services required even a few extra steps. For instance, the main database store had to be moved to a different location. Or highly-available services had to be unregistered first in order to stop all instances. All of which required adding additional logic before step four in the form of Ansible code. Moreover, going through this iterative process on the productive system would have been impossible without disrupting the users, thus a test environment was used. Our test environment is a scaled-down version of the productive HPC-Cloud, with fewer, smaller servers, but with the same set of deployed services, configured with settings as similar as possible at all times. #### Conclusions Following a successful primary stage of the migration in early November, 17 out of 20 services have been migrated to Kolla-ansible. A secondary phase will be scheduled for early 2026 to finish up the remaining services, leaving us prepared for a major upgrade to the next OpenStack release later in the year. Apart from solving the issues mentioned above, there are already two user-visible changes: A more featureful version of the noVNC embedded graphical console and a new version of the web dashboard with 2FA support. Although not yet mandatory during this transition phase, the 2FA-enabled "MPCDF Login" authentication will become the only accepted method for the dashboard. This new configuration puts us in a much better position to roll out major release upgrades as well as new services, which will result in a more robust HPC-Cloud with more features and better performance. _Brian Standley, Maximiliano Geier_ ## Globus Migration to SelfService/MPCDF SSO In January 2026 the Globus services at MPCDF will be re-configured to use the MPCDF Single sign on (SSO) and the opt-in service model via SelfService. These changes will impact all Globus Services at MPCDF (DataHub, GO-Nexus, GO-S3) and in some cases the association to Globus Groups within the Globus Web Portal. The migration to MPCDF SSO will provide both, 2FA and SSO functionality, for improved security and a better user experience. Moreover, these changes bring Globus in line with other MPCDF services such as GitLab and DataShare, providing a common user experience. A transition period is foreseen for January 12th until February 9th and we ask users to actively test their accounts during this period. More details regarding each aspect of the re-configuration are provided here. **Opt-In:** In the future users will be required to opt-in to the Globus Services via SelfService (see screenshot in Fig. 6). Where possible this option has already been enabled for existing Globus users. However, we kindly ask users to check their status in the [SelfService](https://selfservice.mpcdf.mpg.de). Please note that opt-in is only available for standard MPCDF user accounts, invited guests (g-account) do not have this option. ![SelfService Opt-In for Globus](220/Globus-selfservice-opt-in-closeup.png) **SSO:** From January 12th to February 9th access to the Globus endpoints at MPCDF will be possible using either login.datahub.mpcdf.mpg.de or mpcdf.mpg.de domains. The existing login domain (login.datahub.mpcdf.mpg.de) will be decommissioned on February 9th. After that date only the new domain (mpcdf.mpg.de) which is provided by the MPCDF SSO will be available for login. **Groups within the Globus Web Portal:** Users who have registered for access to the _Max Planck Computing and Data Facility_ and _MPCDF Flows Users_ Groups within the Globus web portal will be required to re-register, ideally using their primary ID in Globus. On January 12th existing Group access to user with login.datahub.mpcdf.mpg.de as domain will be revoked, forcing users to re-register. On February 9th the login.datahub.mpcdf.mpg.de domain will be decommissioned and access to groups via that ID will no longer be possible. We ask all Globus users to please check their Opt-In Status, test access to Globus services via MPCDF SSO and cross-check their group membership in the Globus web portal (re-applying for membership if needed). **Note:** During the overlap period, January 12th to February 9th, both login domains may be used. However, only users who have opted in to the Globus service will be able to access the Globus endpoints at MPCDF. _John Alan Kennedy_ ## News & Events ### Multifactor authentication: deactivation of E-mail tokens The deactivation of E-mail tokens that was [announced in the last issue Bits & Bytes](https://docs.mpcdf.mpg.de/bnb/219.html#multifactor-authentication-deactivation-of-e-mail-tokens) had to be postponed. The plan is now to inform all users directly and then to deactivate in the course of the first quarter of 2026, after all users have switched over to one of the supported token types: app, external hardware tokens, SMS, and TAN list. Note, that SMS and TAN list are intended only as fallback. Users are kindly asked to ensure that they have a valid app or hardware token, ideally in combination with an SMS token or a TAN list as backup. _Kathrin Beck, Andreas Schott_ ### International HPC Summer School 2026 The International HPC Summer School (IHPCSS) 2026 will take place from July 12th to July 17th in Perth, Australia. The series of these annual events started 2010 in Sicily, Italy and provides advanced HPC knowledge to computational scientists, focusing on postdocs and PhD students. Through the participation of Canada, the USA, South Africa, Japan, Australia and Europe a truly international group of highly motivated students is meeting each year. Interested students and postdoctoral fellows are invited to apply at the [school's website](https://ss26.ihpcss.org) by January 31st, 2026. School fees, travel, meals and housing will be covered for all accepted applicants through funds from the European Union and EuroHPC. For further information, please visit the website of the summer school. _Erwin Laure_ ### AMD workshop Next spring, MPCDF will organize another AMD GPU workshop with a focus on the MI300A technology in _Viper-GPU_. This time, the workshop is split into two parts: A first part in collaboration with [HLRS](https://www.hlrs.de) (they also have AMD MI300A APUs in their Hunter system) with four half-day sessions of lectures and exercises in the afternoons of April 21st to 24th, 2026 , and a second part with a hackathon on _Viper-GPU_ from April 27th to 29th. The online lectures of the first part will be given by AMD, and the accompanying exercises will be done on AMD cloud resources. The program will soon be published on our webpage, and attendees may select parts of the program according to their knowledge and topics of interest. For the online hackathon on the MPCDF _Viper-GPU_ system the week after users are invited to apply with their code to bring in and to work on profiling and optimization aspects or specific porting issues, supported by experts from AMD and MPCDF. Participants are expected to have a good understanding of their code and they should also be familiar with the relevant parts of the lectures of the previous week, as there will be no introductory lectures for the week of the hackathon. Registration for both events will open early next year and will be announced on our website. _Tilman Dannert_ ### Meet MPCDF In our "Meet MPCDF" series, one seminar is planned so far for the beginning of next year: - February 5th, 15:30-16:30, Efficient usage of the tape archive We encourage our users to propose further topics of their interest, e.g. in the domains of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### Introduction to MPCDF Services The next session of our introductory online course, which is designed to familiarize new users with the MPCDF compute and data services, will be held on April 30th, 2026, 14:00-16:30, online. No registration is necessary, you can just join with the link published on our website. The link is only active at the time of the workshop. _Tilman Dannert_ Bits and Bytes Logo # No.219, August 2025 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_219.pdf) ## High-performance Computing ### HPC system _Viper_ The new HPC system _Viper_ was deployed in two phases, _Viper-CPU_ and _Viper-GPU_. _Viper-CPU_ has been in stable operation and fully utilized for over a year. The machine recently passed a number of final robustness and performance tests. These included a High-Performance Linpack (HPL) benchmark which delivered 4.3 PFlop/s, and a successful verification of the contracted application performance, based on the MPG benchmark suite consisting of major HPC application codes developed and used in the MPG. The machine is now formally accepted, resulting in the removal of the formal warning of "user-risk" mode operation. The second part of the deployment, _Viper-GPU_, has been in early-operation mode since February, and was successfully [benchmarked and ranked in the June 2025 issue of the Top500 list](https://www.mpcdf.mpg.de/115266/news20250610-viperintop500) with an HPL performance of 31.1 PFlop/s using all 300 nodes (600 AMD MI300A APUs). The Slurm-based scheduling and accounting was integrated with _Viper-CPU_ during a maintenance at the end of July. This also marked the beginning of official accounting for _Viper-GPU_, with a weighting factor of two to be applied per node-hour on _Viper-GPU_ relative to _Viper-CPU_. This part of the machine will continue to operate in "user-risk" mode until it passes final robustness and performance tests, which are planned for the next few weeks. MPCDF continuously updates the software stack on both _Viper_ machines and provides application support for users, in collaboration with software engineers and application experts from AMD and Eviden. _Markus Rampp_ ### Using Visual Studio Code (VSCode) with the Remote-SSH extension Visual Studio Code (VSCode) is a popular code editor that can be enhanced with a wide variety of extensions. For working with MPCDF systems, the "Remote-SSH" extension is particularly useful. It allows users to connect to a remote machine over SSH, enabling them to edit files and run commands directly on the remote system as if it were local, combining the convenience of a local editor with the capabilities of the remote HPC environment. Users can find up-to-date instructions on the necessary configuration for Linux and MacOS as part of the [MPCDF FAQ](https://docs.mpcdf.mpg.de/faq/connecting.html#how-can-i-connect-to-hpc-systems-and-clusters-using-visual-studio-code-vscode). Please be aware that we cannot guarantee that the Remote-SSH extension will work in all cases due to the plethora of specifics in the configurations, operating systems, and versions across users. _Klaus Reuter_ ## Software News ### New ELPA version 2025.06.001 The latest [ELPA eigensolver library](https://elpa.mpcdf.mpg.de) (version 2025.06) delivers significant performance enhancements, particularly for GPU-based computations at MPCDF. Notably, the new release features improved support for AMD GPUs via ROCm 6.4.2 recently made available on _Viper_. Optimized rocSOLVER functions provided by this ROCm update achieve up to a 5x performance boost when using ELPA with RCCL (ROCm Collective Communication Library). Additionally, general GPU performance has increased by up to 15% for both standard and generalized eigenproblems. This ELPA version also addresses a bug that previously prevented NCCL/RCCL builds from handling more than one MPI process per GPU in generalized eigenproblems and auxiliary ELPA routines. Furthermore, ELPA is now available on _Raven_ with a specialized toolchain combining the Intel compiler, CUDA, and excluding NCCL support, which is especially suitable for applications such as FHI-aims. To access this new toolchain on _Raven_, load the following modules: `intel/2025.2 impi/2021.16 mkl/2025.2 cuda/12.6` and use the module `elpa/mpi/standard/gpu/2025.06.001`. All other available module combinations can be queried with `find-module elpa.*2025.06.001`. _Petr Karpov, Tobias Melson, Andreas Marek_ ### Compilers New compilers have been made available on _Raven_, _Viper-CPU_, _Viper-GPU_, and other clusters: the GNU compiler collection 15.1 and Intel oneAPI 2025.2. The former can be used by loading the module `gcc/15`. The latter provides the compiler module `intel/2025.2`, the corresponding MPI module `impi/2021.16`, the MKL module `mkl/2025.2`, and modules for the Intel profiling tools. As usual, the full scientific software stack is compiled with these toolchains. _Tobias Melson_ ### Water Boa Python 2025.06 To address a significant change in the licensing of the popular Anaconda Python Distribution, MPCDF has been providing a free drop-in replacement for a scientific Python software stack since June 2024, labeled "Water Boa Python". This ensures that all users have access to a comprehensive scientific Python environment without licensing concerns. Recently, version 2025.06 was rolled out on the HPC systems and clusters. It is built entirely from the open-source `conda-forge` channel and offers a robust and up-to-date collection of packages for scientific computing, data analysis, and visualization. It is based on cPython 3.13. The package list and versions are very similar to those of the commercial counterpart, making it easy to transition existing workflows from Anaconda. On the MPCDF systems, it is available via the environment module `python-waterboa/2025.06`. After loading the environment module, running `conda list` will show the full list of the available Python packages. We encourage users to adopt Water Boa Python for their scientific workflows. The project's scripts and package lists are available internally on [MPCDF GitLab](https://gitlab.mpcdf.mpg.de/mpcdf/water-boa-python). _Klaus Reuter_ ### Accelerated prediction of protein structures and complexes with ColabFold [ColabFold](https://github.com/sokrypton/ColabFold) is a user-friendly tool that streamlines protein structure prediction by combining the capabilities of AlphaFold2 with the rapid Multiple Sequence Alignment (MSA) computation of MMseqs2. MPCDF's local installations of ColabFold have recently been upgraded to include GPU-accelerated MSA based on [MMseqs2-GPU](https://github.com/soedinglab/MMseqs2) for Nvidia hardware. The databases are stored on special NVMe-backed file systems separate from the '/ptmp' and '/u' file systems, to optimize for the intense disk-IO during the MSA phase and to avoid a slowdown of the general-purpose file systems. ColabFold runs [locally](https://github.com/YoshitakaMo/localcolabfold) on MPCDF resources, without putting load on ColabFold's public MSA servers. Users of the plain AlphaFold2 installations might be interested in considering a transition to ColabFold. Information on how to get started is available via the command `module help colabfold/202507` on _Raven_ as well as on some institute clusters. Technical information on how the accelerated MSA works is given in a post on the [Nvidia Developer Blog](https://developer.nvidia.com/blog/boost-alphafold2-protein-structure-prediction-with-gpu-accelerated-mmseqs2/). Please note that the speedups reported in the blog were achieved under ideal conditions on optimized hardware and will be more modest in multi-user HPC environments. _Klaus Reuter_ ### AlphaFold2 available on _Viper-GPU_ Similarly to the frequently used installations for Nvidia GPUs on _Raven_ and on selected clusters, AlphaFold2 is now available on _Viper-GPU_ as well. To port the original codebase from Nvidia GPUs to AMD, several core dependencies such as JAX and OpenMM have been replaced with specific builds for the AMD MI300A APUs. Information on how to use AlphaFold2 on _Viper-GPU_ can be obtained from running the command `module help alphafold/2.3.2-2025`. _Klaus Reuter_ ### DataShare command-line client (`ds`/`pocli`) modernized In preparation for the migration of the MPCDF DataShare service to the Nextcloud platform (see announcement below), the lightweight command-line client `ds` (also known as `pocli`) has been updated to ensure compatibility with the new backend. It is crucial that all users of the `ds` client upgrade, as older versions will no longer function correctly after the migration. - On MPCDF systems, the updated client is provided via the `datashare` environment module. Please ensure you are loading the latest available version of this module. - On local machines (e.g., laptops), you must upgrade your installation by running the command: `pip install --upgrade pocli`. This will fetch the latest version from the Python Package Index ([PyPI](https://pypi.org/project/pocli/)). Further information including the source code for `pocli` is available on [MPCDF GitLab](https://gitlab.mpcdf.mpg.de/mpcdf/pocli). Be aware that the use of an application password (also known as device-specific password) is mandatory, entering the regular user password will not work. More details are given [below](#multi-factor-authentication-mandatory). _Florian Kaiser, Klaus Reuter_ ## Using Access Tokens in GitLab _GitLab access tokens_ (GAT) are an easy and secure way to access your GitLab repositories. They are meant to establish a machine-to-machine communication between GitLab on the one side and scripts or workflow engines on the other one. Think of a GAT as a substitute for your user account, but you can exactly specify in which context the token is allowed to do what. Beside some special kinds of access tokens, GitLab supports the following main and most important types of access tokens (left side of the graphic): * __Personal Access Token:__ these are the most powerful tokens, as they can act directly "in the name of the user" with all his permissions * __Group Access Token:__ tokens assigned to a GitLab group, allowing for permissions across multiple projects within that group * __Project Access Token:__ created at the project level, allowing restricted access to that specific project ![Overview of the GitLab Access Tokens](219/gat_01.png) When creating a token, the user can specify in a fine-graned manner which permissions - so called scopes - the token should have. For example, the scopes _read_repository_ and _write_repository_ can be used to give the token read/write access to the Git repositories it belongs to. Depending on the permissions the user has in the current context, also setting the _User Role_ of an access token needs to be done accordingly. When creating an access token in GitLab, all of these three dimensions (type, scope and role) have to be taken into account. Choosing the proper configuration of an access token can be somewhat tricky depending on the use case. ### Using access tokens on the command line One main use case for access tokens is authentication on the command line. This can be either a human interaction (avoiding SSH keys) or a script (maybe as part of a continuous-integration pipeline). To allow an access token to _clone/pull_ and _push_ a repository via the HTTPS protocol, the following setup is necessary: * __Type:__ Could be a personal, group or project access token. Preferable is always a token with lowest capabilities. * __Scope:__ _read_registry_ (clone, pull) and its analogue, _write_registry_ (push) are necessary for accessing a repository. * __User Role:__ As the user roles _guest_ and _planner_ have no write access to a repository, at least the user role _reporter_ needs to be assigned for write permissions. In addition to these settings, the user can set an expiration date for any token. Once the token is saved in GitLab, the user has only on the resulting web page the possibility to copy the token to the clipboard. Once the web page is closed, GitLab will never show the token again. During its life time, the token can be used on the command line, in continuous-integration pipelines or scripts to act as a proxy for the user himself. When asked for username and password, the user's account name and instead of his real password the token can be used. More information on GitLab's access token system can be found in the [official documentation](https://docs.gitlab.com/security/tokens/). _Thomas Zastrow_ ## News/Announcements ### DataShare: Migration to Nextcloud The MPCDF DataShare sync&share service based on the _ownCloud_ product has been in operation for more than 10 years, being used daily by hundreds of users to store, edit and share their documents and data. Already in 2016, ownCloud was forked into a new product called _Nextcloud_. Both, ownCloud and Nextcloud co-existed in parallel for most of the time, focussing on different niches in the market. Recently however, momentum around ownCloud has slowed significantly, with the vendor Kiteworks finally announcing its end of life for the end of 2026. For this reason and after careful deliberation and testing, the MPCDF **DataShare service will be migrated to Nextcloud** on **September 13th and 14th, 2025.** **On this weekend, the service will be unavailable!** All data, shares, calendars etc. will be preserved by the migration, and the general functionality and look&feel should remain very similar. Nevertheless, there will be some important changes after the migration: ##### New desktop and mobile clients: If you used the *ownCloud* or branded *DataShare* client on your device before, you will need to **install the [Nextcloud client](https://nextcloud.com/install/#install-clients)** instead in order to continue being able to synchronize your data on that device. You may re-use the ownCloud data directory on your machine in order to avoid downloading all of the files again, provided you **ensure that it is no longer accessed by the ownCloud client** by either uninstalling it, or removing the DataShare account configuration from it. ##### Shares by expired users no longer accessible: Currently, data belonging to an expired user can still be accessed via public links (e.g. `https://datashare.mpcdf.mpg.de/s/`) or by other DataShare users it was shared with. **After the migration to Nextcloud, this will no longer be the case!** If you or your collaborators are still using data belonging to an expired user, **please transfer it to an active user** if possible and re-share it from there. In special cases where this is not feasible, for example if the shared link must remain the same, contact support@mpcdf.mpg.de for assistance. ##### Expired accounts will be deleted after 6 months: Expired accounts including all data, shares, calendars etc. associated with them will be deleted after 6 months. Please make sure to transfer data that should be preserved to another user before your account expires. This will be possible via the Nextcloud web interface after the migration. ##### Pending or rejected shares will not be migrated: Pending shares that you rejected or never accepted will not be migrated, e.g. can no longer be accepted afterwards. ##### Old style "v1" chunking API no longer supported: The old-style "v1" [chunking API](https://github.com/owncloud/core/wiki/spec:-big-file-chunking) for uploading large files will no longer be supported. This should only impact a small number of users using old versions of the *pocli*, *pyocclient* or similar. ##### Custom groups will be converted to Nextcloud Teams: Any ownCloud "custom groups" that you may have created will be converted to *Nextcloud Teams*, also called *Circles*. They generally work very similar and can be managed through the new _Contacts_ app. ##### Multi-factor authentication mandatory: Multi-factor authentication via MPCDF Login will become mandatory for enhanced security. This is the same Single sign-on (SSO) login that was already introduced for Gitlab earlier this year, i.e. you will only have to sign in once per day for any of DataShare, Gitlab, or other MPCDF services that will use SSO in the future. In case you have not set up 2FA in our SelfService for other services such as Gitlab already, we recommend doing so any time in advance in order to ensure you will be able to login to DataShare after the maintenance. If you use third-party clients that do not support MFA, you may create [device specific passwords](https://docs.nextcloud.com/server/30/user_manual/en/session_management.html#managing-devices) for them. _Florian Kaiser, Michele Compostella_ ### A farewell to the AFS cell "ipp-garching.mpg.de" The [Andrew Filesystem](https://en.wikipedia.org/wiki/Andrew_File_System), commonly known as AFS, is reaching its end of life at MPCDF. Introduced to the Max Planck Institute for Plasma Physics (IPP) by Hartmut Reuter and publicly announced in the Bits&Bytes of June 1993 by the then-director of RZG, Stefan Heinzel, it quickly gained importance at RZG, IPP and the neighbouring institutes at the Garching Campus. AFS had several features of a cloud file system: global access, redundant metadata, and fine-grained access control lists. This allowed users to log in to any machine and find the same home directory. Scientists could write their programs on their office computers and compile them on the systems they were meant to run on without needing to copy them around. In 1994, Hartmut Reuter introduced Multi-Resident AFS (MR-AFS), which managed the storage of large data not only on disk, but also on tape. At that time (and actually also today), the available disk space was insufficient to store all the experimental data. Therefore, it could move data to a tape backend while still being visible in the filesystem. When a file was accessed, it was automatically copied back to disk. This HSM (Hierachical Storage Management) system had been in operation until 2008, when it then was replaced by AFS-RXOSD, an RZG/MPCDF in-house development. Later on, IBM's GHI/HPSS took over the role of automatic file transfer to tape, and MPCDF switched back to plain OpenAFS, also since Hartmut had already retired, and further development of AFS-RXOSD had come to an end. Throughout its lifetime at MPCDF, AFS provided many services: users' home directories, storage for experimental data, and software for clusters and office machines alike. However, all kind of technologies eventually come to an end. The AFS network protocol, amongst other things, no longer meets today's standards. The AFS cell "ipp-garching.mpg.de" will therefore be set to **read-only** at the end of **November 2025** and will be **finally shut down in November 2026**. MPCDF would not be where it is today without AFS. Hence, we would like to again thank Hartmut Reuter and all involved colleagues, particularly also from IPP, for their vision and dedication over more than three decades. _Christof Hanke_ ### Multifactor authentication: deactivation of E-mail tokens All MPCDF services are meanwhile protected by two-factor (2FA) authentication. Until now the following token types can be used: app, external hardware tokens, SMS, E-mail, and TAN list. To improve overall security, MPCDF will deactivate the usage of E-mail tokens. Since May, the creation of new E-mail tokens has been disabled, but existing E-mail tokens can still be used until end of September 2025. While you still have access to your E-mail token, please make sure that you also have a valid app or hardware token, ideally in combination with an SMS token as backup. [See also our documentation on 2FA](https://docs.mpcdf.mpg.de/faq/2fa.html). In case you loose access to all registered tokens, a token reset process can be triggered through the MPCDF SelfService interface. For security reasons, in order to make sure that your E-mail account has not been hacked, the contact person registered for your account needs to authenticate your identity and confirm your request. _Kathrin Beck, Andreas Schott_ ### Account re-applications The workflow for MPCDF-account applications and for the reopening of expired accounts has been updated mainly to enable a more comfortable way to reactivate existing accounts. If an MPCDF user account has been deactivated less than six months ago and the user metadata are still valid, it can easily be reactivated by the user through opening a ticket in the helpdesk. If an account is closed for longer than six months, it should be reopened by filling a new account application instead of sending a ticket. This will update the user’s metadata automatically instead of manual adjustment. Thus, users can now submit multiple applications, regardless of whether they have a closed account or a previously rejected account. _Kathrin Beck, Andreas Schott_ ### Export control and updated Terms of Use Export control regulations impose restrictions on the access to and usage of high-performance computing (HPC) systems whose components (compute nodes) exceed certain performance thresholds. The [new AI machine for several Max Planck Institutes](https://docs.mpcdf.mpg.de/bnb/217.html#a-shared-ai-system-for-eleven-mpis), which is about to become operational, is the first system of such kind at MPCDF, and we anticipate that more HPC systems at the MPCDF will fall under this category in the future. Users with access to such systems must now accept an updated version of our Terms of Use. The main relevant change is the new § 7 Sec. 2p), which covers new obligations. In particular, users are required _to comply with all currently applicable export control regulations under EU and national law, including sanctions and embargoes, and, in the event of intended access from outside the European Union, have a corresponding export control review carried out by the export control authorities in good time prior to access and, if approval by the competent authorities is required, only access the data after such approvals have been obtained._ Details can be found in the [OHB of MPG](https://ohb.mpg.de/Policies/MPG%20VII.201%20en%204.0/VII.2.01_Establishment%20of%20an%20MPG-wide%20Export%20Control%20System_en.pdf). In connection with the new Terms of Use, we have introduced a new workflow for accepting the Terms of Use. Users must accept the new Terms of Use in the MPCDF SelfService within one month of their release. If this does not happen, the password's lifetime will be shortened to one month, during which time the user can accept the Terms of Use and extend the password for another year. Furthermore, any extension or change of the password will require re-acceptance of the current Terms of Use. _Kathrin Beck, Andreas Schott_ ## Events ### WAMTA 2026 MPCDF is co-organizing and hosting the 2026 Workshop on Asynchronous Many-Task Systems and Applications (WAMTA) which will take place from Feb 16-18, 2026 at MPCDF in Garching. The objectives of this workshop are to bring together experts in asynchronous many-task frameworks, developers of science codes, performance experts, and hardware vendors to discuss the state-of-the-art techniques needed to program, analyze, benchmark, and profile these codes to achieve maximum performance possible from modern machines. For the first time in this well-established series, the WAMTA 2026 workshop will be held in Europe. Invited speakers include Rosa M. Badia (BSC-CNS), Michael Klemm (AMD, OpenMP ARB), and Nick Brown (EPCC). Further details, including registration information and the call for papers can be found on the [workshop webpage](https://wamta26.stellar-group.org/). _Erwin Laure, Markus Rampp_ ### MPG-DV-Treffen Save-the-date: The 42nd edition of the MPG-DV-Treffen is scheduled for September 23-25, 2025, as part of the _IT4Science Days_ in Bremen. Details are available from the [event page.](https://plan.events.mpg.de/event/474/) _Raphael Ritz_ ### MPG-NFDI and Research Data Management Workshops Registration is open for the third MPG-NFDI Workshop (October 27-28) providing a forum for MPG researchers engaged in NFDI to exchange experiences and successes. It is followed by the 7th MPG-Workshop on Reseach Data Management (October 28-30). Both events take place in Leipzig at the MPI for Evolutionary Anthropology. Further information is available from the [event page.](https://rdm.mpdl.mpg.de/mpdl-services/workshops/7-fdm-workshop-2025-forschungsdatenmanagement-in-der-max-planck-gesellschaft/) _Raphael Ritz_ ### Introduction to MPCDF services The next issue of our semi-annual seminar series "Introduction to MPCDF services" will be given on October 16th, 14:00-16:30 online. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and DataShare, together with a concluding question & answer session. No registration is required, just connect at the time of the workshop via the zoom link provided on our webpage. _Tilman Dannert_ ### Meet MPCDF In our "Meet MPCDF" series, two seminars are planned in autumn: - October 2nd, 15:30-16:30, Diagonalization of Sparse Matrices - November 6th, 15:30-16:30, Possibilities for Enhanced Security in Gitlab We encourage our users to propose further topics of their interest, e.g. in the domains of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### Python for HPC On popular demand, the next MPCDF course on "Python for HPC" will be given in November or early December 2025. Presented online via Zoom, the course will cover essential tools and techniques for using the Python ecosystem efficiently on HPC clusters. The exact date will be announced in September on the MPCDF website, together with a detailed list of topics, a schedule, and a registration link. _Tilman Dannert_ Bits and Bytes Logo # No.218, April 2025 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_218.pdf) ## High-performance Computing ### _Viper-GPU_ operational ![Deployment of Viper-GPU. Image credit: K. Zilker, MPCDF](218/218_viper-gpu-1600.jpg) The second, GPU-powered phase of the new supercomputer _Viper_ of the Max Planck Society was opened for early user operation (still in "user-risk" mode, and with some hardware and software tuning ongoing) in February 2025. The machine currently comprises 228 compute nodes with two AMD Instinct MI300A APUs (accelerated processing unit), each with 128 GB of high-bandwidth memory (HBM3) which is shared coherently between the 228 CDNA3 GPU compute units and 24 Zen4 CPU cores. Another 72 nodes are added in the course of April. The nodes are interconnected with an Nvidia InfiniBand (NDR 400 Gb/s) network with a non-blocking fat-tree topology, and are connected to online storage based on IBM Storage Scale (aka GPFS) parallel file systems (`/u` and `/ptmp`) with a total capacity of ca. 11 PB, and a 480 TB NVMe-based filesystem (`/ri`) for projects with read-intensive applications. A high-level overview of _Viper-GPU_ was given in a [MeetMPCDF seminar](https://www.mpcdf.mpg.de/training/meetmpcdf) on February 6th. Users can find all relevant technical details on the hardware and software and its usage, including example batch submission scripts, in a [comprehensive user guide](https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-user-guide.html). Users are invited to register for our AMD-GPU workshop and _Viper-GPU_ hackathon in May (see "Events" below). Among the most notable differences to the _Raven_ supercomputer, users should be aware that - _Viper-CPU_ and _Viper-GPU_ each have their individual sets of login nodes, file systems, and software (module) stacks - transitioning from an Nvidia-GPU programming environment requires adapting build systems and compiler tool chains. Technical details and practical tips can be found in a detailed [migration guide](https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-user-guide.html#migration-guide-for-users-coming-from-intel-and-nvidia-based-hpc-systems). For code developers a new gitlab shared runner with an AMD GPU is available in the MPCDF GitLab-CI environment (see below). _Markus Rampp_ ### HPC performance monitoring on _Viper-GPU_ The MPCDF is running a comprehensive [performance monitoring system](https://docs.mpcdf.mpg.de/doc/computing/performance-monitoring.html) on the HPC systems that allows support staff as well as users to check on a plethora of performance metrics of compute jobs. Recently, the system was deployed to the _Viper-GPU_ supercomputer. Unlike _Raven_ which uses Nvidia GPUs, _Viper-GPU_ features AMD MI300A APUs that combine a GPU, CPU cores, and high-bandwidth memory on the same socket. These APUs and the available software tools currently do not support high-level metrics such as GFLOPs and memory bandwidths. Users can access their performance data under these limitations, with the metric "GPU utilization" currently being the most important indicator if a job actually makes use of the GPUs. _Klaus Reuter, Sebastian Kehl_ ## Software News ### Activation of python-waterboa as the new Python default As announced in Bits and Bytes No.216, Aug 2024, new versions of the Anaconda Python distribution cannot be provided anymore due to licensing restrictions. Therefore, MPCDF has come up with its own Python basis "Water Boa Python" which is not dissimilar to the original but entirely based on the "conda-forge" project from which packages are freely available. Comprehensive information on the topic was presented in [MeetMPCDF of Oct 2024](https://datashare.mpcdf.mpg.de/s/CZR64ip6fvi6TqA). The environment module "python-waterboa/2024.06" has been available since summer 2024 on the HPC systems and clusters. Please be aware that it meanwhile also serves as the basis for the central software builds provided by MPCDF, replacing Anaconda Python 2023.03. Integrated into the hierarchical environment modules this affects new versions of 'mpi4py', 'h5py-mpi', and other domain-specific software. As usual, existing branches within the modules hierarchy and hence software already installed on the clusters stay unchanged. _Klaus Reuter_ ### New ELPA version 2025.01.001 The new 2025.01 release of the ELPA eigensolver library features significant optimizations for GPUs supporting NCCL/RCCL (Nvidia/ROCm Collective Communication Libraries), which enables efficient direct memory transfers between GPU devices. NCCL/RCCL requires the number of MPI processes to match the total number of GPUs; otherwise, ELPA automatically falls back to the previous GPU+MPI (no NCCL/RCCL) implementation. The new ELPA is available as a module, for example, `elpa/mpi/standard/gpu/2025.01.001` after loading `gcc/13 openmpi/5.0 cuda/12.6` on _Raven_ or `gcc/14 openmpi/5.0 rocm/6.3` on _Viper-GPU_. All available module combinations can be queried with `find-module elpa.*2025.01.001`. _Petr Karpov, Andreas Marek, Tobias Melson_ ## GitLab shared CI runner with AMD GPU available The MPCDF offers a new shared GitLab runner in its GitLab CI environment. It is based on a virtual machine with 8 CPU cores, 32 GB of RAM, 1 TB of disk space and an AMD Instinct MI210 GPU. The runner is named "MPCDF-GPU-05" and will accept only CI-jobs tagged with `amd-mi200`. Accordingly, the MPCDF CI module images were recently extended with AMD software. Currently, the `gcc_14` flavour of these images contains the environment modules `rocm/6.3` and `amd-llvm/6.0`. As both packages are undergoing rapid development, updates will be regularly applied to the CI module images. Unlike our usual policy of retaining all versions, older versions of rocm and amd-llvm will need to be removed from the image after some time. In order to always load the latest versions, simply run `module load rocm` or `module load amd-llvm` without specifying a version number. An up-to-date list of all available images and their contents can be inspected and searched in our [documentation](https://mpcdf.pages.mpcdf.de/ci-module-image/). A snippet from a file `.gitlab-ci.yaml` testing a simple HIP code on the CI runner could look as follows: ```yaml hip_hello_world: tags: - amd-mi200 image: gitlab-registry.mpcdf.mpg.de/mpcdf/ci-module-image/gcc_14:latest script: - module load gcc/14 - module load rocm - rocm-smi --showhw - hipcc --offload-arch=native -o hello.x hip_hello_world.cpp - ./hello.x ``` This example uses the flag `--offload-arch=native` which automatically infers the correct architecture for the locally available GPU (which is `gfx90a` for the MI200 generation of GPUs available with the new CI runner). Please be aware that the flag `--offload-arch=gfx942` which targets the MI300A GPUs on _Viper-GPU_ must not be used on these CI runners. While code compiled using that flag might even run it will likely produce bogus results. _Francesco Turcinovich, Nicolas Fabas, Tobias Melson, Klaus Reuter, Thomas Zastrow_ ## HowTo: External Usage of GitLab Wikis If you need to write project documentation or any other technical document, the "GitLab Wiki" is the right place to do it. Integrated into a GitLab repository, it offers the common Wiki functionalities (please see GitLab's documentation for a complete overview): * Support for several common markup languages (Markdown, RDoc, AsciiDoc, Org) * Nesting of pages * Integration of visualization libraries (LateX math formulas, graph structures, ...) * Searching for a page title Nevertheless, working with a Wiki in GitLab's web interface is not very comfortable: some functionality like a full-text search is still missing. Therefore, it makes sense to clone your Wiki and work on your local machine on it. Cloning a GitLab Wiki? Yes, because behind the scenes a GitLab Wiki is nothing else than another Git repository. Independent of your project's repository, it can be cloned with the command: ``` git clone git@gitlab.mpcdf.mpg.de:NAMESPACE/PROJECT-NAME.wiki.git ``` where NAMESPACE is the path to your GitLab repository (normally your user name or the group(s) of the repository) and PROJECT-NAME is the name of your GitLab project. You can also find the link to the Wiki repository in the web interface: ![Cloning a GitLab Wiki as repository to your local system](218/218_gitlab_wikis.png) Once you have cloned the Wiki repository to your local file system, you can work with the pages and folders in your preferred text editor or IDE. ### Note-taking Apps As GitLab Wikis consist of markup formatted files organized in a folder structure, it is also possible to integrate them into common note-taking apps like Obsidian or Joplin. For example in Obsidian, you can open the local folder which contains the cloned Wiki as a "Vault" (File menu / Open Vault). The full functionality of Obsidian like full-text search, linking between pages etc. is now available for working with your GitLab Wiki. Don't forget to add, commit and push your changes once you are done. The screenshot in Fig. 3 shows the current article, being part of a GitLab Wiki and edited locally in Obsidian. ![Working with the cloned Wiki in the Obsidian note-taking app](218/218_gitlab_wikis_02.png) _Thomas Zastrow_ ## Enhanced SSH Configuration OpenSSH is the most commonly used SSH client on the command line. It is available for Linux, MacOS and Windows (via Powershell). A useful feature is the [configuration file](https://www.man7.org/linux/man-pages/man5/ssh_config.5.html) located in `$HOME/.ssh/config`, which is an alternative to using flags with the ssh command. It allows to define an alias for each host, along with specific connection parameters. Such aliases are then available to all programs based on SSH, like sftp, scp, Ansible or VS Code. Since OpenSSH 7.3p1, separate configuration files can be included in the main config file via the include directive. ```xorg.conf include config.d/*.conf include config.d/cloud/*.conf ``` Each remote host can be defined with the Host directive. It introduces a block containing several connection options, e.g.: ```xorg.conf Host webserver Hostname webserver.mpcdf.mpg.de User MPCDF_USER_NAME IdentityFile /home/user/.ssh/my-key.pem ``` This is equivalent to: ```bash ssh -i /home/user/.ssh/my-key.pem MPCDF_USER_NAME@webserver.mpcdf.mpg.de ``` Now, with the configuration block defined, we can simply connect to webserver with: ```bash ssh webserver ``` If the same option is reused often, it can be defined separately in a wildcard block, e.g.: ```xorg.conf Host *.mpcdf.mpg.de User MPCDF_USER_NAME ``` This means that all connections will implicitely be as the user given in the wildcard block. Wildcards do not work in Windows. You can refer to the [MPCDF documentation](https://docs.mpcdf.mpg.de/faq/connecting.html) about login to machines at MPCDF with SSH. _Nicolas Fabas_ ## News ### MPCDF Status Page On a new [service-status webpage](https://status.mpcdf.mpg.de), the MPCDF provides real-time information about: * the status of the most common web services of the MPCDF: the homepage, SelfService, Helpdesk, GitLab, DataShare, the MPCDF documentation and the HPC Cloud Admin Interface * the availability of the HPC systems _Raven_ and _Viper_. At the bottom of the page, upcoming maintenances and emergency incidents are announced. The new status page is also linked at the MPCDF homepage under "operational information". ![The new MPCDF status page](218/218_statuspage.png) _Thomas Zastrow_ ### Mandatory 2FA for GitLab Starting June 2nd, it will no longer be possible to login to the [MPCDF GitLab webinterface](https://gitlab.mpcdf.mpg.de) without a second authentification factor. The current login via username and password will be deactivated and only the "2FA Login" (username and password in combination with a second factor, see our [FAQ](https://docs.mpcdf.mpg.de/faq/2fa.html) for further information) will be possible. Please note that this login functionality is handled by the MPCDF user management and is independent of GitLab's internal 2FA functionality (which you can additionally activate in your profile). Usage of access tokens and the SSH-based Git functionality on the command line are not affected. If you want to try out the 2FA Login, you can choose it already now on GitLab's login page: ![2FA Login button on MPCDF GitLab](218/218_gitlab_login.png) _Thomas Zastrow_ ## Events ### AMD workshop Very similar to our last AMD-GPU workshop in November 2024, we will host a second workshop and hackathon with a focus on the new _Viper-GPU_ system. It will be held online on three afternoons from May 13th to May 15th, 2025. On the first day, AMD will give presentations on how to use the system with different programming approaches and introduce their tools. The second and third day are dedicated to hands-on work on the participants' applications, using the _Viper-GPU_ system. Experts from AMD and MPCDF will assist in this hackathon. More information and the registration can be found [here](https://plan.events.mpg.de/e/amd-mpcdf-workshop-2025). _Tilman Dannert_ ### Introduction to MPCDF services The next issue of our semi-annual seminar series on the introduction to MPCDF services will be given on May 8th, 14:00-16:30 online. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and DataShare, together with a concluding question & answer session. No registration is required, just connect at the time of the workshop via the zoom link given on our webpage. _Tilman Dannert_ ### MeetMPCDF Save-the-date: The next editions of our monthly online-seminar series "MeetMPCDF" are scheduled for May 8th, June 5th, July 3rd, 15:30 (CEST). Topics will be announced in due time via the all-users mailing list. All announcements and presentations of previous seminars can be found on our [training webpage](https://www.mpcdf.mpg.de/services/training). We encourage our users to propose further topics of their interest, e.g. in the domains of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### MPG-DV-Treffen Save-the-date: The 42nd edition of the MPG-DV-Treffen is scheduled for September 23-25, 2025, as part of the _IT4Science Days_ in Bremen. Details will follow as they become available. _Raphael Ritz_ ### Research Data Alliance - Deutschland-Tagung 2025 As in previous years, MPCDF contributed to the organization of the RDA Deutschland-Tagung 2025 which took place at the Geoforschungszentrum in Potsdam on February 18th and 19th this year. Various topics in the area of research data management were presented and discussed. Focus areas included the newly established "Datenkompetenzzentren" as well as the role of research data management in artificial intelligence. The program and presentation materials are available from the [conference website](https://indico.desy.de/event/47204/timetable/#all.detailed). _Raphael Ritz_ Bits and Bytes Logo # No.217, December 2024 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_217.pdf) ## High-performance Computing ### AlphaFold3 available on _Raven_ Recently, Google DeepMind released the source code of [AlphaFold3](https://github.com/google-deepmind/alphafold3), shortly after Demis Hassabis and John Jumper from the same company had received half of the [Nobel Prize in Chemistry 2024](https://www.nobelprize.org/prizes/chemistry/2024/press-release/) for the development of AlphaFold2. AlphaFold3 extends the capabilities of AlphaFold2 by enabling predictions of interactions of biomolecules in addition to inferring their structure. AlphaFold3 is now available on _Raven_, complementing installations of AlphaFold2 that have been provided and regularly updated since summer 2021. To get started, execute the command `module help alphafold/3.0.0` on _Raven_ and follow the instructions. An important difference to AlphaFold2 is that the AI model behind AlphaFold3 is not publicly available. Users must register with Google DeepMind and download their personal copy after approval. By default, for the software installation provided by MPCDF, the weights file 'af3.bin' has to be placed into the home directory at '~/alphafold_3_0_0/model/'. It is the responsibility of the user to comply with the terms of use of the AI model. The MPCDF is interested in getting feedback from users on the usability and performance of AlphaFold3 on the A100 GPUs of _Raven_. The memory requirements of AlphaFold3 are higher than those of AlphaFold2, and therefore out-of-memory conditions are more likely to happen. The scripts provided by our installation try to mitigate this situation by enabling CUDA unified memory for the inference step, logically extending the GPU memory with host memory. _Klaus Reuter_ ### Resource limits on the HPC machines In order to maintain the responsiveness of the login nodes on the HPC machines, per-user [resource limits were introduced on _Raven_](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html#resource-limits) and also on _Viper_ earlier this year. The per-user limit is currently two cores on _raven01/02_ and _viper01/02_ and 6 cores on _raven03/04_ and _viper03/04_, respectively. A hard memory limit is also enforced, which is 10% of the available memory on the first two login nodes of _Raven_ and 50% of the available memory for the login nodes 3 and 4 of _Raven_ and all login nodes of _Viper_. The following table summarizes these limits. | |raven01/02|raven03/04|viper01/02|viper03/04| |------|----------|----------|----------|----------| |cores | 2 |6 |2 |6 | |memory|50 GB |256 GB |256 GB |256 GB | As a consequence, this limits running multi-threaded or distributed jobs (which is the intention), but it may also affect the (parallel) performance of the build procedures of large HPC codes. Usually the builds are done in parallel with the build system spawning threads (`make -j` or `cmake --build . --parallel` are typical examples). The number of threads spawned should be limited in the build procedure to the number of cores available (2 or 6) by passing this number to the build command (`make -j 6` or `cmake --parallel 6`). It is important to note that these resource limits also apply to CI jobs executed by a GitLab runner which has been launched by the user on the login nodes. Hence, also such runners should take the above mentioned resource limits into account when launching parallel builds, otherwise the build jobs may slow down significantly. There are the following options to increase the build performance for such CI jobs: - Move your GitLab runners which use local builds from _raven01/02_ and _viper01/02_ to _raven03/04_ and _viper03/04_ and restrict the build procedure's parallelism to 6. - You can keep your runners on the first two _Raven_ nodes, but submit the build job into the "interactive" queue via the Slurm system. There you can use up to 8 cores for your job and hence use 8 threads for the build procedure. `salloc --partition=interactive -n 1 --cpus-per-task=8 --time=00:20:00 --mem=32G srun ` - Change your build tests to use the shared runners of our GitLab instance. _Tilman Dannert_ ### HPC monitoring on _Viper_ The MPCDF is running a comprehensive [performance monitoring system](https://docs.mpcdf.mpg.de/doc/computing/performance-monitoring.html) on the HPC systems that allows support staff as well as users to check on a plethora of performance metrics of compute jobs. Recently, the system was deployed to the _Viper_ supercomputer. Unlike _Raven_ and previous HPC systems, _Viper_ features AMD EPYC Genoa CPUs with Zen4 cores that have somewhat less-capable Performance Monitoring Units (PMUs). For instance, while on Intel-based CPUs, GFLOP rates can be obtained individually for each precision and vector width, only total GFLOP rates independent of the precision can be measured on the AMD Zen4 processors. Similarly, the support for uncore events such as the memory bandwidth is still limited, but expected to continuously improve with more recent kernel versions. Users can access their performance data under these limitations. We're working on improving the support for the _Viper_ system over time. _Klaus Reuter_ ### Routine transition to a new set of CI module images in 2025 Since late 2023 the MPCDF has been providing Docker images with software stacks that are installed in an essentially identical fashion on the HPC systems, enabling users to test their software consistently with various compiler and library toolchains on the GitLab shared CI runners. Interested readers can find the full announcement in [Bits & Bytes No. 214, December 2023](https://docs.mpcdf.mpg.de/bnb/214.html#no-214-december-2023). We would like to remind the CI users of the strategy we are employing to tag and manage these CI images. Starting with the year 2025, the images tagged with '2024' will not receive any updates and will hence stay unchanged. At the same time, we will start with a new set of images tagged '2025' (then identical to 'latest') that contain more recent software. Users can find up-to-date lists of the available images and the software therein [here](https://mpcdf.pages.mpcdf.de/ci-module-image/). Please note that users do not need to take action unless they want to access more recent software stacks for their CI tests. _Klaus Reuter, Tobias Melson_ ### Checks for uninitialized variables disabled in latest Intel Fortran compiler Many Fortran developers rely on [correctness-checking capabilities of the compiler](https://docs.mpcdf.mpg.de/doc/computing/software/debugging-tools.html#run-time-checks), for example in their CI pipelines or other non-regression-checking and debugging strategies. The Intel Fortran compilers, for example, by using the option `-check`, can instrument an executable with [various runtime checks](https://www.intel.com/content/www/us/en/docs/fortran-compiler/developer-guide-reference/2025-0/check.html) including the commonly used array-bounds check (`-check bounds`), or uninitialized-variables check (`-check uninit`). With the latter option, however, the new Intel compiler, ifx, is producing false positives, in particular in combination with MPI, which is why Intel decided to disable the `-check uninit` option. Note also, that `-check all` effectively translates to `-check all,nouninit` in the [latest ifx (2025) version](https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-fortran-compiler-release-notes.html) which may be perceived as a silent relaxation of the overall strictness. In general, we recommend to use the following set of options for enabling a restrictive set of runtime checks for Intel and GNU Fortran compilers, respectively, which includes an effective check for uninitialized variables by pre-setting variables with "nan" and catching resulting floating-point exceptions. - `ifx -g -traceback -check all -fpe0 -init=arrays -init=snan` - `gfortran -g -Wall -fcheck=all -finit-real=snan -ffpe-trap=invalid,zero,overflow` Note, however, that some of these options can significantly increase the execution time of the generated executable and hence these should only be used for debugging and correctness checking. _Markus Rampp_ ## A Shared AI System for Eleven MPIs The demand for using and developing AI methods is significantly growing in many Max-Planck Institutes and the recent adoption of generative models has given another major push. MPCDF already supports many institutes in these efforts through its AI support team, which also ensures the efficient availability of major AI frameworks on the MPG supercomputers. A key bottleneck is, however, the availability of high-end GPUs, which are needed for training and increasingly also for inference, particularly in large language models. Hence, eleven Max-Planck Institutes, coordinated by MPCDF, have joined forces to procure a GPU system. A total budget of over 6.5 million Euro has been raised, more than 50% of which contributed by the BAR. The system is being procured in two rounds – the first one has just finished and a system comprising 136 Nvidia H200 GPUs (configured in 17 nodes with 8 GPUs), 1 PB of fast NVMe storage and Infiniband interconnect will be installed in the second quarter of 2025. The second procurement round, potentially targeting even newer GPU generations, will take place mid 2025 and the final system will then feature well over 200 GPUs. The system, available to the participating institutes (Fritz-Haber Institute of the MPG, MPI for Human Development, MPI for Human Cognitive and Brain Sciences, MPI for Informatics, MPI for Software Systems, MPI for Sustainable Materials, MPI for Biochemistry, MPI for Polymer Research, MPI for Biogeochemistry, MPI for Geoanthropology, and MPI for Multidisciplinary Sciences) will not only provide an economical, shared resource, but also facilitate the exchange of experiences between the participating MPIs and beyond. It is anticipated to become a nucleus and hub for further collaborations among MPIs in the extremely rapidly growing field of AI. The technical design of the machine allows for future expansion with additional funding. Interested groups can already now contribute to the procurement in 2025. In addition, we plan to organize regular AI roundtables, workshops and trainings, open also to groups not yet participating in the system. If you are interested joining the discussions or even the investment, please contact support@mpcdf.mpg.de _Erwin Laure_ ## Events ### International HPC Summer School 2025 The International HPC Summer School (IHPCSS) 2025 will take place from July 6th to July 11th in Lisbon, Portugal. The series of these annual events started 2010 in Sicily, Italy and provides advanced HPC knowledge to computational scientists, focusing on postdocs and PhD students. Through the participation of Canada, the USA, South Africa, Japan, Australia and Europe a truly international group of highly motivated students is meeting each year. Interested students and postdoctoral fellows should monitor the school’s website ([https://ss25.ihpcss.org](https://ss25.ihpcss.org)) where registration opens on December 16th. School fees, travel, meals and housing will be covered for all accepted applicants through funds from the European Union and EuroHPC. For further information and application, please visit the website of the summer school. _Erwin Laure_ ### Introduction to MPCDF services The next issue of our semi-annual seminar series on the introduction to our services will be given on May 15th, 14:00-16:30 online. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and DataShare, together with a concluding question & answer session. No registration is required, just connect at the time of the workshop via the zoom link given on our webpage. ### Meet MPCDF The next editions of our monthly online-seminar series "Meet MPCDF" are scheduled for - February 6th, 15:30 "Introducing the Viper GPU system with AMD MI300A APUs" - March 6th, 15:30 Topic to be announced - April 3rd, 15:30 Topic to be announced All announcements and material can be found on our [training webpage](https://www.mpcdf.mpg.de/services/training) and the "Meet MPCDF" invitations will be sent to the all-users mailing list. We encourage our users to propose further topics of their interest, e.g. in the domains of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ Bits and Bytes Logo # No.216, August 2024 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_216.pdf) ## High-performance Computing ### New HPC system _Viper_ (Phase-1: CPU) ![Deployment of Viper-CPU (on premises of the LRZ). Image credit: H. Huber (LRZ)](216/viper-cpu.jpg) The deployment of the first phase of the new HPC system of the Max Planck Society, _Viper_, has been completed by Eviden and MPCDF and the CPU-based part of the machine is operational (still in "user-risk" mode, and with some hardware repairs ongoing) since June 2024. The machine comprises 768 compute nodes, each with two [AMD EPYC 9554 "Genoa" CPUs](https://www.amd.com/en/products/cpu/amd-epyc-9554), providing 128 Zen4 cores per node and 512 GiB (609 nodes), 768 GiB (90 nodes), 1024 GiB (66 nodes), or 2048 GiB (3 nodes) of DDR5 memory per node, respectively. The nodes are interconnected by an Nvidia/Mellanox InfiniBand (NDR 200 Gb/s) network with a non-blocking fat-tree topology, and are connected to online storage based on IBM SpectrumScale (aka GPFS) parallel file systems with a total capacity of ca. 11 PB. The new machine replaces _Cobra_ which was decommissioned in July after more than 6 years of operation, and is already productively used by a large number of users. A high-level overview of _Viper_ was given in a [MeetMPCDF seminar](https://www.mpcdf.mpg.de/training/meetmpcdf) on July 4. Users can find all relevant technical details on the hardware and software and its usage, including example batch submission scripts, in a [comprehensive user guide](https://docs.mpcdf.mpg.de/doc/computing/viper-user-guide.html). Among the most notable changes, users transitioning from _Cobra_ or _Raven_ should be aware that [compiler optimization settings](https://docs.mpcdf.mpg.de/doc/computing/viper-user-guide.html#migration-guide-for-users-coming-from-intel-based-hpc-systems) need to be adapted from Intel CPUs to the specifics of an AMD x86 CPU, and that some [resource limits have been introduced on the login and interactive nodes](https://docs.mpcdf.mpg.de/doc/computing/viper-user-guide.html#login) in order to keep the "front-end" nodes responsive for a large number of users and different (interactive) use cases. The shipment of the second, GPU-accelerated, phase of _Viper_ with more than 600 [AMD MI300A APUs](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300a.html) (Accelerated Processing Units) has just started. This part of the machine is expected to be operational in autumn 2024. _Markus Rampp_ ## HPC Software News ### MPCDF GitLab `module-image` to be discontinued on October 31 Users can run their continuous integration (CI) pipelines on the shared runners of the MPCDF GitLab instance using the same software stack as present on the HPC clusters. Last year, [new CI module images](https://docs.mpcdf.mpg.de/bnb/214.html#announcing-legacy-status-and-later-discontinuation-of-the-module-image) were introduced, replacing the deprecated `module-image` that has been available for a long time. The latter will now finally be discontinued and cannot be used anymore after October 31, 2024. All users still referencing the `module-image` (i.e., the `.gitlab-ci.yml` file contains the line `image: gitlab-registry.mpcdf.mpg.de/mpcdf/module-image`) need to adapt their CI pipelines to use the new images by then. Further information can be found in the [documentation](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html#docker-images-for-ci-with-mpcdf-environment-modules). ### New module images available in MPCDF GitLab Additional [CI module images](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html#docker-images-for-ci-with-mpcdf-environment-modules) (see also previous section) were deployed based on the `intel/2024.0` software stack. The tags `intel:latest`, `intel-impi:latest`, and `intel-openmpi:latest` point to `intel/2024.0` now. ### Nvidia HPC SDK version 24.3 available on _Raven_ The Nvidia HPC SDK version 24.3 was installed on _Raven_ and is available by loading the module `nvhpcsdk/24`. It ships its own CUDA versions 11.8 and 12.3, similar to the previous `nvhpcsdk/23`. An update of the `nvhpcsdk/24` module from the currently installed version 24.3 to a more recent minor version is in preparation. _Tobias Melson_ ## Major Change in the Python Infrastructure on the HPC Clusters For nearly a decade, MPCDF has been providing the Anaconda Python Distribution as the foundation for Python user applications, offering a large selection of important packages such as NumPy, SciPy, matplotlib, etc. in compatible versions and using optimized builds and libraries. However, earlier this year, Anaconda Inc. has changed its software licensing model such that MPCDF is not allowed to install *new versions* of Anaconda Python any more. Moreover, the package channels 'default' and 'anaconda' had to be disabled in the global `.condarc` config files for the existing installations, in order to prevent newly created conda environments to download from these channels which would require licensing. From now on, MPCDF will deploy its own comprehensive Python stack entirely based on software from [conda-forge](https://conda-forge.org), a community-driven initiative that develops conda packages which do not fall under the strict licensing of Anaconda Inc. and can therefore be used freely. For each release, the versions of the key packages such as Python, NumPy, SciPy, matplotlib, Numba, pandas will be the same as the versions in the respective release of the commercial Anaconda distribution, whereas the versions of less important dependencies may vary for dependency-resolution reasons. Along these lines, we start with *Water Boa Python 2024.06* (environment module `python-waterboa/2024.06`) which has been rolled out on the HPC clusters recently and might still experience minor modifications or extensions. Existing Anaconda installations, including 2023.03, stay available unchanged. _Klaus Reuter_ ## Introducing containerized Applications as Part of the MPCDF Software Stack Software containers are ubiquitous, e.g. in cloud computing, and are also gaining popularity on HPC systems for various reasons and with certain advantages. On MPCDF systems, the Apptainer container platform is provided to support containers in user space. A typical use case is, for example, the installation of a complex software package that expects a certain operating system version together with the respective libraries (e.g. Ubuntu 22.04), and is therefore incompatible with the host operating system of the HPC cluster (e.g. SLES 15). Once installed successfully into a container image, the execution of the software is then largely portable between host Linux operating systems. From a system administration and software deployment point of view, another advantage of containerization is that the application and its complete set of dependencies are contained within a single compressed image file (e.g. Apptainer `.sif`). This saves disk space, inodes, and installation time, compared to a regular installation of the application directly in the cluster file system. Large software packages can easily consume O(100,000) inodes, in the case of MATLAB R2023bU5 the inode count even amounts to about 680,000, for example. To mitigate the associated negative impacts on the file system, MPCDF is going to provide certain large software packages in containerized form in the future, starting with MATLAB R2024aU2 which is available already now as an environment module. The fact that the software is running within a container is largely hidden from the user by providing executable wrappers for the containerized executables (e.g. `matlab`, `mex`, `mcc`). These wrappers mount the cluster file systems into the container as they would appear on the host operating system. The command `module help matlab/R2024aU2` gives further hints, advanced users may explicitly launch e.g. a containerized shell and adapt the execution command line of the containerized application, if necessary. _Klaus Reuter_ ## Nexus-S3: Object Storage in the HPC-Cloud and beyond Nexus-S3 is a new, scaleable object storage service by MPCDF, compatible with the Amazon S3 protocol. MPCDF users can opt-in to Nexus-S3 via the SelfService portal, which provides a free 1TB (1M objects) quota (see opt-in instructions below). Data can be accessed using standard S3 clients and libraries such as [minio-client](https://min.io), [s3cmd](https://s3tools.org/s3cmd), [rclone](https://rclone.org/) and python-boto3 as well as via Globus (MPCDF GO Nexus S3 Collection) or via a web browser/curl. _Note:_ The minio-client and rclone are both available via the modules system on MPCDF clusters. ![The Object Storage System (including Globus Endpoint)](216/Object-Storage-MPCDF-Use-Cases-2.png) Nexus-S3 also supports object storage functionality such as versioning, life-cycle policies and temporary URL generation to allow users to download files with an expiry date. Together with the transfer and sharing functionality available via Globus this provides many solutions to use cases such as large-scale data sharing and publishing. ### Example use case An example use case would be the generation of data via computational jobs in a batch system with the following consolidation and sharing of the data via Object storage and Globus. 1. Submit batch jobs to produce data 2. Each job uses S3 cli tools, such as minio-client, to upload data to Object Storage 3. Data in Object Storage is shared with collaborators via Globus. Benefiting from the use of groups and sharing provided by Globus. ### Opt-in via SelfService Access to Object Storage is possible via the [MPCDF SelfService](https://selfservice.mpcdf.mpg.de). Log in with your MPCDF account and go to “My account / Services” to opt-in for Nexus-S3. _Note:_ After opt-in, it can take up to 60 minutes until the accounts are created and for the access/secret keys to become available. Once the account has been created in the S3 service you can access your access/secret keys by clicking "View Access Token". These access/secret keys are used by S3 clients and Globus to access your S3 storage, please keep them secure and treat them as you would do with a password. If you feel these keys may have been exposed, please create a helpdesk ticket and request that new keys be generated. ### Object storage for larger projects For larger projects it is possible to, cost-effectively, rent object storage in the 10s-100s TiB region via the HPC-Cloud. For more information please see [our documentation](https://docs.mpcdf.mpg.de/doc/cloud/renting/index.html) _John Kennedy, Robert Hish, Kathrin Beck, Bolarinwa Adeoye_ ## The MetaStore Research Data Publication Platform MetaStore is the catch-all data publishing platform of the MPCDF. It is meant as a place to create and publish metadata which describes already existing data stored in the various storage systems at MPCDF (e.g. Nexus-S3). MetaStore provides a landing page and a Digital Object Identifier (DOI) for the linked data, irrespective of where exactly the data is stored or how it is accessible. The data set on MetaStore will be then used as a landing page for the DOI. DOIs can also be created later in time, and not necessarily directly at the time when the data set is uploaded. Please be aware that DOI creation is irreversible. ### Managing data sets and resources in MetaStore Data sets and resources can be managed via the Web UI or a REST-style API. MetaStore’s [default](https://docs.mpcdf.mpg.de/doc/data/publication/metastore/docs/datacite-standard-format.html) metadata schema is the DataCite metadata schema in a simple key-value form. In addition, the [extended](https://docs.mpcdf.mpg.de/doc/data/publication/metastore/docs/datacite-extended-format.html) metadata schema supports the full functionality of DataCite’s metadata schema, including controlled vocabularies. Each data set can contain one or more resources. A resource can be either an URI or an uploaded file, typically corresponding to supplementary data like papers or visualizations. Uploads larger than 1GB are not supported. ### Who can use it MetaStore is not meant to be used by individual users, but by Max Planck Institutes. If your institute has already an account on MetaStore, you can use it with this account. For more details, please look at the [user documentation](https://docs.mpcdf.mpg.de/doc/data/publication/metastore-documentation.html). You can contact us at support@mpcdf.mpg.de. _Nicolas Fabas, Thomas Zastrow_ ## News ### Password policy #### MPG regulations New password regulations have been defined in the MPG, which can be found in the [OHB (XIX.04)](https://ohb.mpg.de). According to chapter 2.1 of the password-policy document the minimal length of a password is now 12 characters (for privileged accounts 14), and it must consist of at least three different character sets out of these four categories: lower case, upper case, digits, and special characters. Beside that, a minimal complexity is required, and the use of other information linked to the account, like names, telephone numbers, birthdays or similar are not allowed to be used in passwords. Some of these requirements can be enforced by checking passwords when setting them. Furthermore, passwords can be checked against databases containing exposed passwords, like those found in HaveIbeenPwnd. #### No expiration, but checking Following the latest NIST recommendations there is no longer the need to regularly change the password. Therefore it is wise to go for a sufficiently complex password. MPCDF will no longer enforce a yearly change of the passwords. In order to assure that the password still fulfills the MPG rules and is not found in a database of broken passwords, a yearly check of the passwords will be performed instead. This means, that MPCDF still will set yearly expiration dates on passwords, but there is no need to change them, if the check turns out to be OK. This check has to be done in the SelfService and if it succeeds, the expiration date will simply be shifted one year into the future. In case the check fails, either because the password does not comply with the latest MPG rules or because it is found in the database of broken passwords, a password change is enforced. This change to the previous password policy of MPCDF should allow all users to use complex passwords, which then will be valid forever. Therefore, already sufficiently complex passwords which were set up in the past, will also no longer require a change, but only a check via SelfService. _Andreas Schott_ ### 2FA for DataShare and GitLab MPCDF will introduce two-factor authentication (2FA) for the two externally exposed services DataShare and GitLab later this month. In the beginning there will be an additional login option which enforces the 2FA with the MPCDF-2FA server. This will be available for the regular accounts as well as for guest accounts. Later this year the standard login method for DataShare and GitLab will be switched over to this method as the only way to login. We therefore ask everybody to try out this new login method, which as a bonus provides a single-sign-on for the two services. So logging into DataShare will automatically log you in into GitLab and vice versa. _Andreas Schott_ ## Events ### AMD GPU workshop & hackathon (November 5-7) In order to prepare for the second phase of the new HPC system _Viper_ of the MPG with a large number of AMD MI300A APUs, the MPCDF in collaboration with AMD offers an online course with hands-on for this new architecture. The workshop comprises two and a half days, starting on November 5th with an afternoon of lectures by experts from AMD, followed by two full days of expert-guided hands-on work on individual codes participants are invited to bring in. The workshop targets intermediate to advanced developers who can start out with a code that already uses GPUs (for example on the _Raven_ GPU partition with Nvidia A100 GPUs), and who would like to also leverage the new _Viper_ system with MI300A APUs. For registration and further details please visit the [workshop registration page](https://plan.events.mpg.de/e/amd-gpu-workshop-24). _Markus Rampp_, _Tilman Dannert_ ### Introduction to MPCDF services (October 24) The next issue of our semi-annual online course "Introduction to MPCDF services" is scheduled for October 24th, 14:00-16:30 via Zoom. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and DataShare, together with a concluding question & answer session. No registration is required, just connect at the time of the workshop via the [zoom link](https://mpcdf-mpg-de.zoom-x.de/j/69488919156?pwd=T256S1dXSjhXcU1hNGhZNExkeVVsQT09). _Tilman Dannert_ ### Meet MPCDF The next editions of our monthly online-seminar series "Meet MPCDF" are scheduled for - September 5th, 15:30 "Basic profiling of HPC applications" by Sebastian Ohlmann (MPCDF) - October 10th, 15:30 (Topic to be announced) - November 7th, 15:30 "The _Viper_ GPU system" All announcements and material can be found on our [training webpage](https://www.mpcdf.mpg.de/services/training). We encourage our users to propose further topics of their interest, e.g. in the domains of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### MPCDF at Garching Campus Open Doors (October 3) For the first time MPCDF will open its doors for the general public at the [_Open Day Campus Garching_](https://forschungscampus-garching.de/) at October 3, 10:00-17:00. We will provide short talks, posters about scientific high-performance computing, data science, and artificial intelligence and offer the opportunity to have a peek into the machine hall. [The program](https://www.mpcdf.mpg.de/opendoors2024) is targeted at the general public, but we, at MPCDF, always appreciate exchange with our friends and expert users who might take the opportunity of the campus event to meet in person with MPCDF staff. _Friederike Neu, Markus Rampp_ ### HPC-Cloud workshop (September 10-12) Following the growing popularity of the HPC-Cloud the MPCDF will host a cloud workshop which aims to establish a community of cloud users from the scientific projects. Projects will share ideas and real-world experience gained in the past months and years. The cloud experts from MPCDF will be present to facilitate the exchange and discuss future directions for the HPC-Cloud. For more information and to register please visit Agenda and Registration at [https://plan.events.mpg.de/e/hpccloudws](https://plan.events.mpg.de/e/hpccloudws). _Raphael Ritz_ ### Course on "Python for HPC" (November 26-28) The next iteration of our popular course on "Python for High-performance Computing" is scheduled for November 26th to 28th, 2024. The event takes place online via Zoom and teaches how to use the Python ecosystem efficiently on HPC systems. We detail on topics such as using NumPy, SciPy, Cython, Numba, JAX, writing compiled extensions in C, C++, Fortran, making use of multithreading, GPU programming, and leveraging distributed memory parallelization using mpi4py and Dask. The lectures in the morning are complemented by exercises and Q&A sessions in the afternoon. [Registration is now open](https://plan.events.mpg.de/e/mpcdf-python-for-hpc-2024). _Sebastian Kehl, Sebastian Ohlmann, Klaus Reuter_ Bits and Bytes Logo # No.215, April 2024 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_215.pdf) ## High-performance computing ### Licensed software in Slurm (Comsol) The Slurm workload manager can handle licensed software by assigning available licenses to jobs at scheduling time. If licenses are not available, jobs are kept pending until specified licenses become available, rather than failing at runtime due to a lack of available licenses. This can be particularly useful for software that only has a few licenses. For example, for the Comsol software, MPCDF currently maintains only two licenses for each Comsol module for batch use. In order to use this feature, you need to add the `-L, --licenses` option to the batch job scripts with a comma-separated list of required license names. To find out which licenses the `mph` comsol file requires, you can check it in the comsol GUI or simply run the `mphlicenses` command after loading the comsol module on HPC _Cobra_ or _Raven_ systems: ``` ~> module load comsol/6.2 ~> mphlicenses test_file.mph comsol_nonlinearstructmaterialsbatch@lserv,comsol_comsolbatch@lserv,comsol_structuralmechanicsbatch@lserv,comsol_clusternode@lserv,comsol_comsoluser@lserv ``` The complete Slurm #SBATCH directive to run this comsol test_file on the clusters then reads ``` #SBATCH -L comsol_nonlinearstructmaterialsbatch@lserv,comsol_comsolbatch@lserv,comsol_structuralmechanicsbatch@lserv,comsol_clusternode@lserv,comsol_comsoluser@lserv ``` In order to get a list of all available comsol licenses on HPC systems, run the command: ``` scontrol show licenses ``` _Mykola Petrov_ ## HPC Software News ### Improved workflow for multimer predictions with AlphaFold2 on _Raven_ AlphaFold2 (AF2) is an artificial-intelligence program released by Google DeepMind which performs predictions of protein structure based on the sequence of amino acids. AF2 has been provided by MPCDF since summer 2021 on _Raven_ and on several institute clusters, and is regularly updated and customized to achieve optimal performance on the MPCDF systems. Initially, the prediction of the structure of single proteins (monomers) has been a main application of AF2, however more recently the users' focus shifted towards predicting protein-protein complexes (multimers). As the prediction of multimers is considerably more expensive than most monomer cases, users sometimes hit the 24-hour job time limit on _Raven_ with the original AF2 program, which internally loops over multiple models and random seeds. To overcome this limitation, MPCDF has refactored the internal double loop over the models and predictions into individual Slurm jobs. In particular, individual Slurm array jobs are now used for each model, each array containing a configurable number of tasks calculating individual predictions using individual random seeds. As a result, each individual prediction task now has a maximum wall clock time of 24 hours. On _Raven_, run `module help alphafold/2.3.2-2024` to get further instructions. For multimer cases, copy, adapt and run the script `submit_alphafold_jobs.sh` to submit the job-array-based predictions. Users are encouraged to approach us via the helpdesk with their feedback, which is valuable to further improve this service. _Klaus Reuter_ ### New version of Intel oneAPI with deprecation of ifort compiler Intel oneAPI version 2024.0 has been installed on _Raven_ and other HPC clusters recently. With this installation, we drop the `.x` suffix and the patch-level version from the compiler module name. Thus, the Intel compiler module is now called `intel/2024.0`. Bugfixes provided by Intel incrementing the patch level number will be installed under the hood without further notification. Run `module show intel/2024.0` in order to see which exact compiler version is present. The corresponding MPI module is `impi/2021.11`, and the latest MKL module is `mkl/2024.0`. As [announced previously](https://docs.mpcdf.mpg.de/bnb/214.html#intel-oneapi-transition-from-ifort-to-ifx), MPCDF, starting with `intel/2024.0`, is using `ifx` as the default Intel Fortran compiler, together with its MPI wrapper `mpiifx`. The “classic” Fortran compiler `ifort` is still present, but is now formally tagged as deprecated by Intel. _Tobias Melson_ ### CUDA modules on _Raven_ MPCDF offers various CUDA modules on _Raven_. It depends on the desired compiler, which of these modules should be picked. If an application code is compiled with the GNU compiler (`gcc/11` or `gcc/12`), the CUDA modules `cuda/*` are suitable. In case the code is compiled with the Nvidia compiler (`nvhpcsdk/23`), the matching CUDA modules are named `cuda/*-nvhpcsdk` (note the suffix here). The following table lists the compatible CUDA modules for the recent GNU and Nvidia compilers: | Compiler | Compatible CUDA modules | | ----------- | -------------------------------------- | | gcc/11 | cuda/11.4, cuda/11.6 | | gcc/12 | cuda/12.1, cuda/12.2 | | nvhpcsdk/23 | cuda/11.8-nvhpcsdk, cuda/12.3-nvhpcsdk | As [announced previously](https://docs.mpcdf.mpg.de/bnb/214.html#cuda-aware-openmpi-on-raven), CUDA-aware OpenMPI (`openmpi_gpu/4.1`) is available after having loaded one valid combination of compiler and CUDA modules. _Tobias Melson, Tilman Dannert_ ### New AMD-GPU ELPA release The [ELPA library](https://elpa.mpcdf.mpg.de) provides highly optimized, scalable solvers for generalized and standard, dense symmetric (Hermitian) eigenproblems. Since 2022, the ELPA library supports AMD GPUs and on Europe's first pre-exascale system _LUMI_ (CSC, Finland) ELPA was successfully employed for solving huge standard eigenvalue problems with a matrix size of up to 3.2 million (leading dimension) on more than 8000 AMD MI250x GPUs. In preparation for the new HPC system _Viper_ at the MPCDF we have released a new version 2024.03.001 of the ELPA library which has been further extensively optimized for AMD GPUs. Among others, support of the GPU-to-GPU collective communications library RCCL (the equivalent to Nvidia's NCCL library) has been implemented. In addition, the GPU implementation of the routines for the generalized eigenvalue problem has been reworked and speedups of up to a factor of 10 have been achieved. The latest version of ELPA is available to all users as a pre-built software package in the module environment of the MPCDF software stack. _Andreas Marek_ ## Kubernetes in the HPC-Cloud Containers have been widely adopted as a way to develop, distribute, and deploy applications. Kubernetes provides a framework to run containerized applications in a reliable and scalable fashion. Kubernetes is based on cluster management technology developed at Google ([cf. Verma, Abhishek, et al.: "Large-scale cluster management at Google with Borg", Proceedings of the tenth European conference on computer systems, 2015](https://dl.acm.org/doi/abs/10.1145/2741948.2741964)) and donated to the Linux Foundation in 2015. It has enjoyed wide adoption since. At MPCDF projects can deploy Kubernetes clusters hosted in the HPC-Cloud based on an [OpenStack Heat Orchestration Template](https://docs.openstack.org/heat/latest/template_guide/hot_guide.html) as [provided by the MPCDF Cloud enabling team](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/kubernetes/). The team maintains two templates: one for a small and simple cluster aimed at projects that wish to test and evaluate Kubernetes, and the second providing a production-grade cluster aimed at projects looking to deploy their services and applications using Kubernetes. ### Usage The canonical way to control a Kubernetes cluster is with [`kubectl`](https://kubernetes.io/docs/reference/kubectl/). The credentials required to connect to the cluster are provided on the control plane nodes set up by the template. The most important feature is that users define desired states for pods executing their containers and any peripheral resources in configuration files. Many software projects publish the configuration states required to run their products using [`Helm`](https://helm.sh), the package manager for Kubernetes. Helm charts define the desired state of many interacting Kubernetes components. Users can configure charts by setting desired values of chart variables defining the specific features of their deployment. In productive deployments, it is imperative to store and track the configuration files of the services orchestrated in Kubernetes. This can be accomplished, for example, using GitLab. In addition, GitLab offers integrations for Kubernetes that deploy the state defined in the configuration files stored in a repository to Kubernetes as defined in the GitLab CI/CD pipelines. Taking advantage of this feature of GitLab provides a concise way to manage applications for teams of administrators and developers. ### Function Kubernetes is best suited for orchestrating long-running services that are implemented as many inter-dependent microservices. It is also possible to run applications on a schedule, or execute single commands in their containers. #### Deploying a web service Here is a configuration file you can apply with `kubectl create -f .yaml` ```yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: svc-demo-deployment labels: app: svc-demo-app spec: replicas: 2 selector: matchLabels: app: svc-demo-app template: metadata: labels: app: svc-demo-app spec: containers: - name: svc-demo-container image: jmalloc/echo-server ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: svc-demo spec: selector: app: svc-demo-app type: LoadBalancer ports: - port: 80 targetPort: 8080 protocol: TCP ``` The first section defines a deployment with a container image that runs the application, in this case a server that parrots your HTTP requests. The second section defines a service, exposing the application using a load balancer. Now running `kubectl get all` will give you: ``` NAME READY STATUS RESTARTS AGE pod/svc-demo-deployment-587b7c9974-4z5wn 1/1 Running 0 66s pod/svc-demo-deployment-587b7c9974-lz7pp 1/1 Running 0 66s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/svc-demo LoadBalancer 10.101.113.187 XXX.XXX.XXX.XXX 80:32260/TCP 66s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/svc-demo-deployment 2/2 2 2 66s NAME DESIRED CURRENT READY AGE replicaset.apps/svc-demo-deployment-587b7c9974 2 2 2 66s ``` You can see two pods running the application and the service making the application available at `http://XXX.XXX.XXX.XXX:80`. The application can be removed with `kubectl delete -f .yaml`. ### Existing applications Kubernetes on the HPC-Cloud is already used at MPCDF to host a number of prominent applications, for example [MoveApps](https://www.moveapps.org/), [NOMAD](https://nomad-lab.eu/nomad-lab/), or [GlycoSHIELD](https://dioscuri-biophysics.pages.mpcdf.de/glycoshield-md/). _Frank Berghaus_ ## GitLab: Graphs & Diagrams In addition to classic Git functionality, MPCDF's GitLab instance offers a wide range of capabilities around code organisation and project management. In many of these scopes, for example Wiki pages or issues, the user can enter formatted text in Markdown format. After saving, GitLab will render the Markdown annotations which results in a nicely formatted human readable document. Beside standard Markdown annotations, GitLab integrates out-of-the-box libraries for rendering graph structures and diagrams in Markdown texts. In this article, the creation of graphs and diagrams with the help of the Mermaid library is introduced. Inside any Markdown-formatted document in GitLab, the following code fragment encapsulates a Mermaid sub document: ```mermaid graph TD A --> B A --> C C --> D C --> E E --> A ``` After saving the Markdown document, GitLab would create the following graph from the code above: ![](215/215_gitlab_01.png) The Mermaid library supports different types of graphs and diagrams. The image below shows some examples, top to bottom: a timeline, a mindmap, a pie chart and - maybe particularly relevant for GitLab users - a Git graph, showing the branch and commit structure of a Git repository: ![](215/215_gitlab_02_hochkant.png) The Markdown and Mermaid code of the examples examples can be found [here](https://gitlab.mpcdf.mpg.de/thomz/graphs-and-diagrams). Further information about supported graph and diagram types can be found in the [Mermaid Documentation](https://mermaid.js.org/intro/). ### Alternatives In the same way GitLab supports the Mermaid library, also [PlantUML](https://plantuml.com/en/) for UML diagrams and [Kroki](https://kroki.io/) are supported. And if you don't want to dive into another descriptive language, GitLab allows you to insert manually drawn diagrams via the external service [draw.io](https://draw.io). You can find this function as "Insert or edit diagram" button in the Markdown editor (red circle): ![](215/215_gitlab_03.png) _Thomas Zastrow_ ## LLMs meet MPCDF ![A GPT4 interpretation of Max Planck using LLMs at the MPCDF](215/llms_meet_mpcdf.png) The term "Large Language Model" (LLM), has been part of public discourse since at least the end of 2022, following the release of OpenAI's ChatGPT. The general-purpose capabilities of these large AI models to process and store unstructured data have ignited a wave of new applications, also in the world of science (cf. [_The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4_. Microsoft Research](https://arxiv.org/pdf/2311.07361.pdf)). However, most state-of-the-art models are provided by major tech companies and are accessible only through paid APIs. This paywall, along with the requirement to transfer research data to third parties, often hinders the use of so-called "closed" LLMs. In contrast, "open" LLMs, which can be downloaded and used on-premise, are rapidly narrowing the gap in capabilities compared to their closed counterparts (cf. [_Open LLM Leaderboard_. Hugging Face](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)). However, the immense size and complexity of LLMs require significant computational resources and pose technical challenges for less experienced users, both in terms of operation and training. Given the computational resources and expertise at the MPCDF, an increasing number of scientists from the Max Planck Society are seeking our advice and support on utilizing LLMs. The inquiries come from a wide range of scientific domains, from natural sciences to the humanities, underscoring the broad impact these models have on the scientific community. The AI group at the MPCDF is actively enhancing its expertise in large language models and expanding its support for various use cases. To this end, we have created the ["LLMs Meet MPCDF" GitLab repository](https://gitlab.mpcdf.mpg.de/dcfidalgo/llms-meet-mpcdf), where we start offering examples and guidance on deploying and fine-tuning LLMs on our HPC systems. For now we showcase how to set up an inference server for open models with [TGI](https://github.com/huggingface/text-generation-inference), and how to supervise-fine-tune a 70B Llama2 model by means of [FSDP](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html). If you have unaddressed use cases or projects involving LLMs, please don't hesitate to reach out to us, we would be more than happy to hear from and collaborate with you. _David Carreto Fidalgo, Andreas Marek_ ## News & Events ### Introduction to MPCDF services The next issue of our semi-annual online course "Introduction to MPCDF services" will be held on April 25th, 14:00-16:30 via Zoom. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and datashare, together with a concluding question & answer session. Basic knowledge of Linux is required. No registration is required, just connect at the time of the workshop via the [zoom link](https://mpcdf-mpg-de.zoom-x.de/j/69488919156?pwd=T256S1dXSjhXcU1hNGhZNExkeVVsQT09). ### Meet MPCDF The next editions of our monthly online-seminar series "Meet MPCDF" are scheduled for - May 2nd, 15:30 "CMake - Cross-supercomputer Make" by Vedran Miletic (MPCDF) - June 6th, tba All announcements and material can be found on our [training webpage](https://www.mpcdf.mpg.de/services/training). We encourage our users to propose further topics of their interest, e.g. in the fields of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ Bits and Bytes Logo # No.214, December 2023 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_214.pdf) ## HPC Software News ### CUDA-aware OpenMPI on _Raven_ MPCDF provides CUDA-aware OpenMPI on _Raven_ based on different compilers and CUDA versions. The complete list can be inspected by running `find-module openmpi_gpu`. Below, we highlight some relevant combinations of compiler and CUDA modules that can be used with the `openmpi_gpu/4.1` module. GCC-based CUDA-aware OpenMPI builds are available after loading `gcc/11 cuda/11.6` or `gcc/12 cuda/12.1`. Recently, a CUDA-aware OpenMPI module has been added which works with the CUDA version and the compilers provided by the Nvidia SDK. To access it, the modules `nvhpcsdk/23 cuda/11.8-nvhpcsdk` must be loaded. _Tobias Melson, Tilman Dannert_ ### GPU-accelerated VASP With the deployment of CUDA-aware OpenMPI for Nvidia compilers (nvhpcsdk, see above) MPCDF provides GPU-accelerated builds of the [VASP](https://www.vasp.at/) software package for atomic-scale materials modelling from first principles. Currently, a `vasp-gpu/6.4.2` module is available on _Raven_ and selected institute clusters. Note that MPCDF does not hold a license for VASP. Individual users have to bring in their own license (via MPCDF helpdesk) in order to be enabled for using VASP at MPCDF. _Markus Rampp_ ### Intel oneAPI: transition from ifort to ifx The transition to the new LLVM-based compilers in the Intel oneAPI package is progressing. Already in the currently installed module `intel/2023.1.0.x`, `icx` and `icpx` are the default compilers for C and C++, respectively, replacing the "classic" compilers `icc` and `icpc`. As the next step, MPCDF will follow Intel's recommendation to set `ifx` together with its MPI wrapper `mpiifx` as the default Fortran compiler in the upcoming intel module corresponding to the oneAPI release 2024.0. The "classic" Fortran compiler `ifort` will still be present for some time, but should be considered deprecated, because its development had effectively been frozen some time ago. Users are advised to adjust all Fortran builds to use the new `ifx` compiler. A [porting guide](https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html) exists with detailed information on this transition. Further support is provided at the [MPCDF Helpdesk](https://helpdesk.mpcdf.mpg.de/). _Tobias Melson, Markus Rampp_ ## Module Software Stacks for Continuous Integration Pipelines on MPCDF GitLab Shared Cloud Runners In order to provide the developers of HPC applications with a familiar and comprehensive software environment also within GitLab-based continuous integration (CI) pipelines, the MPCDF is maintaining special Docker images. These images use environment modules to make software accessible, very similar to how [software is handled on the HPC systems](https://docs.mpcdf.mpg.de/doc/computing/software/environment-modules.html), and they can be used on the [Shared Cloud Runners](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html) offered by the MPCDF. Hence, e.g. build scripts would work on both the HPC systems and the CI cloud environment in a consistent way. This article introduces a redesign of the module-enabled Docker image infrastructure which is eventually going to replace the currently used `module-image`. [Documentation](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html) and a [software list](https://mpcdf.pages.mpcdf.de/module-image/modules.list) for the current `module-image` are available online. ### Introducing a novel module-enabled Docker image infrastructure To address some limitations of the `module-image` we have developed a more flexible infrastructure composed of various Docker images, each of which provides a toolchain based on a _single_ combination of compiler and MPI variant. The access to the software is implemented via environment modules. Up-to-date lists of the images together with lists of the software contained are [documented in GitLab](https://mpcdf.pages.mpcdf.de/ci-module-image/). Currently, the images are based on *openSUSE Leap 15.5* which is largely compatible with the SLES 15 operating system used on many HPC clusters at MPCDF. As indicated by its tag, each image only contains a single toolchain, namely one compiler with optionally one MPI library plus a selection of widely used additional libraries. The initial list of software can be extended upon request. Arbitrary further software from the official OpenSUSE repos may be installed by the users individually, if necessary. Users are encouraged to migrate to the new CI images soon, report potential issues and request additional software modules via the helpdesk, if necessary. Essentially, the `module-image` can simply be replaced in the user's `gitlab-ci.yml` file with one of the new images that provides the desired software stack for the respective CI job. Please note that the modules inside the images will be updated and extended regularly, similarly to the software modules updated on the HPC systems. To limit the individual growth of these Docker images over time, we will put the following tagging-and-purging strategy in place: Essentially, all images are tagged using 'latest' and/or the calendar year. In the course of a year, say 2023, the images tagged with 'latest' and the year ('2023') are identical and receive regular updates and additions of software. With the beginning of the new year, all images tagged with the previous year stay unchanged (frozen). The newly created images for 2024, say, will start out in early January again in a slim state and will be tagged 'latest'. Users can then choose to migrate to the more recent images (tagged '2024' and 'latest' in our example) or stick with the older (but static!) images (tagged '2023') for a while. In case a user opts for using the tag 'latest', please be warned that the software environment will change at the beginning of each year. ### Announcing legacy status and later discontinuation of the `module-image` The current `module-image` has been provided for several years now to provide easy access to the familiar modules environment also from within GitLab CI jobs. As the image and infrastructure are based on the outdated CentOS 7, the `module-image` will be considered legacy after the test phase of the new Docker image infrastructure and will then not be updated any more. Ultimately, the image will have to be removed (to be announced in due time). _Tobias Melson, Klaus Reuter_ ## Compressed Portable Conda Environments for HPC Systems ### Introduction and Motivation The Conda package manager and the related workflows have become an accepted standard when it comes to distributing scientific software for easy installation by end users. Using `conda`, complex software environments can be defined by means of simple descriptive `environment.yml` files. On MPCDF systems, users may use Conda environments, but without support from MPCDF for the software therein. Once installed, large Conda environments can easily amount to several 100k individual (small) files. On the local file systems of a laptop or PC this is typically not an issue. However, in particular on the large shared parallel file systems of HPC systems the vast amount of small files may cause issues, as these file systems are optimized for other scenarios. Inode exhaustion and heavy load due to (millions of) file opens, short reads, and closes happening during the startup phase of Python jobs from the different users on the system are only two examples. ### Move Conda environments into compressed image files MPCDF developed the new open-source tool *Condainer*, which adresses these issues by moving Conda environments into compressed squashfs images, reducing the number of files stored directly on the host file system by orders of magnitude. Condainer images are standalone and portable: They can be copied between different systems, improving reproducibility and reusability of proven-to-work software environments. In particular, they sidestep the integration of a specific `conda` executable into the user's `.bashrc` file, which often causes issues and is orthogonal to the module-based software environments provided on HPC systems. Technically, Condainer uses a Python basis from Miniforge (which is a free alternative to Miniconda) and then installs the user-defined software stack from the usual `environment.yml` file. Package dependency resolution and installation are extremely fast thanks to the `mamba` package manager (an optimized replacement for `conda`). As a second step, Condainer creates a compressed squashfs image file from the staging installation, before it deletes the latter to save disk space. Subsequently, the compressed image is mounted (using `squashfuse`) at the very same directory, providing the full Conda environment to the user who can `activate` or `deactivate` it, just as usual. Moreover, Condainer provides functionality to run executables from the Conda environment directly and transparently, without the need to explicitly mount and unmount the image. Please note that the squashfs images used by Condainer are not "containers" in the strict terminology of Docker, Apptainer, or alike. With Condainer, there is no process isolation or similar, rather Condainer is an easy-to-use and highly efficient wrapper around the building, compressing, mounting, and unmounting of Conda environments on top of compressed image files. ### Basic usage examples #### Build a compressed environment Follow along the following commands once in order to build a compressed image of a Conda environment that is defined in 'environment.yml': ```bash # on MPCDF systems, e.g. Raven: module load condainer # create specific project directory: mkdir my_cnd_env && cd my_cnd_env # initialize project directory with a skeleton: cnd init ls # edit the 'environment.yml' example file, # or copy your own file here # build the environment and compressed image: cnd build ls ``` #### Activate a compressed environment After building, you can activate the environment for your current shell session, similar to plain Conda or a Python virtual environment: ```bash source activate ``` Please note that `source activate` will only work with bourne shells (e.g. `bash` or `zsh`), not with the older C shells and korn shells. #### Alternatively, run an executable from a compressed environment directly In case you do not want to activate the environment, you can run individual executables from the environment directly, e.g. ```bash cnd exec -- python3 ``` The `cnd` command supports the flag `--directory` to specify a certain Condainer project directory, allowing for arbitrary current working directories. ### Limitations As the squashfs fuse mounts are specific to an individual compute node, Condainer currently (v0.1.8) does not support multi-node batch jobs. ### Availability The software including its documentation is freely available via the [MPCDF gitlab](https://gitlab.mpcdf.mpg.de/mpcdf/condainer). Moreover, it is provided via the environment module `condainer` on the _Raven_ HPC system, and will be offered on more systems in the near future. _Klaus Reuter_ ## New Features in the HPC-Cloud After commissioning the initial set of compute and storage resources in [2021](https://docs.mpcdf.mpg.de/bnb/207.html#hpc-cloud), deploying the GPU- and NVMe-focused [extension](https://docs.mpcdf.mpg.de/bnb/212.html#mpcdf-hpc-cloud) earlier this year, and rolling-out integrated [object storage](https://docs.mpcdf.mpg.de/bnb/213.html#hpc-cloud-object-storage), MPCDF has recently deployed several new features to better support the diverse technical requirements of current and future projects: ### Expanded menu of flavors and images Projects now have access by default to more combinations of vCPUs and memory, known as *flavors* within the cloud environment, so that virtual machines can be sized to match the application. In practice, MPCDF now offers flavors up to 24 vCPUs and 64 GB of memory, subject to certain boundary conditions which ensure they can be efficiently mapped to a physical compute node. As before, even larger flavors as well as local SSD-, GPU-, and NVMe-enabled versions of the default flavors can be created on request. Several new virtual machine templates, known as *images* within the cloud environment, have been prepared, including AlmaLinux as well as newer releases of CentOS Stream, Debian, and openSUSE Leap. Commercial operating systems such as Red Hat Enterprise Linux and SUSE Linux Enterprise Server are also available on a BYOL (bring your own license) basis. ### SSD-based block volumes There is now an SSD-based block volume type *CephSSD* representing a middle-ground option between highly-performant local SSDs and the highly-flexible and scalable network-based HDD storage associated with the default volume type. While I/O performance cannot be guaranteed in a shared-resource environment, one can expect a roughly 2X speedup in terms of small I/O operations per second and large transfer bandwidth, as well as a significant reduction in latency. To evaluate whether the new volume type is a good option for your project, please contact the cloud enabling team via the [helpdesk](https://helpdesk.mpcdf.mpg.de). As a tip, existing block volumes can be migrated between types *online*, making it relatively simple to test an already-deployed application. In addition to the new volume type, all Ceph-based system disks, i.e. the OS root of the VMs not hosted on local SSDs, have been transparently migrated to an SSD pool "for free", so that routine tasks such as software installation and updates complete more quickly. ### Automated domain name service Hostnames are now automatically generated for most devices attached to the public or local cloud networks, including virtual machines and floating IP addresses. The system works like this: 1. Each virtual machine is assigned a hostname of the following form: `VM_NAME.PROJECT_NAME.hpccloud.mpg.de` If the name of the virtual machine is invalid according to the requirements of DNS, then a unique hostname based on the fixed IP address will be substituted automatically. 2. Each floating IP is assigned a hostname of the following form: `FIP_DESCRIPTION.PROJECT_NAME.hpccloud.mpg.de` If the description field is empty or invalid, then a unique hostname based on the floating IP address will be substituted automatically. 3. Hostnames are synchronized with the MPCDF DNS servers every five minutes. For devices on the public cloud network, both forward and reverse entries are propagated to the global DNS, whereas on local networks only the forward (i.e. hostname->IP address) entries are published. Thus, within the framework described above it is possible to deploy and configure many applications on the HPC-Cloud without tracking individual IP addresses. ### Shared filesystem service The HPC-Cloud now implements shared filesystem-as-a-service (FSaaS), also known as OpenStack Manila or simply *Share* within the cloud dashboard, as an alternative to Nexus-Posix project directories. The two technologies are complementary, with differing strengths and weaknesses: | Service | Technology | Protocol | External access | Typical scale | Lifetime | Provisioning | | ----------- | ------------------------ | --------------------- | -------------------------- | ----------------------------- | --------------- | ---------------------------------- | | FSaaS | CephFS | NFS, or native CephFS | *none* | 100 GB up to 50 TB | *arbitrary* | self-provisioned via dashboard/API | | Nexus-Posix | IBM Storage Scale (GPFS) | NFS | _Raven_, _Robin_, GO-Nexus | 10 TB up to 100 TB, or larger | months to years | on request via helpdesk ticket | The key advantages of Manila-based shares are that they can be provisioned quickly by project admins and, being logically isolated from other systems, can easily handle arbitrary UIDs and GIDs including root and/or service users. On the other hand, Nexus-Posix as an MPCDF-administered filesystem supports secure interoperation with _Raven_ and other systems as well as built-in backups. To evaluate filesystem-as-a-service for your project please get in touch with the cloud enabling team. More details about these new features can be found in the [technical documentation](https://docs.mpcdf.mpg.de/doc/cloud/technical/). _John Alan Kennedy, Brian Standley, Maximiliano Geier_ ### The Robin cluster The Remote Visualization Service at MPCDF has recently been expanded with a new cluster called _Robin_. _Robin_ is the first compute cluster of MPCDF in the HPC-Cloud and its resources are available to all users with an access to the HPC systems (i.e. _Cobra_ and _Raven_). One of the main advantages of the _Robin_ cluster is its flexibility: new nodes can be easily deployed in the HPC-Cloud, automatically configured and added to the Slurm cluster, allowing for a convenient scaling of the compute resources on _Robin_ depending on the current demand. _Robin_ uses Slurm as a job scheduler and it can currently host up to 20 CPU sessions and 12 GPU sessions, concurrently. Access to the cluster is restricted via our Remote Visualization Service web interface, so that users are not allowed to connect directly via ssh to the login or compute nodes of _Robin_. Each session on _Robin_ provides 12 virtual CPUs and 64 GB of RAM, with GPU sessions having access to a shared Nvidia A30 GPU (up to 4 sessions can share a single GPU). _Robin_ mounts the _Raven_ file systems, providing access to all the software and data available on the _Raven_ cluster, including the user’s home directory and ptmp folder. A runtime of up to 7 days is currently allowed, with a plan to increase to up to 28 days of maximum runtime in the future, but users are encouraged to stop their sessions once their calculations are completed and should be aware that long-running jobs can be killed in case of maintenance of the cluster. Users requesting GPU sessions are encouraged to limit the memory used by their code to roughly 1/4 of the available GPU memory (~6 GB out of the 24 GB available), in order to avoid disrupting the calculations of other users sharing the same GPU. This is particularly important for Machine Learning software (e.g. Tensorflow, Pytorch) that can allocate the entire available GPU memory for a single process. _Robin_ is designed to provide a single solution for the remote visualization needs of future HPC clusters at MPCDF: the filesystem of new clusters (like the upcoming _Viper_) can be made available on _Robin_, providing easy access to software and data without the need of a dedicated installation of the Remote Visualization Service on each cluster. Users interested in using the Remote Visualization Service on _Robin_ are reminded to initialize their sessions on the cluster once (before submitting their first session), as described in our [documentation](https://docs.mpcdf.mpg.de/doc/visualization/index.html). _Michele Compostella_ ## News & Events ### AMD-GPU development workshop In preparation for the [new supercomputer _Viper_](https://docs.mpcdf.mpg.de/bnb/212.html#new-supercomputer-of-the-mpg-cobra-successor) of the MPG with AMD MI300A GPUs to be installed in 2024, the MPCDF in collaboration with AMD offered an online course on AMD Instinct GPU architecture and the corresponding ROCm software ecosystem, including the tools to develop or port HPC or AI applications to AMD GPUs. The workshop was held as an online event, spanning three afternoons on November 28-30, 2023. The workshop material can be found on the [MPCDF training website](https://www.mpcdf.mpg.de/services/training). _Tilman Dannert_ ### Meet MPCDF The monthly online-seminar series "Meet MPCDF" skips the talk in January 2024. The next edition will take place on February 1st, 2024 with a talk on "ScaLAPACK and ELPA: how to diagonalize really large dense matrices" given by Petr Karpov from the MPCDF. Subsequent dates will be March 7th and April 4th (topics to be announced). All announcements and material can be found on our [training webpage](https://www.mpcdf.mpg.de/services/training). We encourage our users to propose further topics of their interest, e.g. in the fields of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### RDA Deutschland Tagung 2024 The [German chapter of the Research Data Alliance](https://www.rda-deutschland.de/) will have its next conference in Potsdam, February 20-21, 2024. This year's focus is on legal, administrative and organizational topics concerning research data management in Germany and Europe. The early registration deadline is January 12, 2024. Further details including the program are available from . As in previous years, MPCDF is contributing to the organization of the event. _Raphael Ritz_ Bits and Bytes Logo # No.213, August 2023 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_213.pdf) ## High-performance Computing ### New GPU development partition on _Raven_ A gpu development partition "gpudev" was created on the set of GPU nodes on _Raven_ in order to facilitate development and testing of GPU codes. In order to use the "gpudev" partition you have to specify ``` #SBATCH --partition=gpudev ``` in your submit script. The maximum number of nodes available with "gpudev" is 1, the maximum execution time is 15 minutes, and you can choose to use between one and four Nvidia A100 GPUs like for usual GPU jobs. _Renate Dohmen, Mykola Petrov_ ### Memory profiling with heaptrack The memory profiler heaptrack has recently been installed on _Raven_. It can be used to measure memory usage, find memory allocation hotspots and memory leaks in a C, C++, or Fortran code. Heaptrack traces the memory allocation size and frequency as well as the call stack. More information about heaptrack can be found [here](https://github.com/KDE/heaptrack). To use heaptrack, you first have to collect the memory profiling data while running your executable. There is no need for recompiling (instrumenting) your executable. Simply run it through `heaptrack` in your SLURM job script as shown below (assuming your executable is named `a.out`): ``` module load heaptrack srun hpcmd_suspend heaptrack ./a.out ``` A `.gz` file containing the profiling data will be generated in your job submission directory. To view a short analysis of it in the console, run `heaptrack --analyze` on that file. You also have the option to download the data file to your workstation and analyze it locally. This is especially useful if you install the graphical interface for heaptrack which may be possible via your system's package manager. Note that the version of heaptrack used for data collection and analysis should match in this case. Heaptrack does not natively support MPI. However, it can analyze MPI codes and generate a separate output file for each MPI rank. _Tobias Melson_ ### New compilers and libraries: Intel oneAPI 2023.1 Intel oneAPI 2023.1 has been made available on _Raven_, _Cobra_, and other clusters. It provides the compiler module `intel/2023.1.0.x`, the MPI module `impi/2021.9`, the MKL module `mkl/2023.1`, and the corresponding Intel profiling tools. Also the software stack on these clusters has been compiled with this toolchain. With this oneAPI version, MPCDF follows Intel's recommendation to set the new LLVM-based C and C++ compilers, `icx` and `icpx`, respectively, as the default. They replace the deprecated "classic" compilers `icc` and `icpc`. With this transition, the corresponding MPI wrappers are called `mpiicx` and `mpiicpx`, respectively. The "classic" Fortran compiler `ifort` is still the default, however, the new LLVM-based Fortran compiler `ifx` and its MPI wrapper `mpiifx` are already available and can be tested thoroughly. We propose compiling your Fortran code with both `ifort` and `ifx` to compare performance and unravel possible issues. Please contact us via the [MPCDF Helpdesk](https://helpdesk.mpcdf.mpg.de/) in case you encounter compiler-related problems . _Tobias Melson_ ## Using linters to improve and maintain code quality Linters are static code analyzers that are commonly used for detecting programming errors, [code smells](https://en.wikipedia.org/wiki/Code_smell), compatibility issues, stylistic errors, etc. Some linters also have automatic patch generation or in-place code fixing capabilities. The term linter originates from S. C. Johnson’s [Lint](https://citeseerx.ist.psu.edu/doc/10.1.1.56.1841) tool for C source code, released in 1978. In the example section, you can see a short list of open-source linters for various syntaxes. Modern compilers can also be used as linters by [activating warnings](https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html). Using external linters is a part of [OpenSSF best practices criteria](https://bestpractices.coreinfrastructure.org/en/criteria/0?details=true&rationale=true#static_analysis). As an example, here's a typical output from [pylint](https://pylint.readthedocs.io/en/latest/): ``` > pylint not_bad.py ************* Module not_bad not_bad.py:1:0: C0114: Missing module docstring (missing-module-docstring) not_bad.py:6:0: E0401: Unable to import 'wrongpy' (import-error) not_bad.py:9:0: C0116: Missing function or method docstring (missing-function-docstring) not_bad.py:10:4: W0621: Redefining name 'timestep' from outer scope (line 32) (redefinedouter-name) not_bad.py:18:13: W1514: Using open without explicitly specifying an encoding (unspecifiedencoding) ----------------------------------- Your code has been rated at 6.40/10 ``` Depending on the syntax of your code, you may have multiple options when choosing a linter for your project. For example, in a python project, you can use [isort](https://pypi.org/project/isort/) and [autoflake](https://pypi.org/project/autoflake/) for cleaning up imports, [vulture](https://pypi.org/project/vulture/) for finding dead code, [bandit](https://pypi.org/project/bandit/) for checking security issues, [pyroma](https://pypi.org/project/pyroma/) for checking packaging, and [black](https://pypi.org/project/black/) for formatting. Your project’s size and complexity are also important factors in choosing linters. If your project is large, [ruff](https://beta.ruff.rs/docs/) together with a type checker such as [mypy](https://pypi.org/project/mypy/) can be much faster than [pylint](https://pypi.org/project/pylint/). Linters are usually designed to be very flexible and easily controllable using a configuration file. You must keep your linter's configuration together with your code under version control. One can manually run the linters to detect the issues and fix them. However, to ensure code quality, these linters must be integrated into your workflow to be triggered automatically. This is commonly done by adding them to the build system, integrating them into IDEs, or using [git pre-commit](https://pre-commit.com). Specialized runners such as [lintrunner](https://pypi.org/project/lintrunner/) can simplify setting up linters for complex projects. To maintain code quality, linters must be added to automated tests, e.g., in CI pipelines. The output of the linters can also be integrated into git forges such as [GitLab](https://docs.gitlab.com/ee/ci/testing/code_quality.html) and inspected in each merge request. As an example, you can see pylint integration in the Code Quality section of [this merge request](https://gitlab.mpcdf.mpg.de/mpcdf/training/pylint/-/merge_requests/2). The following table lists a few popular open-source linters as a starting point for your projects. |Linter|Syntax|Availability|Notable features| |--|--|--|--| |[pylint](https://pylint.readthedocs.io/en/latest/)|Python|[PyPI](https://pypi.org/project/pylint/)|Supports advanced type inference and detects code duplication but can be slow. Also supports various libraries such as [pydantic](https://docs.pydantic.dev/latest/) via [plugins](https://pypi.org/project/pylint-pydantic/). See [features list](https://pylint.pycqa.org/en/latest/user_guide/checkers/features.html).| |[flake8](https://flake8.pycqa.org/en/latest/index.html)|Python|[PyPI](https://pypi.org/project/flake8/)|A wrapper around [PyFlakes](https://pypi.org/project/pyflakes/), [pycodestyle](https://pypi.org/project/pycodestyle/), and [McCabe complexity checker](https://pypi.org/project/mccabe/). See [features list](https://flake8.pycqa.org/en/latest/user/error-codes.html).| |[nbQA](https://nbqa.readthedocs.io/en/latest/index.html)|Jupyter Notebook|[PyPI](https://pypi.org/project/nbqa/)|Includes pylint, flake8, isort, mypy, black, and various other python linters for Jupyter Notebooks.| |[clang-tidy](https://clang.llvm.org/extra/clang-tidy/)|C/C++|[various package managers](https://pkgs.org/search/?q=clang-tidy&on=files), [static binary](https://github.com/llvm/llvm-project/releases)|Includes checks for [C++ Core Guidelines](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines), OpenMP, MPI, etc. Accepts [compilation database](https://clang.llvm.org/docs/JSONCompilationDatabase.html).| |[cppcheck](https://cppcheck.sourceforge.io/)|C/C++|[various package managers](https://pkgs.org/search/?q=cppcheck), [static binary](https://github.com/danmar/cppcheck/releases)|A static data-flow analyzer with focus on detecting undefined behaviour. Can check for [MISRA Compliance](https://www.misra.org.uk/misra-c/).| |[fortran-linter](https://github.com/cphyc/fortran-linter)|Fortran|[PyPI](https://pypi.org/project/fortran-linter/) |A very basic line-by-line fortran linter.| |[lintr](https://lintr.r-lib.org/)|R|[CRAN](https://cran.r-project.org/web/packages/lintr/)|Can be used together with [styler](https://styler.r-lib.org/) and is included in [goodpractice](https://cran.r-project.org/web/packages/goodpractice/index.html). See [features list](https://lintr.r-lib.org/reference/linters.html).| |[JET.jl](https://aviatesk.github.io/JET.jl)|Julia|[_built-in package manager_]|Supports optimization analysis and error checking by abstract interpretation. Uses Julia's type inference system. | |[codeIssues](https://www.mathworks.com/help/matlab/ref/codeissues.html)|MATLAB| [_built-in_] |Provides functionality for interacting with [MATLAB Code Analyzer](https://www.mathworks.com/help/matlab/matlab_prog/check-code-for-errors-and-warnings.html). Supports [automatic fixing](https://www.mathworks.com/help/matlab/ref/codeissues.fix.html).| |[shellcheck](https://github.com/koalaman/shellcheck)|Shell script| [various package managers](https://pkgs.org/search/?q=shellcheck), [static binary](https://github.com/koalaman/shellcheck/releases)|Detects syntax issues, semantic problems, corner cases and pitfalls. Can be used together with [shfmt](https://github.com/mvdan/sh) for formatting scripts. See [online demo](https://www.shellcheck.net).| |[unmake](https://github.com/mcandre/unmake)|Makefile| [static binary](https://github.com/mcandre/unmake/releases) |Makefile linter with emphasis on portability.| |[markdownlint](https://github.com/DavidAnson/markdownlint)|Markdown|[npm](https://www.npmjs.com/package/markdownlint) |Can be used through [markdownlint-cli](https://www.npmjs.com/package/markdownlint-cli) or [markdownlint-cli2](https://www.npmjs.com/package/markdownlint-cli2). See [online demo](https://dlaa.me/markdownlint/).| |[ChkTeX](https://www.nongnu.org/chktex/)|LaTeX| [CTAN](https://www.ctan.org/pkg/chktex) |A part of [TeXLive](https://www.tug.org/texlive/). See [manual and features list](https://www.nongnu.org/chktex/ChkTeX.pdf).| |[restructuredtext-lint](https://github.com/twolfson/restructuredtext-lint)|reStructuredText| [PyPI](https://pypi.org/project/restructuredtext-lint/) | Can be used as a python library or through its cli named `rst-lint`. For linting Sphinx files use [sphinx-lint](https://pypi.org/project/sphinx-lint/).| |[commitlint](https://commitlint.js.org)|Commit message| [npm](https://www.npmjs.com/package/@commitlint/cli) |Enforces conformance to [conventional commit format](https://conventionalcommits.org).| |[megaLinter](https://megalinter.io/latest/)|[*multiple*]| [npm](https://www.npmjs.com/package/mega-linter-runner) | Includes many linters with a CI/CD focus. You can use [mega-linter-runner](https://www.npmjs.com/package/mega-linter-runner) to run it locally. See the [full list of linters](https://megalinter.io/latest/supported-linters/).| _Meisam Farzalipour Tabriz_ ## HPC-Cloud Object Storage On July 1st, 2023 a new Petabyte-scale Object Storage system was commissioned at MPCDF as part of the HPC-Cloud. The Object Storage system is based upon CEPH, a software-defined storage solution, and comprises 11 servers with a total of 11 PiB storage. One of the major advantages of the Object Storage system is that it offers global access, i.e. it may be accessed from MPCDF clusters and servers at Max Planck Institutes as well as user desktops/laptops (see Figure 1). ![HPC Cloud Object Storage](213/Object-Storage-MPCDF-Use-Cases-Small.png) In addition to global data access other benefits include S3 compatible API, life-cycle policies, temporary data sharing via temp URLs, multiple clients and a flexible python SDK. The SDK allows the storage to be accessed and managed directly from within applications, opening numerous possibilities for projects to automate and encapsulate data access in their workflows and services. The Object Storage system complements existing storage solutions at MPCDF offering an alternative to standard POSIX-based storage systems. Two possible data storage classes exist, replication and erasure coding, which allow the underlying data storage to be tuned to best suit a project's use-cases. An example use-case would be using the storage as an output sink for batch-based data production: Temporary access keys could be generated for the batch processing, these may be revoked after the batch processing. Then a set of batch jobs could be run with results being PUT into the object storage. Post processing jobs, at remote MPI clusters and/or desktops, could then GET the data for processing. Final results could again be PUT into object storage and could also be shared with collaborators via temporary access URLs if required. The Object Storage can be rented by projects in the range of 10s-100s of TB and is primarily designed to support scientific datasets with objects in the multi-MB range with a PUT/GET access pattern. The storage can be used together with cloud compute services or stand-alone as a storage silo. More information about the HPC-Cloud and the rental model can be found [here](https://docs.mpcdf.mpg.de/doc/cloud/index.html). _John Alan Kennedy, Florian Kaiser, Robert Hish_ ## JADE - Automated Slurm deployments in the HPC-Cloud A growing number of HPC-Cloud projects are deploying complex systems in the cloud to support various use-cases. One such system is a Slurm cluster, either as a backend for a service or to help better utilise cloud resources. To address this need an example solution has been created in the form of JADE. JADE uses Infrastructure as Code solutions including Terraform, Packer and Ansible to provide an automated deployment of Slurm clusters within the MPCDF HPC-Cloud. ![High-level JADE architecture](213/Jade.png) Terraform allows a JADE deployment to be managed as a complete stack, ensuring that a cluster can be reliably deployed as a whole, but also can be reliably deleted (removing all dependencies). This helps to maintain clean deployments within a cloud project and avoids possible wastes of resources. Packer and Ansible allow JADE images to be generated in a reproducible and well-understood manner. This is a cloud best practice that can be utilised in many other projects. JADE aims to be simple to deploy, elastic, i.e. it can scale in and out (scale the number of workers), and ephemeral, i.e. a cluster can be destroyed and re-created without loss of persistent data. This is ideal for projects which require small to medium Slurm cluster deployments or which wish to evaluate deploying a batch system within the cloud either for testing or to better utilise cloud resources. Using JADE a Slurm cluster can be deployed within approximately 10 minutes. The standard deployment (see Figure 2) provides a Slurm master and worker nodes, which only cluster admins can log in to, and a UI node for users to submit batch jobs. Cluster admins have complete freedom w.r.t. software installations and can flavour the UI and worker nodes to suit the user communities (a popular choice is the deployment of cluster specific software via modules in a similar fashion to the MPCDF clusters and HPC systems). The integration of MPCDF user management systems and Nexus-Posix allows JADE to provide users with an experience similar to standard cluster deployments at MPCDF. Moreover, since Nexus-Posix can be mounted on both Cloud resources and Raven, a hybrid solution can use _Raven_ for large-scale HPC processing and a JADE-based cluster for long-running post-processing jobs, sharing the data via Nexus-Posix. In addition to using JADE as a solution projects can use the Terraform-based deployment as an example to help understand how other complex services can be deployed within the HPC-Cloud. Further information about JADE can be found in [our gitlab repository](https://gitlab.mpcdf.mpg.de/mpcdf/cloud/jade). _John Alan Kennedy_ ## GitLab: Tips & Tricks ### Online editing of source code revisited Five years ago, in 2018, [Bits&Bytes published an article about GitLab's integrated Web IDE](https://docs.mpcdf.mpg.de/bnb/pdf/bits_and_bytes_issue_199.pdf) and how it can be used to edit code online. Today, GitLab contains a completely different Web IDE which is based on MS Visual Code. If you are already familiar with MS Visual Code on your desktop, you will now find the same behaviour and user experience directly integrated into GitLab. In future versions, GitLab's Web IDE will also support the native MS Visual Code plugins, which can be used to enhance the IDE's functionality in many ways. From any GitLab repository, you can reach the Web IDE via the button "Web IDE" on the repository's start page or from any open file. ![GitLab's new Web IDE](213/screenshot_web_ide.png) ### Custom badges GitLab offers [badges](https://gitlab.mpcdf.mpg.de/help/user/project/badges) to display short pieces of information in a graphical way. Badges are small images, displayed under the header of a GitLab project or group. By default, GitLab supports badges to display information about the current status of the CI Pipeline, test coverage and the latest release. ![Pipeline Badge](213/pipeline.png) With custom badges, it is also possible to upload any image and use it as a badge. You can find the badge settings under "Settings / General / Badges" in any GitLab repository. Here, you can create individual badges from uploaded images; for example, upload a logo and use it as eye catcher for your repository: ![Custom Badge](213/Screenshot_badges.png) ### Security warning GitLab's image registry is a convenient way of managing Docker images. You can upload and tag self-created Docker images, use them in Continous Integration Pipelines or make them accessible from outside GitLab. To work properly, the software inside the images needs sometimes user credentials, access tokens or other secret information to access services or data somewhere else. Storing these credentials in a Docker image which is publicly available can be a high security risk. In a recently published [article by M. Dahlmanns et al.](https://arxiv.org/abs/2307.03958), the authors found thousands of private credentials stored in publicly available Docker images. If you store self-created Docker images in GitLab's image registry and make them publicly available, please make sure that none of your usernames, passwords or other access tokens are stored inside the image! _Thomas Zastrow_ ## New IBM tape library and tape drives installed at MPCDF This year, the two Oracle SL8500 tape libraries at MPCDF will be taken out of service. To replace them and also enhance capacity and performance, one of the existing IBM TS4500 tape libraries has been expanded. Additionally, a new IBM TS4500 library has been installed, along with a total of 96 new LTO9 tape drives. The initial installation of the Oracle SL8500 tape library took place at MPCDF back in 2006, more than 17 years ago. Subsequently, it underwent multiple expansion phases, reaching a capacity of 20,000 tapes. Throughout the years, various generations of tapes and tape drives have been utilized, ranging from LTO-3 with a native capacity of 400 GB to LTO-8 with a native capacity of 12 TB. This tape library has served for many years as the primary storage location for backup and archive data generated by users of different Max Planck Institutes, ensuring its retention over an extended period. Due to the library model's discontinuation by Oracle and its prolonged period of operation, a decision was made to seek a suitable replacement. In 2013, MPCDF (formerly RZG) installed an IBM TS3500 tape library at the Leibniz computing centre (LRZ) with the purpose of storing a second copy of long-term archive data. In the following years, to accommodate the continuously growing volume of data, three additional IBM TS4500 tape libraries were installed at both MPCDF and LRZ. These additional installations were essential in meeting the expanding data storage demands. Now, the old Oracle library gets replaced by expanding one of the existing IBM TS4500 tape libraries with approximately 10,000 tape slots and procuring an additional IBM TS4500 library to further meet growing storage requirements. Furthermore, a total of 96 of latest generation LTO-9 tape drives have been installed across all IBM libraries. Here are some features of the IBM TS4500 tape library model: - One base frame and up to 17 expansion frames with a total capacity of over 22,000 LTO tapes per library. - Up to 128 tape drives per library. - Dual robotic accessors. - Automatic control-path and data-path failover. - Support for multiple logical libraries. - Tape-drive encryption and WORM media support. - Persistent worldwide names, multipath architecture, drive/media exception reporting, remote drive/media management. ![IBM TS4500 tape library](213/IBM-Tapelibrary.jpg) After the installation of the new systems, the current MPCDF tape storage infrastructure looks like this: - 5 IBM TS4500 tape libraries (3 at MPCDF + 2 at LRZ) with a total capacity of more than 105,000 LTO tape slots, of which about 60,000 are currently in use. - 200 LTO tape drives, of which 106 are latest-generation LTO-9 tape drives, while the remaining drives consist mostly of LTO-8. A small number of older LTO-7 and LTO-6 drives are still operational. - The tape drives are all integrated in two Fibre Channel Storage Area Networks (SANs), one at MPCDF and another one at LRZ. These SANs employ 16 and 32 Gb/s capable Broadcom Fibre Channel switches laid out in a multiple-path meshed topology. Over 30 server machines (IBM Spectrum Protect and HPSS servers) have access to these tape SANs to store data on tape. - Currently, the total data stored on tape, comprising backups in Spectrum Protect and long-term archives in HPSS, amounts to approximately 330 Petabytes. _Manuel Panea Doblado_ ## News & Events ### Open positions at MPCDF The MPCDF currently has three open positions in the Systems, Basic IT-services, and HPC application support division, respectively. Specifically, we are looking for: - Systems expert - design and operation of complex cloud and storage infrastructures for scientific applications - Computer Scientist - system and user management development - HPC application expert - development and optimization of new methods in the electronic-structure software package Octopus For details and directions to apply, please visit the [MPCDF career webpage]( https://www.mpcdf.mpg.de/career). _Markus Rampp_ ### Meet MPCDF Our monthly online seminar series "Meet MPCDF" has developed to a well-attended and valued training event. On every first Thursday of the month at 15:30 you can participate in an online seminar with a talk usually given by a member of the MPCDF and subsequent discussion. All material can later be found on our [training webpage](https://www.mpcdf.mpg.de/services/training). The schedule of upcoming talks is: - September, 7th: The MPCDF Metadata Tools - October, 5th: To be decided - November, 2nd: To be decided - December, 7th: Introduction to the new HPC system Viper We encourage our users to propose further topics of their interest, e.g. in the fields of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### AMD-GPU development workshop In preparation for the [new supercomputer of the MPG with AMD MI300A GPUs in 2024](https://docs.mpcdf.mpg.de/bnb/212.html#new-supercomputer-of-the-mpg-cobra-successor), the MPCDF in collaboration with AMD and Atos offers an online course on AMD Instinct GPU architecture and the corresponding ROCm software ecosystem, including the tools to develop or port HPC or AI applications to AMD GPUs. The workshop will be held as an online event, spanning three afternoons on **November 28-30, 2023**, please save the date. Further details including the agenda and a registration link will be published on the [MPCDF training website](https://www.mpcdf.mpg.de/services/training) in due course. _Tilman Dannert, Markus Rampp_ Bits and Bytes Logo # No.212, April 2023 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_212.pdf) ## High-performance Computing ### New Supercomputer of the MPG - _Cobra_ successor In the course of the last year MPCDF, in collaboration with MPG-GV and in close consultation with committees of the MPG-BAR and the MPCDF board, conducted a procurement for the next-generation HPC system of the Max Planck Society which will replace the current _Cobra_ machine by the end of this year. The corresponding proposal to the president of the MPG was supported by 58 departments and 19 research groups of 37 Max Planck Institutes from all three sections of the MPG. On January 16th, 2023 a [contract was signed with the winner of the European tender, Atos](https://atos.net/en/2023/press-release_2023_02_09/atos-to-build-max-planck-societys-new-bullsequana-xh3000-based-supercomputer). The new machine consists of a CPU-only and a GPU-accelerated system, both based on AMD processors of the latest generations, an Nvidia/Mellanox InfiniBand (NDR) network, as well as disk (ca. 20 PiB) and NVMe (ca. 500 TiB) storage with IBM SpectrumScale file system technology. The CPU-only system consists of 768 compute nodes, each with two AMD EPYC "Genoa" processors, providing 128 Zen4 cores per node and 512 GiB (609 nodes), 768 GiB (90 nodes), 1024 GiB (66 nodes), or 2048 GiB (3 nodes) of DDR5 memory per node, respectively. This system is scheduled for installation in the second half of this year. The GPU-accelerated system comprises 192 compute nodes, each with two of the new AMD Instinct MI300A "APU" processors with CPU cores and GPU compute units integrated on the same chip and coherently sharing the same high-bandwidth memory (128 GiB HBM3 per APU). This system is scheduled for installation during the first half of 2024. In order to help preparing applications for the new system, MPCDF will provide further technical details to its users as they become available and will organize dedicated trainings and workshops in due course, in particular covering the new APU technology. For more details and support please contact [Markus Rampp](mailto:markus.rampp@mpcdf.mpg.de). The procurement of the _Raven_ successor system is scheduled for 2025/2026. _Erwin Laure & Markus Rampp_ ### Documentation of HPC hardware characteristics Recently, the [_Raven_ user guide](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html) was extended with more detailed information on the [_Raven_ hardware characteristics](https://docs.mpcdf.mpg.de/doc/computing/raven-details.html), including a schematic of a GPU node showing its building blocks and the theoretical bandwidths between them. Moreover, roofline plots for the CPUs and GPUs and other performance numbers from measurements based on microbenchmarks are presented which may be helpful to the advanced user when analyzing and optimizing HPC code. _Klaus Reuter_ ### CMake Recipes Repository Have you ever asked yourself questions like "How do I link against third-party libraries correctly with CMake?", "How do I create custom CMake targets to build my documentation?" or "How do I get the current Git hash in my source code to print it to the log files?". We are here to help! The MPCDF maintains a repository with a growing number of CMake recipes, i.e. ready-to-use CMake code snippets for various tasks. You can check them out in our [GitLab](https://gitlab.mpcdf.mpg.de/mpcdf/training/cmake-recipes). And, if there is something missing or you need further clarifications, feel free to open an issue in our helpdesk. _Sebastian Eibl_ ## MPCDF HPC Cloud #### Introduction The MPCDF HPC Cloud provides on-demand computing and storage resources to research projects of Max Planck Institutes. Implementing the infrastructure-as-a-service model, the HPC cloud offers servers, networking features, and storage resources through a high-level API, CLI, or browser-based GUI. The moniker _HPC Cloud_ highlights the co-location of the cloud and HPC systems at MPCDF. While the HPC systems provide massive computing power, they are limited to applications that can be run within a batch system. The cloud complements the HPC workflows by enabling projects to define flexible computing solutions, including the deployment of workflow engines, databases and long-running jobs (exceeding the 24hr limit of the HPC batch system). Interaction with the HPC systems can be achieved, most notably, using the Nexus storage systems: the Nexus-POSIX file system which is mounted on _Raven_ and can be mounted on the HPC Cloud on demand, as well as Nexus-S3, the globally accessible object store. The HPC Cloud and Nexus storage systems are implemented with OpenStack, Ceph, and IBM Spectrum Scale. #### Hardware Resources ![HPC Cloud Hardware at MPCDF](212/HPC-Cloud-Hardware.jpg) The initial deployment of the HPC Cloud took place in 2021 in collaboration with the Fritz Haber Institute (FHI), and the Max Planck Institutes (MPI) for Human Cognitive and Brain Sciences, and for Iron Research, comprising 60 Intel Icelake-based compute nodes for a total 4320 cores, 44 TB of main memory, 12 Nvidia A30 GPUs, and 80 TB of local SSDs. An extension in collaboration with FHI, MPI of Animal Behavior, and the MPDL, is currently being deployed and will provide additional resources including 2688 cores, 96 TB of main memory, 60 Nvidia A100 GPUs, and 130 TB of NVMe-based local storage. All compute nodes have redundant 25 Gb ethernet links to a 100 Gb backbone, which itself has 100 Gb uplinks to the MPCDF core network as well as a direct connection to _Raven_. This second installment provides significant new resources and also allows for better coverage of use cases such as machine learning and memory or I/O intensive applications. #### Project Support The HPC Cloud is open to Max Planck Institutes and already hosts projects from numerous institutes including Biblioteca Hertziana, MPI for Biology of Ageing, MPI of Neurobiology and MPI for Chemistry. In collaboration with the MPCDF Cloud team, research teams can design, deploy and manage solutions within the HPC Cloud, taking advantage of the proximity to the _Raven_ HPC system when needed. In addition MPCDF has developed a set of recipes covering common requests, for example deployment of Kubernetes clusters "on top" of cloud-based resources. Starting in spring 2023 Max Planck Institutes can rent resources within the HPC Cloud based on a flexible and cost-effective renting model. Projects may start with an evaluation phase in which free tier resources are used to test project-specific use cases for a period of three to six months. Upon the completion of an evaluation phase, projects may transition into production where resources are rented on a rolling basis. More information about the MPCDF HPC Cloud and rental model can be found [in our documentation](https://docs.mpcdf.mpg.de/doc/cloud/index.html). #### Summary The HPC Cloud has been supporting cloud and hybrid Cloud-plus-HPC projects since 2021 and the opportunity now exists for new projects to evaluate use cases and rent cloud resources. The core aspects of the MPCDF HPC Cloud offerings are: - Standard cloud services (compute, storage, networking) - Solutions for hybrid Cloud-HPC projects (best of both worlds) - Enabling/design support for MPG research projects (and evaluation projects) - A flexible and cost-effective billing model _John Alan Kennedy, Frank Berghaus, Brian Standley_ ## News ### MPCDF SelfService Over the past months we have received a number of user requests concerning the registration pages for new MPCDF accounts. These pages and the entire registration workflow have now been modernized to provide a more comfortable experience to both applicants and approvers and to fit in with the overall aesthetics of the platform. Both applicants and approvers are informed about the registration process in more detail. Applicants can now also edit their application data up until its acceptance or rejection. Furthermore, phone numbers and E-mail addresses now need to be verified before they can be used for 2FA token creation to avoid typos and accidental registration of unauthorized addresses. E-mail addresses used for 2FA must differ from the main account E-mail address to avoid possible circumvention of the 2FA mechanism. _Amazigh Zerzour_ ### Pushing Fusion-Plasma Simulations Towards Exascale Together with the Max Planck Institute for Plasma Physics (IPP), the MPCDF engages in two projects to improve the performance and scalability of fusion-plasma simulations (particularly the [GENE code](https://genecode.org/)) towards exascale, that is 1018 floating-point operations per second. Precise simulations of fusion plasmas are essential for the development of fusion reactors, like the European ITER experiment or IPP’s ASDEX-Upgrade and Wendelstein-7X. In the Darexa-F (data reduction for exascale applications in fusion research) project, funded by the BMBF as part of the Scalexa program, MPCDF, who is also leading the projects, works together with IPP, the Technical University Munich, the Friedrich-Alexander-University Erlangen-Nürnberg, and ParTec on improving data handling in fusion simulations. We are particularly looking at compression techniques as well as mixed-precision data formats to reduce the overhead introduced by I/O, communication, and memory access. The exploitation of novel hardware, e.g. Data Processing Units (DPUs) is also an aim of the project. Darexa-F started on December 1st, 2022 and will run for three years. Under the leadership of the Royal Institute of Technology (KTH) in Stockholm, Sweden, a consortium of ten European partners is improving simulations of different plasma applications (space, laser, fusion) towards exascale in a EuroHPC Centre of Excellence called [Plasma-PEPSC](https://plasma-pepsc.eu). The project applies different techniques to improve performance and scalability, including optimizations for GPUs, advanced memory management, novel communication mechanisms, and the exploitation of novel hardware, including upcoming European processors. These techniques are applied to four codes: BIT (Czech Academy of Sciences), GENE (IPP), PIConGPU (Helmholtz-Zentrum Dresden-Rossendorf), and Vlasiator (Univ. of Helsinki). MPCDF is collaborating with IPP on the GENE code and Dr. Tilman Dannert from MPCDF is responsible for the overall technical developments as the project’s Technical Director. Plasma-PEPSC started on January 1st, 2023 and will run for four years. _Erwin Laure_ ### Base4NFDI: Creating NFDI-wide basic services in a world of specific domains NFDI is a German initiative to set up research data infrastructures within all disciplines, covering humanities and social sciences, life sciences, natural sciences and engineering sciences. To ensure sustainability, it will integrate national with international activities. In addition to domain-specific NFDI consortia, [Base4NFDI](https://base4nfdi.de/) has been formed. Base4NFDI is a unique joint effort of all NFDI consortia to develop and deploy NFDI-wide basic services. These services will be integrated into the emerging infrastructures at the European level, especially the EOSC. The target group for basic services is the wider NFDI community and, in particular, operators of community-specific services. The resulting NFDI-wide basic service portfolio will be beneficial for all disciplines. MPCDF has a co-leading role in Task Area 2: service integration and ramping-up for operation. MPCDF is also represented on the _Technical Expert committee_ which evaluates proposals for basic service development and gives recommendations on funding those. _Raphael Ritz_ ## Events ### Meet MPCDF Our monthly online-seminar series "Meet MPCDF" has developed to a well-visited and valued training event. On every first Thursday of the month at 15:30 you can participate in an online seminar consisting of a talk usually given by a member of the MPCDF and subsequent discussion. All material can later be found on our [training webpage](https://www.mpcdf.mpg.de/services/training). The schedule of upcoming talks is: - April, 6th, _The HPC Cloud at the MPCDF_ - May, 4th, _Containers in HPC_ - June, 1st, To be decided - July, 6th, _The MPCDF Metadata Tools_ We encourage our users to propose further topics of their interest, e.g. in the fields of high-performance computing, data management, artificial intelligence or high-performance data analytics. Please send an E-mail to . _Tilman Dannert_ ### Introduction to MPCDF services The next issue of our semi-annual workshop "Introduction to MPCDF services" will be held on April 20th, 14:00-16:30 via Zoom. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and datashare, together with a concluding question & answer session. Basic knowledge of Linux is required. [Registration is open](https://plan.events.mpg.de/event/56/). _Tilman Dannert_ ### AI Training Course In May the MPCDF, in collaboration with Nvidia, will host an "AI for Science Bootcamp" training event. At May 12th-13th, you will learn in an online event how to apply AI tools, techniques, and algorithms to real-life problems. Among others, you will study the key concepts of deep neural networks, how to build deep-learning models, and how to measure and improve the accuracy of your models. This online bootcamp is a hands-on learning experience where you'll be guided by step-by-step instructions with mentors on hand to help throughout the process. [Registration is open](https://www.mpcdf.mpg.de/events/33495/14192). Since the capacity for the hands-on session is limited, a "first-come-first-serve" policy has to be applied for the registrations. _Andreas Marek_ ### Course on "Python for HPC" The next iteration of our well-established course on "Python for High Performance Computing" is scheduled for July 25th to 27th, 2023. The event takes place online via Zoom and teaches how to use the Python ecosystem efficiently on HPC systems. We detail on topics such as using NumPy, SciPy, Cython, Numba, JAX, writing compiled extensions in C, C++, Fortran, making use of multithreading, GPU programming, and leveraging distributed memory parallelization using mpi4py and Dask. The lectures in the morning are complemented by exercises and Q&A sessions in the late afternoon. [Registration is open](https://plan.events.mpg.de/e/mpcdf-python-for-hpc-2023). _Klaus Reuter_ ### RDA-Deutschland Conference During this year's [Love Data Week](https://forschungsdaten.info/fr/fdm-im-deutschsprachigen-raum/deutschland/love-data-week/) (February 13th-14th, 2023) the annual conference of the [Research Data Alliance Germany](https://www.rda-deutschland.de) took place. The online event featured numerous contributions organized in 14 sessions ranging from [RDA for Newbies](https://indico.desy.de/event/37011/contributions/132884/) to [Data-Driven Decision Making](https://indico.desy.de/event/37011/contributions/132898/). Former MPCDF colleague Peter Wittenburg presented the opening keynote reviewing [10 years of RDA and 5 years of RDA Germany](https://indico.desy.de/event/37011/contributions/132897/). The conference program as well as summaries of the sessions and most of the slides presented are available from the [conference website](https://indico.desy.de/event/37011/). As in previous years MPCDF contributed to the organization of the event. _Raphael Ritz_ Bits and Bytes Logo # No.211, December 2022 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_211.pdf) ## High-performance Computing ### Anaconda Python modules Starting early next year, users will need to specify an explicit version when loading the 'anaconda' Python module, similar to the compiler, MPI, and CUDA modules. This is supposed to avoid common issues due to changes of the default version of a module. In case you need to load the anaconda module for your jobs, please adapt your submit scripts already now and use ``` module load anaconda/3/2021.11 ``` to load version 3/2021.11 explicitly, for example. This new behaviour will be enforced with an upcoming maintenance early 2023 which will be announced in due time. _Sebastian Ohlmann_ ### Hints for architecture-specific and optimized CUDA compilation Compute nodes at the MPCDF that are equipped with Nvidia GPUs typically run a long-term-supported (LTS) major version of the CUDA device driver. At the same time, more recent CUDA SDK versions are provided (e.g. `cuda/11.6`) that rely on the [forward-compatibility feature](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#deployment-consideration-forward) of the device driver. In case you are using the Nvidia CUDA compiler to compile source code, make sure to specify the correct target architecture via the `-arch` flag, e.g. `sm_80` for the Ampere (A100) GPUs on _Raven_ or `sm_70` for the Volta (V100) GPUs on _Cobra_: ``` nvcc -arch sm_80 source_file.cu ``` The proper `-arch` flag makes the binary match the actual GPU architecture and avoids PTX-related errors that might occur with most recent CUDA versions on LTS drivers if the architecture is not specified. Moreover, this flag enables compiler optimizations specific to the target microarchitecture which may improve performance. _Klaus Reuter_ ### New Intel C/C++ compilers and associated MPCDF software stack Intel is replacing their current, so-called "classic" C and C++ compilers (icc and icpc, respectively) by new LLVM-based compilers (icx and icpx, respectively). The classic compilers were already deprecated and will be removed from new OneAPI releases in the course of the second half of 2023, as mentioned in the [release notes](https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-c-compiler-release-notes.html). The same transition is planned for the classic Fortran compiler (ifort) to be replaced by the new LLVM-based compiler (ifx), but only at a later time when the new compiler matches the functionality and performance of the classic one. MPCDF will offer the new compilers and a full software stack compiled with icx, icpx and ifort on the HPC systems and clusters starting early next year. The corresponding intel modules are named with a version number ending with "x" (the first will be `intel/2022.2.1.x`). We recommend users to test and adopt this new software stack as soon as possible. In case of issues regarding functionality or performance, please let us know by opening a ticket in the helpdesk or via E-mail to . _Sebastian Ohlmann_ ### Turbomole license for MPG renewed The existing licensing agreement of the MPG for the quantum-chemistry software package [Turbomole](https://www.3ds.com/products-services/biovia/products/molecular-modeling-simulation/solvation-chemistry/turbomoler/) was recently renewed with the new owner of Turbomole, Dassault Systems. The agreement has been negotiated by the [Max Planck Digital Library](https://www.soli.mpdl.mpg.de/en/) in collaboration with MPCDF. MPCDF maintains latest versions of Turbomole on its HPC systems and provides the software for download by Max Planck Institutes on request. For obtaining Turbomole please open a ticket in the MPCDF helpdesk or send an E-mail to . _Markus Rampp_ ### Apptainer on HPC clusters, the Linux Foundation successor to Singularity #### Introduction [Apptainer]() is an open-source, container virtualization software designed to execute software in a secure, portable and reproducible environment. Just like its predecessor Singularity, Apptainer has been developed with the idea of providing container technologies on HPC systems. The software, for example, gives users an easy way to use different operating systems on the HPC systems while still ensuring that containers run in an established user environment, without a pathway for privilege escalation on the host. Apptainer was born in 2021, when the Singularity open-source project split into two distinct projects: Apptainer and SingularityCE. The Apptainer branch has joined the Linux Foundation, while the Sylabs' fork of Singularity, dedicated to commercial use, was renamed SingularityCE. While, at least at the beginning, there has been continual alignment between Sylabs' SingularityCE and Apptainer, over time the projects will likely diverge as both continue to mature and new features are included in the releases. As part of the transition, only open community standard interfaces will be supported in Apptainer. This includes removing the "Library", the Sylabs repository (similar to DockerHub) where you can push your containers to or pull containers from, and the "Remote Builder" support (but see the "Notes on the Sylabs Cloud endpoint" below). In the event these become open community maintained standards (and not corporate controlled), these features might be re-added at a later date. #### Apptainer at MPCDF In its current version, Apptainer provides backwards compatibility offering `singularity` as a command line link. It is also committed to maintain as much of the command-line interface (CLI) and environment functionality available in the old Singularity software as possible. From the user's perspective, very little, if anything, should change and the wrapper around the `singularity` command allows users to run commands like `singularity pull`, `singularity run`, etc. just as before. On the HPC clusters at MPCDF, an environment module is available in order to load the Apptainer software. For backward compatibility, a Singularity module (singularity/link2apptainer) is also provided and will print a warning message and load the default Apptainer module. Users are encouraged to switch from the old Singularity module to the new Apptainer one, adjusting their scripts as needed. Please, note that support for the old Singularity software has already been discontinued and the new SingularityCE software will not be supported on the HPC clusters at MPCDF. #### Notes on the Sylabs Cloud endpoint As mentioned above, Apptainer removed support for the old library remote endpoint provided by Sylabs Cloud. Practically, this means that commands like ``` apptainer pull library://lolcow ``` would fail with the error message "FATAL: Unable to get library client configuration: remote has no library client". The `apptainer remote` command group allows users to manage the service endpoints Apptainer will interact with for many common command flows. If access to the Sylabs Cloud remote is required, users can follow the instructions [here](https://apptainer.org/docs/user/1.0/endpoint.html#restoring-pre-apptainer-library-behavior) and run the commands ``` apptainer remote add --no-login SylabsCloud cloud.sylabs.io apptainer remote use SylabsCloud ``` in order to restore the library behaviour of the old Singularity software. It is possible to check the current list of remote endpoints using ``` apptainer remote list ``` #### Example of common Apptainer commands In the following, we illustrate some of the most common Apptainer commands (based on version 1.0.3). For more information and a complete description of all the commands, see the [Apptainer User Guide](https://www.apptainer.org/docs/). **Help command:** ``` apptainer help [] ``` Note that this command also shows the available options for each apptainer command. **Pull an image (_lolcow_ in the following examples) from an online repository:** ``` # Pull from the Sylabs Cloud (see notes above) apptainer pull ./lolcow.sif library://lolcow # Pull from the docker registry apptainer pull ./lolcow.sif docker://godlovedc/lolcow ``` **Build locally an image from a recipe:** ``` sudo apptainer build lolcow.sif lolcow.def ``` where the recipe file _lolcow.def_ contains the following ``` Bootstrap: docker From: ubuntu:16.04 %post apt-get -y update apt-get -y install cowsay lolcat %environment export LC_ALL=C export PATH=/usr/games:$PATH %runscript date | cowsay | lolcat ``` **Important**: the build command requires `sudo` just as installing software on your local machine requires root privileges. For this reason, users should build images from a recipe on their local machine (where they have sudo privileges) and then transfer the image to HPC clusters in order to run it. **Run the image _lolcow.sif_:** ``` apptainer run lolcow.sif ``` This command will run the user-defined default command within a container (i.e. the section `%runscript` in the apptainer recipe file). Some useful flags supported by the `apptainer run` command are: * `-C/--containall` this option ensures that not only file systems, but also PID, IPC, and environment are completely contained and separated from the user environment on the HPC cluster. In order to access files and directory on the host when this flag is used, you need to explicitly bind them (see the `-B/--bind` option below). * `-B/--bind /path_outside_container/:/path_inside_container/` is used to bind a user-specified path on the host (path before the `:` symbol) to a path inside the container (path after the `:` symbol). Multiple bind paths can be provided using a comma-separated list. * `--nv` is used in order to instruct the container's environment to use an Nvidia GPU and the basic CUDA libraries to run a CUDA enabled application. See [here](https://apptainer.org/docs/user/1.0/gpu.html) for details about the GPU support in Apptainer. * `--net --network=none --network-args "portmap=8080:80/tcp"` maps a port from inside the container to a different port on the host. Note that unprivileged users on the HPC clusters need to include the `--network=none` option. For example: In order to mount the `/ptmp/` folder available in the HPC cluster inside the container, so that your containerized code can access data or store results in your `/ptmp/$USER` space, you can use: ``` apptainer run --containall --bind /ptmp/$USER/:/path_inside/ lolcow.sif ``` where `/path_inside/` is the new path inside the container where the `/ptmp/` folder is mounted. If your code needs access to the GPUs available on the compute node, you can run ``` apptainer run --nv lolcow.sif ``` In order to expose port 80 inside of the container and map it to port 8080 outside of the container, use ``` apptainer run --net --network=none --network-args "portmap=8080:80/tcp" lolcow.sif ``` **Execute a specific command (`whoami` in the example) in the container:** ``` apptainer exec lolcow.sif whoami ``` **Open a shell in the container:** ``` apptainer shell lolcow.sif ``` Note that the flags presented above for the `apptainer run` command can also be used for the `apptainer exec` and the `apptainer shell` commands. _Michele Compostella_ ## GitLab Tips & Tricks: Use of Docker Images in GitLab CI Continous-integration (CI) pipelines are a well-adopted feature of the MPCDF GitLab instance, and many of our users are utilizing CI pipelines in their daily work. To execute these pipelines, MPCDF hosts several servers as so-called "shared GitLab runners" which means that every user of the MPCDF GitLab instance can execute pipelines on these servers. You can find more detailed information about our GitLab runners [here](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html). For every defined job in a CI pipeline, a Docker container with a default Docker image is executed on one of the runners. This default Docker image is __python:3.9__ on all shared Gitlab runners currently. For other programming languages or if the MPCDF module system is required, you can easily base your pipeline on a different image. The image can be taken from a public container registry like [Docker Hub](https://hub.docker.com) or be a self-built image stored in MPCDF's GitLab instance ([here](https://docs.gitlab.com/ee/user/packages/container_registry/) you can find further documentation on how to use the GitLab container registry). You can specify the Docker image to be used for creating your Docker container by modifying your pipeline definition file `.gitlab-ci.yml`. Insert a line at the top of the file, so that it looks, e.g., like this: ``` image: gitlab-registry.mpcdf.mpg.de/mpcdf/module-image ``` This example would provide an environment that is aligned with the software stack provided by the MPCDF module system on HPC systems and clusters. As another example, with ``` image: golang:latest ``` the latest Docker image with the environment for the Go programming language will be downloaded from Docker Hub, cached on our GitLab servers, and used as a basis for the container. To achieve reproducibility of the results, we recommend to explicitly specify a version instead of `latest`, because the latter might continuously be overwritten by newer images: ``` image: golang:1.19.1 ``` _Thomas Zastrow, Tobias Melson_ ## GO-Nexus This article presents GO-Nexus, an enhanced data transfer and sharing service for MPCDF's Nexus-Posix storage system based on Globus Online (GO). Nexus-Posix is an IBM Spectrum Scale filesystem which is commonly used by projects in MPCDF’s HPC-Cloud. Projects can rent a reservation on Nexus-Posix which can be scaled up as the project grows. The reservations are generally in the range of 10-100 TB and are accessed via mount points on the HPC system _Raven_ and/or HPC-Cloud VMs. Until now external access to Nexus-Posix has required either project specific solutions or the use of tunneled SFTP connections. Both of these solutions have significant limitations which lead to extra project overheads, slow and possibly unreliable data transfers, custom-made solutions etc. GO-Nexus is designed to address this by providing a Globus connect server on top of Nexus-Posix. Globus provides a fast, reliable and user-friendly way to transfer or share large amounts of data. Additionally, Globus can aid projects in publishing findable data for their communities. These qualities make Globus an ideal service to enable access to the large-scale data stored in Nexus-Posix. Combining these two services provides a solution for several core use cases and opens extra possibilities for projects which make use of Nexus-Posix. Two of the primary use cases are highlighted in Fig. 1, namely: 1. Transfer and Sharing service with mount points on Raven and possibly HPC-Cloud VMs. 2. Standalone Transfer and Sharing service, for data collection, publishing and sharing. The first use case highlights how projects can expose the Nexus-Posix filesystem mounted on the Raven HPC system and/or HPC-Cloud VMs, with users possibly performing large-scale simulations at MPCDF and then transferring results back to their home institute or even sharing them with colleagues in world-wide collaborations. ![GO-Nexus Example Use-Cases](211/Use-cases-Globus-Go-Nexus-2022.11.29.png) The second use case shows how standalone storage can be made globally available via GO-Nexus. This could be used when gathering data in the field for processing at a later date and/or for distributed collaborations where GO-Nexus would act as a central datastore, benefiting from the high-speed network connection at MPCDF. In both cases the reservations can be exposed either as findable or as private data collections via Globus, where users and community members can easily search for the data via the Globus web portal. In addition to the reliable transfer capabilities GO-Nexus benefits from all the advanced functionality which is available via the MPCDF’s globus subscription, enabling actions such as sharing and the use of Globus flows for automation. Several cloud projects have already adopted GO-Nexus for large-scale data transfers and to regularly sync data to and from Nexus-Posix by using “Globus timers”, a cron like service offered through the Globus web portal. _John Alan Kennedy_ ## News & Events ### Discontinuation of General VPN MPCDF has been offering a virtual private network (VPN) named "General VPN" for all its users, which allowed the access to the MPCDF network from everywhere. For security reasons this VPN will be discontinued as of February 1, 2023. Users are recommended to employ the VPN of their home institute in order to access the MPG network and to use the [MPCDF gateway machines](https://docs.mpcdf.mpg.de/doc/computing/gateways.html?highlight=tunnel#gateway-machines) for login or for tunnelling connections to MPCDF services. _Andreas Schott_ ### Meet MPCDF The monthly seminar series "Meet MPCDF" which was newly launched earlier this year, is receiving great interest by many MPCDF users. So far, the series covered "AI services at the MPCDF" (June), "CMake for HPC" (July), "MPCDF Gitlab Features" (September), "Data Transfer and Sharing" (October), "Going Public with your Code" (November) and "Trends in high-performance computing" (December). The corresponding announcements and slides can be found on the [MPCDF training portal](https://www.mpcdf.mpg.de/services/training). We gratefully acknowledge the attention of so many participants and look forward to continuing the series with talks and discussions next year. After a christmas break the series continues on Thursday, February 2, with a talk on basic debugging tools and strategies. Further topics envisaged for subsequent events (first Thursday of the month, at 15:30) include containers for HPC, strategies for testing software, and the Jupyter ecosystem. Our users are particularly welcome to suggest topics of their interest and to raise discussions and specific requests to MPCDF during the seminar. Registration is not required, the zoom link is distributed in advance via our all-users mailing list and is also posted on the MPCDF website. ### Python for HPC MPCDF held another issue of its well-established online course on "*Python for HPC*" from November 15 to November 17, which was attended by around 130 participants from several institutes. This annual workshop consists of lectures in the morning and exercise sessions in the afternoon and teaches how to combine the advantages of python for quickly developing new code with techniques for achieving good performance on HPC systems. The next issue is planned for November 2023. ### Advanced HPC Workshop From November 22 to 24, MPCDF organized the annual "Advanced HPC workshop" with talks from experts from MPCDF, Intel and Nvidia about profiling, debugging and porting HPC codes. The material of the talks can be found on the [MPCDF training portal](https://www.mpcdf.mpg.de/services/training). _Tilman Dannert_ Bits and Bytes Logo # No.210, August 2022 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_210.pdf) ## High-performance Computing ### Cobra successor procurement Earlier this year, MPCDF in collaboration with administrative headquarters of the Max-Planck Society (MPG) launched a Europe-wide call for tenders for procuring a successor for the HPC system _Cobra_ of the MPG. The corresponding proposal to the president of the MPG is supported by 37 Max-Planck Institutes of all three sections and expresses the need for significant CPU-only and GPU-accelerated computing capacity and storage in the time frame of the next five years. Bids from different vendors and with different technical characteristics will be gauged primarily by the compute performance and energy efficiency the offered system delivers for a representative set of HPC applications of the MPG. This benchmark suite comprises a mix of major simulation codes and machine-learning applications developed or extensively used in the MPG on both, CPU-only and GPU-accelerated HPC platforms. Delivery of the new system is expected by the end of next year, or early 2024. _Erwin Laure, Hermann Lederer, Markus Rampp_ ### CO2 footprint of MPCDF With more and more Max-Planck Institutes requesting the CO2 "footprint" associated with their use of MPCDF computing resources, we now provide a [web page](https://www.mpcdf.mpg.de/about/co2-footprint) which documents the total annual power consumption of MPCDF, together with the fraction used by the central HPC systems (currently, _Raven_ and _Cobra_) and the corresponding CO2 generation, as communicated by the electricity provider. In addition, the web page shows the electricity mix of nuclear, fossile, and "green" sources. _Andreas Schott_ ### Software news #### MPI compiler wrappers consolidation Starting with the Intel MPI module version `impi/2021.6`, only MPI compiler wrappers named according to a scheme with the prefix `mpi` followed by the name of the underlying (non-MPI) compiler executable are available on MPCDF systems (e.g., `mpiicc`, `mpiicpc`, `mpiifort` for Intel compilers and `mpigcc`, `mpig++`, `mpigfortran` for GNU compilers). Compiler wrappers named differently, e.g. `mpicc`, `mpicxx`, `mpif77`, ..., which are unsupported and have been deprecated by MPCDF before will no more be available under Intel MPI. The command-line option `-show` informs about the underlying compiler and the options actually used by the MPI wrapper. #### COMSOL multiphysics available Upon request by a few Max-Planck Institutes, MPCDF now provides the commercial software package [COMSOL multiphysics](https://www.comsol.com/) on the HPC system _Raven_. The licensing is based on the [software licensing service](https://www.soli.mpdl.mpg.de/en/) of the Max-Planck Digital Library, and currently includes a few of the commonly used "add-ons" with specialized COMSOL functionality. Additional functionality or an extension of the pool of concurrent licenses can be accommodated on request. A general overview of free and commercial scientific-software packages available on MPCDF systems is provided in our [documentation](https://docs.mpcdf.mpg.de/doc/computing/software/hpc-application-packages.html). Specific software packages can be located on MPCDF machines with the help of the command `find-module ` _Tobias Melson, Sebastian Ohlmann, Markus Rampp_ ## GitLab CI Distributed Cache ### Introduction GitLab's continuous integration (CI) infrastructure supports the caching of files and directories for reuse between different jobs of a pipeline and also between subsequently launched pipelines of the same repository. For certain CI setups that rely on a specific (e.g. static or rarely changing) set of files the GitLab CI cache should be used to speed up the pipeline execution considerably. The MPCDF shared runners (for both CPU and GPU) are configured to use distributed caching, enabling them to access the same caches from different runners consistently. After a pipeline is done, the created caches will not be deleted but stay available for further runs of the same pipeline. ![Schema distributed cache](210/distributedCacheSchema_3.png) ### Adding a cache to your CI configuration For a CI job caching is enabled by adding a `cache` section to its configuration in the `.gitlab-ci.yml` file: ```yaml my_job: cache: key: my-cache-key paths: - bin/ ``` The `cache:key` keyword can be used to give each cache a unique identifier. Note that all jobs that use the same cache key access the same cache, which applies to the same pipeline and also to subsequent pipelines of the same branch of a project. The `cache:paths` section lists paths to be included into the cache. In order to enforce a rebuild of the cache, the corresponding key needs to be changed in the pipeline definition. In the following, two example use cases are discussed. The first use case demonstrates the advantages of creating a cache only once and then reusing it in subsequent jobs and further pipeline runs. The second one demonstrates the use of a distributed cache over different shared runners during a pipeline run. A [GitLab project](https://gitlab.mpcdf.mpg.de/khr/gitlab-ci-cache-usage-examples) contains the setups for both examples. ### CI of software that depends on third-party packages Consider a Python package that requires a certain third-party package as a dependency. A naive (but frequently adopted) approach would be to `pip install` such a dependency within the CI script right before the actual tests of the primary package are run. Clearly, this creates unnecessary overhead and Internet traffic when done repeatedly. The GitLab CI cache can be used to store the third-party packages between invocations of the pipeline such that `pip` does not repeatedly download them. Once the cache has been created it will stay on the storage back end, being available for further runs of the same pipeline. For compiled codes an analogous use case would be to cache the download and build of a third-party library your primary code relies on. ### CI of a complex HPC code that requires CPU and partly GPU resources Consider the prototypical CI pipeline of an HPC code that first builds the code and then runs tests on it. Often, the build takes a considerable amount of time whereas the tests are comparably quick. On the GPU runners this becomes inefficient because during the build the GPU is almost always idle. Using the GitLab CI cache the build can be performed on the (less expensive and more available) CPU runners, and only the tests can be executed on GPU runners, thereby improving the overall throughput of CI jobs. For some codes it may also be possible to use a single build for both CPU and GPU tests. ### Concluding remarks The `cache:key` keyword allows to cover a variety of use cases, in particular in combination with predefined variables. [Several examples](https://docs.gitlab.com/ee/ci/caching/#common-use-cases-for-caches) are provided in the comprehensive [GitLab cache documentation](https://docs.gitlab.com/ee/ci/caching). Unlike GitLab artifacts which are exposed per pipeline run and are downloadable by users via the GitLab web interface for a predefined amount of time, the GitLab cache is considered a mere optimization. A cache is not guaranteed to exist and might need to be regenerated (automatically). In practice, this would be the case e.g. after some time (e.g. weeks) when the cache was cleaned automatically on the server side and the user would launch the pipeline once more. _Thomas Zastrow, Klaus Reuter_ ## Globus Flows ### Introduction In recent years Globus Online has continued to evolve, adding functionality such as data sharing and publishing as well as automation services. One of the latest additions is Globus Flows which we will introduce in this article. Globus Flows is a service which allows users to define and execute data workflows using the Globus Subscription attached to servers at MPCDF, such as DataHub. These workflows consist of Actions which can be combined into a single logical operation, be that as simple as staging data across several servers or more complex, for instance processing data on ingest with the need for a human in the loop to review and confirm results before the data is finally published to end users. ### Flows in detail Each Action in a Flow is enabled by an Action Provider which is a REST resource. Globus provides a wide selection of general Action Providers such as transfer, delete and Datacite minting, which can be used by all authorized users to form Flows. In addition, a Python SDK exists to allow projects to create and register their own Action Providers, thus they can wrap their own analysis or infrastructure services to make them available to Flows. The Flows themselves are written using the Amazon State Language which allow Actions and States to be linked to form complex pipelines which involve both data transfer and processing. Once a Flow has been defined it can be deployed to the Globus Automate platform where it can be made visible to other Globus Online users or kept private so that it is only visible to the author. The Flow can be run from the Globus Online Web app and a form for input values can be generated for users as an interface to the Flow (see example below). In most cases users will not need to write Flows themselves, but instead they will either use standard Flows defined already by Globus or specific Flows defined by their project or community. A [library of Flows](https://app.globus.org/flows/library) already exists and can be found via the Globus Online Web app. ### Example A simple, and yet often useful, example is that of a two-stage data transfer. In this case data is to be transferred between source and destination via an intermediate server. This workflow is often used when users stage data to/from the MPCDF HPC system via DataHub. The flow encompasses the following steps and can be found [online](https://app.globus.org/flows/99791f7d-6c2c-4675-af4b-b927db68bad0): 1. Copy data from source to intermediate 2. Copy data from intermediate to destination 3. Delete data on intermediate server Once a user clicks "Start Flow" they are presented with a form where they can fill in the details about the source, intermediate and destination. The screenshot (Figure 2) shows this form where the intermediate has already been selected as the DataHub. Note that the intermediate step is highlighted in green to show it has been defined. ![Two-stage Globus Flow](210/dark-small-2stage-flow-datahub-selected.png) Without Globus Flows these steps need to be taken manually by the user, which in the best case leads to a need to "babysit" transfers and in the worst case can lead to human error and the need to re-transfer data. Using a Flow allows this to be wrapped up into a single logical operation. Moreover, similar to transfers in Globus Online, Flows may be given labels and the results from previous Flows can be viewed in the Web app. This means users can debug Flows and also see their history of previously run Flows. ### Summary Globus Flows provide a means to automate data transfers, simplifying this process for users while also making multi-stage operations more reliable. Many useful Flows, which can be used or taken as templates, already exist in the Flows library. In addition, custom Flows can be defined which can even include project-specific Action Providers. _John Alan Kennedy, Frank Berghaus_ ## News & Events For latest training events, course material, as well as for more details and updates on the courses listed below, please visit the [MPCDF training website](https://www.mpcdf.mpg.de/services/training). ### Introduction to MPCDF services The next issue of our semi-annual workshop "Introduction to MPCDF services" will be held on October 13th, 14:00-16:30 on zoom. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and datashare, together with a concluding question & answer session. Basic knowledge of Linux is required. [Registration](https://events.gwdg.de/e/intro-mpcdf-2022-10-13) is open. ### Meet MPCDF The next seminar in our monthly online-lecture series "Meet MPCDF" will be held on September 1st, 15:30-16:30 on zoom. During this event our GitLab instance will be introduced, addressing its known and less-well-known features. Also the subsequent issue of "Meet MPCDF" (on October 6th, 15:30-16:30) is related to GitLab, when we will present a talk about "Going public with your code". The seminar will address various questions and decisions that may become relevant in preparation of sharing your own code with other developers and the public, such as licensing, software management and quality assurance (automatic tests, continuous integration, etc.). For "Meet MPCDF" events, no registration is necessary, the connection details can be found on the MPCDF training website. ### Advanced HPC workshop MPCDF will again organize an advanced HPC workshop for developers and users of the MPG and of the [EU Centre of Excellence NOMAD](https://www.nomad-coe.eu/) from Tuesday, November 22nd until Wednesday, November 23rd, 2022, with an additional hands-on day (participation is optional and by application) on Thursday, November 24th. If the pandemic permits, we will have the hands-on day in presence in Garching and the lectures in a hybrid fashion onsite and online. The main topics of the lectures are * Debugging and profiling of CPU and GPU codes * Porting codes to GPU-accelerated systems As a prerequisite, we require participants to have an account for the HPC machines of the MPCDF and to be already familiar with accessing, building and running their codes there. If you are interested in bringing in your own code to work "hands-on" with the experts and to use the techniques and tools taught in the lectures, please apply by adding a short description of your code and your specific goals in the registration form. The entire Thursday, November 24th is dedicated to working on the selected code projects. The lectures will be given by members of the application group of the MPCDF, together with experts from Intel and Nvidia. [Registration](https://events.gwdg.de/e/hpc-workshop-2022) is open until November 12th. Applicants for the hands-on day are kindly asked to register as early as possible and to prepare a representative test case for their code on _Raven_ (ideally the test can be run on a single _Raven_ node, either GPU-accelerated or CPU-only) already in advance of the workshop. Assistance by MPCDF is provided on request. ### Python for HPC From November 15th to November 17th, 2022, MPCDF offers another iteration of the course "Python for HPC" to participants of the MPG. The course will be given online via zoom with lectures in the morning (9:00-12:00) and exercises in the late afternoon (16:00-17:00). The course teaches approaches to use Python efficiently and appropriately in an HPC environment, covering performance-related topics such as NumPy, Cython, Numba, compiled C and Fortran extensions, profiling of Python and compiled code, parallelism using multiprocessing and mpi4py, and efficient I/O with HDF5. In addition, topics related to software engineering will be addressed, such as packaging, publishing, testing, and the semi-automated generation of documentation. The lectures will be given based on Jupyter notebooks which include many reusable code examples. For each topic, hands-on exercises will be provided and discussed in separate sessions. On the last day, there will be time for a general question & answer session. [Registration](https://events.gwdg.de/e/python4hpc-2022) is open. _Tilman Dannert_ Bits and Bytes Logo # No.209, April 2022 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_209.pdf) ## High-performance Computing ### AlphaFold2 on the HPC system _Raven_ [AlphaFold2](https://www.nature.com/articles/s41586-021-03819-2) (AF2) is an AI system that is able to predict the 3d structure of a protein from its amino acid sequence. At runtime, the AF2 system initially performs multiple sequence alignments (MSA) on the CPU, followed by the actual structure predictions on the GPU. The MPCDF has been providing installations and job scripts on _Raven_ since early August 2021, shortly after Deepmind/Alphabet Inc. had publicly released version 2.0.0. This article summarizes recent changes and advancements that are important to run AF2 efficiently on the HPC system _Raven_. With the deployment of version 2.2.0 (environment module 'alphafold/2.2.0') on _Raven_, several improvements were implemented: The AF2 software now runs natively, unlike previous installations that are enclosed in a software container based on the original Docker image. In addition, the job scripts provided by the MPCDF are now split into a plain CPU job for the MSA phase and a dependent GPU job for the subsequent prediction phase. This split helps to minimize idle times of the GPUs. Moreover, to cover the memory footprint of large protein setups it is necessary to allocate more than one GPU together with the host memory via CUDA Unified Memory, as explained in the job scripts. The command `module help alphafold/2.2.0` gives further instructions to users and information on how to access and use these scripts. The MSA phase of the AF2 pipeline in particular is I/O bound and puts high load on the file system when reading from the databases. To this end, the MPCDF has stored these databases on a separate file system from the regular user file systems '/ptmp' and '/u' in order to avoid performance impacts on other HPC jobs while maximizing the I/O performance for AF2. In late March 2022, the databases were migrated from spinning disks to a more advanced NVMe-based storage system that is read-only mounted on each _Raven_ node. With that change users should experience a significant performance improvement for the MSA phase. The path to the AF2 databases is provided by the environment variable 'ALPHAFOLD\_DATA' which is set when an 'alphafold' environment module is loaded. Please, only read the databases from the directory referenced via 'ALPHAFOLD\_DATA', do not create your own copies in '/ptmp' or '/u'. Moreover, please do not run AF2 on the HPC system _Cobra_ as there is no optimized storage for the databases, and hence the performance is inferior compared to _Raven_. _Klaus Reuter_ ## GitLab CI ### GitLab shared runners on GPUs The MPCDF GitLab instance offers a wide variety of _DevOps_ functionalities. One common _DevOps_ functionality is _Continous Integration (CI)_. CI allows the user to define _job pipelines_ which are executed after new data was pushed into a GitLab repository. A pipeline can compile code, execute tests or render images, just to name some common use cases. These pipelines are executed asynchronously on _GitLab runners_. In principle, every GitLab user can set up a GitLab runner, e.g. on a local laptop or a remote machine the user has access to. In addition, the MPCDF offers several _shared GitLab runners_ in the [HPC-Cloud](https://docs.mpcdf.mpg.de/doc/cloud/index.html) which can be readily employed by any MPCDF GitLab user. You can find a list of the currently available runners in the [MPCDF documentation](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html). Besides the well-established shared runners on CPUs, the MPCDF now offers two additional runners supporting GPU-enabled applications. These runners are labeled _MPCDF-GPU-01_ and _MPCDF-GPU02_, and each runner has access to a MIG partition of an Nvidia A30 GPU. They are labeled with the tags _cloud-gpu_ and _nvidia-cc80_ and are configured to only run correctly tagged jobs. A CI job that wants to make use of the GPU runners needs to explicitly define at least one of these two tags. #### Example: Continuous integration testing of CUDA code The following example _.gitlab-ci.yml_ file demonstrates how to use the GPU shared runners to perform continuous integration testing of a CUDA-enabled HPC code. ```yaml # .gitlab-ci.yml cuda-basic-ci: image: gitlab-registry.mpcdf.mpg.de/mpcdf/module-image tags: - cloud-gpu - nvidia-cc80 script: - nvidia-smi - module avail - module load gcc/11 cuda/11.4 - nvcc --version #- ... compile and test the CUDA code as you would do on the HPC system ``` The example uses the 'module-image' provided by the MPCDF that offers a software environment largely consistent with the software environments on the HPC systems. As shown the CUDA GPU toolkit and other software are pulled into the environment via `module load` commands. ### Continuous integration testing for HPC codes on MPCDF GitLab Continuous integration (CI) and in particular continuous unit and integration testing are indispensable ingredients for today's HPC software development workflows. The MPCDF GitLab offers shared runners in the HPC-Cloud that enable teams or individual users to easily perform automated tests for each code commit. Technically, there are shared runners with access to CPUs and GPUs, where each runner offers 4 virtual cores based on the Intel IceLake architecture to the CI jobs, and each GPU-enabled runner offers in addition a virtual GPU corresponding to about 50 % of an A30 GPU (Ampere architecture). Particularly useful to HPC users is the environment-module enabled Docker image which the MPCDF provides to be used on the shared runners. With that image CI job scripts can simply issue the familiar `module load` commands to get access to virtually the same software as offered directly on the HPC systems. Hence, tests can easily be implemented for different compilers (e.g. Intel, GNU, Nvidia) or MPI libraries (Intel MPI, OpenMPI). Moreover, within the limits of the 4 virtual cores per runner, sequential vs. parallel execution can be tested based on threads (OpenMP) or processes (MPI), or the correctness of different vectorization levels (e.g. AVX2 or AVX512) can be checked for. Note that these cloud-based resources are less well suited for continuous benchmarking. The MPCDF recommends to use the shared runners in combination with the 'module-image' and not to set up custom individual runners directly on the HPC systems. The latter would not only have to interact properly with the Slurm batch system, but also has an intrinsic security issue for multi-user repositories, because it would execute code committed by any user who can push to the repository in the context of the user who has set up the runner. An example of a '.gitlab-ci.yml' file that uses the 'module-image' is given in the previous section on the GPU runners. General information about the shared runners is available at the [MPCDF documentation pages](https://docs.mpcdf.mpg.de/doc/data/gitlab/gitlabrunners.html). _Thomas Zastrow & Klaus Reuter_ ## Globus Online The MPCDF DataHub service has provided a staging area for multi-terabyte (TB) data transfers for the past several years. In recent times the MPCDF has observed an increasing need for researchers to transfer and share multi-TB datasets, specifically with non-MPG collaborators from around the world. To address this trend the MPCDF obtained a Globus Online Subscription and on the 9th of March the DataHub's Globus Online service was upgraded to version 5.4 and this subscription was enabled. This upgrade means that extra functionality is now available for MPCDF users via the subscription and that the DataHub now appears in the Globus Online Portal in a slightly different way. In brief: * Data is now exposed via collections in the Globus Web Portal (mpcdf#datahub is no longer available). * Sharing data with any Globus user is now possible. * Enhanced client functionality is now available to users who join the MPCDF Globus Plus Group. More detailed information is provided below. #### DataHub access via the Globus Online Portal The upgade to v5.4 means that the old endpoint "mpcdf#datahub" is no longer available. This was replaced by two new "collections" (logical collections for accessing data). In the Globus Portal these collections are: 1. "MPCDF DataHub Stage-and-Share Area" -- The same scratch-based /data area that was mounted on mpcdf#datahub 2. "MPCDF DataHub CBS Project Space" -- An explicit collection for the project space of the MPI for Human Cognitive and Brain Sciences The collections can be found by using the search function in the File Manager or Bookmarks section of the Globus Online Portal. Accessing the collections remains similar. Simply follow the usual login steps, then link an identity from "MPCDF DataHub OIDC Server (login.datahub.mpcdf.mpg.de)" and once this is linked use the identity (`username@login.datahub.mpcdf.mpg.de`) to access the collection. #### Enhanced functionality The new subscription allows the use of the following enhanced functionality: 1. __Data sharing__: In the File Manager section of the Portal a directory can be selected for sharing. This is called a "Guest Collection" and can be shared with individual Globus Online users or groups of users. The guest users can be any user with a Globus Online account, they do not need to have an MPCDF account. 2. __Globus Plus__ (for Globus Personal Clients): MPCDF users can now enable sharing from a Globus Connect Personal Endpoint and also perform client-to-client data transfers. This functionality may be enabled by requesting membership of the group "Max Planck Computing and Data Facility Globus Plus". Simply search for this group in the Groups section of the Portal and click "Join Group" to make a request for membership. #### More information More information can be found in the MPCDF documentation pages: [MPCDF DataHub and Globus Online](https://docs.mpcdf.mpg.de/doc/data/data-transfer/mpcdf-datahub-and-globus-online.html). General information on Globus Online and Globus Connect Personal can be found in the Globus Documentation: [How-to](https://docs.globus.org/how-to/), [FAQ](https://docs.globus.org/faq/), [Videos](https://www.globus.org/videos). For specific questions about the MPCDF support for Globus Online please create a [helpdesk ticket](https://helpdesk.mpcdf.mpg.de) or mail . _John Alan Kennedy_ ## New SelfService Features and Improvements The [MPCDF SelfService](https://selfservice.mpcdf.mpg.de) is constantly evolving. By the end of April version 5.0.0 will be released including additional functionality and improvements of existing workflows to create a better user experience. All changes mentioned below will be made available with this new version. ### Redesign of the login process The login page has been redesigned to be more intuitive and visually appealing. It now simply features an input field for the username and password, respectively, without the need to first specify the type of account. The page also allows users to request a new MPCDF account via a registration button, following well-established design patterns to provide more clarity. If [two-factor authentication (2FA)](https://docs.mpcdf.mpg.de/faq/2fa.html) is activated for your account, the SelfService will ask for the OTP in a second step. In case no OTP can be provided due to a lost or defective token the user can now initiate an automated access restoration workflow without the need to contact the MPCDF support. This will allow users to regain access to their accounts more quickly while maintaining the existing security level. To avoid getting locked out in the first place we strongly recommend [creating a backup token](https://docs.mpcdf.mpg.de/faq/2fa.html#how-do-i-enroll-and-use-a-secondary-backup-token) to anyone with 2FA enabled. ### Viewing accounting data All users with a regular MPCDF account can now check how many computing and storage resources they used in a given month or time range. This includes computation on our HPC systems as well as storage volume on AFS and other file servers. The data is displayed in multiple tables highlighting different aspects for maximum clarity and control over one's resource usage: * Ungrouped: this is the raw data that may be used for custom analysis * By type: see which systems you used most * By cost center: see how your usage will get billed * By month: see how your usage changed over time * By account: in case you own secondary accounts, see which one used how many resources Institute responsibles and accounting departments are able to access the information and tables for their entire institute. The SelfService offers filtering for specific users and/or cost centers. All tables can be downloaded separately in different formats such as PDF and CSV for documentation or further analysis. Note, however, that the SelfService provides preliminary data for informational purposes only which may differ from the official accounting and billing. ### Additional improvements The following smaller features will be released with the new SelfService version: * Users can now see more of their account details as well as their secondary accounts and associated information at "My Account > My data". * Each account under "My Account > My data" now shows a button "Change password" for easier navigation. * Supervisors can now filter the list of their supervised users by whether they are locked or not. * Supervisors can now bulk-edit their supervised users (extend or lock multiple accounts). * Supervisors can now see the month of the last successful and failed logins for each supervised account. This serves to help supervisors decide whether the account is still needed while keeping the necessary vagueness to disallow user monitoring. We are aiming to provide a pleasant user experience and are always happy to receive suggestions for improval and comments on UI from our users. Please send us your comments: [support@mpcdf.mpg.de](mailto:support@mpcdf.mpg.de). _Amazigh Zerzour, Andreas Schott_ ## Access to AFS restricted for local Access only As already announced in [Bits&Bytes issue 206](https://docs.mpcdf.mpg.de/bnb/206.html#decommissioning-of-afs), the AFS cell _ipp-garching.mpg.de_ jointly operated by IPP and MPCDF will be decommissioned in the course of the next few years. As a first step and protective measure the worldwide access to the AFS cell _ipp-garching.mpg.de_ will be blocked, while the access will remain possible from the local networks of the IPP (including IPP-HGW), MPA, MPE, MPQ, MPCDF, and a few external collaboration partners. Also MPP, which is moving to the campus soon, will not be restricted. Respective VPN connections will allow the access, too. This **blocking of AFS-connections** by the firewall will be activated on **May 16th, 2022.** _Andreas Schott_ ## News & Events ### AI bootcamp Together with Nvidia, the MPCDF organizes an online bootcamp "AI for Science", which will take place on May 23rd-24th. The event targets scientists who do not have any prior knowledge of AI methods. During the two-days, hands-on workshop the participants will learn how to apply AI tools, techniques, and algorithms to real-world problems and will study the key concepts of deep neural networks, how to build deep learning models, and how to assess and improve their accuracy. Since the number of registrations already exceeds the capacity of the workshop, it is planned to organize another "AI for Science" bootcamp towards the end of 2022. _Andreas Marek_ ### International HPC Summer School 2022 The International HPC Summer School (IHPCSS) 2022 is planned as an in-person event from June 19th to June 24th in Athens, Greece. The series of these annual events started 2010 in Sicily, Italy. After cancellation of the IHPCSS 2020 event due to the Covid-19 pandemic and conducting IHPCSS 2021 as a pure virtual event, the in-person event in June shall take place with fully vaccinated participants only and under adequate health measures. For the 2022 event, the organizing partners XSEDE for the US, PRACE for Europe, RIKEN CCS for Japan and the SciNet HPC Consortium for Canada have carried out a joint call for applications. After the reviews by all partners according to the same selection criteria, up to 90 participants have been selected and invited, thereof 30 from European institutions. From Max Planck institutes, four applicants made it into the final selection. School fees, meals and housing will be covered for all accepted applicants. For further information please visit the [website of the summer school](https://ss22.ihpcss.org/). _Hermann Lederer_ ### Workshop "Introduction to MPCDF services (online)" The next issue of our semi-annual introductory workshop will be held on April 28th, 14:00-16:30 on zoom. Topics comprise login, file systems, HPC systems, the Slurm batch system, and the MPCDF services remote visualization, Jupyter notebooks and datashare. Basic knowledge of Linux is required. Registration is necessary and can be done [here](https://events.gwdg.de/e/intro-mpcdf-2022-04-28). _Tilman Dannert_ ### "Meet MPCDF": New online forum and lectures for MPCDF users In June 2022, the MPCDF will launch a new series of monthly online lectures together with a Q&A forum for its users. The event will be held on the first Thursday of the month, from 15:30 to 16:30, and features a technical talk (ca. 20-30 minutes) about an AI, HPC or data-related topic given by a staff member of the MPCDF. In addition, this meeting offers the opportunity for the users to informally interact with MPCDF staff, in order to discuss relevant kinds of technical topics. Optionally, questions or requests for specific topics to be covered in more depth can be raised in advance via [e-mail](mailto:training@mpcdf.mpg.de). Target audience are intermediate to advanced users of MPCDF services as well as computational scientists and software developers of the MPG. Users seeking a basic introduction to MPCDF services instead, are referred to our semi-annual online workshop "Introduction to MPCDF services" (see above) which offers a consistent introduction for new users of the MPCDF. The first "Meet MPCDF" event will take place on June 2nd at 15:30 with the talk "Introduction to the AI tools at the MPCDF". The second issue is planned for July 7th and will cover a topic on HPC software engineering. Connection details and updates can be found on the [MPCDF webpage](https://www.mpcdf.mpg.de/services/training). _Tilman Dannert, Markus Rampp_ ### RDA-Deutschland-Tagung 2022 February 21st-25th, more than 500 people attended this year's online conference of the [German chapter](https://www.rda-deutschland.de/) of the [Research Data Alliance](https://www.rd-alliance.org/). Various topics around research data management were discussed with a specific focus on the [FAIR](https://force11.org/info/the-fair-data-principles/) (_Findable, Accessible, Interoperable and Reuseable_) as well as the [CARE principles](https://www.gida-global.org/care) -- _Collective Benefit, Authority to control, Responsibility and Ethics_. The schedule and some of the slides presented are available from the [conference website](https://indico.desy.de/event/31380/timetable/#20220221). As in previous years the MPCDF was involved in the organization of the conference. Next year's conference is scheduled for February 13th-17th, 2023. _Raphael Ritz_ Bits and Bytes Logo # No.208, December 2021 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_208.pdf) ## High-performance Computing ### Termination of general user operation for _Draco_ login nodes As announced previously, starting from January 10, 2022 access to the _Draco_ login nodes will be restricted to the users belonging to institutes which own dedicated _Draco_ nodes; namely the Fritz Haber Institute, the Max Planck Institute for the Structure and Dynamics of Matter, and the Max Planck Institute for Animal Behaviour. The file systems /draco/u and /draco/ptmp will stay available on _Cobra_ and _Raven_ until further notice. _Renate Dohmen_ ### Announcement of CUDA no-defaults on _Cobra_ and _Raven_ Please note that in the near future you will need to specify an explicit version when loading the "cuda" module, just as is already required for the Intel compiler and MPI modules. This will be enforced after the next maintenance window and a plain `module load cuda` will fail, then. Instead, use ```bash module load cuda/11.2 ``` to load version 11.2 explicitly, for example. In case you need to load the "cuda" module for your jobs, please adapt your job scripts already now. The actual dates of the maintenances will be announced in due time. _Sebastian Ohlmann, Klaus Reuter_ ### Usage of /tmp and /dev/shm on _Cobra_ and _Raven_ On the HPC systems at MPCDF neither the /tmp file system nor the `TMPDIR` environment variable should be used for storing scratch data. Instead, the /ptmp directory which is accessible as a parallel file system from all compute nodes is provided for such purposes. On the other hand some applications require access to the local file system on the compute nodes for storing temporary files. In this case the /tmp or /dev/shm directories can be used. Since RAM is significantly faster than disk storage, it is advantageous to use /dev/shm instead of /tmp for higher I/O performance. This becomes important when an application extensively uses temporary files, e.g. for interprocess communication through files. For such cases users can use the variables `JOB_TMPDIR` and `JOB_SHMTMPDIR` in their batch scripts, which are set individually for each job. For codes which use the variable `TMPDIR` it is recommended to set it like `TMPDIR=$JOB_TMPDIR`. Using the variables `JOB_TMPDIR` and `JOB_SHMTMPDIR` guarantees that all temporary files stored in these temporarily created directories will be cleaned after the job has finished. Note, that the _Raven_ HPC cluster is a diskless system, therefore the /tmp directory can be used only for files which do not exceed 2 GB in total. _Mykola Petrov_ ### Eigensolver library ELPA Further enhancements of the eigensolver library ELPA can be found in the ELPA release 2021.05.001. This version includes extensions of the infrastructure for GPU usage, such that AMD GPUs are now fully supported, and an initial (experimental) support for Intel GPUs has been added. The hybrid usage of MPI and OpenMP has also been improved: ELPA now can automatically detect which level of thread support (such as "MPI_THREAD_SERIALIZED" or "MPI_THREAD_MULTIPLE") is available in the MPI library used, and ELPA adapts the OpenMP parallelisation accordingly. The ELPA library is publicly available as open-source software and can be downloaded from the [ELPA git repository](https://elpa.mpcdf.mpg.de) hosted by the MPCDF. A new release 2021.11.001 of ELPA is currently being prepared and the new version will be available on the MPCDF systems within the next days. Among others, the new release will feature imporved support for Nvidia A100 GPUs, the option to use non-blocking MPI collectives, and a faster implementation of the autotuning. _Andreas Marek, Hermann Lederer_ ### Using Python-based hybrid-parallel codes on HPC systems NumPy and SciPy are arguably the most important base packages when it comes to scientific computing with Python. Most Python-based packages in HPC and in HPDA/AI use them, and in doing so leverage the high-performance and implicit thread parallelization these packages provide. Typically, NumPy is linked to a highly optimized math library such as Intel MKL which automatically parallelizes using threads. Care has to be taken when additional process-based layers of parallelism are employed on top. Python's 'multiprocessing' and 'mpi4py' are to be named in this context, and moreover high-level parallelization frameworks such as 'dask' or 'ray'. In each case, processes are spawned by such packages to distribute and parallelize work. It is crucial to limit the number of threads used by each of these processes in order to avoid overloading of the compute resources. In many cases, on each process the NumPy-internal threading would just use the total number of cores logically available on the system, independently of the other processes running on the same system. To give a simple example for the Raven system with 72 cores per node, a multiprocessing-based code with 72 worker processes would use several thousand threads in total if each worker internally used NumPy naively, leading to very bad overall performance and to potential harm to the stability of the compute node. Obviously, in this example each worker process would need to cap the number of threads to 1. To limit the number of threads NumPy and similar threaded packages are using, set the influential environment variables accordingly in your job scripts before launching the Python code. Such variables are for example `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, `NUMEXPR_NUM_THREADS`, and `NUMBA_NUM_THREADS`. Some packages support function calls to set the number of threads. For more details, please consult the documentation of the packages you're using. Example scripts for important use cases are given in the user guides for the HPC systems, e.g. for [Raven](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html#single-node-example-job-scripts-for-sequential-programs-plain-openmp-cases-python-julia-matlab). Similar to the Python cases for which such issues have been seen many times on the HPC systems recently, the same argument applies to Julia-based codes and in general to any hybrid code including the canonical MPI+OpenMP setup. _Klaus Reuter_ ## Python bindings for C++ using pybind11 and scikit-build With the rising popularity of the Python programming language it has become increasingly important for computational scientists to be able to make their software easily available to the Python ecosystem and community. Historically, exposing compiled extensions from C/C++ to Python has often been cumbersome, error prone and technically challenging, given the plethora of compilers, libraries and relevant target platforms developers have to deal with. The present article introduces a combination of two Python packages that promise to make this daunting task much easier and more stable. First, the [pybind11](https://github.com/pybind/pybind11) header-only library provides a convenient approach to generate Python bindings for existing or newly developed C++ code. Second, the [scikit-build](https://github.com/scikit-build/scikit-build) package can be used to bridge Python's setuptools with [CMake](https://cmake.org/), leveraging the power of CMake for the build process of the Python extension. As a result, CMake's native features such as discovering and linking of numerical libraries, dependency management, support of various build-generators, or even cross-compilation can easily be taken advantage of during the build process of the Python extension. A key advantage is that the file 'setup.py' stays minimal and simple, instead the aforementioned complexities are handled by CMake. The basic usage of pybind11 in combination with scikit-build is demonstrated below by means of a simple Python extension package. The code example can be obtained from the [MPCDF GitLab](https://gitlab.mpcdf.mpg.de/sebak/pybind11-hello-world). #### Interfacing Python/NumPy with C++ using pybind11 pybind11 is a header-only library that provides conversion from C/C++ types to Python, and vice versa. The following C++ Python extension module demonstrates its use in combination with NumPy arrays. ```c++ // Python example module 'cumsum' #include #include #include #include namespace py = pybind11; // numpy-like cumulative sum, taking a NumPy // array as input and returning a NumPy array py::array_t cumsum(py::array_t a) { // obtain information about the n-d // input array auto shape = a.request().shape; size_t count = std::accumulate(shape.begin(), shape.end(), 1, std::multiplies()); // create output array py::array_t b(count); // obtain raw pointers double * a_p = (double*) a.request().ptr; double * b_p = (double*) b.request().ptr; // compute cumulative sum into b double cs = 0.0; for (size_t i = 0; i=42", "wheel", "pybind11[global]>=2.6.0", "cmake>=3.18", "scikit-build", ] build-backend = "setuptools.build_meta" ``` Now the Python module can be compiled and installed by running the command `pip install --user .` in the root directory of the project. Similarly Wheel archives can be created for distribution. Note that with modern Python packaging tools it is not necessary to manually install pybind11 and scikit-build, instead all build dependencies will be installed into an isolated build environment by `pip`. The '[global]' feature of the pybind11 requirement is necessary to install the include and cmake files correctly into the dedicated build environment, it does not affect the Python installation or environment in use and can thus be used safely. _Sebastian Kehl, Klaus Reuter_ ## The Gitlab Package Registry Ready-to-use applications or libraries are often published via _package portals_. Nearly every programming ecosystem has such a common and widely used web portal: for example, the "Python Package Index (PyPi)" in the Python world, "Maven Central" for Java. With the _GitLab Package Registry_, you can now publish application packages in various formats directly via the MPCDF GitLab instance. The _GitLab Package Registry_ was introduced to the open-source variant of GitLab in version 13.3. The Package Registry can be used by any GitLab user to publish packages in various formats. In contrast to publicly available package management portals, GitLab's Package Registry allows the user to keep a package completely private or share it just with the other members of the current repository or group. In addition, GitLab's _Continous Integration_ capabilities are a convenient way of building and testing a package automatically from source code stored in a GitLab repository. The Package Registry should not be confused with GitLab's _Container Registry_, which can be used to store and distribute Docker images, but not application packages. GitLab's Package Registry supports currently a wide variety of package formats, including Maven (for Java/JDK based applications), npm (JavaScript) and PyPi (Python). Further package formats are under development and some are still in beta or alpha status, you can find the whole list of supported package formats in the [GitLab documentation](https://docs.gitlab.com/ee/user/packages/package_registry/index.html#supported-package-managers). If you want to use the package registry in your GitLab repository, you need to enable it under "Settings / General / Visibility, project features, permissions / Packages": ![](208/settings.png) After activation, you can access the package registry under "Packages & Registries / Package Registry" in the left menu. The concrete procedure how to build and upload a package to the package registry depends on the format of the package. The [GitLab documentation](https://gitlab.mpcdf.mpg.de/help/user/packages/package_registry/index) has examples for the most common package formats. #### Example: Publishing Python packages For building and publishing Python packages, you can find a detailed tutorial in the MPCDF documentation: [Poetry and GitLab: Devops for Python developers](https://docs.mpcdf.mpg.de/doc/data/gitlab/devop-tutorial.html). The tutorial makes use of Poetry, a packaging and dependency management tool for Python. It was already introduced in the previous edition of Bits&Bytes ([Poetry: Packaging and Dependency Management for Python](https://docs.mpcdf.mpg.de/bnb/207.html#poetry-packaging-and-dependency-management-for-python)). Testing, creation and uploading a PyPi package is done via GitLab's _Continous Integration_ pipelines. After you have successfully created and uploaded a PyPi Package to the package registry, GitLab shows the package details and how a user can download and install it into his local Python environment: ![](208/publishedPackage.png) _Thomas Zastrow_ ## Using Application Tokens instead of Passwords The MPCDF offers several services which can be used on a wide variety of (electronic) devices. For example, the DataShare client can be installed on any smartphone, tablet or laptop: the client makes uploading and downloading files from or to your device easy and convenient. But using these kind of services on mobile devices has a disadvantage: if you don't want to type it every time, you need to save your MPCDF password on the mobile device. And even if the client stores the password encrypted, if you loose your device or someone steals it, your password may be in the wild. Another, but similar use case are server based applications which need to access services like DataShare or GitLab. If such an application should run automatically and unsupervised, your personal MPCDF password needs to be made available to it. In order to avoid such security issues, services like DataShare and GitLab are offering _application specific tokens_ (sometimes also called _device tokens_). These tokens are additional credentials which can be created by the user himself -- no help from a service administrator is necessary. Every device can get its own application token: if your laptop got stolen, you just need to delete its application tokens and go on with all other devices and MPCDF services without any change. ![](208/appTokens.png) __Its strongly recommended that you create for every device its own token!__ To clarify: unless your hard disk is encrypted, a potential thief still has access to the data _locally_ stored on the device. But he can't access the application itself anymore and update, change or delete data on the server. The following sections describe the procedure of creating application tokens in DataShare and Gitlab. ### DataShare After logging in to DataShare, go to your account settings (top right of the screen). In the menu on the left, there is an entry "Security". On the bottom of this page, you can find the option "App passwords / tokens". Enter a name for your device into the text box and click "Create new app passcode": ![](208/app-tokens-ds-1.png) DataShare will now create a secure and safe token for you - make sure that you copy and paste it! DataShare will never be able to show you the token again. If you forgot to save the token, you need to delete the entry for the device and create a new app token. In any DataShare client, you can now use the app token in combination with your MPCDF user name to log in. ### GitLab In GitLab, you can find the application tokens under your personal account, "Access tokens": ![](208/app-tokens-gl-1.png) In contrast to DataShare's App Tokens, GitLab's Access Tokens have more functionality. Every token can have an expiration date and one or more scopes. Via scopes, you can set the token permissions to GitLab in a fine-granular way. _Thomas Zastrow_ ## Software Publishing Software written in the context of research receives more and more attention and is increasingly considered as genuine research output that is publishable in its own right. In this article we outline three ways of publishing software, thereby making it referenceable and citable. ### Do it yourself In order to make software citable one needs at a minimum an identifier such as a digital object identifier (DOI) pointing to a place on the web from where the software can be obtained. The latter can be any website of your choosing or just a tagged revision of your code in a publicly accessible version control system such as MPCDF's Gitlab. Being affiliated with the MPG you can request a DOI from the Max Planck Digital Library (MPDL) through their [DOI service](https://doi.mpdl.mpg.de/). You [fill in the form](https://doi.mpdl.mpg.de/request-doi) and thereby specify the URL of your code plus some basic metadata and that's it. ### Publish via a data archiving site In many cases, however, you want a copy of your code to be available from a 3rd-party data repository thereby delegating the responsibility for the long-term preservation of your code. You also often want to publish multiple versions of your code over time and you want to be able to refer to individual revisions as well as your coding project in general and expect that the metadata associated with the identifiers reflect these relationships. An example of how this can be achieved is Github in concert with Zenodo as [explained on Github](https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content). While there is no such tight integration of Zenodo with Gitlab you can achieve essentially the same by setting up your own code publication pipeline using, for example, [gitlab2zenodo](https://pypi.org/project/gitlab2zenodo/). ### Software Heritage A further option to archive and publish your code is via [Software Heritage](https://www.softwareheritage.org/). Software Heritage maintains an infrastructure and services that will crawl your public code repository (no matter whether it is based on git, subversion, or any other common revision control system) on a regular basis once you have prepared your code repository and registered it as [explained on their website](https://www.softwareheritage.org/howto-archive-and-reference-your-code/). They will store a copy of your code, preserve it and assign a unique intrinsic identifier which can then be used much in the same way as a DOI. ### Final remarks No matter which way you publish your code it is a recommended best practice to also make the repository of the (to-be) published code publicly accessible. Some of the approaches mentioned above even require that. In all cases you are expected to add metadata including authorship and usage rights (aka a license). And as a general recommendation: a license should be chosen under all circumstance and it is advisable to do this as early as possible. If in doubt ask your peers or seek advice, e.g., on [websites such as "Choose a license"](https://choosealicense.com/). _Raphael Ritz_ ## News & Events ### International HPC Summer School 2022 The international HPC summer school (IHPCSS) 2022 is planned as an in-person event from June 19th to June 24th in Athens, Greece. The series of these annual events started 2010 in Sicily, Italy. Due to the Covid-19 pandemic, IHPCSS 2020 had to be cancelled and was carried out as a virtual event in 2021, with mirrored sessions in two different time zones to allow for convenient participation from any part of the world. Now for 2022, the organizing partners XSEDE for the US, PRACE for Europe, RIKEN CCS for Japan and the SciNet HPC Consortium for Canada are inviting again for applications for participation in Greece. In case pandemic conditions will not allow to ensure health-safety, the organizers will switch to a virtual event. A final decision is expected by March 2022. Eligible candidates for applications are graduate students and postdoctoral scholars from institutions in Canada, Europe, Japan and the United States. Interested students are invited to apply by the end of January 2022. School fees, meals and housing will be covered for all accepted applicants. 30 seats out of the total number of 80 are reserved for applicants from European institutions, and 50 seats are given to students from the US, Canada and Japan. Traditionally, students from Max Planck Institutes were participating. For further information and application, please visit the [website of the summer school](https://ss22.ihpcss.org/). _Hermann Lederer_ ### Advanced HPC workshop 2021 From November 22nd to 25th, the MPCDF hosted its annual Advanced HPC Workshop for the MPG, as an online event. Around 20 participants listened to 21 lectures given by members of the applications group, the AI group and by experts from Intel and Nvidia. The topics included software carpentry, debugging, profiling and optimizing codes for CPUs and GPUs. The last day was dedicated to five code projects brought in by the participants. Together with the code owners and the experts from Intel and Nvidia, various tools were applied to the codes and optimization strategies were developed. Material of the MPCDF training programs, including semi-annual introductory courses for new users, as well as upcoming events can be found at the MPCDF webpage under ["Training & Education"](https://www.mpcdf.mpg.de/services/training) _Tilman Dannert_ ### 60 Years Max Planck Computing Centre in Garching In August 1961 the Institute for Plasmaphysics (IPP) in Garching procured one of the most powerful computers at that time -- an IBM 7090 system with a performance of 100 kFlop/s. The IPP had been founded in 1960 by Werner Heisenberg and the Max Planck Society. To the first users besides IPP belonged the Max Planck Institutes for Physics and Astrophysics, for Biochemistry, and both Munich Universities. The German Computing Centre in Darmstadt procured the same system und used the Garching computer in addition in phases of capacity shortage. In 1969 an IBM 360/91 system was installed which also belonged to the world's top systems. In 1979 the first vector computer worldwide for general basic science was installed at the "Rechenzentrum Garching" (RZG). In the following years and decades the RZG has evolved from a local to a central facility of the Max Planck Society, and in 2015 it was renamed to Max Planck Computing and Data Facility (MPCDF), underlining that it also belongs to the world's largest academic data centers. More than 50 Max Planck Institutes make use of the services for computing, storage, leading-edge HPC and AI application development, and carry out data management projects in collaboration with MPCDF. In addition, the MPCDF is engaged in many national and international projects. On October 14th, the 60-year anniversary was celebrated with a [scientific symposium](https://www.mpcdf.mpg.de/anniversary-mpcdf.html) in Garching. Vice president of the Max Planck Society, Prof. Blaum, gave a honorific speech, and renowned scientists from Plasma und Astrophysics, Materials and Life Sciences, Quantum Physics and Computer Science inspired the auditorium with brilliant presentations on the state of the art of computer-based basic science. The celebration act was finished with a dinner in the rotating restaurant of the Munich Olympic Tower, enabled by sponsorships of technology partner companies IBM, Lenovo and Nvidia. _Hermann Lederer_ Bits and Bytes Logo # No.207, August 2021 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_207.pdf) ## High-performance Computing ### HPC system _Raven_ fully operational ![HPC system Raven](207/raven_final.jpg) The deployment of the new HPC system of the Max Planck Society, _Raven_, has been completed by Lenovo and MPCDF in June 2021. The machine now comprises 1592 CPU compute nodes with the new Intel Xeon IceLake-SP processor (Platinum 8360Y with 72 cores per node). In addition, _Raven_ provides 192 GPU-accelerated compute nodes, each with 4 Nvidia A100 GPUs (4 x 40 GB HBM2 memory per node and NVLink 3). _Raven_ entered the most recent [June 2021 edition of the Top500 list](https://www.top500.org/lists/top500/list/2021/06/) of fastest supercomputers with a measured HPL (high-performance linpack) benchmark performance of 8620 TFlop/s (rank 47) for the GPU-accelerated part, and 5416 TFlop/s (rank 79) for the CPU-only part. On the HPL benchmark, the GPU-accelerated part of _Raven_ with 768 Nvidia A100 GPUs achieves a power efficiency of 22.9 GFlop/s/W which puts it on rank 12 of the [Green500 ranking](https://www.top500.org/lists/green500/list/2021/06/) of the most energy-efficient supercomputers. Together with Cobra (rank 77, 5613 TFlop/s) and its (unranked) GPU-partition, researchers of the Max Planck Society have an aggregate HPL performance of ca. 22 PFlop/s at their disposal, which is roughly the equivalent of a Top-15 supercomputer. More details about _Raven_ can be found below, on the [MPCDF webpage](https://www.mpcdf.mpg.de/services/supercomputing/raven) and in a [previous Bits & Bytes article](https://docs.mpcdf.mpg.de/bnb/206.html#hpc-system-raven-deployment-of-the-final-system). _Raven_ users are referred to the technical documentation and to specific trainings offered by the MPCDF (see below). _Hermann Lederer, Markus Rampp_ ### GPU Computing on _Raven_ ##### Overview The GPU-accelerated part of _Raven_ comprises 192 nodes, each with 4 Nvidia A100 GPUs ("Ampere" architecture) which are mutually interlinked with the high-speed interconnect NVLink 3 (100 GB/s per direction for every pair out of the 4 GPUs in a node) and are connected to the host CPUs (which are of the same type as the CPU-only part of _Raven_, Intel Xeon Platinum 8360Y) via the PCIe-4 bus at a speed of 32 GB/s per direction for every GPU. The GPU-accelerated nodes in _Raven_ are connected to the Infiniband network at 200 Gbit/s which is twice the bandwidth available in the CPU-only part. A subset of 32 nodes is connected with 400 Gbit/s (at the expense of a reduced internal PCIe bandwidth). These nodes can be requested by a special slurm option (`--constraint="gpu-bw"`). The MPCDF documentation provides [example scripts for the slurm batch system](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html#batch-jobs-using-gpus). Users may request 1, 2, or 4 GPUs and a corresponding fraction of cores of the host CPUs (18, 36, or 72 cores). If multiple nodes are requested, all nodes including all cores and GPUs are allocated exclusively for the job. Computing time on GPU-accelerated nodes is accounted using a weighting factor of 4 relative to CPU-only jobs. Users are advised to check the performance reports of their jobs in order to monitor adequate utilization of the resources. In addition to the standard software stack, the MPCDF module system provides CUDA and related GPU libraries like cuFFT, cuBLAS, ..., the Nvidia HPC SDK (nvhpcsdk, formerly known as PGI) with OpenACC and OpenMP-5 capable C, C++ and Fortran compilers, as well as further GPU-enabled HPC libraries and applications like MAGMA, ELPA, GROMACS, NAMD, OCTOPUS, and also GPU-optimized machine-learning software like TensorFlow. In addition to the standard Intel-MPI (impi) library an optimized installation of OpenMPI is provided for enabling faster GPU-to-GPU transfers (see article below). Further software of common interest can be installed on request. Dedicated training material and upcoming events can be found on the MPCDF webpage under Training & Education (see also below). ##### CUDA-aware MPI on _Raven_ GPU nodes An easy way to leverage the fast NVLink interconnect in an HPC code using MPI (Message-Passing Interface) communication is to employ CUDA-aware MPI which allows to transfer data between the GPUs of a node via NVLink rather than transferring via PCIe and through the host CPUs. CUDA-aware MPI is a feature of certain MPI libraries such as OpenMPI and MVAPICH and it allows to transfer data directly between GPUs. This is achieved by handling pointers to GPU buffers transparently in MPI calls: if the MPI library detects that a pointer to a buffer in an MPI call points to GPU memory, it will initiate a transfer directly from or to that GPU. This also makes the code easier: the user does not have to transfer the data to the CPU before calling MPI routines. Currently, peer-to-peer calls (i.e., send and receive) have the best support and will lead to direct GPU-GPU transfers. If the communicating ranks are placed on different nodes, using CUDA-aware MPI has the advantage that no additional buffer on the CPU memory is needed. Instead, the transfer can be initiated from the GPU memory over the network. Collectives are also supported, but for most of them a transfer to the CPU will happen internally. In the future, the MPI libraries will probably be able to leverage NCCL (Nvidia collective communication library), which provides also collectives that use GPU buffers and also launch kernels (e.g., for reductions). On our _Raven_ system, we support a CUDA-aware version of OpenMPI that can be loaded using ```bash module load gcc/10 cuda/11.2 openmpi_gpu/4 ``` It also includes support for the low-level drivers "gdrcopy" and "nvpeermem" via UCX. Moreover, we recommend to profile your code using "Nsight systems" (`module load nsight_systems`) which will yield useful information on kernel launches and data transfers, including transfer size and speed, and whether it is a CPU-GPU or GPU-GPU transfer. A profiling run can be started using ```bash nsys profile -t cuda,nvtx,mpi srun ./app ``` in a batch script. This will create a report file that can later be opened using the GUI (`nsight-sys`) either remotely or locally. The GUI will show a timeline of the execution including all kernel launches and data transfers. This can be used to check which transfers already employ the NVLink interconnect. _Sebastian Ohlmann, Markus Rampp_ ### HPC system _Cobra_ - Module system to be aligned with _Raven_ Currently on Cobra, a default set of Intel environment modules is loaded automatically during login and during the startup of a batch job. Moreover, default versions are currently configured for the environment modules of the Intel Compiler and Intel MPI. Please note that with an upcoming Cobra maintenance in September this will change. After that maintenance no defaults will be defined for the Intel compiler and MPI modules and no modules will be loaded automatically at login. This change aligns the configuration with _Raven_ where users already have to specify module versions, and no default modules are loaded. The exact date will be announced in due time, but users are encouraged to take notice and adapt their scripts already now. What kind of adaptations of user scripts are necessary? Please load a specific set of environment modules with explicit versions consistently when compiling and running your codes, e.g. use ```bash module purge module load intel/19.1.3 impi/2019.9 mkl/2020.4 ``` in your job scripts as well as in interactive shell sessions. Note that you must specify a specific version for the 'intel' and the 'impi' modules (no defaults), otherwise the command will fail. Please note that for your convenience, pre-compiled applications provided as modules like vasp or gromacs will continue to load the necessary intel and impi modules automatically, i.e. no changes of the batch scripts are required for these applications. The Intel versions that are currently recommended are specified in the documentation, but other versions provided by the MPCDF work as well. User codes compiled prior to the maintenance will continue to work provided that the user loads the correct environment modules in the job script. _Klaus Reuter, Sebastian Ohlmann_ ### Decommissioning of _Draco_ On July 15th, 2021 after more than 5 years of operation, the main part of the HPC extension system _Draco_ has been decommissioned. The dedicated nodes which were added to _Draco_ by individual institutes (NOMAD laboratory at the FHI, Max Planck Institute for the Structure and Dynamics of Matter, Max Planck Institute for Animal Behaviour) at a later time continue to operate exclusively for their owners. For the other users, resources on the HPC systems _Cobra_ and _Raven_ are offered as a replacement. Users of the _Draco_ GPU nodes will find comparable resources in the RTX5000 GPUs of _Cobra_ (partitions gpu_rtx5000 and gpu1_rtx5000), and moreover GPUs of type V100 on _Cobra_ and of type A100 on _Raven_. Access to data in /draco/u, /draco/ptmp and /draco/projects remains possible via the login nodes at least until the end of 2021. Note, however, that no external file systems are mounted on _Draco_ any more. Data transfer from _Draco_ to _Cobra_ or _Raven_ has to be initiated from _Cobra_ or _Raven_, respectively. _Renate Dohmen, Mykola Petrov_ ## HPC Cloud In collaboration with the main funders, the Fritz Haber Institute, the Max-Planck-Institut für Eisenforschung, and the Max Planck Institute for Human Cognitive and Brain Sciences, MPCDF has designed and built a hybrid cloud to complement the HPC system _Raven_. The HPC Cloud system, based on OpenStack, Ceph, and IBM Spectrum Scale, comprises the following hardware: ![HPC Cloud](207/cloud-img1.jpg) - 60 Intel IceLake-based (Xeon Platinum 8360Y @ 2.4 GHz) compute nodes totaling 4320 cores and 44 TB of RAM. Included are eight nodes each with 2 TB of RAM to support TB-scale “huge” virtual machines and four nodes each with 3 Nvidia A30 GPUs. - 460 TB of block and object storage including 80 TB of network-attached SSDs and 80 TB of host-attached SSDs. - 3.5 PB of file storage accessible from both _Raven_ and cloud servers. - Dual 25 Gb/s Ethernet uplinks from all compute nodes to a fully-redundant 100 Gb/s backbone. Conceptually, the HPC Cloud enables scientists to combine batch and cloud-based computing within the same pipeline, taking advantage of both massive HPC cluster resources and highly flexible software environments. This enables projects to realize novel hybrid solutions while also benefiting from simple scaling within the cloud and the ability to rapidly prototype new solutions. Practically, the system offers standard cloud computing “building blocks”, including virtual machines based on common Linux operating systems, software-defined networks, routers, firewalls, and load balancers, as well as integrated block and S3-compatible object storage services. All resources can be provisioned and managed via a web browser or industry-standard RESTful APIs. Each project will receive a quota for each resource type, within which institute-based admins have the freedom to allocate their resources as necessary to realize their projects. The MPCDF Cloud Team is available to provide consulting and advice during both the project planning and realization phases. Moreover, to ensure the seamless flow of data between the HPC Cloud and _Raven_, an IBM Spectrum Scale filesystem has been deployed. Each project may request space within the filesystem which can then be mounted on both systems. This provides one data repository for the users and enables simple data flows between, e.g., high-performance simulations and post processing or data analytics and machine learning on cloud servers. The current activities are focused primarily on hardware commissioning and realizing solutions for the initial partners and main funders. In parallel, best practices are being developed for new projects, for which the MPCDF-funded parts of the HPC Cloud will be open in late 2021. _Brian Standley, John Alan Kennedy_ ## Poetry: Packaging and Dependency Management for Python #### Introduction Poetry is a tool for _"Python packaging and dependency management made easy"_ (). It supports Python developers during the process of code writing in several ways: * __Dependency Management__: Poetry is tracking and managing all package dependencies of a Python project. * __Virtual Environment__: By creating a virtual environment, Poetry takes care of an appropriate development and runtime system. * __Publishing__: Poetry is tightly integrated into the _Python Package Index (PyPI)_. Poetry projects can be easily published to PyPI. Poetry can be installed into a user's home directory, so there is no need for a system-wide installation. More information on installing Poetry can be found [here](https://python-poetry.org/docs/#installation). #### A Poetry project and initial configuration After successful installation, a new Poetry project can be created: ``` poetry new my-poetry-project ``` This command creates a new folder, which contains already the basic structure of a Poetry project: * __README.rst__: the README file for the project, will be displayed as an overview page of GitLab or GitHub projects * __pyproject.toml__: configuration file * __my_poetry_project__ (folder): the place for your code, already as Python Package declared via an empty file \__init__.py * __tests__ (folder): Python unit tests are going here The configuration file _pyproject.toml_ after the initialization of a new project consists of several sections: ``` [tool.poetry] name = "my-poetry-project" version = "0.1.0" description = "" authors = ["Tom Zastrow"] [tool.poetry.dependencies] python = "^3.8" [tool.poetry.dev-dependencies] pytest = "^5.2" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" ``` #### Dependency Management The section _tool.poetry_ takes some generic metadata. The sections _tool.poetry.dependencies_ and _tool.poetry.dev-dependencies_ are declaring the dependencies of the project during development and deployment. After adding external packages, they need to be installed (Poetry is fetching the packages from the _Python Package Index_): ``` poetry install ``` In the background, Poetry has now created a virtual environment for the current project and installed the dependent packages. Besides manually editing the dependencies, it is also possible to let Poetry do the work. For example, the following command installs the _Pandas_ library and all its dependencies into the current project: ``` poetry add pandas ``` #### Running your code An individual Python script can be executed in the current virtual environment: ``` poetry run python my-script.py ``` It is also possible to permanently activate the virtual environment as Poetry shell: ``` poetry shell ``` #### Package creation and publishing After you are done with programming, Poetry can create a package out of the current project: ``` poetry build ``` You will find the result in the folder _dist_. As a last step, Poetry helps you publishing your project on the _Python Package Index (PyPI)_: ``` poetry publish ``` This command will ask you for your PyPI credentials and upload your package. After a short time, the package will appear in the PyPI catalogue. #### Conclusion Poetry is a tool which simplifies some common steps during the development of a Python application. Besides the commands demonstrated in this article, Poetry's full list of commands can be displayed: ``` poetry --help ``` _Thomas Zastrow_ ## More functionality for the SelfService - Call to action for 2FA users ### Migration of MyMPCDF functionality Following the switch to the new website in March the MPCDF SelfService platform has received a range of additional features that had previously been located in the "MyMPCDF" section of the MPCDF website. The following services were migrated from MyMPCDF to the SelfService: * Account creation * Password change * Mailing list administration * Administration of supervised users * TSM backup form (release later this month) * Accounting view (release planned for next month) There is also a new link to the vacation notice e-mail setting located under "My Account". The corresponding forms now have a more modern feel and some extra functionality has been added. Please, send suggestions on how to further improve the SelfService to [support@mpcdf.mpg.de](mailto:support@mpcdf.mpg.de). ### Updated password policy The migration of the password change feature to the SelfService entailed a stricter validation of new passwords. Passwords now have to pass a check against the cracklib library which rejects passwords that are based on dictionary words or are too simplistic. Long passphrases are still accepted. In a later iteration passwords will also get checked against the haveibeenpwned database. This database contains passwords that have been leaked in attacks against other websites and must be considered insecure. Our users interact with a plethora of different client software to connect to our services and not all software accepts all characters in passwords. While we try to allow as many characters as possible we were forced to limit the set of allowed characters to increase compatibility. ### Two-Factor Authentication (2FA) - Call to action **It is important that our users enroll at least two different tokens** to prevent getting locked out if their token gets lost. A common scenario is the switch to a new smartphone since the existing app token does not automatically transfer to the new phone. When the user realizes this the old phone is often already reset to factory settings. This is why the SelfService now enforces enrolling at least one secondary token after the enrollment of a primary token (app or hardware). The available secondary token types are: * **SMS token** (protects against loss of app or hardware token) * **E-mail token** (protects against loss of phone or hardware token) * **TAN list** (protects against loss of phone or hardware token; needs to be printed and stored at a secure location) **If you don't have a secondary token as a backup mechanism yet, please enroll one to avoid losing access to our services if you lose your primary token.** You can learn how to enroll a backup token in our [FAQs](https://docs.mpcdf.mpg.de/faq/2fa.html#how-do-i-enroll-and-use-a-secondary-backup-token). _Amazigh Zerzour, Andreas Schott_ ## News & Events ### Brochure "High-Performance Computing and Data Science in the MPG" The MPCDF has issued a brochure with examples of science that is currently being supported by the MPCDF services. The by-no-means comprehensive selection includes 28 articles from astrophysics, brain research, materials and bio science, high-energy physics, plasma physics and fusion research, turbulence research, demographics, contributed by various Max Planck Institutes and is available [on our webpage](https://www.mpcdf.mpg.de/MPCDF_Brochure_2021). _Erwin Laure, Markus Rampp_ ### GPU bootcamp For the first time, Nvidia and MPCDF organize a so-called GPU Bootcamp from October 19th to 20th. In two half-days the basics of GPU programming (architecture, libraries, OpenACC) will be introduced and extensive hands-on sessions with OpenACC example codes shall help the participants become familiar with GPU programming. Registration and further details can be found at the [event website](https://gpuhackathons.org/event/mpcdf-n-ways-gpu-programming-bootcamp). ### Introductory course for new users of MPCDF Another edition of our biannual introductory course for new users of the MPCDF will be given as online course on October 13th, 14:00 to 16:30 (CEST). Registration is open until October 1st, via the [MPCDF website](https://www.mpcdf.mpg.de/services/training). ### Advanced HPC workshop MPCDF will again organize an advanced HPC workshop for users of the MPG from Monday, November 22nd until Wednesday, November 24th, 2021 with an optional day with hands-on on Thursday, November 25th. The workshop will be given online. The main topics of the lectures are * Software engineering for HPC codes (git, gitlab, CI, testing) * Debugging and profiling of CPU and GPU codes * Porting codes to GPU-accelerated systems As a prerequisite, we require participants to have an account for the HPC machines of MPCDF and are already familiar with accessing, building and running their codes. If you are interested in bringing in your own code to work with the experts and to apply the techniques and tools taught in the lectures, please apply by adding a short description of your code and specific goals in the registration form. The entire Thursday, November 25th is dedicated to working on the selected code projects. The workshop will be given by members of the application group of the MPCDF together with experts from Intel and Nvidia. The registration will open soon and can be accessed via the [MPCDF training website](https://www.mpcdf.mpg.de/services/training). The deadline for registration for the lectures is November 12th. Please note the earlier deadline October 15th for the additional hands-on day. The applicants for the hands-on day are asked to test the building of the code on _Raven_ and to prepare a representative test case for the problem that they want to inspect (ideally the test can be run on a single _Raven_ node, either GPU or CPU) in advance of the workshop. Assistance by MPCDF is provided on request. ### Python for HPC From October 5th to October 7th, 2021, MPCDF offers another iteration of the course on _Python for HPC_ to participants of the MPG. The course will be given online via Zoom with lectures in the morning and exercises in the later afternoon. The course teaches approaches to use Python efficiently and appropriately in an HPC environment, covering performance-related topics such as NumPy, Cython, Numba, compiled C and Fortran extensions, profiling of Python and compiled code, parallelism using multiprocessing and mpi4py, and efficient I/O with HDF5. In addition, topics related to software engineering will be addressed, such as packaging, publishing, testing, and the semi-automated generation of documentation. The lectures will be given based on Jupyter notebooks and will include many reusable code examples. For each topic, hands-on exercises will be provided and discussed in separate sessions. On the last day, there will be time for a general question & answer session. The registration is open and can be accessed via the [MPCDF training website](https://www.mpcdf.mpg.de/services/training). _Tilman Dannert, Sebastian Ohlmann, Klaus Reuter_ Bits and Bytes Logo # No.206, April 2021 ```{contents} Contents :local: :depth: 2 ``` [PDF version](./pdf/bits_and_bytes_issue_206.pdf) ## High-performance Computing ### HPC System _Raven_: deployment of the final system ![Deploying Raven](206/raven.jpg) The _Raven_-interim HPC system is currently being dismantled to make way for the final system. The new machine will eventually comprise more than 1400 compute nodes with the brand new Intel Xeon IceLake-SP processor (72 cores per node arranged in 2 "packages"/NUMA domains, and 256 GB RAM per node). A subset of 64 nodes is equipped with 512 GB RAM and 4 nodes with 2 TB RAM. In addition, Raven will provide 192 GPU-accelerated compute nodes, each with 4 Nvidia A100 GPUs (4 x 40 GB HBM2 memory per node and Nvlink 3) connected to the IceLake host CPUs with PCIe Gen 4. All nodes are interconnected with a Mellanox HDR InfiniBand network (100 Gbit/s) using a pruned fat-tree topology with three non-blocking islands (2 CPU-only islands, 1 GPU island). The GPU nodes are interconnected with at least 200 Gbit/s. The first half of the final system will become operational at the beginning of May, the second half by July, 2021. The new IceLake CPU on _Raven_ ([Intel Xeon Platinum 8360Y](https://ark.intel.com/content/www/us/en/ark/products/212459/intel-xeon-platinum-8360y-processor-54m-cache-2-40-ghz.html)) is based on the familiar SkyLake core architecture and the corresponding software stack, which MPCDF users are already familiar with on the interim system as well as on _Cobra_ and several clusters. Notably, the IceLake platform provides an increased memory bandwidth (8 memory channels per package, compared to 6 channels on Skylake and its sibling CascadeLake). First benchmarks have shown a STREAM Triad performance of about 320 GB/s for a new _Raven_ node (compared to ca. 190 GB/s measured on _Cobra_). The IceLake CPU is based on 10 nm technology and shows a significant energy efficiency increase over the interim CPU. More information about _Raven_ can be found on the [MPCDF webpage](https://www.mpcdf.mpg.de/) and in the [technical documentation](https://docs.mpcdf.mpg.de/doc/computing). Details about the migration schedule and necessary user actions will be announced to all HPC users of MPCDF facilities in due time. Basically, users of the interim system will only be required to recompile and relink their codes and to adapt their Slurm submission scripts to match the new node-level resources. As the new machine will use the same file systems (/raven/u, /raven/ptmp, /raven/r) as the interim system, migration of user data is not needed. _Hermann Lederer, Markus Rampp_ ### Charliecloud and Singularity containers supported on _Cobra_ and _Raven_ The Singularity and Charliecloud container engines have been recently deployed on the HPC clusters of the MPCDF, offering additional opportunities to run scientific applications at our computing centre. Through containers, users have full control of the operating system and on the software stack included in their environment, so that applications, libraries and other dependencies can be packaged and transferred together. Containers also supply an operating system virtualization to run software. This level of isolation, provided via cgroups and namespaces of the Linux kernel, offers a logical mechanism to abstract applications from the environment in which they run, promoting software portability between different hosts. Introducing just a small overhead with respect to bare metal runs, applications in containers have increased reproducibility, running identically regardless of where they are deployed. This makes the use of containers particularly compelling when porting software with complex dependencies or executing applications that require system libraries different from the ones available on the host system (or even a completely different operating system). Containers also provide an easy way to access and run pre-packaged applications that are available online, usually in the form of Docker containers that can be easily converted into a Singularity or Charliecloud container image. Additional information on the use of Singularity and Charliecloud at MPCDF can be found at the [technical documentation page of the MPCDF](https://docs.mpcdf.mpg.de/doc/computing/software/containers.html) and in [Bits&Bytes No. 205](https://docs.mpcdf.mpg.de/bnb/pdf/bits_and_bytes_issue_205.pdf). _Michele Compostella_ ### Control and verification of the CPU affinity of processes and threads #### Introduction The correct mapping of processes and threads to processors is of paramount importance to get the best possible performance from the HPC hardware. That mapping is often called pinning and handled via CPU affinities. Likewise, wrong pinning is very often the cause for inferior performance, especially on systems one uses for the first time. In the worst cases of incorrect pinning, some processors would stay idle whereas other processors would be overloaded with more tasks than they are actually able to run simultaneously. This article gives some information on how to check and control the pinning on MPCDF systems. The `pincheck` library and tool developed at MPCDF is introduced, before the article concludes with some technical background for those readers who are interested. #### Checking CPU affinities at runtime In practice it is unfortunately cumbersome to learn about the actual pinning of a job, as different batch systems, MPI libraries, and OpenMP runtimes offer different ways to turn on verbose output. For example, setting the environment variable `SLURM_CPU_BIND=verbose` will instruct Slurm's `srun` to print the process pinning it performs. Similarly, setting `I_MPI_DEBUG=4` will enable verbose output from the Intel MPI library that includes some pinning information. Third, for example, the variable `KMP_AFFINITY=verbose,compact` will enable pinning output for OpenMP codes compiled with the Intel compilers, but please be aware that `verbose` cannot be specified alone but always needs a type specifier (here `compact`), otherwise no thread pinning would be applied. As each of these outputs depends on individual software they each need to be read and interpreted differently. To reduce that complexity, MPCDF has developed a simple library and tool that yields the pinning information of codes at runtime in a unified and human-readable fashion. #### The `pincheck` library and tool To give developers and HPC users the possibility to easily check the CPU affinities of the processes and threads of their actual HPC jobs at runtime, MPCDF has developed a lightweight C++ library and tool named [`pincheck`](https://gitlab.mpcdf.mpg.de/khr/pincheck). It collects and returns the processor affinities from all MPI ranks in `MPI_COMM_WORLD` and from all the related OpenMP threads. The affinities are obtained in a portable way via system calls from the kernel, and no dependency on specific compilers or runtimes exists. `pincheck` is publicly available under a permissive MIT license. For C++ codes, there is a header file ('pincheck.hpp') available that can be easily included and used from existing codes. In this case, the C++ header already includes the implementation, and no linking to a library is necessary. For C/C++ and Fortran codes, we will provide a library in combination with a C header file and a Fortran module with the next release in the near future. Alternatively, `pincheck` can be compiled and used as a stand-alone program to check the CPU affinities one gets based on certain batch scripts, environment variables, MPI and OpenMP runtimes, etc. Detailed information on how to use `pincheck` from an existing code, and on how to compile and run it as a stand-alone program is available in the git repository. #### Processor and thread affinities on Slurm-based systems at MPCDF On the HPC systems and clusters at MPCDF, processes are typically started via the `srun` launcher of Slurm. Based on the resources requested for a batch job, Slurm takes care of the CPU affinities of *processes* (which are typically the MPI tasks) by applying useful defaults (i.e., the `block` distribution method). For example, for a pure MPI job without threading, `srun` will pin the tasks to individual cores such that consecutive tasks share a socket. For hybrid (MPI/OpenMP) jobs that use one MPI task per socket and (per task) a number of threads equal to the number of cores per socket, `srun` will pin each MPI task to an individual socket. The *threads* spawned by these processes inherit the affinity mask, and the user has the option to further restrict the pinning of these individual threads. For OpenMP codes, this can be done by setting the environment variable `OMP_PLACES`, for example to the string `cores` which will pin each OpenMP thread to an individual core. Other threading models (e.g. pthreads) typically offer certain functions to achieve similar functionality. The MPCDF documentation provides example submit scripts that already include proper settings for the pinning of MPI processes and OpenMP threads, see for example the [section on Slurm scripts for the Raven system](https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html#slurm-example-batch-scripts). #### Technical background The compute nodes of today's HPC systems typically contain two or more multi-core chips (sockets) where each chip consists of multiple individual processors (cores) -- a design that implies a complex memory hierarchy: each core has its private caches (typically L1 and L2), but shares a last-level cache (typically L3) with a set of other cores that are linked via a fast on-die bus. That bus links to a memory controller to which DIMM modules are connected. Each socket contains one or more such sets of cores that are called NUMA (non-uniform memory access) domains for the following reason: a core may logically access any memory attached to the compute node, however, at different bandwidths and latencies depending on which NUMA domain a particular part of the memory is physically attached to. Different NUMA domains are connected via bus systems that are slower than the intra-domain buses. On a NUMA system it is therefore desirable that a core accesses physical memory local to its NUMA domain. Memory allocation and use is managed by the Linux operating system in chunks (pages) that are typically 4 kB in size. A first-touch policy applies, and, if possible, memory pages are placed closest to the core on which they were first used. HPC developers must therefore write their threaded programs in a NUMA-aware fashion to optimize for caches and minimize inter-domain memory accesses, and make sure that a process or thread stays within the domain or on the particular core. For codes that implement non-ideal memory access patterns (e.g., thread 0 touches all memory first, and then other threads access that memory across NUMA domains), the automatic NUMA balancing of the Linux OS may improve the performance during the runtime. By default, the scheduler of the Linux operating system may move processes and threads ("tasks") between the available processors. In case such moves occur within a NUMA domain, a task may suffer a temporary performance penalty when it is moved to a core which initially does not have relevant data cached in L1 or L2. In case a task is moved from one NUMA domain to another, there is in addition a more severe performance penalty caused by non-local memory accesses, i.e., when the moved task accesses memory pages physically located on another NUMA domain from where these pages had been touched first by the same task. In most HPC scenarios it is advantageous to restrict that moving activity in order to improve the overall temporal and spatial locality of the caches and memory accesses. To enable programmers and users to control the placement of tasks relative to NUMA domains and cores, the operating system supports setting so-called affinity masks which are taken into account by the scheduler. Using such masks, tasks can be "pinned" to sets of cores (e.g. NUMA domains) or even to individual cores or hardware threads, such that they stay there and are not moved. On a low level these masks are actually bit masks, but fortunately users can mostly work on a higher level by using e.g. the variable `OMP_PLACES=cores` to instruct the OpenMP runtime to pin individual threads to individual cores. #### References * [The `pincheck` library on MPCDF GitLab](https://gitlab.mpcdf.mpg.de/khr/pincheck) * [Slurm `srun` manpage](https://slurm.schedmd.com/srun.html) * [Intel MPI Library Developer Guide for Linux OS](https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top.html) * [Intel C++ Compiler Classic Developer Guide and Reference, OpenMP Library Support](https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support.html) * [OpenMP Reference Guides](https://www.openmp.org/resources/refguides/) * [U. Drepper, What Every Programmer Should Know About Memory, 2007](https://www.akkadia.org/drepper/cpumemory.pdf) * [C. Lameter, NUMA (Non-Uniform Memory Access): An Overview, 2013](https://doi.org/10.1145/2508834.2513149) * [Optimizing Applications for NUMA, Intel Corporation, 2011](https://software.intel.com/content/www/us/en/develop/articles/optimizing-applications-for-numa.html) * [Linux' automatic NUMA balancing](https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-tuning-numactl.html) _Klaus Reuter_ ## High-performance data analytics and AI software stack at MPCDF In the last years we have been observing an ever-growing number of researchers who want to use institute clusters and the HPC systems at MPCDF for data analytics and especially for machine-learning and deep-learning projects. This wish stems from the fact that the extremely powerful resources of HPC servers, especially if equipped with high-end GPU devices, can substantially boost the performance of data-analytics and AI workloads. Furthermore, the possibility to use multi-node setups to parallelize the workflows can reduce the time to solution by orders of magnitude. However, for users it is a non-trivial task to obtain a software stack that really does exploit the hardware features of HPC systems (SIMD vectorisation, Tensor cores of the GPUs, high-bandwidth fabrics, to mention a few) and does run with a reasonable fraction of the theoretical performance of the systems. Especially the builds of frameworks which the users can obtain via the usual distribution ways of the ML/DL community, such as Python-based installation methods like "pip", usually do not run efficiently on HPC hardware. In order to address the needs of its users for such workflows, MPCDF provides an HPC-optimized software stack for data-analytics and AI applications. Among other things, this software stack comprises * basic ML and AI libraries such as Nvidia's and Intel's DNN implementations * Nvidia NCCL and cuDNN * Intel MKL-DNN * opencv * popular frameworks * Tensorflow * Pytorch * Mxnet * scikit-learn * parallelization frameworks * Horovod (for Tensorflow, Pytorch and MxNet) * Apache Spark * tools for image and NLP processing See the MPCDF documentation for a [detailed list](../doc/computing/software/data_analytics-machine_learning#list-of-supported-software) and for some [examples](../doc/computing/software/data_analytics-machine_learning) of how to use the software together with Slurm. Whenever possible, a CPU and a GPU variant of the software is provided, which gives the user the freedom of choice and allows a seamless migration between different nodes and even clusters. As usual, the software is provided on MPCDF systems via the module environment. Please note that MPCDF uses a hierachical software stack (see Bits & Bytes No. [198](https://docs.mpcdf.mpg.de/bnb/pdf/bits_and_bytes_issue_198.pdf)) and not all software is always visible with the "module avail" command. We recommend to use the "find-modules" command, which will help users to find whether a software is available and which modules have to be loaded before the respective module will be visible. Example: ``` user@cobra01:~> find-module tensorflow/gpu tensorflow/gpu/1.14.0 (after loading anaconda/3/2019.03) tensorflow/gpu/2.1.0 (after loading anaconda/3/2019.03) tensorflow/gpu/2.1.0 (after loading anaconda/3/2020.02) tensorflow/gpu/2.2.0 (after loading anaconda/3/2019.03) tensorflow/gpu/2.2.0 (after loading anaconda/3/2020.02) tensorflow/gpu/2.3.0 (after loading anaconda/3/2020.02) ``` After the desired modules have been loaded, the software can be used in the usual way and for example can be used with Jupyter Notebooks. Further readings: * Bits & Bytes [No.203](https://docs.mpcdf.mpg.de/bnb/pdf/bits_and_bytes_issue_203.pdf) for Jupyter Notebooks as a Service * Bits & Bytes [No.200](https://docs.mpcdf.mpg.de/bnb/pdf/bits_and_bytes_issue_200.pdf) for Data Analytics at MPCDF _Andreas Marek_ ## Decommissioning of AFS After many years of acting as the central file system in MPCDF, the time has come to say goodbye to the Andrew File System (AFS). This does not mean that AFS will disappear immediately, but as a first step it is planned that home directories will no longer be set up in AFS and not all login nodes of the Linux clusters will provide access to AFS, as it is already the case on _gatezero_. The lack of support for Windows forces the use of alternatives. For most users the Sync&Share functionality provided by our datashare is a good solution. For experiment data and software distribution other ways are already established or still have to be determined. Thus, we kindly ask all our users to no longer consider AFS as the one and only filesystem for data exchange, but to implement alternatives and not to store new data in AFS home directories. _Andreas Schott_ ## Relaunch of MPCDF website and new technical documentation platform In March 2021, MPCDF relaunched its main website, adopting the corporate design of the Max Planck Society. The technical documentation for users of MPCDF services, including a comprehensive and continuously extended FAQ, as well as the MPCDF computer bulletin Bits&Bytes has been refurbished and is now available at [https://docs.mpcdf.mpg.de/](https://docs.mpcdf.mpg.de/) _Markus Rampp on behalf of the MPCDF Webteam_ ## Events ### New online introductory course for new users of MPCDF The MPCDF has started offering a new online introductory course targeting new users. The first issue was held on April 13th with over 100 registered users from more than 30 Max Planck Institutes. In the future, it will be repeated on a semi-annual schedule. The 2.5 hour online course is given by application experts of MPCDF and includes an interactive chat option and concluding Q&A sessions. It provides a basic introduction to the essential compute and data services available at MPCDF, and is intended specifically for lowering the bar for their first-time usage. This course is the basis for more advanced courses such as the annual "Advanced HPC workshop" organised by MPCDF (next issue: autumn 2021, see below). Major topics of the online introductory course include an overview and practical hints for connecting to the HPC compute and storage facilities and using them via the Slurm batch system. The course material can be found at the MPCDF webpage. ### Advanced HPC workshop: save the date Our annual Advanced High-performance Computing Workshop is scheduled for November 22nd to 24th, 2021, with an additional day of hands-on sessions for accepted projects on the 25th. The main topics will be software engineering, debugging and profiling for CPU and GPU. The talks will be given by members of the application group and by experts from Intel and Nvidia. Further details and registration options will be announced in the next issue of Bits & Bytes. _Klaus Reuter, Sebastian Ohlmann, Tilman Dannert_ Bits and Bytes Logo # Previous Editions * [No.205](pdf/bits_and_bytes_issue_205.pdf) * High-performance Computing * FAQ 4 2FA * Charliecloud: containers for HPC * repo2docker: Running Jupyter Notebooks via Docker * News & Events * [No.204](pdf/bits_and_bytes_issue_204.pdf) * Two-factor authentication at the MPCDF * High-performance Computing * Rclone - The Swiss army knife of cloud storage * ELPA eigensolvers further enhanced * News & Events * [No.203](pdf/bits_and_bytes_issue_203.pdf) * New Director of the MPCDF * High-performance Computing * Jupyter Notebook as a service * FTP decommissioning * Tips & Tricks * Ecosystem Data Management * News & Events * [No.202](pdf/bits_and_bytes_issue_202.pdf) * High-performance Computing * Python on HPC systems * Cluster hosting * The MPCDF SelfService: Guest-User and Self-Management for GitLab and DataShare * Archival * News & Events * [No.201](pdf/bits_and_bytes_issue_201.pdf) * High-performance Computing * HPC Performance Monitoring System * ELPA eigensolvers further pushed to unexcelled performance and scaling behavior * New Web-based Remote Visualization Service * Collaborative Document Editing in DataShare * News & Events * [No.200](pdf/bits_and_bytes_issue_200.pdf) * Bits & Bytes turns 50! * High-performance Computing * Data-Analytics on MPCDF HPC systems * Keeper - Archive the way you work * New Projects and Collaborations * News * [No.199](pdf/bits_and_bytes_issue_199.pdf) * High-performance Computing * The evolution of the Anonymous FTP server and data sharing at MPCDF * Lightweight performance measurement tools * GitLab: Online Editing of Source Code * [No.198](pdf/bits_and_bytes_issue_198.pdf) * HPC * Spark on Draco * Major Upgrade of DataShare Service * SSH security considerations * [No.197](pdf/bits_and_bytes_issue_197.pdf) * DataHub * Yacora on the Web * High-performance Computing * Inastemp: a Vectorization Library to Accelerate C++ Codes * [No.196](pdf/bits_and_bytes_issue_196.pdf) * Singularity: containers for HPC * Transferring data to and from MPCDF * DataShare via WebDAV * System and software upgrade on the HPC extension system Draco * MPCDF's GitLab now supports Continuous Integration & Delivery * [No.195](pdf/bits_and_bytes_issue_195.pdf) * Draco cluster extended by large-memory nodes * A simple command-line client for the MPCDF DataShare (ownCloud) service * BagIt - command-line tool supporting the BagIt compound data format * Archiving your data with HPSS * Visualizing data from molecular simulations with VisIt * [No.194](pdf/bits_and_bytes_issue_194.pdf) * HPC * Galaxy instance for the MPG * Python infrastructure * Visualization * Recent optimizations of ELPA eigensolvers * csvkit - A command line suite of tools for converting and working with CSV data * Events * [No.193](pdf/bits_and_bytes_issue_193.pdf) * DRACO as an extension of HYDRA * VM Backups * TeD-T: The Term Definition Tool * iRODS * Uberftp - A powerful command line tool for gridftp based data management * [No.192](pdf/bits_and_bytes_issue_192.pdf) * Remote visualization on Hydra * VNC remote desktops on MPCDF Linux systems * BagIt: smart packaging of files * Gitlab update * DataShare update * python infrastructure at the MPCDF * Change in Forcheck * MPCDF archive system -- expansion of GHI for projects * ELPA: improved eigensolvers for computational materials science * Vagrant -- Virtual Development Environments made easy * [No.191](pdf/bits_and_bytes_issue_191.pdf) * High-Reformance Computing * The MPCDF GitLab Service * Upgrade of the Visualization Infrastructure * New Centre of Excellence: NOMAD Laboratory * Mailing problems with new DSL * Debugging Memory Corruption with the Address Sanitizer Library * [No.190](pdf/bits_and_bytes_issue_190.pdf) * MPCDF, the new name of the RZG * High-throughput data transfers using Globus Online * Interactive graphics with x3d(om) * Projects * 100-Gbps wide area network connection * Tips and Tricks * [No.189](pdf/bits_and_bytes_issue_189.pdf) * Sync & Share * HPC system Hydra * [No.188](pdf/bits_and_bytes_issue_188.pdf) * Taco - A Metadata System for Hierarchically Structured Data Collections * HPC System Hydra * Virtual Hosting Environment * Extreme Scaling and Visualization of HPC Applications * [No.187](pdf/bits_and_bytes_issue_187.pdf) * Status of the Max Planck Supercomputer Hydra * Hydra Environment for GPU and MIC Applications * GIT Hosting Service * Long-Term Archiving * [No.186](pdf/bits_and_bytes_issue_186.pdf) * New Max Planck Supercomputer Hydra: Main Installation * Intel Software License Agreement for MPG * Sofware News * Storage/Archive Systems * Redundant connection to the German Research Network * [No.185](pdf/bits_and_bytes_issue_185.pdf) * Next Generation Max Planck Supercomputer at RZG * HPSS - High Performance Storage System at RZG * VisIt, a parallel visualization and data analysis tool * Software News * [No.184](pdf/bits_and_bytes_issue_184.pdf) * Next Generation Supercomputer * Software updates on HPC systems * New Scalable Eigenvalue Solver * HPSS - High Performance Storage System * Performance Analysis with IBM HPC toolkit * [No.183](pdf/bits_and_bytes_issue_183.pdf) * New Linux cluster for remote visualization * Code validation tools * Power6 software upgrade * [No.182](pdf/bits_and_bytes_issue_182.pdf) * New Bits & Bytes * Introducing Environment Modules * GPGPU Computing * Applications: Performance Analysis with Scalasca * Network Configuration Templates for PCs * Update on Mass Storage * System Environments for HPC * [No.181](pdf/bits_and_bytes_issue_181.pdf) * vip - The new IBM Power6 Supercomputer at RZG * genius - The IBM Blue Gene/P at RZG * DEISA2 * Archiving at RZG * Video Conferencing * [No.180](pdf/bits_and_bytes_issue_180.pdf) * Genius - The IBM Blue Gene/P system at RZG * The Munich-ATLAS-Tier2 project * Long Time Data Preservation * [No.179](pdf/bits_and_bytes_issue_179.pdf) * The Next Generation Supercomputer of the Max Planck Society at RZG * Refurbishment of the Old RZG Computer Machine Hall * Emailservices at RZG * Virtual Private Network - Accessing Local Resources from the Internet * New Tape Library at RZG * Linux-Clusters: Consolidation of the Used Distribution * Data Visualization Support * [No.178](pdf/bits_and_bytes_issue_178.pdf) * New architectures doubling compute power at RZG * DEISA - Supercomputing at European scale under way * No virus-check on AFS-volumes * [No.177](pdf/bits_and_bytes_issue_177.pdf) * Xeon-based Linux cluster vs. IBM Regatta for small jobs * [No.176](pdf/bits_and_bytes_issue_176.pdf) * IBM p690 Supercomputer * Migrating Filesystem (HSM) * Linux Cluster Expansion * Numerical Libraries * [No.175](pdf/bits_and_bytes_issue_175.pdf) * IBM p690 Supercomputer * Notes on virus attacks * Active Directory * Mozilla as substitute for Netscape * [No.174](pdf/bits_and_bytes_issue_174.pdf) * Trends in Videoconferencing in IPP & MPG * IBM p690 Supercomputer * GPFS on the Linux Clusters * Using the NAG library from C and C++ programs * New Color Printer at RZG dispatcher room * [No.173](pdf/bits_and_bytes_issue_173.pdf) * Status IBM Regatta * New Linux Cluster * Migration of Kerberos Server # Getting Access As the MPCDF Cloud service is still in the pilot phase, there is no formalized procedure for requesting and being granted access. In practical terms, access entails one or more _users_, each linked to a personal MPCDF user account, who administer resources belonging to a _project_. The scope of a single project may range from a small research group up to an entire institution. Details such as resource quotas, network access, etc. will be discussed on a per-project basis. If you would like to use the service for a project carried out at, or in partnership with, a Max Planck Institute, please get in touch with us via the [helpdesk](../../../faq/help.html#how-can-i-get-help-and-support). # Orchestration Orchestration refers to the programmatic creation, maintenance, and deletion of complex systems of resources in IaaS clouds. Here we provide examples on how to accomplish this using OpenStack's internal orchestration engine, HEAT, and the popular third party tool, Terraform. Terraform is a very popular open source toolkit that plugs into many cloud environments in addition to OpenStack. It however lacks some advanced features of OpenStack HEAT. That means, HEAT allows the HPCCloud to continuously maintain your infrastructure on the side of the cloud. This enables features such as automatic scaling in response to demand. ## OpenStack HEAT In addition to the web dashboard and CLI client, OpenStack includes an orchestration service which is particularly suited for large and/or complex collections of resources (stacks). The service takes as input a template combined with a set of parameters, which can be passed individually or via an environment file, and then automatically creates the requested resources. 1. Create a template describing the resources and parameters. Examples: * quickstart.yml, based on the above scenario * mpcdf.yml, utilizing some of the more-advanced resources detailed below 2. Launch a new stack on Project / Orchestration / Stacks, specifying the template file and parameters. Quickstart-equivalent commands: ``` (STACK) wget https://home.mpcdf.mpg.de/~brian/quickstart{,-env}.yml (STACK) edit quickstart-env.yml (STACK) openstack stack create -t quickstart.yml -e quickstart-env.yml ``` Note that best practice is to treat stacks as a first-class objects, e.g. by making all modifications (including deletion) of stack resources through the orchestration service. ### Defining a Template Here's a simple example of a stack deployment using [OpenStack HEAT](https://wiki.openstack.org/wiki/Heat). The stack itself is defined in a template (and we use an environment file also to add flexibility). The template defines the resources you create ( in this case we are using the simple-server.yml template). ``` heat_template_version: 2013-05-23 description: > A simple HOT template to deploy a server and assign a floating IP address. parameters: key_name: type: string default: nobody description: Name of an existing key pair to use for the instance flavor: type: string description: Instance type for the instance to be created default: m1.small constraints: - allowed_values: [m1.tiny, m1.small, m1.medium, m1.large] description: Value must be one of 'm1.tiny', 'm1.small', 'm1.medium' or 'm1.large' image: type: string default: none description: ID or name of the image to use for the instance floating_ip_pool: type: string default: none description: Pool to use for Floating IP address resources: instance: type: OS::Nova::Server properties: name: instance image: { get_param: image } flavor: { get_param: flavor } key_name: { get_param: key_name } instance_floating_ip: type: OS::Nova::FloatingIP properties: pool: { get_param: floating_ip_pool } association: type: OS::Nova::FloatingIPAssociation properties: floating_ip: { get_resource: instance_floating_ip } server_id: { get_resource: instance } ``` The environment is defined in the env file (in this case simple-env.yml). When you create a stack you select the template and env (this lets you create stacks from the same templates but with different configuration parameters) ``` parameters: image: 96bbbecc-b96d-4622-8dd5-0ae314d18cca flavor: m1.small floating_ip_pool: cloud-dmz key_name: suse-stack-jk ``` ### Deploying a stack #### GUI You can start a "stack" in openstack via the GUI. Go to Orchestration->Stacks and then click on the Launch Stack button. This will ask you for the template and env: * simple-server.yml is the template * simple-env.yml is the env file The rest is very intuitive. It takes a few seconds to create the stack - after that you can click on the stack in the GUI and you can look at the topology/resources etc etc. (just try it an click around in the gui) #### Command Line > **Info**: > To follow the information below, you need to have your [RC file](clients.md) activated! If you are using the command line openstack client you need to ensure that the heat plugin/extension is installed. Then either run heat directly: ``` heat stack-create -f Templates/simple-server.yml -e Templates/simple-env.yml simplestack ``` or run the command via the openstack client ``` openstack stack create -t Templates/simple-server.yml -e Templates/simple-env.yml simplestack ``` ### The Future These templates can be extended to create several servers - including the bootstrap of the configuration. ## Terraform This simple example shows how to use Terraform to orchestrate resources on the MPCDF HPCCloud. To use this example, you need access to have a project in the HPCCloud and install Terraform from [Hashicorp](https://www.terraform.io/downloads). Below is a Terraform manifest set-up to connect with the HPCCloud. Similarly as above it creates and instances with a floating IP and allows you to SSH into it. ```terraform terraform { required_providers { openstack = { source = "terraform-provider-openstack/openstack" version = "~> 1.43" } } } variable "auth_url" { description = "OpenStack authentication URL" type = string default = "https://hpccloud.mpcdf.mpg.de:13000" } variable "application_credential_id" { type = string } variable "application_credential_secret" { sensitive = true type = string } provider "openstack" { auth_url = var.auth_url application_credential_id = var.application_credential_id application_credential_secret = var.application_credential_secret use_octavia = true } provider "http" { } data "http" "ipme" { url = "http://ifconfig.me" } resource "openstack_compute_secgroup_v2" "allow_ssh" { name = "allow_ssh" description = "Allow SSH from host running this manifest" rule { from_port = 22 to_port = 22 ip_protocol = "tcp" cidr = "${data.http.ipme.response_body}/32" } } resource "openstack_compute_keypair_v2" "example-key" { name = "example-key" public_key = "SSH_PUBLIC_KEY" } resource "openstack_compute_instance_v2" "example_server" { name = "example_server" flavor_name = "mpcdf.tiny" image_id = "14d96e58-f435-4794-95d9-e664bbef6d5c" key_pair = openstack_compute_keypair_v2.example-key.name security_groups = ["default", openstack_compute_secgroup_v2.allow_ssh.name] network { name = "cloud-local-1" } } resource "openstack_compute_floatingip_v2" "floating_ip" { pool = "cloud-public" } resource "openstack_compute_floatingip_associate_v2" "server_floating_ip" { floating_ip = openstack_compute_floatingip_v2.floating_ip.address instance_id = openstack_compute_instance_v2.example_server.id } ``` Please replace `SSH_PUBLIC_KEY`, that is the content of your public key file. It should be a long string following this pattern: `ssh-rsa AAA.....== ` (the name is optional). Now it is time to initialize terraform: ```bash terraform init ``` This will install the OpenStack provider for Terraform and prepare your environment. Before you can execute the manifest, you need to create some application credentials for Terraform. You can generate those like by calling: ```bash openstack application credential create terraform --format json |tee terraform_ac.json ``` ```{eval-rst} .. note:: You need to have the OpenStack clients installed and set up as explained in :doc:`clients` ``` That will save your application credentials in the `terraform_ac.json` file. Keep it safe, keep it secret. You can grab the credentials from this file later with: ```bash jq "{id, secret}" terraform_ac.json ``` ```{eval-rst} .. note:: You may have to install JSON parser utility jq using your package manager ``` Now you are ready to apply your manifest. ```bash terraform apply ``` ```{eval-rst} .. warning:: You need to do this in a fresh shell. The Go OpenStack client used by Terraform gets confused if you have sourced the rc file for your cloud environment. ``` Terraform will ask for the application credentials you generated above and then tell you the operations it is planning to execute. If you hit yes, you should see the new keypair and instance pop up in your HPCCloud project. Once the instance is up and running, terraform will finish. You can see more about what Terraform did with: ```bash terraform show ``` For example, you'll find an IP address of the machine you can ssh to now. Refer to the Terraform OpenStack provider [docs](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs) to learn how to expand on this simple example. .. ------------- .. authors: mykp .. plone_url: https://www.mpcdf.mpg.de/services/computing/linux/migration-from-sge-to-slurm .. ------------- .. raw:: html .. role:: bolditalic :class: bolditalic =========================== Migration from SGE to Slurm =========================== ---- Overview ^^^^^^^^ HPC clusters at MPCDF use Slurm job scheduler for batch job management and execution. This reference guide provides information on migrating from SGE to Slurm. Common job commands """"""""""""""""""" +--------------------------------------+-------------------+--------------------------------------+ | Command | SGE | Slurm | +======================================+===================+======================================+ | Cluster status | -- | sinfo | +--------------------------------------+-------------------+--------------------------------------+ | Job submission | qsub | sbatch | +--------------------------------------+-------------------+--------------------------------------+ | Start an interactive job | qlogin or qrsh | srun \-\-pty bash | +--------------------------------------+-------------------+--------------------------------------+ | Job deletion | qdel | scancel | +--------------------------------------+-------------------+--------------------------------------+ | Job status (all) | qstat or show | squeue | +--------------------------------------+-------------------+--------------------------------------+ | Job status by job | qstat -j | squeue -j | +--------------------------------------+-------------------+--------------------------------------+ | Job status by user | qstat -u | squeue -u | +--------------------------------------+-------------------+--------------------------------------+ | Job status detailed | qstat -j | scontrol show job | +--------------------------------------+-------------------+--------------------------------------+ | Show expected start time | qstat -j | squeue -j \-\-start | +--------------------------------------+-------------------+--------------------------------------+ | Hold a job | qhold | scontrol hold | +--------------------------------------+-------------------+--------------------------------------+ | Release a job | qrls | scontrol release | +--------------------------------------+-------------------+--------------------------------------+ | Queue list / information | qconf -sql | scontrol show partition | +--------------------------------------+-------------------+--------------------------------------+ | Queue details | qconf -sq | scontrol show partition | +--------------------------------------+-------------------+--------------------------------------+ | Node list | qhost | scontrol show nodes | +--------------------------------------+-------------------+--------------------------------------+ | Node details | qhost -F | scontrol show node | +--------------------------------------+-------------------+--------------------------------------+ | X forwarding | qsh | salloc or srun \-\-pty | +--------------------------------------+-------------------+--------------------------------------+ | Monitor or review job resource usage | qacct -j | sacct -j | +--------------------------------------+-------------------+--------------------------------------+ | GUI | qmon | sview | +--------------------------------------+-------------------+--------------------------------------+ Job submission options in scripts """"""""""""""""""""""""""""""""" +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Option | SGE (qsub) | Slurm (sbatch) | +=================================+===============================================+========================================================+ | Script directive | #$ | #SBATCH | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Job name | -N | \-\-job-name= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Standard output file | -o | \-\-output= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Standard error file | -e | \-\-error= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Combine stdout/stderr to stdout | -j yes | \-\-output= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Working directory | -wd | \-\-workdir= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Request notification | -m | \-\-mail-type= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Email address | -M | \-\-mail-user= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Job dependency | -hold_jid [job_ID \| job_name] | \-\-dependency=after:job_JD[:job_JD...] | | | | | | | | \-\-dependency=afterok:job_JD[:job_JD...] | | | | | | | | \-\-dependency=afternotok:job_JD[:job_JD...] | | | | | | | | \-\-dependency=afterany:job_JD[:job_JD...] | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Copy environment | -V | \-\-export=ALL (default) | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Copy environment variable | -v | \-\-export= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Node count | -- | \-\-nodes= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Request specific nodes | -l hostname= | \-\-nodelist= | | | | | | | | \-\-nodefile= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Processor count per node | -pe | \-\-ntasks-per-node= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Processor count per task | -- | \-\-cpus-per-task= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Memory limit | -l mem_free= | \-\-mem= (in mega bytes -MB) | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Minimum memory per processor | -- | \-\-mem-per-cpu= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Wall time limit | -l h_rt= | \-\-time= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Queue | -q | \-\-partition= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Request specific resource | -l resource= | \-\-gres=gpu: or \-\-gres=mic: | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Job array | -t | \-\-array= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Licences | -l licence= | \-\-licences= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ | Assign job to the project | -P | \-\-account= | +---------------------------------+-----------------------------------------------+--------------------------------------------------------+ Job environments """""""""""""""" +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Information | SGE | Slurm | Comments | +==============================+===================+=========================+=================================================+ | Version | -- | -- | Can be extracted by | | | | | | | | | | ``sbatch --version`` | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Job name | $JOBNAME | $SLURM_JOB_NAME | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Job ID | $JOBID | $SLURM_JOB_ID | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Batch or interactive | $ENVIRONMENT | -- | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Submit host | $SGE_O_HOST | $SLURM_SUBMIT_HOST | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Submit directory | $SGE_O_WORKDIR | $SLURM_SUBMIT_DIR | Slurm jobs start from the submit | | | | | | | | | | directory by default | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Node file | $PE_HOSTLIST | -- | File and path that lists the nodes | | | | | | | | | | where a job has been allocated | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Node list | cat $PE_HOSTLIST | $SLURM_JOB_NODELIST | To get a list of nodes: | | | | | | | | | | ``scontrol show hostnames $SLURM_JOB_NODELIST`` | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Hostname | $HOSTNAME | $SLURM_SUBMIT_HOST | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Job user | $USER | $SLURM_JOB_USER | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Job array index | $SGE_TASK_ID | $SLURM_ARRAY_TASK_ID | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Queue name | $QUEUE | $SLURM_JOB_PARTITION | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Number of allocated nodes | $NHOSTS | $SLURM_JOB_NUM_NODES | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Number of procecces | $NSLOTS | $SLURM_NTASKS | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Number of procecces per node | -- | $SLURM_TASKS_PER_NODE | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Requested tasks per node | -- | $SLURM_NTASKS_PER_NODE | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Requested cpus per task | -- | $SLURM_CPUS_PER_TASK | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ | Scheduling priority | -- | $SLURM_PRIO_PROCESS | | +------------------------------+-------------------+-------------------------+-------------------------------------------------+ The OpenMP can require a variable **OMP_NUM_THREADS** to be set what can be obtained from the Slurm environment variable **$SLURM_CPUS_PER_TASK** that is set when *--cpus-per-task* is specified in a sbatch script +---------------------------------------------------------------------------------------+ | Set OMP_NUM_THREADS | +=======================================================================================+ | .. code:: bash | | | | # Set the number of cores available per process if the $SLURM_CPUS_PER_TASK is set | | if [ ! -z $SLURM_CPUS_PER_TASK ] ; then | | export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK | | else | | export OMP_NUM_THREADS=1 | | fi | +---------------------------------------------------------------------------------------+ Sample job scripts """""""""""""""""" +------------------------------------+-------------------------------------------+ | SGE script | Slurm script [#R1]_ | +====================================+===========================================+ | .. code:: bash | .. code:: bash | | | | | #!/bin/bash | #!/bin/bash -l | | # | # NOTE the -l flag! | | # | # | | #$ -N sge_test | #SBATCH -J slurm_test | | #$ -j y | #SBATCH -o test.output | | #$ -o test.output | #SBATCH -e test.output | | # Current working directory | # Default in slurm | | #$ -cwd | #SBATCH -D ./ | | #$ -M YourID@some.domain | #SBATCH --mail-user YourID@some.domain | | #$ -m bea | #SBATCH --mail-type=ALL | | # Request for 8 hours run time | # Request 8 hours run time | | #$ -l h_rt=8:0:0 | #SBATCH -t 8:0:0 | | # Specify the project for job | # Specify the project for job | | #$ -P your_project_name_here | #SBATCH -A your_project_name_here | | # Set Memory for job | # Set Memory for job | | #$ -l mem=4G | #SBATCH --mem=4000 | | echo "start job" | echo "start job" | | sleep 120 | sleep 120 | | echo "bye" | echo "bye" | +------------------------------------+-------------------------------------------+ .. rubric:: Remarks .. [#R1] #SBATCH -A can be simply ignored as not used in the same way as in SGE at MPCDF. More examples can be found at home page of institute general-purpose compute cluster `Cobra `__ and on the page with `sample scripts `__ Further Reading """"""""""""""" Links to external resources: - `Slurm home page `__ - `Slurm documentation `__ - `Slurm cheat sheet `__ - `Rosetta Stone of Workload Managers `__ Slurm batch script for Science of Light ======================================= Sample batch script for embarrassingly parallel CPU jobs -------------------------------------------------------- ```bash #! /bin/bash -l # # This file is a sample batch script for "embarrassingly parallel" CPU applications via Slurm's job array mechanism. # For more information on job arrays, see: https://slurm.schedmd.com/job_array.html # # Standard output and error: #SBATCH -o ./tjob_%A_%a_out.txt #SBATCH -e ./tjob_%A_%a_err.txt # # Initial working directory: #SBATCH -D ./ # # Job Name: #SBATCH -J test_slurm # # Queue (Partition): #SBATCH --partition= # # Process management (number of parallel executions is specified using the --array option): # * possible formats: `--array=0-9`, `--array=1,3,5,7`, `--array=1-7:2` # * reduce maximum number of simultaneously running tasks using a "%" separator (e.g. `--array=0-9%4`) # * to start only one instance, use --array=0 or (better) leave the --array option away completely #SBATCH --array=0-9 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 # # Explicitly specify memory requirement (default is maximum available on node): #SBATCH --mem=1024MB # # Wall clock limit: #SBATCH --time=24:00:00 # # Configure notification via mail: #SBATCH --mail-type=none #SBATCH --mail-user=@mpl.mpg.de # Run the program srun $SLURM_ARRAY_TASK_ID $SLURM_ARRAY_TASK_COUNT ``` Sample batch script for multithreaded CPU jobs without hypertheading -------------------------------------------------------------------- ```bash #! /bin/bash -l # # This file is a sample batch script for multi-threaded CPU applications (e.g. with pthread, OpenMP, ...). # # Standard output and error: #SBATCH -o ./tjob_%j_out.txt #SBATCH -e ./tjob_%j_err.txt # # Initial working directory: #SBATCH -D ./ # # Job Name: #SBATCH -J test_slurm # # Queue (Partition): #SBATCH --partition= # # Process management: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=16 # specify number of CPU cores (maximum: 16 on highfreq, 32 on highmem) # # Explicitly specify memory (default is maximum available on node) #SBATCH --mem=64GB # # Wall clock limit: #SBATCH --time=24:00:00 # # Configure notification via mail: #SBATCH --mail-type=none #SBATCH --mail-user=@mpl.mpg.de # Load necessary modules here # module load ... # Set number of CPUs per tasks for OpenMP programs if [ ! -z $SLURM_CPUS_PER_TASK ] ; then export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK else export OMP_NUM_THREADS=1 fi # Disable hyperthreading. Disabled by default. Set it after modules load. # export SLURM_HINT=nomultithread # Run the program srun ``` Sample batch script for multithreaded CPU jobs in hypertheading mode -------------------------------------------------------------------- ```bash #! /bin/bash -l # # This file is a sample batch script for multi-threaded CPU applications (e.g. with pthread, OpenMP, ...). # # Standard output and error: #SBATCH -o ./tjob_%j_out.txt #SBATCH -e ./tjob_%j_err.txt # # Initial working directory: #SBATCH -D ./ # # Job Name: #SBATCH -J test_slurm # # Queue (Partition): #SBATCH --partition= # # Process management: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=32 # specify number of CPU cores (maximum: 32 on highfreq, 64 on highmem) # # Explicitly specify memory (default is maximum on node): #SBATCH --mem=64GB # # Wall clock limit: #SBATCH --time=24:00:00 # # Configure notification via mail: #SBATCH --mail-type=none #SBATCH --mail-user=@mpl.mpg.de # Load necessaru modules here # module load ... # Set number of CPUs per tasks for OpenMP programs if [ ! -z $SLURM_CPUS_PER_TASK ] ; then export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK else export OMP_NUM_THREADS=1 fi # Enable hyperthreading. Set it after modules load. export SLURM_HINT=multithread # Run the program srun ``` Sample batch script for embarrassingly parallel GPU jobs -------------------------------------------------------- ```bash #! /bin/bash -l # # This file is a sample batch script for "embarrassingly parallel" GPU applications via Slurm's job array mechanism. # For more information on job arrays, see: https://slurm.schedmd.com/job_array.html # # Standard output and error: #SBATCH -o ./tjob_%A_%a_out.txt #SBATCH -e ./tjob_%A_%a_err.txt # # Initial working directory: #SBATCH -D ./ # # Job Name: #SBATCH -J test_slurm # # Queue (Partition): #SBATCH --partition=dgx # # Process management (number of parallel executions is specified using the --array option): # * possible formats: `--array=0-9`, `--array=1,3,5,7`, `--array=1-7:2` # * reduce maximum number of simultaneously running tasks using a "%" separator (e.g. `--array=0-9%4`) # * to start only one instance, use --array=0 or (better) leave the --array option away completely #SBATCH --array=0-3 #SBATCH --gres=gpu:1 # specify number of GPUs #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=4 # specify number of CPU cores (as a rule of thumb, 4 per GPU) # # Memory requirement (default is 64GB): #SBATCH --mem=32GB # # Wall clock limit: #SBATCH --time=24:00:00 # # Configure notification via mail: #SBATCH --mail-type=none #SBATCH --mail-user=@mpl.mpg.de # Choose container image SINGULARITY_IMAGE_FILE="/ptmp/simg/nvidia-theano:18.03-python3-v1.simg" if [ ! -f /singularity ]; then # This branch is executed when in Slurm; essentially, it calls Singularity. # Better do not edit this branch unless you know exactly what you are doing. COPY_OF_THIS_SCRIPT=$(mktemp --suffix=.sh) trap "{ rm -f ${COPY_OF_THIS_SCRIPT}; }" EXIT chmod 700 ${COPY_OF_THIS_SCRIPT} cp $0 ${COPY_OF_THIS_SCRIPT} srun singularity run --nv ${SINGULARITY_IMAGE_FILE} ${COPY_OF_THIS_SCRIPT} else # This branch is executed when in Singularity. # From here, you can start your computation (feel free to modify this branch). $SLURM_ARRAY_TASK_ID $SLURM_ARRAY_TASK_COUNT fi ``` Sample batch script for parallel COMSOL jobs -------------------------------------------- ```bash #!/bin/bash -l # # This file is a sample batch script for parallel multi-threaded (MPI/OpenMP) comsol run. # # Standard output and error: #SBATCH -o ./comsol_%j_out.txt #SBATCH -e ./comsol_%j_err.txt # # Initial working directory: #SBATCH -D ./ # # Job Name: #SBATCH --job-name="COMSOL" # # Queue (Partition): #SBATCH --partition=highmem # specify partition # # Process management: ##SBATCH --array=0-9 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 # # Always set number of cores to be used by each ntask (ntasks-per-node * cpus-per-task <= maximum on node) #SBATCH --cpus-per-task=4 # specify number of CPU cores (maximum: 16 on highfreq, 32 on highmem) # # Explicitly specify memory (default is maximum available on node) #SBATCH --mem=128GB # # Wall clock limit: #SBATCH --time=04:00:00 # # Configure notification via mail: #SBATCH --mail-type=none #SBATCH --mail-user=@mpl.mpg.de # Load necessary modules here module load comsol # choose suitable model MODELTOCOMPUTE="comsol_smalltest.mph" INPUTFILE="input/$MODELTOCOMPUTE" DIR_OUTPUT="output/$SLURM_JOB_ID" DIR_LOGS="logs/$SLURM_JOB_ID" mkdir -p $DIR_OUTPUT mkdir -p $DIR_LOGS OUTPUTFILE="$DIR_OUTPUT/$MODELTOCOMPUTE" BATCHLOG="$DIR_LOGS/${MODELTOCOMPUTE}.log" # Run the COMSOL command, using -nn 8 and -nnhost 2 deduced from SLURM comsol batch -np $SLURM_CPUS_PER_TASK -mpibootstrap slurm -mpifabrics shm:tcp \ -inputfile ${INPUTFILE} -outputfile ${OUTPUTFILE} \ -batchlog ${BATCHLOG} -alivetime 15 -prefermph -recover ``` Sample batch script for COMSOL with Matlab jobs ----------------------------------------------- ```bash #!/bin/bash -l # # This file is a sample batch script to run Comsol models from Matlab via LiveLink. # # Standard output and error: #SBATCH -o ./comsol_%j_out.txt #SBATCH -e ./comsol_%j_err.txt # # Initial working directory: #SBATCH -D ./ # # Job Name: #SBATCH --job-name="COMSOL" # # Queue (Partition): #SBATCH --partition=highmem # specify partition # # Process management: ##SBATCH --array=0-9 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 # # Explicitly specify memory (default is maximum available on node) #SBATCH --mem=128GB # # Always set number of cores to be used by each ntask (ntasks-per-node * cpus-per-task <= maximum on node) #SBATCH --cpus-per-task=10 # specify number of CPU cores (maximum: 16 on highfreq, 32 on highmem) # # Wall clock limit: #SBATCH --time=04:00:00 # # Configure notification via mail: #SBATCH --mail-type=none #SBATCH --mail-user=@mpl.mpg.de # # Use resources on node exclusively ##SBATCH --exclusive # Load necessary modules here module load comsol matlab # Define name of the running script MSCRIPT="comsol_livelink_matlab_script.m" # Set free port for Matlab-Comsol communication. Needed if several Comsol servers run on the same node # Not necessary if node is used exclusively PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1])') # Start Comsol server in background comsol mphserver -port $PORT & # Wait until server will start sleep 10s # Run Matlab script matlab -nosplash -nodisplay -r " addpath $COMSOL_HOME/mli; mphstart($PORT); run $MSCRIPT; exit " ``` # GPFS on Linux clusters GPFS, IBM's General Parallel File System, is available on all Linux cluster machines operated by the MPCDF. It provides shared file space accessible from all nodes in a cluster, typically the users' home (/u) and scratch space (/ptmp). GPFS allows sharing of files among users from different groups. # Cobra User Guide ```{eval-rst} .. warning:: - Cobra batch job processing has ended on July 1st, 2024 - Cobra login nodes have been decommissioned on July 19th, 2024 ``` ```{contents} Contents :local: :depth: 2 ``` ## System Overview The Supercomputer Cobra was installed in spring 2018, was expanded with NVIDIA Tesla V100 GPUs in Dec 2018 and with NVIDIA Quadro RTX 5000 GPUs in July 2019. All compute nodes contain two Intel Xeon Gold 6148 processors (Skylake (SKL), 20 cores @ 2.4 GHz) and are connected through a 100 Gb/s OmniPath interconnect. Each island (~ 636 nodes) has a non-blocking, full fat tree network topology, while among islands a blocking factor of 1:8 applies. Therefore, batch jobs are restricted to a single island. In addition, there are 6 login nodes and an I/O subsystem that serves 5 PetaByte of disk storage with direct HSM access (via GHI). ### Overall configuration * 1284 compute nodes (2 × SKL), 96 GB RAM DDR4 each * 1908 compute nodes (2 × SKL), 192 GB RAM DDR4 each * 16 compute nodes (2 × SKL), 384 GB RAM DDR4 each * 8 compute nodes (2 × SKL), 768 GB RAM DDR4 each * 64 compute nodes (2 × SKL + 2 × NVIDIA Tesla V100-32) * 120 compute nodes (2 × SKL + 2 × NVIDIA Quadro RTX 5000) * 24 compute nodes (2 × SKL), 192 GB RAM DDR4 each (dedicated to MPSD) ### Summary 3424 compute nodes, 136,960 CPU-cores, 128 Tesla V100-32 GPUs, 240 Quadro RTX 5000 GPUs, 529 TB RAM DDR4, 7.9 TB HBM2, 11.4 PFlop/s peak DP, 2.64 PFlop/s peak SP ![MPCDF Cobra](_static/cobra-2018.jpg "MPCDF Cobra") ## Access ### Login For security reasons, direct login to the HPC cluster Cobra is allowed only from within the MPG networks. Users from other locations have to log in to one of our [gateway systems](gateways) first. Use ssh to connect to Cobra: ```bash ssh cobra.mpcdf.mpg.de ``` You will be directed to one of the Cobra login nodes (cobra01i, cobra02i). You have to provide your (Kerberos) password and an OTP on the Cobra login nodes. SSH keys are not allowed. Secure copy (scp) can be used to transfer data to or from cobra.mpcdf.mpg.de Cobra's (all login/interactive nodes) ssh key fingerprints (SHA256) are: ```text G45rl+n9MWi/TWQA3bYXoVxBI/wiOviJXe99H4SacWU (RSA) KcGJxKBfrsVyexByJFgbuFDigfvGfrgZ5Urvmh/ZJLI (ED25519) ``` ### Using compute resources The pool of login nodes cobra.mpcdf.mpg.de is mainly intended for editing, compiling and submitting your parallel programs. Running parallel programs interactively in production mode on the login nodes is not allowed. Jobs have to be submitted to the Slurm batch system which reserves and allocates the resources (e.g. compute nodes) required for your job. Further information on the batch system is provided [below](#slurm-batch-system). ### Interactive (debug) runs If you need to test or debug your code, you may login to 'cobra-i.mpcdf.mpg.de' (cobra03i-cobra06i) and run your code interactively (2 hours at most) with the command: ```bash srun -n NUMBER_OF_CORES -p interactive --time=TIME_LESS_THAN_2HOURS --mem=MEMORY_LESS_THAN_32G ./EXECUTABLE ``` But please, take care that the machine does *not become overloaded*. Don't use more than 8 cores in total and do not request more than 32 GB of main memory. Neglecting these recommendations may cause a system crash or hangup! ### Internet access Connections to the Internet are only permitted from the login nodes in outgoing direction; Internet access from within batch jobs is not possible. To download source code or other data, command line tools such as `wget`, `curl`, `rsync`, `scp`, `pip`, `git`, or similar may be used interactively on the login nodes. In case the transfer is expected to take a long time it is useful to run it inside a `screen` or `tmux` session. ## Hardware configuration ### Compute nodes * 3240 compute nodes * Processor type: [Intel Skylake 6148](https://ark.intel.com/products/120489/Intel-Xeon-Gold-6148-Processor-27_5M-Cache-2_40-GHz) * Processor clock: 2.4 GHz * *Theoretical* peak performance per node: 2.4 GHz \* 32 DP Flops/cycle \* 40 = 3072 DP GFlop/s * Cores per node: 40 (each with 2 hyperthreads, thus 80 logical CPUs per node) * Node topology: 2 NUMA domains with 20 physical cores each * Main memory * standard nodes: 1284 × 96 GB * large memory nodes: 1932 × 192 GB * very large memory nodes: 16 × 384 GB, 8 × 768 GB Accelerator part of Cobra: * 64 nodes, each hosting 2 V100 GPUs (*Tesla V100-PCIE-32GB*: 32 GB HBM2, 5120 CUDA cores + 640 Tensor cores @ 1380 MHz, compute capability 7.0 / "Volta") * 120 nodes, each hosting 2 RTX5000 GPUs (*Quadro RTX 5000*: 16 GB GDDR6, 3072 CUDA cores + 384 Tensor cores + 48 RT units @ 1935 MHz, compute capability 7.5 / "Turing") ### Login and interactive nodes * 2 nodes for login (Hostname `cobra.mpcdf.mpg.de`) * 4 nodes for interactive program development and testing (Hostname cobra-i.mpcdf.mpg.de) * Main memory: 4 × 192 GB Batch access is possible via the Slurm batch system from the login nodes `cobra.mpcdf.mpg.de` and `cobra-i.mpcdf.mpg.de`. ### Interconnect * fast OmniPath (100 Gb/s) network connecting all the nodes The compute nodes and GPU nodes are bundled into 6 domains (islands) with 636 nodes each (or 64 nodes in the case of a GPU island). Within one domain, the OmniPath network topology is a 'fat tree' topology for highly efficient communication. The OmniPath connection between the islands is much weaker, so batch jobs are restricted to a single island, that is 636 nodes. ### I/O subsystem * 8 I/O nodes * 5 PB of online disk space ## File systems ### $HOME Your home directory is in the GPFS file system `/u` (see below). ### AFS AFS is only available on the login nodes `cobra.mpcdf.mpg.de` and on the interactive nodes `cobra-i.mpcdf.mpg.de` in order to access software that is distributed by AFS. If you don't automatically get an AFS token during login, you can get an AFS token with the command `/usr/bin/klog.krb5`. Note that there is no AFS on the compute nodes, so you have to avoid any dependencies on AFS in your job. ### GPFS There are two global, parallel file systems of type [GPFS](https://www.ibm.com/products/spectrum-scale) (`/u` and `/ptmp`), symmetrically accessible from all Cobra cluster nodes, plus the migrating file system `/r` interfacing to the HPSS archive system. #### File system `/u` The file system `/u` (a symbolic link to `/cobra/u`) is designed for permanent user data such as source files, config files, etc. The size of `/u` is 0.6 PB mirrored (RAID 6). Note that **no system backups** are performed. Your home directory is in `/u`. The default disk quota in `/u` is 2.5 TB, the file quota is 2 million files. You can check your disk quota in `/u` with the command: ```bash /usr/lpp/mmfs/bin/mmlsquota cobra_u ``` #### File system `/ptmp` The file system `/ptmp` (a symbolic link to /cobra/ptmp) is designed for batch job I/O (4.5 PB mirrored, RAID 6, **no system backups**). Files in `/ptmp` that have not been accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary (with prior notification). As a current policy, no quotas are applied on `/ptmp`. This gives users the freedom to manage their data according to their actual needs without administrative overhead. This liberal policy presumes a fair usage of the common file space. So, please do a regular housekeeping of your data and archive/remove files that are not currently in use. Archiving data from the GPFS file systems to tape can be done using the migrating file system `/r` (see below). #### File system `/r` The `/r` file system (a symbolic link to `/ghi/r`) stages archive data. It is available only on the login nodes `cobra.mpcdf.mpg.de` and on the interactive nodes `cobra-i.mpcdf.mpg.de`. Each user has a subdirectory `/r/*initial*/*userid*` to store data. For efficiency, files should be packed to tar files (with a size of about 1 GB to 1 TB) before archiving them in `/r`, i.e., please avoid archiving small files. When the file system `/r` gets filled above a certain value, files will be transferred from disk to tape, beginning with the largest files which have not been used for the longest time. For documentation on how to use the MPCDF archive system, please see the [backup and archive section](../data/backup-archive/index.md). #### /tmp Please, don't use the file system `/tmp` for scratch data. Instead, use `/ptmp` which is accessible from all Cobra cluster nodes. In cases where an application really depends on node-local storage, you can use the variables `JOB_TMPDIR` and `JOB_SHMTMPDIR`, which are set individually for each job. ## Software ### Access to software via environment modules Environment modules are used at MPCDF to provide software packages and enable switching between different software versions. Use the command ```bash module avail ``` to list the available software packages on the HPC system. Note that you can search for a certain module by using the `find-module` tool (see below). Use the command ```bash module load package_name/version ``` to actually load a software package at a specific version. Further information on the environment modules on Cobra and their hierarchical organization is given below. Information on the software packages provided by the MPCDF is available [here](software/index.md). ### Recommended compiler and MPI stack on Cobra We currently (as of 2021/07) recommend using the following versions on Cobra: ```bash module load intel/19.1.3 impi/2019.9 mkl/2020.4 ``` ### Hierarchical module environment To manage the plethora of software packages resulting from all the relevant combinations of compilers and MPI libraries, we organize the environment module system for accessing these packages in a natural hierarchical manner. Compilers (gcc, intel) are located on the uppermost level, depending libraries (e.g., MPI) on the second level, more depending libraries on a third level. This means that not all the modules are visible initially: only after loading a compiler module, the modules depending on this will become available. And similarly, loading an MPI module in addition will make the modules depending on the MPI library available. Starting with the maintenance on Sep 22 2021, no defaults are defined for the compiler and MPI modules, and no modules are loaded automatically at login. This forces users to specify explicit versions for those modules during compilation and in the batch scripts to ensure that the same MPI library is loaded. This also means that users can decide themselves when they use newer compiler and MPI versions for their code which avoids compatibility problems when changing defaults centrally. For example, the FFTW library compiled with the Intel compiler and the Intel MPI library can be loaded as follows: First, load the Intel compiler module using the command ```bash module load intel/19.1.3 ``` second, the Intel MPI module with ```bash module load impi/2019.9 ``` and, finally, the FFTW module fitting exactly to the compiler and MPI library via ```bash module load fftw-mpi ``` You may check by using the command ```bash module avail ``` that after the first and second steps the depending environment modules become visible, in the present example impi and fftw-mpi. Moreover, note that the environment modules can be loaded via a single 'module load' statement as long as the order given by the hierarchy is correct, e.g., ```bash module load intel/19.1.3 impi/2019.9 fftw-mpi ``` It is important to point out that a large fraction of the available software is not affected by the hierarchy, e.g., certain HPC applications, tools such as git or cmake, mathematical software (maple, matlab, mathematica), visualization software (visit, paraview, idl) are visible at the uppermost hierarchy. Note that a hierarchy exists for depending Python modules via the 'anaconda' module files on the top level, and similarly for CUDA via the 'cuda' module files. To start at the root of the environment modules hierarchy, run `module purge`. Because of the hierarchy, some modules only appear after other modules (such as compiler and MPI) have been loaded. One can search all available combinations of a certain software (e.g. fftw-mpi) by using ```bash find-module fftw-mpi ``` Further information on using environment modules is given [here](software/environment-modules.md). ### Transition to no-default Intel modules in September 2021 Please note that with the Cobra maintenance on Sep 22, 2021, the default-related configuration of the Intel modules was removed, as announced by email on Aug 02, 2021. After that maintenance, no defaults are defined for the Intel compiler and MPI modules, and no modules are loaded automatically at login. The motivation for introducing these changes is to avoid the accidental use of different versions of Intel compilers and MPI libraries at compile time and at run time. Please note that this will align the configuration on Cobra with the configuration on Raven where users have to specify full versions and no default modules are loaded. What kind of adaptations of user scripts are necessary? Please load a specific set of environment modules with explicit versions consistently when compiling and running your codes, e.g. use ```bash module purge module load intel/19.1.3 impi/2019.9 mkl/2020.4 ``` in your job scripts as well as in interactive shell sessions. Note that you must specify a full version for the 'intel' and the 'impi' modules, otherwise the command will fail. Please note that for your convenience, pre-compiled applications provided as modules like 'vasp' or 'gromacs' will continue to load the necessary 'intel' and 'impi' modules automatically, i.e. no changes of the batch scripts are required for these applications. We do, however, recommend adding a `module purge` in those cases. ## Slurm batch system The batch system on the HPC cluster Cobra is the open-source workload manager Slurm (Simple Linux Utility for Resource management). To run test or production jobs, submit a job script (see below) to Slurm, which will find and allocate the resources required for your job (e.g. the compute nodes to run your job on). By default, the job run limit is set to 8 on Cobra, the default job submit limit is 300. If your batch jobs can't run independently from each other, please use job steps or contact the helpdesk on the MPCDF web page. The Intel processors on Cobra support the hyperthreading mode which *might* increase the performance of your application by up to 20%. With hyperthreading, you have to increase the number of MPI tasks per node from 40 to 80 in your job script. Please be aware that with 80 MPI tasks per node each process gets only half of the memory by default. If you need more memory, you have to specify it in your job script (see example batch scripts). If you want to test or debug your code interactively on `cobra-i.mpcdf.mpg.de` (cobra03i-cobra06i), you can use the command: ```bash srun -n N_TASKS -p interactive ./EXECUTABLE ``` For detailed information about the Slurm batch system, please see [Slurm Workload Manager](https://slurm.schedmd.com/). Overview of batch queues (partitions) on Cobra: ```text Partition Processor Max. CPUs Max. Memory Max. Nr. Max. Run type per Node per Node of Nodes Time std.| large ----------------------------------------------------------------------------- tiny Skylake 20 42 GB 0.5 24:00:00 express Skylake 40 / 80 in HT mode 85 | 180 GB 32 30:00 medium Skylake 40 / 80 in HT mode 85 | 180 GB 32 24:00:00 n0064 Skylake 40 / 80 in HT mode 85 | 180 GB 64 24:00:00 n0128 Skylake 40 / 80 in HT mode 85 | 180 GB 128 24:00:00 n0256 Skylake 40 / 80 in HT mode 85 | 180 GB 256 24:00:00 n0512 Skylake 40 / 80 in HT mode 85 | 180 GB 512 24:00:00 n0620 Skylake 40 / 80 in HT mode 85 | 180 GB 620 24:00:00 fat Skylake 40 / 80 in HT mode 748 GB 8 24:00:00 chubby Skylake 40 / 80 in HT mode 368 GB 16 24:00:00 gpu_v100 Skylake 40 / 80 (host cpus) 180 GB 64 24:00:00 gpu1_v100 Skylake 40 / 80 (host cpus) 90 GB 0.5 24:00:00 gpu_rtx5000 Skylake 40 / 80 (host cpus) 180 GB 120 24:00:00 gpu1_rtx5000 Skylake 40 / 80 (host cpus) 90 GB 0.5 24:00:00 Remote visualization: rvs Skylake 40 / 80 (host cpus) 180 GB 2 24:00:00 ``` The most important Slurm commands are * `sbatch ` Submit a job script for execution * `squeue` Check the status of your job(s) * `scancel ` Cancel a job * `sinfo` List the available batch queues (partitions). Sample Batch job scripts can be found below. Notes on job scripts: * The directive ```text SBATCH --nodes= ``` in your job script sets the number of compute nodes that your program will use. * The directive ```text SBATCH --ntasks-per-node= ``` specifies the number of MPI processes for the job. The parameter tasks-per-node cannot be greater than 80 because one compute node on Cobra has 40 cores with 2 threads each, thus 80 logical CPUs in hyperthreading mode. * The directive ```text SBATCH --cpus-per-task= ``` specifies the number of threads per MPI process if you are using OpenMP. * The expression ```text tasks-per-node * cpus-per-task ``` may not exceed 80. * The expression ```text nodes * tasks-per-node * cpus-per-task ``` gives the total number of CPUs that your job will use. * Jobs that need less than a half compute node have to specify a reasonable memory limit so that they can share a node! * A job submit filter will automatically choose the right partition/queue from the resource specification. * Please note that setting the environment variable 'SLURM_HINT' in job scripts is not necessary and discouraged on Cobra. ## Slurm example batch scripts ### MPI and MPI/OpenMP batch scripts #### MPI batch job without hyperthreading ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob.out.%j #SBATCH -e ./tjob.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=40 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 # Run the program: srun ./myprog > prog.out ``` #### Hybrid MPI/OpenMP batch job without hyperthreading ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob_hybrid.out.%j #SBATCH -e ./tjob_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=4 # for OpenMP: #SBATCH --cpus-per-task=10 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly: export OMP_PLACES=cores # Run the program: srun ./myprog > prog.out ``` #### Hybrid MPI/OpenMP batch job in hyperthreading mode ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob_hybrid.out.%j #SBATCH -e ./tjob_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=4 # Enable Hyperthreading: #SBATCH --ntasks-per-core=2 # for OpenMP: #SBATCH --cpus-per-task=20 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock Limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly: export OMP_PLACES=threads # Run the program: srun ./myprog > prog.out ``` #### MPI batch job in hyperthreading mode using 180 GB of memory per node ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob.out.%j #SBATCH -e ./tjob.err.%j # Initial working directory: #SBATCH -D ./ # Job Name : #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=16 #SBATCH --ntasks-per-node=80 # Enable Hyperthreading: #SBATCH --ntasks-per-core=2 # # Request 180 GB of main memory per node in units of MB: #SBATCH --mem=185000 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 # enable over-subscription of physical cores by MPI ranks export PSM2_MULTI_EP=0 # Run the program: srun ./myprog > prog.out ``` #### OpenMP batch job in hyperthreading mode using 180 GB of memory per node ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob_hybrid.out.%j #SBATCH -e ./tjob_hybrid.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of nodes and MPI tasks per node: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 # Enable Hyperthreading: #SBATCH --ntasks-per-core=2 # for OpenMP: #SBATCH --cpus-per-task=80 # # Request 180 GB of main memory per node in units of MB: #SBATCH --mem=185000 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock Limit: #SBATCH --time=24:00:00 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK # For pinning threads correctly export OMP_PLACES=threads # Run the program: srun ./myprog > prog.out ``` #### Small MPI batch job on 1 - 20 cores (using a shared node) ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob.out.%j #SBATCH -e ./tjob.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_slurm # # Number of MPI Tasks, e.g. 8: #SBATCH --ntasks=8 #SBATCH --ntasks-per-core=1 # Memory usage [MB] of the job is required, 2200 MB per task: #SBATCH --mem=17600 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 # Run the program: srun ./myprog > prog.out ``` ### Batch jobs using GPUs #### MPI batch job on GPUs ```bash #!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob.out.%j #SBATCH -e ./tjob.err.%j # Initial working directory: #SBATCH -D ./ # #SBATCH -J test_slurm # # Node feature: #SBATCH --constraint="gpu" # Specify type and number of GPUs to use: # GPU type can be v100 or rtx5000 #SBATCH --gres=gpu:v100:2 # If using both GPUs of a node # #SBATCH --gres=gpu:v100:1 # If using only 1 GPU of a shared node # #SBATCH --mem=92500 # Memory is necessary if using only 1 GPU # # Number of nodes and MPI tasks per node: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=40 # If using both GPUs of a node # #SBATCH --ntasks-per-node=20 # If using only 1 GPU of a shared node # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # wall clock limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 module load cuda/11.2 # Run the program: srun ./my_gpu_prog > prog.out ``` ### Batch jobs with dependencies The following script generates a sequence of jobs, each job running the given job script. The start of each individual job depends on its dependency, where possible values for the `--dependency` flag are, e.g. * `afterany:job_id` This job starts after the previous job has terminated * `afterok:job_id` This job starts after previous job has successfully executed ```bash #!/bin/bash # Submit a sequence of batch jobs with dependencies # # Number of jobs to submit: NR_OF_JOBS=6 # Batch job script: JOB_SCRIPT=./my_batch_script echo "Submitting job chain of ${NR_OF_JOBS} jobs for batch script ${JOB_SCRIPT}:" JOBID=$(sbatch ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} I=1 while [ ${I} -lt ${NR_OF_JOBS} ]; do JOBID=$(sbatch --dependency=afterany:${JOBID} ${JOB_SCRIPT} 2>&1 | awk '{print $(NF)}') echo " " ${JOBID} let I=${I}+1 done ``` ### Batch job using a job array ```bash #!/bin/bash -l #SBATCH --array=1-20 # specify the indexes of the job array elements # Standard output and error: #SBATCH -o job_%A_%a.out # Standard output, %A = job ID, %a = job array index #SBATCH -e job_%A_%a.err # Standard error, %A = job ID, %a = job array index # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J test_array # # Number of nodes and MPI tasks per node: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=40 # #SBATCH --mail-type=none #SBATCH --mail-user=userid@example.mpg.de # # Wall clock limit: #SBATCH --time=24:00:00 # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load intel/19.1.3 impi/2019.9 # The environment variable $SLURM_ARRAY_TASK_ID holds the index of the job array and # can be used to discriminate between individual elements of the job array: srun ./myprog $SLURM_ARRAY_TASK_ID >prog.out ``` ### Single-node example job scripts for sequential programs, plain-OpenMP cases, Python, Julia, Matlab In the following, example job scripts are given for jobs that use at maximum one full node. Use cases are sequential programs, threaded programs using OpenMP or similar models, and programs written in languages such as Python, Julia, Matlab, etc. The Python example programs referred to below are available for [download](https://datashare.mpcdf.mpg.de/s/KCEtd0tP3zLypq4). #### Single-core job ```bash #!/bin/bash -l # # Single-core example job script for MPCDF Cobra. # In addition to the Python example shown here, the script # is valid for any single-threaded program, including # sequential Matlab, Mathematica, Julia, and similar cases. # #SBATCH -J PYTHON_SEQ #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH --ntasks=1 # launch job on a single core #SBATCH --cpus-per-task=1 # on a shared node #SBATCH --mem=2000MB # memory limit for the job #SBATCH --time=0:10:00 module purge module load gcc/10 impi/2019.9 module load anaconda/3/2021.05 # Set number of OMP threads to fit the number of available cpus, if applicable. export OMP_NUM_THREADS=1 # Run single-core program srun python3 ./python_sequential.py ``` #### Small job with multithreading, applicable to Python, Julia and Matlab, plain OpenMP, or any threaded application ```bash #!/bin/bash -l # # Multithreading example job script for MPCDF Cobra. # In addition to the Python example shown here, the script # is valid for any multi-threaded program, including # Matlab, Mathematica, Julia, and similar cases. # #SBATCH -J PYTHON_MT #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH --ntasks=1 # launch job on #SBATCH --cpus-per-task=8 # 8 cores on a shared node #SBATCH --mem=16000MB # memory limit for the job #SBATCH --time=0:10:00 module purge module load gcc/10 impi/2019.9 module load anaconda/3/2021.05 # Set number of OMP threads to fit the number of available cpus, if applicable. export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun python3 ./python_multithreading.py ``` #### Python/NumPy multithreading, applicable to Julia and Matlab, plain OpenMP, or any threaded application ```bash #!/bin/bash -l # # Multithreading example job script for MPCDF Cobra. # In addition to the Python example shown here, the script # is valid for any multi-threaded program, including # plain OpenMP, parallel Matlab, Julia, and similar cases. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J PY_MULTITHREADING #SBATCH --nodes=1 # request a full node #SBATCH --ntasks-per-node=1 # only start 1 task via srun because Python multiprocessing starts more tasks internally #SBATCH --cpus-per-task=40 # assign all the cores to that first task to make room for multithreading #SBATCH --time=00:10:00 module purge module load gcc/10 impi/2019.9 module load anaconda/3/2021.05 # set number of OMP threads *per process* export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} srun python3 ./python_multithreading.py ``` #### Python multiprocessing ```bash #!/bin/bash -l # # Python multiprocessing example job script for MPCDF Cobra. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J PYTHON_MP #SBATCH --nodes=1 # request a full node #SBATCH --ntasks-per-node=1 # only start 1 task via srun because Python multiprocessing starts more tasks internally #SBATCH --cpus-per-task=40 # assign all the cores to that first task to make room for Python's multiprocessing tasks #SBATCH --time=00:10:00 module purge module load gcc/10 impi/2019.9 module load anaconda/3/2021.05 # Important: # Set the number of OMP threads *per process* to avoid overloading of the node! export OMP_NUM_THREADS=1 # Use the environment variable SLURM_CPUS_PER_TASK to have multiprocessing # spawn exactly as many processes as you have CPUs available. srun python3 ./python_multiprocessing.py $SLURM_CPUS_PER_TASK ``` #### Python mpi4py ```bash #!/bin/bash -l # # Python MPI4PY example job script for MPCDF Cobra. # Plain MPI. May use more than one node. # #SBATCH -o ./out.%j #SBATCH -e ./err.%j #SBATCH -D ./ #SBATCH -J MPI4PY #SBATCH --nodes=1 #SBATCH --ntasks-per-node=40 #SBATCH --time=00:10:00 module purge module load gcc/10 impi/2019.9 module load anaconda/3/2021.05 module load mpi4py/3.0.3 # Important: # Set the number of OMP threads *per process* to avoid overloading of the node! export OMP_NUM_THREADS=1 srun python3 ./python_mpi4py.py ``` # Raven hardware details This page summarizes additional details on the Raven hardware and presents performance measurements from microbenchmarks. ## Node architecture The Raven system comprises compute nodes powered by dual Intel Xeon IceLake-SP processors ([Platinum 8360Y](https://ark.intel.com/content/www/us/en/ark/products/212459/intel-xeon-platinum-8360y-processor-54m-cache-2-40-ghz.html)) with 36 physical CPU cores per socket (i.e. 72 per node). These nodes feature 256 GB of RAM, with 64 nodes having 512 GB and 4 nodes featuring 2048 GB. Additionally, there are 192 GPU-accelerated compute nodes, each equipped with 4 [Nvidia A100 40GB-SXM GPUs](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf) connected via NVLINK3 and connected to the host via PCIe. These GPU nodes also feature 512 GB RAM and use the same Intel Xeon IceLake-SP CPUs. The CPU nodes are interconnected with a Mellanox HDR InfiniBand network at 100 Gbit/s, whereas the GPU nodes are linked at a rate of 200 Gbit/s. The nominal bandwidths between the components on a Raven GPU node are approximately: * 100 GB/s per direction between each of the four A100 GPUs using NVLINK3 * 67 GB/s per direction between the two CPU sockets using UPI * 32 GB/s per direction between each A100 GPU and the host using PCIe4 x16 * 25 GB/s per direction via the InfiniBand network interface The following schematic highlights the topology of a Raven GPU node (where the notation '2x X GB/s' refers to full duplex, enabling a bandwidth of X GB/s in each direction simultaneously): ![Raven GPU node schematic](_static/raven-gpu-schematic.png "Raven GPU node schematic") A set of 32 Raven GPU nodes is equipped with a second InfiniBand interface that doubles the bandwidth into the network to 50 GB/s per direction. On Slurm, these nodes can be selected via the `--constraint="gpu-bw"` flag of `sbatch`. Due to a limited number of available PCIe lanes, the GPUs on these nodes are connected to the host at half the bandwidth compared to the regular Raven GPU nodes. The following schematic shows the topology of a Raven GPU node of the 'gpu-bw' type: ![Raven GPU-BW node schematic](_static/raven-gpu-schematic-bw.png "Raven GPU-BW node schematic") ## Empirical Roofline Models Below, empirical roofline plots are presented that are based on measurements made on a CPU node and on an individual GPU of Raven. [Roofline models](https://zenodo.org/record/1236156) illustrate the limitations of a computational kernel due to either the memory bandwidth or the maximum floating point performance of the hardware, depending on the arithmetic intensity of that kernel. ![CPU Empirical Roofline Plot](_static/roofline_raven_cpu.png "CPU Empirical Roofline Plot") ![GPU Empirical Roofline Plot](_static/roofline_raven_gpu.png "GPU Empirical Roofline Plot") ## Performance measurements using microbenchmarks The [likwid suite](https://github.com/RRZE-HPC/likwid) implements a set of microbenchmarks to measure, e.g., the flops and the memory bandwidth a system may achieve. The numbers below are based on actual measurements on a full node, i.e. utilizing all CPU cores and memory channels. For the measurements on the GPU the [BabelStream](https://github.com/UoB-HPC/BabelStream) microbenchmarks were used. ### CPU ```text Currently Loaded Modulefiles: 1) intel/21.6.0 2) likwid/5.2(default) ``` #### Flops ```text instruction set GFlops/s ----------------- ---------- scalar 304.660 SSE 640.023 AVX 1357.607 AVX-FMA 2716.209 AVX512 2685.938 AVX512-FMA 5370.171 ``` #### Memory Bandwidth ```text load instruction set GBytes/s ----------------- ---------- scalar 321.435 SSE 336.826 AVX 338.035 copy instruction set GBytes/s ----------------- ---------- scalar 260.375 SSE 290.084 AVX 294.560 stream instruction set GBytes/s ----------------- ---------- scalar 300.476 SSE 303.429 AVX 303.799 triad instruction set GBytes/s ----------------- ---------- scalar 306.859 SSE 307.839 AVX 307.724 ``` To complement the previously presented numbers, the following plot shows measurements of the memory bandwidth under variation of the number of threads. Each thread is bound ("pinned") to an individual physical core. ![Memory bandwidth vs. number of physical cores employed](_static/raven_stream_scaling.png "Memory bandwidth vs. number of physical cores employed") The orange curve depicts measured results based on a scattered pinning, i.e., threads are pinned to physical cores on the two CPU sockets in a round-robin fashion, thereby making use of all available memory channels in a balanced way. As a rule of thumb and evident from the plot, at least half of the physical cores per socket are required to make efficient use of the available memory bandwidth. In contrast, the blue curve shows results based on a compact pinning, i.e., threads are pinned to the first socket until it is fully occupied (36) before the second socket is populated with threads as well (72). That transition illustrates the memory bandwidth a single socket is able to deliver. ### GPU ```text BabelStream Version: 4.0 Implementation: CUDA Running kernels 100 times Precision: double Array size: 268.4 MB (=0.3 GB) Total size: 805.3 MB (=0.8 GB) Using CUDA device NVIDIA A100-SXM4-40GB Driver: 11040 Function MBytes/sec Min (sec) Max Average Copy 1403896.061 0.00038 0.00039 0.00038 Mul 1360548.080 0.00039 0.00040 0.00040 Add 1357798.755 0.00059 0.00060 0.00060 Triad 1362270.774 0.00059 0.00069 0.00060 Dot 1229347.744 0.00044 0.00045 0.00044 ``` # Viper hardware details This page summarizes additional details on the Viper hardware and presents performance measurements from microbenchmarks. ## Empirical Roofline Models Below, empirical roofline plots are presented that are based on measurements made on a CPU node. There are two different memory configurations available. The standard nodes have 512GB per node and show a lower memory bandwidth than the larger 768GB nodes. [Roofline models](https://zenodo.org/record/1236156) illustrate the limitations of a computational kernel due to either the memory bandwidth or the maximum floating point performance of the hardware, depending on the arithmetic intensity of that kernel. ![CPU Empirical Roofline Plot](_static/roofline_viper_cpu_standard_nodes.png "Empirical Roofline Plot for a 512GB node") ![CPU Empirical Roofline Plot](_static/roofline_viper_cpu_largemem_nodes.png "Empirical Roofline Plot for a largemem node") # Partitioning of MI300A GPU resources The AMD MI300A APU allows its physical GPU compute resources to be divided into multiple logical devices, enabling more flexible workload scheduling and resource allocation. The MI300A contains six Accelerator Complex Dies (XCDs), each housing 38 CDNA3 Compute Units (CUs), for a total of 228 CUs per socket (note that a Viper-GPU node has two sockets). These XCDs can be grouped into partitions in three ways. In SPX (Single Partition X-celerator) mode — the default — all six XCDs are presented to the system as a single monolithic GPU with 228 CUs and access to the full 128 GB of unified HBM3 memory. This is optimal for large workloads such as full-scale HPC simulations or deep-learning training runs that benefit from the highest possible compute throughput and memory capacity. In TPX (Triple Partition X-celerator) mode, the six XCDs are grouped into three logical devices of two XCDs each; this offers a middle ground between unified execution and fine-grained partitioning, although most workloads benefit more from either SPX or CPX mode. In CPX (Core Partitioned X-celerator) mode, each of the six XCDs is exposed as an independent logical GPU device, yielding six separate devices per socket, each with 38 CUs and approximately 21 GB of HBM. ![Overview of the three partitioning modes of the AMD MI300A](_static/amd_xcd.png) The partitioning mode can be reconfigured dynamically via Slurm using the `--mi300-partition=MODE` option of `sbatch`, `salloc`, or `srun`, where `MODE` is `tpx` or `cpx`. Alternatively, the option can be set in the batch script: ```bash #SBATCH --mi300-partition=cpx # AMD MI300 compute partition: tpx|cpx ``` On the Viper-GPU `apu` Slurm partition, where nodes are allocated exclusively, this sets the partitioning mode on both MI300A APUs. For code testing, a single APU can also be partitioned in the `apudev` Slurm partition. When the option is used, the `ROCR_VISIBLE_DEVICES` and `HIP_VISIBLE_DEVICES` environment variables are set accordingly. When the APUs are reconfigured via `sbatch` or `salloc`, there is no need to repeat the option on `srun` for individual job steps, since the APUs are already divided into multiple logical devices. The selected partitioning mode directly affects both job placement and GPU kernel scheduling, and therefore application performance. SPX mode is best suited to workloads that can efficiently utilize the full GPU. CPX mode, in contrast, benefits applications that do not require the full memory capacity and can exploit additional parallelism, for example by running one MPI rank per XCD partition. Stencil codes are a typical example of this class of applications, and the HPC community has observed significant speedups when running them in CPX mode compared to SPX (see [this study](https://dl.acm.org/doi/10.1145/3773656.3773680)). Readers interested in the underlying architecture and partitioning concepts are encouraged to consult the official [AMD Instinct MI300A APU Overview](https://instinct.docs.amd.com/projects/amdgpu-docs/en/latest/gpu-partitioning/mi300a/overview.html) documentation. # Viper-GPU hardware details This page summarizes additional details on the Viper-GPU hardware and presents performance measurements from microbenchmarks. ## Node architecture ## Empirical Roofline Models ## Performance measurements using microbenchmarks For the measurements on the GPU the [BabelStream](https://github.com/UoB-HPC/BabelStream) microbenchmarks with the hip backend were used. ### GPU ```text BabelStream Version: 5.0 Implementation: HIP Running kernels 100 times Precision: double Array size: 8589.9 MB (=8.6 GB) Total size: 25769.8 MB (=25.8 GB) Using HIP device AMD Instinct MI300A Driver: 60342131 Memory: DEFAULT Init: 0.728338 s (=35381.643594 MBytes/sec) Read: 0.231641 s (=111248.902886 MBytes/sec) Function MBytes/sec Min (sec) Max Average Copy 3721708.772 0.00462 0.00521 0.00469 Mul 3710036.456 0.00463 0.00508 0.00470 Add 3630681.987 0.00710 0.01663 0.00725 Triad 3604594.358 0.00715 0.01064 0.00724 Dot 3255586.355 0.00528 0.00553 0.00536 ``` # Using Flash-based I/O-Accelerators on Viper-GPU ## Introduction The Viper-GPU HPC system is equipped with flash accelerators, a solution that combines specialized hardware and software to leverage the high throughput and low latency of Non-Volatile Memory Express (NVMe) solid-state drives for batch jobs. These accelerators are designed to mitigate the I/O bottleneck of HPC and AI applications during intense read and write phases. The deployment on Viper-GPU is based on the *Smart Bunch of Flash* (SBF) product by Eviden. On Viper-GPU, SBF provides two modes of operation that differ in the lifetime of the storage buffer: * An **ephemeral buffer** has a lifetime that ends as soon as the job that uses it ends. This is the default mode. * A **persistent buffer** survives the job that creates it. It can then be used by subsequent jobs until it is destroyed explicitly. SBF cannot be used for jobs on shared nodes. Jobs requesting SBF must allocate full node(s) and run on the **gpu** partition. ## Getting started To use SBF, a line starting with `#BB_LUA` has to be added to the top of the Slurm job script, e.g. as follows: ```bash #!/bin/bash -l #BB_LUA SBF # (SBATCH lines below) #SBATCH # ... # (executable section of job script below) ``` The `#BB_LUA SBF` line has to be placed directly after the first line (`#!/bin/bash -l`) of the script. The following mandatory parameters have to be specified: | Option | Mandatory | Description | |:-------|:----------|:------------| | `StorageSize=` | Yes | The size of each storage buffer per compute node. If `-N ` nodes are requested the job will consume ` * ` disk space from the servers. | | `Path=/var/local/bb` | Yes | The mountpoint where each individual storage buffer will be mounted on each compute node. | The job steps of the job script must be started with the `srun` command to get full access to the storage buffers. Otherwise, only the master compute node that runs the job script will be able to use the SBF feature. ### Ephemeral buffer example The following example job script requests 100 GB of temporary storage space, where the application `myApp` has access to the storage via the mount point `/var/local/bb`. ``` #!/bin/bash -l #BB_LUA SBF StorageSize=100GB Path=/var/local/bb # #SBATCH -J BB #SBATCH --gres=gpu:2 #SBATCH --nodes 1 # this is -N #SBATCH --ntasks-per-node 1 #SBATCH --cpus-per-task 48 #SBATCH -t 0:01:00 srun myApp ``` A typical use case would be an application that requires fast scratch space to write and read temporary files to a local file system. ### Persistent buffer example In persistent mode, SBF knows the three stages "create", "use" and "destroy". Two additional parameters for **#BB_LUA SBF** are relevant, as illustrated below. #### Stage 1: `create_persistent` An initial job can create a persistent buffer using the following parameters in the job script: ``` #BB_LUA SBF create_persistent Name=MyPersistentSBF StorageSize=100GB Path=/var/local/bb ``` where `Name` must be unique and chosen by the user for the jobs that should use the buffer. After the job completes, the `-N` storage buffers of size `` each stay allocated. #### Stage 2: `use_persistent` Subsequent jobs can use the persistent buffer as follows: ``` #BB_LUA SBF use_persistent Name=MyPersistentSBF ``` After the job has completed, the storage buffers are unmounted from the compute nodes, but the buffers still exist on the servers. Subsequent jobs can reuse these storage buffers. Be aware that the same persistent `Name` cannot be used by multiple jobs of the same user at the same time due to the private nature of the storage buffers. The number of compute nodes (`-N` option) of the `use_persistent` request must not exceed the number of compute nodes of the `create_persistent` request. #### Stage 3: `destroy_persistent` Once a persistent buffer becomes unnecessary, the user shall request its destruction (i.e. deallocation) explicitly using the following line in a job script: ``` #BB_LUA SBF destroy_persistent Name=MyPersistentSBF ``` The `destroy_persistent` job can only be executed if no other job is using the persistent buffer anymore. After the job completes, the storage buffers no longer exist and cannot be used by any other job anymore. In case the user specifies the `force=true` parameter in addition after `destroy_persistent`, the job will stop any jobs that are using the persistent buffer and kill them immediately. A typical use case would be an application that relies on the availability of a fast file system to read data from a database file, including random file access. The database could be copied to the persistent buffer during the initial `create_persistent` job, and read from during subsequent jobs. ## Reporting Commands The Slurm commands `squeue` and `scontrol` may display information on the buffer, in particular information on the different phases **stage-in** and **stage-out**. * Use `squeue` to display information about jobs located in the Slurm scheduling queue, including details such as _BurstBufferStageIn_, _BurstBufferResources_, etc. * Use `scontrol show job ` to view detailed information for a specific job, where the **BurstBuffer** and **Reason** fields provide SBF-related data. * The command `scontrol show bbstat` displays the buffer information. The **State** field in the report shows the job's progress regarding the SBF activity. The **Staging-out** state indicates that the resources are not yet released. Once the resources are released, the state becomes **Staged-out**. Only after the state becomes **Staged-out** are the buffer resources fully released and the job considered complete. ## Restrictions * The `-N ` (or `--nodes=`) parameter is mandatory in your Slurm `sbatch` command. SBF creates a separate storage buffer for each node before the job starts, so it requires the node count in advance. * The SBF storage buffers are not shared between nodes. * Using the SBF requires an **exclusive** use of the compute node by a job. * The minimum value of the `StorageSize` parameter is 16MiB. * Heterogeneous jobs are not supported by SBF. * A batch job must be submitted to destroy a persistent buffer. ## Technical Details SBF is a software-defined storage solution by Eviden that allows the creation of fast and temporary storage volumes. These volumes are attached individually to individual compute nodes using NVMeOF (Non-Volatile Memory express Over Fabrics) technology. SBF uses the XFS file system on the volumes. ![banner](../Images/banner7.jpg) # Backup & Archive Systems for any Max Planck Institute If you are an IT Manager at a Max Planck Institute, you can use the MPCDF backup and archive systems to store data. For small amounts of backup or archive data (below 100 TB), a storage area can probably be set up quickly in one of our existing systems. For larger amounts of data, special planning and allocation of resources will be required. If you are interested, the best option is to open a ticket in our [Helpdesk](../../../../faq/help.html#how-can-i-get-help-and-support), briefly describing your needs. You will then be contacted by us to discuss further details. ## MPCDF's policy for long-term preservation of archive data ### Data management with the HSM system HPSS The Hierarchical Storage Management (HSM) system HPSS, introduced at MPCDF (formerly RZG) in 2011, proves to be increasingly essential for managing the archival requirements of many Max Planck Institutes from all three scientific sections. The largest needs, however, arise in the meantime from the Life Sciences. Since late 2018, when the amount of data stored in HPSS at the MPCDF surpassed the 100 PB threshold, the data has more than doubled to over 220 PB. As shown in the list of [publicly disclosed HPSS deployments](https://www.hpss-collaboration.org/customersT.shtml), the MPCDF continues to belong to the top 10 scientific data centers worldwide and remains on rank 1 within Germany. ### Safety of Archival Data In 2006, the president of the Max Planck Society, Prof. Gruß, requested the RZG to ensure that important data can be stored safely for at least 50 years. From the very beginning of mass storage at the RZG with an automated tape library in 1980, a CDC8500 with 2000 8-MB cartridges, the challenge we had to master was to provide bit preservation across multiple generations of archiving technologies, w.r.t both hardware and software. And indeed, the oldest files stored in our archive system are from the 1980s. For archive data, we keep 2 copies of each file on 2 different tapes (in 2 different buildings). For backup data, the standard is only 1 tape copy (more on request). The MPCDF has started to further improve safety of particularly precious archival data which fulfill the following criteria: MPCDF is the master site, and there are no officially maintained data copies at other sites, and in the unlikely case of loss, the data would be unrecoverable or it would require immense efforts to recover the data. For such data, a third tape copy, using different software technology, is now hosted in Berlin on request. This procedure takes into account the updated recommendation for georedundancy which requires a distance larger than 200 km. Click on the account link on the upper right. ![](../images/metastore-api-token-1.png) Click on the API tokens tab. ![](../images/metastore-api-token-2.png) Click on create api token. ![](../images/metastore-api-token-3.png) Copy and store your api token somewhere safe ![](../images/metastore-api-token-4.png) # DataCite Extended Format ```json { "title": "DataCite", "version": "4.4, Complete", "description": "DataCite Schema, version 4.4", "properties": [ { "field_name": "creators", "label": "Creators", "description": "The main researchers involved working on the data, or the authors of the publication in priority order. May be a corporate/institutional or personal name.", "default": "", "optional": true, "repeatable": false, "vocabulary": [], "subfields": [ { "field_name": "nametype", "label": "Name Type", "vocabulary": ["Personal","Organizational"] }, { "field_name": "creatorname", "label": "Name" }, { "field_name": "givenname", "label": "Given Name" }, { "field_name": "familyname", "label": "Family Name" }, { "field_name": "affiliation", "label": "Affiliation" }, { "field_name": "nameidentifier", "label": "Name Identifier", "description": "ORCID ID" }, { "field_name": "", "label": "" }, { "field_name": "", "label": "" } ] }, { "field_name": "titles", "label": "Titles", "description": "A name or title by which the dataset is known.", "default": "", "optional": "false", "repeatable": false, "vocabulary": [] }, { "field_name": "name", "label": "URL", "description": "", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "owner_org", "label": "Institute", "description": "", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "publisher", "label": "Publisher", "description": "The name of the entity that holds, archives, publishes prints, distributes, releases, issues, or produces the dataset. This property will be used to formulate the citation, so consider the prominence of the role.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "publicationyear", "label": "Publication Year", "description": "Year when the data is made publicly available. If an embargo period has been in effect, use the date when the embargo period ends. If there is no standard publication year value, use the date that would be preferred from a citation perspective.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "resourcetype", "label": "Resource Type", "description": "The type of a dataset. You may enter an additional free text description. The format is open, but the preferred format is a single term of some detail so that a pair can be formed with the sub-property.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "subjects", "label": "Subjects", "description": "Subject, keywords, classification codes, or key phrases describing the dataset.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "contributors", "label": "Contributors", "description": "The institution or person responsible for collecting, creating, or otherwise contributing to the developement of the dataset.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "dates", "label": "Dates", "description": "Different dates relevant to the work.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "language", "label": "Language", "description": "Primary language of the resource.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "alternateidentifiers", "label": "Alternate Identifiers", "description": "An identifier or identifiers other than the primary Identifier applied to the dataset being registered. This may be any alphanumeric string which is unique within its domain of issue. May be used for local identifiers. AlternateIdentifier should be used for another identifier of the same instance (same location, same file).", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "relatedidentifiers", "label": "Related Identifiers", "description": "", "default": "Identifiers of related datasets. Use this property to indicate subsets of properties, as appropriate.", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "sizes", "label": "Sizes", "description": "Unstructured size information about the dataset.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "formats", "label": "Formats", "description": "Technical format of the dataset. Use file extension or MIME type where possible.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "version", "label": "Version", "description": "Version number of the dataset. If the primary dataset has changed the version number increases. Register a new identifier for a major version change. Individual stewards need to determine which are major vs. minor versions. May be used in conjunction with alternate identifier and related identifier to indicate various information updates. May be used in conjunction with description to indicate the nature and file/record range of version.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "rightslist", "label": "Rights List", "description": "Any rights information for this resource. Provide a rights management statement for the resource or reference a service providing such information. Include embargo information if applicable. Use the complete title of a license and include version information if applicable.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "descriptions", "label": "Descriptions", "description": "All additional information that does not fit in any of the other categories. May be used for technical information. It is a best practice to supply a description.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "geolocation", "label": "Geolocation", "description": "Spatial region or named place where the data was gathered or about which the data is focused.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "fundingreferences", "label": "Funding References", "description": "Information about financial support (funding) for the dataset being registered.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "relateditems", "label": "Related Item", "description": "Information about a dataset related to the one being registered e.g. a journal or book of which the article or chapter is part.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "type", "label": "Dataset type", "description": "Dataset type", "default": "", "optional": true, "repeatable": false, "vocabulary": [] } ] } ``` # DataCite Standard Format ```json { "title": "DataCite", "version": "4.4, Standard", "description": "DataCite Schema, version 4.4", "properties": [ { "field_name": "creators", "label": "Creators", "description": "The main researchers involved working on the data, or the authors of the publication in priority order. May be a corporate/institutional or personal name.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "title", "label": "Title", "description": "A name or title by which the dataset is known.", "default": "", "optional": "false", "repeatable": false, "vocabulary": [] }, { "field_name": "name", "label": "URL", "description": "", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "owner_org", "label": "Institute", "description": "", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "publisher", "label": "Publisher", "description": "The name of the entity that holds, archives, publishes prints, distributes, releases, issues, or produces the dataset. This property will be used to formulate the citation, so consider the prominence of the role.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "publicationyear", "label": "Publication Year", "description": "Year when the data is made publicly available. If an embargo period has been in effect, use the date when the embargo period ends. If there is no standard publication year value, use the date that would be preferred from a citation perspective.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "resourcetype", "label": "Resource Type", "description": "The type of a dataset. You may enter an additional free text description. The format is open, but the preferred format is a single term of some detail so that a pair can be formed with the sub-property.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "subject", "label": "Subject", "description": "Subject, keywords, classification codes, or key phrases describing the dataset.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "contributor", "label": "contributor", "description": "The institution or person responsible for collecting, creating, or otherwise contributing to the developement of the dataset.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "date", "label": "Date", "description": "Date relevant to the work.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "language", "label": "Language", "description": "Primary language of the resource.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "alternateidentifier", "label": "Alternate Identifier", "description": "An identifier or identifiers other than the primary Identifier applied to the dataset being registered. This may be any alphanumeric string which is unique within its domain of issue. May be used for local identifiers. AlternateIdentifier should be used for another identifier of the same instance (same location, same file).", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "relatedidentifier", "label": "Related Identifier", "description": "Identifiers of related datasets. Use this property to indicate subsets of properties, as appropriate.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "size", "label": "Size", "description": "Unstructured size information about the dataset.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "format", "label": "Format", "description": "Technical format of the dataset. Use file extension or MIME type where possible.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "version", "label": "Version", "description": "Version number of the dataset. If the primary dataset has changed the version number increases. Register a new identifier for a major version change. Individual stewards need to determine which are major vs. minor versions. May be used in conjunction with alternate identifier and related identifier to indicate various information updates. May be used in conjunction with description to indicate the nature and file/record range of version.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "rightslist", "label": "Rights List", "description": "Any rights information for this resource. Provide a rights management statement for the resource or reference a service providing such information. Include embargo information if applicable. Use the complete title of a license and include version information if applicable.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "description", "label": "Description", "description": "All additional information that does not fit in any of the other categories. May be used for technical information. It is a best practice to supply a description.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "geolocation", "label": "Geolocation", "description": "Spatial region or named place where the data was gathered or about which the data is focused.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "fundingreference", "label": "Funding Reference", "description": "Information about financial support (funding) for the dataset being registered.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "relateditem", "label": "Related Item", "description": "Information about a dataset related to the one being registered e.g. a journal or book of which the article or chapter is part.", "default": "", "optional": true, "repeatable": false, "vocabulary": [] }, { "field_name": "type", "label": "Dataset type", "description": "Dataset type", "default": "", "optional": true, "repeatable": false, "vocabulary": [] } ] } ``` # API interaction The API can be used via command-line tools or within scripts. Only datasets with the standard Datacite format can be managed via API. ## Authentication Unless a dataset is public, you will need an API token to be able to interact with it. In order to create one, you need to login into the WebUI of MetaStore and go to your account. Please follow these [instructions](api-tokens.md) before proceeding. ## CKAN Actions Irrespective of the tool you are going to use, you will need to use CKAN actions to achieve what you want to do. Some examples are `package_create`, `package_show`,`resource_update`. Look at the [official documentation](https://docs.ckan.org/en/latest/api/index.html#action-api-reference) for more information. **Note** For legacy reasons, datasets are still called packages in the action names. ### Datacite Metadata fields | Label | Name | Comment | | ------ | ------ | ------ | | Title | `title` | MetaStore will attribute the value of the field `name` when no title is given. | | Name | `name` | URL-friendly, **required by the API** | | UUID | `id` | Generated by MetaStore when not given | | Type |`type` | Must be `standard` for the standard Datacite Format | | Creators | `creators` | | | Organization | `owner_org` | MetaStore organization, **required by the API** | | Publisher | `publisher` | | | Publication Year | `publicationyear` | | | Resource Type | `resourcetype` | | | Subject | `subject` | | | Contributor | `contributor` | | | Date | `date` | | | Language | `language` | | | Alternate Identifier | `alternateidentifier` | | | Related Identifier | `relatedidentifier` | | | Size | `size` | | | Format | `format` | | | Version | `version` | | | Description | `description` | | | Geolocation | `geolocation` | | | Funding Reference | `fundingreference` | | | Related Item | `relateditem` | | | Licence | `rightslist` | | ## Tools ### ckanapi [ckanapi](https://github.com/ckan/ckanapi) is a command line interface and Python module for accessing the CKAN Action API. #### Datasets Listing ```bash ckanapi action package_list -r https://metastore.mpcdf.mpg.de --insecure ``` #### Dataset Creation ```bash ckanapi action package_create -r https://metastore.mpcdf.mpg.de name= owner_org=mpcdf type=standard -a ``` ### curl #### Dataset Creation ```bash curl -XPOST -H "Authorization: " https://metastore.mpcdf.mpg.de/api/3/action/package_create -d '{"name":"final-experiment", "title":"Final Experiment", "creators":"Nicolas Fabas, Thomas Zastrow", "type":"standard", "owner_org":"mpcdf", "publicationyear":"2023", "version":"4.2", "description":"This is our final experiment", "subject": "waves, particles"}' ``` #### Resource Creation ```bash curl -XPOST -H "Authorization: " https://metastore.mpcdf.mpg.de/api/3/action/resource_create -d '{"package_id":"570111cc-66df-4946-b291-5cb99dbd8045", "name":"resource.tar.gz", "format":"tar.gz", "url":"https://objectstore.hpccloud.mpcdf.mpg.de/testbucket/resource.tar.gz"}' ``` ### HTTPie and the CKAN Requestor [HTTPie](https://httpie.io/) is another HTTP client which is easier to use than curl. The [CKAN Requestor](https://gitlab.mpcdf.mpg.de/mpcdf/data-publishing/tools/ckan-requestor) is an MPCDF tool based on HTTPie. Its purpose is to automatically construct an HTTPie query. ```bash https -p hbHBm --verify=no ://metastore.mpcdf.mpg.de/api/3/action/package_create Authorization: name=final-experiment title="Final Experiment" creators="Nicolas Fabas, Thomas Zastrow" type=standard owner_org=mpcdf publicationyear=2023 version=4.2 description="This is our final experiment" subject="waves, particles" ``` With the CKAN Requestor: ```bash ./requestor.sh name=final-experiment title="Final Experiment" creators="Nicolas Fabas, Thomas Zastrow" type=standard owner_org=mpcdf publicationyear=2023 version=4.2 description="This is our final experiment" subject="waves, particles" ``` If you already created a metadata file with the MMD tools, you can simply use it with the CKAN Requestor to publish the dataset like so: ```bash ./requestor.sh < metadata.mmd ``` # List of actions - [Create a Dataset](webui-dataset-create.md) - [Update a Dataset](webui-dataset-update.md) - [Delete a Dataset](webui-dataset-delete.md) First, you need to be logged in on MetaStore. To do so, please click on 'Log In' in the upper right corner of the homepage. ![](../images/metastore-webui-login-1.png) Fill in your user name and password and click 'Login' ![](../images/metastore-webui-login-2.png) In this example, we are going to create a standard dataset. Please click on 'Datasets Standards'. ![](../images/metastore-webui-dataset-create-1.png) You are now seeing a list of standard datasets. Please click on 'Add standard dataset' to create a new one. ![](../images/metastore-webui-dataset-create-3.png) Fill in the form. You cannot create the dataset as long as you have not completed the required fields. ![](../images/metastore-webui-dataset-create-4.png) When you are done, click on 'Next: Add data' ![](../images/metastore-webui-dataset-create-5.png) Now, you need to create at least one resource for the dataset. Fill in at least the required fields, and then click finish. ![](../images/metastore-webui-dataset-create-6.png) When this is done, you can see a view of the created dataset and resources. ![](../images/metastore-webui-dataset-create-7.png) you can delete a dataset in metastore, but if there is already a published doi, the doi will point to an empty page. on the dataset page, click manage on the upper right ![](../images/metastore-webui-dataset-delete-1.png) Click delete at the end of the form ![](../images/metastore-webui-dataset-delete-2.png) Click confirm ![](../images/metastore-webui-dataset-delete-3.png) Now your dataset is deleted ![](../images/metastore-webui-dataset-delete-4.png) updating a dataset entails the following activities: - Adding, updating or deleting metadata of the dataset and resources - Adding, removing or reordering resources If you already have submitted a DOI, you cannot change it. To update a dataset of your choice, click on it on the dataset list. Then click on 'Manage' ![](../images/metastore-webui-dataset-update-1.png) When you are done, click on update dataset ![](../images/metastore-webui-dataset-update-2.png) Now, if you want to update your resources, click on the 'Resources' tab on the dataset update page. ![](../images/metastore-webui-dataset-update-3.png) Now you can either add or reorder the resources of the dataset, or modify already existing ones. We will do the latter. Click on the resource you want to modify. ![](../images/metastore-webui-dataset-update-4.png) When you are done with modifying the metadata of the resource, click on update resource. ![](../images/metastore-webui-dataset-update-5.png) Now you get a summary view of the updated resource. ![](../images/metastore-webui-dataset-update-6.png) # Using an SSH Config File Simplify your SSH workflow by defining connection parameters in a configuration file at `~/.ssh/config`. These settings are also used by other commands like `scp` and `sftp`. ## Host Aliases Create an alias for a remote host using the `Host` directive to specify connection parameters like the hostname, username, and identity file. For example, instead of typing: ```bash ssh -i /home/user/.ssh/my-key.pem YOUR_USERNAME@webserver.mpcdf.mpg.de ``` Add the following to your `~/.ssh/config` file: ``` Host webserver Hostname webserver.mpcdf.mpg.de User YOUR_USERNAME IdentityFile /home/user/.ssh/my-key.pem ``` Now, connect with the much simpler command: ```bash ssh webserver ``` ## Wildcards Use wildcards to apply settings to multiple hosts. For example, to set a default username for all MPCDF systems: ``` Host *.mpcdf.mpg.de User YOUR_USERNAME ``` **Note:** Wildcards are not supported in the Windows `ssh` client. ## Including Other Config Files Since OpenSSH 7.3, you can include other configuration files using the `Include` directive. This is useful for organizing your configurations. For example: ``` Include config.d/*.conf Include config.d/cloud/*.conf ``` This includes all `.conf` files from the `.ssh/config.d/` and `.ssh/config.d/cloud/` directories. Shell access to the clusters from outside of the campus using a Windows machine ======================================================================================= 1. Download the putty.exe (the SSH client itself) from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html 2. Run the downloaded putty.exe file. It will open the PuTTY Configuration window. In the field for the “Host Name” enter the URL for one of the `gateway machines `_. .. image:: /_images/putty-01-config.png :scale: 50 % :align: center 3. Choose the Connection > SSH category from the left list and enter "ssh" + the URL for the cluster you want to access e.g. "ssh cobra.mpcdf.mpg.de" (optional) Choose the "Enable compression", and "Share SSH connections if possible" options. .. image:: /_images/putty-02-cluster.png :scale: 50 % :align: center 4. (optional) Choose the Connection > SSH category in the left list and enter your username in the “Auto-login username” field. .. image:: /_images/putty-03-username.png :scale: 50 % :align: center 5. Go back to the Session category (in the left list). Enter a name for this configuration in the “Saved Sessions” field and click on the “Save” button. From now on, each time you want to connect to the cluster, you just have to run the PuTTY, choose you saved session name from the list and click the “Load” button. All the field will be automatically populated with the saved values. .. image:: /_images/putty-04-save.png :scale: 50 % :align: center 6. Click “Open” button. First time you connect to the gateserver, PuTTY will show you the server's host key and will ask you to confirm it. You can find the host key of our gate machines `here `_. .. image:: /_images/putty-05-gate_key.png :scale: 50 % :align: center 7. After confirming the server key, a connection will be made to the gateway machine and you will be asked for your credentials. .. image:: /_images/putty-06-pass.png :scale: 50 % :align: center 8. Then you need to provide your `2FA token `_ for connecting to the gateserver. .. image:: /_images/putty-07-2FA.png :scale: 50 % :align: center 9. If this is the first time you are connecting to the login node of a cluster from the gate machine, the server’s host key information will be shown. Type “yes” and press Enter to proceed. .. image:: /_images/putty-08-server_key.png :scale: 50 % :align: center 10. A connection will be made through the gate to your requested cluster's login node (added in step 3). You will be asked for the password for this connection. .. image:: /_images/putty-09-server_pass.png :scale: 50 % :align: center 11. You should now see the welcome message on the login node of the cluster. .. image:: /_images/putty-10-shell.png :scale: 50 % :align: center Accessing your files on the clusters from outside of the campus using a Windows machine ======================================================================================= 1. Download the WinSCP from https://winscp.net/eng/downloads.php a. If you have downloaded the portable version (WinSCP-5.*-Portable.zip), just extract the compressed zip file. b. If you have downloaded the WinSCP installer (WinSCP-5.*-Setup.exe), run the installer and follow the steps. Check out the WinSCP installation guide for more detailed info: https://winscp.net/eng/docs/guide_install 2. Run the WinSCP.exe which you extracted/installed in step 1. 3. Click on “Advanced…” button on the login window. .. image:: /_images/winscp-01-welcome.png :scale: 50 % :align: center 4. Go to the connection > tunnel section, click on the “Connect through SSH tunnel” box, enter the URL for one of the `gateway machines `_ in the field for the tunnel host name, enter your username, and click OK. .. image:: /_images/winscp-02-tunnel.png :scale: 50 % :align: center 5. Back in the login window, enter the URL for the cluster you want to access e.g. raven.mpcdf.mpg.de, enter your username and click save button. .. image:: /_images/winscp-03-login.png :scale: 50 % :align: center 6. Choose a name for this session configuration in the “Site name” field, and click OK. .. image:: /_images/winscp-04-save.png :scale: 50 % :align: center 7. Back in the login window, your chosen site name will appear in the list. Choose this site and click on the Login button. .. image:: /_images/winscp-05-start.png :scale: 50 % :align: center 8. You will be connected to the gateway machine. If this is the first time you are connecting to the server, the server’s host key information will be shown. You can find the host key of our gate machines `here `_. Click Yes to proceed. .. image:: /_images/winscp-06-serverkey.png :scale: 50 % :align: center 9. Afterwards, you will be asked for your password. .. image:: /_images/winscp-07-username.png :scale: 50 % :align: center 10. You also need to provide your `2FA token `_ for connecting to the gate machines. .. image:: /_images/winscp-08-2FA.png :scale: 50 % :align: center 11. Another connection will be made through the gateway machine to the login node of the cluster you have chosen in step 5. You will have to enter your password again. (steps 8-9) 12. Finally, in the main window you will see the files on the cluster in the right panel, and the files on your computer in the left one. You can drag and drop files to copy them from the cluster to your computer and vice versa. .. image:: /_images/winscp-09-end.png :scale: 50 % :align: center