Important

Interface change

We will rename the volume types as follows: Ceph to standard, CephBulk to bulk, CephIntensive to fast, and CephSSD to legacy

Storage

The HPC Cloud offers both block storage, in the form of disk volumes which can be directly attached to a server, and file storage, in the form of shared filesystems which can be NFS-mounted by the operating system. In addition, there is an object store providing containers (also called buckets) which allow data to be accessed from multiple clients though standard REST APIs. 1 The block and object storage services are based on Ceph, while file storage offers a choice between CephFS and IBM Storage Scale (GPFS). The latter is also mounted on Raven (by default) as well as Robin (on request) for efficient exchange of data between the various systems.

Block

Storage volumes are logically independent block devices of arbitrary size. They can be attached to a running server and later detached or even reattached to a different server. 2 Typically, volume are used to store data which should persist beyond the lifetime of a single instance, although they are also useful as “scratch” space.

1. Create an empty volume via the button on Project / Volumes / Volumes. The maxiumum allowable size depends on your project’s quota. The default volume type Ceph stores data on an HDD-based pool. There is also type CephSSD, available on request, which targets an SSD-based pool, offering potentially higher performance and lower latency.

   

2. Select a server from Project / Compute / Instances, perform the action, and pick the newly created volume.

3. The device names /dev/vdb of volumes are not guaranteed to be the same each time they are attached to a server. To definitively identify a volume look into /dev/disks/by-id. Volumes will appear as entries virtio-ID where ID is the first 20 characters of the OpenStack volume ID. For example:

     openstack volume list --name VOLUME
     +--------------------------------------+--------+-----------+------+-------------+
     | ID                                   | Name   | Status    | Size | Attached to |
     +--------------------------------------+--------+-----------+------+-------------+
     | 708e61cf-b80a-4488-9a32-eb532715ce38 | VOLUME | available |   10 |             |
     +--------------------------------------+--------+-----------+------+-------------+
   
   Will appear as */dev/disk/by-id/virtio-708e61cf-b80a-4488-9* on the server.
   It is also possible to look up the device name */dev/vdb* from
   `here <https://hpccloud.mpcdf.mpg.de/dashboard/project/volumes/>`_ in the
   "attached to" column).
   

Important

We strongly advice to use the disk ID to positively identify the block device on the server with an OpenStack volume

Create and mount a new filesystem using the following example commands:

mkfs.xfs /dev/disk/by-id/virtio-708e61cf-b80a-4488-9
mkdir /demovol
echo "/dev/disk/by-id/virtio-708e61cf-b80a-4488-9 /demovol xfs defaults 0 0" >> /etc/fstab
mount /demovol

CLI Example

openstack volume create VOLUME --size SIZE [--type TYPE]
openstack server add volume SERVER VOLUME

Tip

It is possible to migrate an existing volume to a new type online, even while still attached to a the server. Simply click the “Change Volume Type” action, choose the target type, and set the migration policy to “On Demand”. The data will then be transparently copied to the new backend storage pool.

openstack volume set VOLUME --type TYPE --retype-policy on-demand

Shared Filesystems (Shares)

Projects can deploy various parallel filesystem within the HPC-cloud. The table below provides some rough guidelines to decide which type of shared filesystem suits your use-case best.

Service	FileSystem	User management	Additional mounts	Strengths	Limitations
NexusPOSIX	NFSv4	MPCDF LDAP	HPCs, Clusters	Performance & Backup	Avg File > 16MB
Manila	NFSv4	project-managed	no	Simplicity	Limited performance
Manila	CephFS	project-managed	no	Performance	Client maintanace
Own NFS	NFSv3/4	project-managed	project-managed	Flexibility	Single server

NexusPOSIX

NexusPOSIX is based on IBM Spectrum Scale, a high performance parallel filesystem. It can be mounted on both HPC Cloud VMs and the HPC systems (Raven & Viper). This cross mounting allows projects to easily access data from both cloud and HPC systems, allowing for hybrid, HPC/Cloud, solutions.

Highlights of NexusPOSIX include:

• May be mounted on Raven HPC system and multiple HPC Cloud VMs

• Data Security - Automatic backup included

• Possible high performance data transfers and sharing via Globus (GO-Nexus)

• Reservations can be increased as projects grow

NexusPOSIX reservations can be requested by projects starting at 5TB and ranging into 100s of TBs. For more information, please make a request via the MPCDF helpdesk.

Because NexusPOSIX is natively mounted on our HPC systems you must use MPCDF UIDs and GIDs in your user management. The filesystem is designed to house large files, meaning your files must be larger than 16MB on average.

Manila

Manila is the HPC Cloud service providing Shares, i.e. shared filesystems. The service is backed by the our cloud Ceph cluster. Access to and from this cluster is limted to the HPC Cloud, therefor no additional mounts can be provided.

As project administeator you have root access to the filesystems provided by the HPC Cloud, and are free to set up user management as suits your project.

These flexible shared filesystems can be created in two configurations:

• NFS: This type of share is a single-threaded export that may be mounted as an NFS share, limiting the number of requests that can be handled. Best suited for shared configuration files.

• CephFS: This type of share is a direct connection to the HPC Cloud Ceph cluster providing increased performance. However the client instances must use a ceph client version compatible with our Ceph servers (v17.2.8, v18.2.4, v19, or greater).

Manila NFS

The Manila client is not included with the standard openstack clients. To get started ensure you have the manila client installed, for example:

pip install --user python-manilaclient

Now you are ready to create a share

openstack share create NFS $SIZE --share-type CephNFS --name $SHARE
openstack share show $SHARE

Note the $SUBDIR from export_locations, i.e. the part coming after /volumes/_nogroup/. You will need it to mount the file system later.

Make sure your client server(s) are connected to the manila-nfs network. If not add the required network connection:

openstack server add network $SERVER manila-nfs

Note the IP of the client server on the manila-nfs network. Access is managed through a whitelist of client IPs:

openstack share access create $SHARE ip $CLIENT_IP

Now ssh into your client server and mount the share:

apt install -y nfs-common
echo "manila-nfs.hpccloud.mpcdf.mpg.de:/volumes/_nogroup/$SUBDIR /share nfs _netdev 0 0" >> /etc/fstab
systemctl daemon-reload
mkdir /share && mount /share

Manila CephFS

The Manila client is not included with the standard openstack clients. To get started ensure you have the manila client installed, for example:

pip install --user python-manilaclient

Now create the new share:

openstack share create CEPHFS $SIZE --share-type CephNative --name $SHARE
openstack share show $SHARE

Note the $SUBDIR from export_locations, i.e. the part coming after /volumes/_nogroup/. You will need it to mount the file system later.

Make sure your client server(s) are connected to the manila-cephfs network. If not add the required network connection:

openstack server add network $SERVER manila-cephfs

With CephFS access is managed through access keys. Create Key for the client:

openstack share access create $SHARE cephx $CEPHUSER

Wait a moment while the credentials are generated and made available, then

openstack share access list $SHARE

Note the access key, which you will need to mount the file system on the client machine. Now ssh into your client machine, there do:

apt install -y ceph-common
cat >> /etc/ceph/keyring <<EOF
[client.$CEPHUSER]
    key = $ACCESS_KEY
EOF
chmod 600 /etc/ceph/keyring
echo "manila-cephfs.hpccloud.mpcdf.mpg.de:3300:/volumes/_nogroup/$SUBDIR /share ceph name=$CEPHUSER,ms_mode=secure,_netdev 0 0" >> /etc/fstab
systemctl daemon-reload
mkdir /share && mount /share

Object

The HPC Cloud includes an object store for saving and retrieving data through a publicly-accessible REST API. Both OpenStack Swift- and S3-style APIs are supported. Objects are stored in containers (or buckets) which in turn belong to the project (or tenant). Folders within buckets are supported but typically handled as a part of the object name.

Important

Please be aware that the contents of public buckets are not only visible but can be modified by anyone on the internet.

1. Create a new bucket via the button on Project / Object Store / Containers. Note that buckets, both public and private, share a global namespace. If a bucket name is already taken by another project you will receive an error message. Please use the S3 bucket naming rules. This will help avoid possible problems when accessing objects via the S3 API.

   

2. Select the bucket and click the button to upload a file. The object name (confusingly labeled “File Name”) defaults to the filename, but may be an arbitrary string. You may create a folder for the object at the same time by prepending one or more names, separated by “/”.

   

3. If you created a public bucket, the contents of the file would then be available at, e.g.: https://objectstore.hpccloud.mpcdf.mpg.de/swift/v1/demobucket/demofolder/demoobject or https://objectstore.hpccloud.mpcdf.mpg.de/demobucket/demofolder/demoobject

CLI Example

openstack container create BUCKET
openstack object create BUCKET FILE --name FOLDER/OBJECT

Object store quotas are separate from those of other storage types, and are not displayed on the dashboard. For questions about your quota, please contact the helpdesk. Note that while the Ceph backend itself supports very large objects, uploads through the dashboard and Swift API are limited to 5GB. 3 Use the S3 API to avoid this limitation.

Tip

The object store supports a subset of the Amazon S3 API. See CLI and Scripting to get started with s3cmd and/or Boto3.

1

Note that the term container is unrelated to Docker containers. Where possible we use bucket to avoid confusion.

2

Attaching a volume to multiple servers simultaneously is not supported.

3

The openstack command also uses the Swift API and is therefore subject to the same limitation. Note that the --segment-size feature of the older swift command splits the file into a collection of smaller objects which cannot easily be downloaded by other clients.