Nexus-S3

Nexus-S3 is a scaleable object storage service compatible with the Amazon S3 protocol.

MPCDF users can opt-in to Nexus-S3 which provide a free 1TB (1M objects) quota (see opt-in below). Data can be accessed using standard S3 clients and libraries such as minio, s3cmd, rclone and python-boto3 as well as via Globus (MPCDF GO Nexus S3 Collection) or via a web browser/curl.

Nexus-S3 also supports object storage functionality such as versioning, life-cycle policies and temporary URL generation to allow users to download files with an expiry date. Together with the transfer and sharing functionality available via Globus this provides many solutions to use-cases such as large scale data sharing and publishing.

Opt-in via SelfService

Access to Object Storage is possible via the MPCDF SelfService. Log in with your MPCDF account and go to “My account / Services” to opt-in for Nexus-S3.

Once the account has been created in the S3 service you can access your access/secret keys by clicking “View Access Token”. These access/secret keys are used by S3 clients and Globus to access you S3 storage, please keep them safe and treat them as you would a password. If you feel these keys may have been exposed please create a helpdesk ticket and request that new keys be generated.

NOTE: After opt-in, is can take up to 60 minutes until the accounts are created and for the access/secret keys to become available.

Accessing Object Storage

Nexus-S3 is globally available and can be accessed from MPCDF and external servers. To access using S3 clients such as minio, s3cmd or rclone simply copy the access and secret keys from selfservice and configure your client to use these with the service host/endpoint set to s3.nexus.mpcdf.mpg.de.

Access via S3

S3cmd: S3cmd is one of the most popular command line tools for accessing S3 based object-storage.

To configure s3cmd run:

s3mcd --configure

Many parameters can be left as default. However, the following need to be set to match the MPCDF systems.

Your access and secret keys (please ensure these stay private).

Access Key: *****************
Secret Key: *****************

Set the S3 endpoint to the endpoint name of the MPCDF storage you are using.

S3 Endpoint [s3.amazonaws.com]: s3.nexus.mpcdf.mpg.de

Object storage at MPCDF does not support DNS style (or virtual bucket) names so set none here

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: none

Once configuration is complete the config file is stored in ~/.s3cfg. This can be viewed and edited as any normal txt file.

S3cmd can be used to manage data in the object storage. See man s3cmd or s3cmd -h for more info.

A simple example set of commands follows, make a bucket, copy data to the bucket, list bucket contents and then query the bucket for info such as access policies :

s3cmd mb s3://<my-bucket>
s3cmd put testfile s3://<my-bucket>/
s3cmd ls s3://<my-bucket>
s3cmd info s3://<my-bucket>

Access via Globus:

  1. Log in to the Globus Web UI

  2. Open the collections tab

  3. Search for the collection: “MPCDF GO Nexus S3 Collection”

  4. Click on the collection (see screenshot for the collection info) Nexus S3 Collection

  5. Click on Credentials

  6. Follow the steps to authenticate and provide consent

  7. Copy and paste your access/secret keys into the relevant fields

  8. Nexus-S3 Objects can then be accessed via the File Manager view as usual

Note: When using Globus the keys will be encrypted on the Globus server at MPCDF and this encrypted form of the keys will be stored in the Globus Services in the Cloud (no decryption keys are stored in the Cloud).

Some words of Caution

Please be aware that the data stored in the Object Storage service is not backed up. Additionally, care should be taken when creating and managing buckets to ensure that access rights are correctly set. When setting buckets as public some S3 clients will set read+write access by default. Please be careful. Publicly accessible buckets will quickly be discovered on the internet and may be abused if writeable.