GO-Nexus
GO-Nexus combines two of MPCDF’s core data solutions, Nexus-Posix and Globus, to enable projects to benefit from permanent online storage which offers relaiable transfer and sharing functionality as well as the ability to create data pipelines to automate data management.
Nexus-Posix is an IBM Spectrum Scale filesystem which is commonly used by projects in MPCDF’s HPC-Cloud. Projects can rent a reservation on Nexus-Posix which can be scaled up as the project grows. The reservations are generally in the range of 10-100 TB and are accessed via mount points on the HPC system Raven and/or HPC-Cloud VMs.
Globus provides a fast, reliable and user-friendly way to transfer or share large amounts of data. Additionally, Globus can aid projects in publishing findable data for their communities. These qualities make Globus an ideal service to enable access to the large-scale data stored in Nexus-Posix. Combining these two services provides a solution for several core use cases and opens extra possibilities for projects which make use of Nexus-Posix.
Two of the primary use cases are highlighted in Fig. 1, namely:
Transfer and Sharing service with mount points on Raven and possibly HPC-Cloud VMs.
Standalone Transfer and Sharing service, for data collection, publishing and sharing.
The first use case highlights how projects can expose the Nexus-Posix filesystem mounted on the Raven HPC system and/or HPC-Cloud VMs, with users possibly performing large-scale simulations at MPCDF and then transferring results back to their home institute or even sharing them with colleagues in world-wide collaborations.
The second use case shows how standalone storage can be made globally available via GO-Nexus. This could be used when gathering data in the field for processing at a later date and/or for distributed collaborations where GO-Nexus would act as a central datastore, benefiting from the high-speed network connection at MPCDF.
In both cases the reservations can be exposed either as findable or as private data collections via Globus, where users and community members can easily search for the data via the Globus web portal.
In addition to the reliable transfer capabilities GO-Nexus benefits from all the advanced functionality which is available via the MPCDF’s globus subscription, enabling actions such as sharing and the use of Globus flows for automation. Several cloud projects have already adopted GO-Nexus for large-scale data transfers and to regularly sync data to and from Nexus-Posix by using “Globus timers”, a cron like service offered through the Globus web portal.