Building a Data Transfer Federation between Research Centers

Cross-site collaboration between research centers is becoming increasingly crucial, with researchers accessing storage or computing resources from other institutions. However, this collaboration results in the need for large data transfers between different storage systems for computation or archival purposes, which can be a challenging and time-consuming task. To address this need, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) designed a data transfer federation in the context of NFDI4Ing project, base services-S4:repositories & storage. A data transfer federation is a collaboration of storage providers across multiple organizations, to enable access to data with federated identities and automated, scalable data access and data transfers between the storage systems. Our proposed federation allows researchers to access and transfer data between different storage systems using their home organization's user account, based on their access rights to the resources in a collaborative project. The storage systems in a federation are either dedicated large-scale systems or systems associated with High-Performance Computing (HPC). ... mehrAt SCC, we have integrated our Large Scale Data Facility: Online Storage (LSDF OS) [1] with WebDAV [2] protocol and OAuth2 [3] authentication to enable access of third-party applications to the storage service. Beyond improved support for programmatic access using OAuth2, the WebDAV server enables users to inspect the LSDF OS filesystem via their web browser. For enabling data movement, we have deployed an instance of the File Transfer Service (FTS) [4] on-premises and integrated it with our identity provider (IdP). FTS is a low-level data management service that orchestrates reliable bulk transfer of files from one storage endpoint to another. It is an open-source software developed by CERN that distributes the majority of the Large Hadron Collider (LHC) data across the Worldwide LHC Computing Grid (WLCG) infrastructure. However, FTS is designed to be used with one identity provider, whereas in a federation, more than one IdP is involved. To address this limitation, we have designed a central IdP to issue tokens that are recognizable by the downstream identity providers which manage users’ access to their corresponding storage service in the federation. At each storage service, tokens issued by the central IdP are resolved to the corresponding user identity at the local IdP via a mapping policy. To achieve this, a unified approach in the federation regarding the information included in the token is necessary. In conclusion, the data transfer federation enables users to have seamless access to a wider range of resources, thereby simplifying collaboration between research centers. [1] https://www.scc.kit.edu/forschung/11843.php [2] https://www.ietf.org/rfc/rfc4918.txt [3] https://datatracker.ietf.org/doc/html/rfc6749 [4] A. A. Ayllon et al, FTS3: New Data Movement Service For WLCG, J. Phys.: Conf. Ser. 513 032081 (2014)