Building a Data Transfer Federation

The Data Transfer Federation allows researchers to access and transfer data between different storage systems using their home organization's user account, based on their access rights to the resources in a collaborative project. The storage systems in a federation are either dedicated large-scale systems or systems associated with High-Performance Computing (HPC). At SCC, we have integrated our Large Scale Data Facility: Online Storage (LSDF OS) [1] with WebDAV [2] protocol and OAuth2 [3] authentication to enable access of third-party applications to the storage service. Beyond improved support for programmatic access using OAuth2, the WebDAV server enables users to inspect the LSDF OS filesystem via their web browser. For enabling data movement, we have deployed an instance of the File Transfer Service (FTS) [4] on-premises and integrated it with our identity provider (IdP). FTS is a low-level data management service that orchestrates reliable bulk transfer of files from one storage endpoint to another. It is an open-source software developed by CERN that distributes the majority of the Large Hadron Collider (LHC) data across the Worldwide LHC Computing Grid (WLCG) infrastructure. ... mehrHowever, FTS is designed to be used with one identity provider, whereas in a federation, more than one IdP is involved. To address this limitation, we have designed a central IdP to issue tokens that are recognizable by the downstream identity providers which manage users’ access to their corresponding storage service in the federation. At each storage service, tokens issued by the central IdP are resolved to the corresponding user identity at the local IdP via a mapping policy. To achieve this, a unified approach in the federation regarding the information included in the token is necessary. In conclusion, the data transfer federation enables users to have seamless access to a wider range of resources, thereby simplifying collaboration between research centers. [1] https://www.scc.kit.edu/forschung/11843.php [2] https://www.ietf.org/rfc/rfc4918.txt [3] https://datatracker.ietf.org/doc/html/rfc6749 [4] A. A. Ayllon et al, FTS3: New Data Movement Service For WLCG, J. Phys.: Conf. Ser. 513 032081 (2014)