KIT | KIT-Bibliothek | Impressum | Datenschutz

Boosting Performance of Data-intensive Analysis Workflows with Distributed Coordinated Caching

Heidecker, C.; Von Cube, R. F.; Giffels, M.; Quast, G.; Sauter, M.; Schnepf, M. J.

Data-intensive end-user analyses in high energy physics require high data throughput to reach short turnaround cycles. This leads to enormous challenges for storage and network infrastructure, especially when facing the tremendously increasing amount of data to be processed during High-Luminosity LHC runs. Including opportunistic resources with volatile storage systems into the traditional HEP computing facilities makes this situation more complex.
Bringing data close to the computing units is a promising approach to solve throughput limitations and improve the overall performance. We focus on coordinated distributed caching by coordinating workows to the most suitable hosts in terms of cached files. This allows optimizing overall processing efficiency of data-intensive workows and efficiently use limited cache volume by reducing replication of data on distributed caches.
We developed a NaviX coordination service at KIT that realizes coordinated distributed caching using XRootD cache proxy server infrastructure and HTCondor batch system. In this paper, we present the experience gained in operating coordinated distributed caches on cloud and HPC resources. ... mehr

Open Access Logo

Verlagsausgabe §
DOI: 10.5445/IR/1000123017
Veröffentlicht am 12.09.2020
DOI: 10.1088/1742-6596/1525/1/012065
Cover der Publikation
Zugehörige Institution(en) am KIT Fakultät für Mathematik (MATH)
Institut für Mechanische Verfahrenstechnik und Mechanik (MVM)
Institut für Theoretische Teilchenphysik (TTP)
Steinbuch Centre for Computing (SCC)
House of Competence (HoC)
Institut für Experimentelle Teilchenphysik (ETP)
KIT-Zentrum Elementarteilchen- und Astroteilchenphysik (KCETA)
Publikationstyp Zeitschriftenaufsatz
Publikationsjahr 2020
Sprache Englisch
Identifikator ISSN: 1742-6588, 1742-6596
KITopen-ID: 1000123017
Erschienen in Journal of physics / Conference series
Band 1525
Heft 1
Seiten Art. Nr.: 012065
Bemerkung zur Veröffentlichung 19th International Workshop on Advanced Computing and Analysis Techniques in Physics Research, ACAT 2019; Steinmatte Conference CentreSaas-Fee; Switzerland; 11 March 2019 through 15 March 2019
Nachgewiesen in Scopus
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page