KIT | KIT-Bibliothek | Impressum | Datenschutz

Boosting Performance of Data-intensive Analysis Workflows with Distributed Coordinated Caching

Heidecker, C. 1; Cube, R. F. von 1; Giffels, M. ORCID iD icon 1; Quast, G. 1; Sauter, M. 1; Schnepf, M. J. 1
1 Karlsruher Institut für Technologie (KIT)


Data-intensive end-user analyses in high energy physics require high data throughput to reach short turnaround cycles. This leads to enormous challenges for storage and network infrastructure, especially when facing the tremendously increasing amount of data to be processed during High-Luminosity LHC runs. Including opportunistic resources with volatile storage systems into the traditional HEP computing facilities makes this situation more complex.
Bringing data close to the computing units is a promising approach to solve throughput limitations and improve the overall performance. We focus on coordinated distributed caching by coordinating workows to the most suitable hosts in terms of cached files. This allows optimizing overall processing efficiency of data-intensive workows and efficiently use limited cache volume by reducing replication of data on distributed caches.
We developed a NaviX coordination service at KIT that realizes coordinated distributed caching using XRootD cache proxy server infrastructure and HTCondor batch system. In this paper, we present the experience gained in operating coordinated distributed caches on cloud and HPC resources. ... mehr

Verlagsausgabe §
DOI: 10.5445/IR/1000123017
Veröffentlicht am 12.09.2020
DOI: 10.1088/1742-6596/1525/1/012065
Zitationen: 1
Cover der Publikation
Zugehörige Institution(en) am KIT House of Competence (HoC)
Institut für Experimentelle Teilchenphysik (ETP)
Institut für Mechanische Verfahrenstechnik und Mechanik (MVM)
Universität Karlsruhe (TH) – Interfakultative Einrichtungen (Interfakultative Einrichtungen)
KIT-Zentrum Elementarteilchen- und Astroteilchenphysik (KCETA)
Steinbuch Centre for Computing (SCC)
Universität Karlsruhe (TH) – Zentrale Einrichtungen (Zentrale Einrichtungen)
Publikationstyp Zeitschriftenaufsatz
Publikationsjahr 2020
Sprache Englisch
Identifikator ISSN: 1742-6588, 1742-6596
KITopen-ID: 1000123017
Erschienen in Journal of physics / Conference series
Verlag Institute of Physics Publishing Ltd (IOP Publishing Ltd)
Band 1525
Heft 1
Seiten Art. Nr.: 012065
Bemerkung zur Veröffentlichung 19th International Workshop on Advanced Computing and Analysis Techniques in Physics Research, ACAT 2019; Steinmatte Conference CentreSaas-Fee; Switzerland; 11 March 2019 through 15 March 2019
Nachgewiesen in Scopus
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page