KIT | KIT-Bibliothek | Impressum | Datenschutz

Data Locality via Coordinated Caching for Distributed Processing

Fischer, M. ORCID iD icon 1; Kuehn, E. ORCID iD icon 1; Gffels, M.; Jung, C. 1
1 Steinbuch Centre for Computing (SCC), Karlsruher Institut für Technologie (KIT)


To enable data locality, we have developed an approach of adding coordinated caches to existing compute clusters. Since the data stored locally is volatile and selected dynamically, only a fraction of local storage space is required. Our approach allows to freely select the degree at which data locality is provided. It may be used to work in conjunction with large network bandwidths, providing only highly used data to reduce peak loads. Alternatively, local storage may be scaled up to perform data analysis even with low network bandwidth. To prove the applicability of our approach, we have developed a prototype implementing all required functionality. It integrates seamlessly into batch systems, requiring practically no adjustments by users. We have now been actively using this prototype on a test cluster for HEP analyses. Specifically, it has been integral to our jet energy calibration analyses for CMS during run 2. The system has proven to be easily usable, while providing substantial performance improvements.
Since confirming the applicability for our use case, we have investigated the design in a more general way. Simulations show that many infrastructure setups can benefit from our approach. ... mehr

Volltext §
DOI: 10.5445/IR/1000063493
DOI: 10.1088/1742-6596/762/1/012011
Zitationen: 1
Zitationen: 2
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Kernphysik (IKP)
Steinbuch Centre for Computing (SCC)
Universität Karlsruhe (TH) – Zentrale Einrichtungen (Zentrale Einrichtungen)
Publikationstyp Zeitschriftenaufsatz
Publikationsjahr 2016
Sprache Englisch
Identifikator ISSN: 1742-6588, 1742-6596
KITopen-ID: 1000063493
HGF-Programm 51.01.01 (POF III, LK 01) Teilchenphysik
Erschienen in Journal of physics / Conference Series
Verlag Institute of Physics Publishing Ltd (IOP Publishing Ltd)
Band 762
Seiten 012011
Nachgewiesen in Dimensions
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page