KIT | KIT-Bibliothek | Impressum | Datenschutz

Tier 3 batch system data locality via managed caches

Fischer, M. ORCID iD icon; Giffels, M. ORCID iD icon; Jung, C.; Kühn, E. ORCID iD icon; Quast, G. ORCID iD icon

Abstract:

Modern data processing increasingly relies on data locality for performance and scalability, whereas the common HEP approaches aim for uniform resource pools with minimal locality, recently even across site boundaries. To combine advantages of both, the High-Performance Data Analysis (HPDA) Tier 3 concept opportunistically establishes data locality via coordinated caches. In accordance with HEP Tier 3 activities, the design incorporates two major assumptions: First, only a fraction of data is accessed regularly and thus the deciding factor for overall throughput. Second, data access may fallback to non-local, making permanent local data availability an ineficient resource usage strategy. Based on this, the HPDA design generically extends available storage hierarchies into the batch system. Using the batch system itself for scheduling file locality, an array of independent caches on the worker nodes is dynamically populated with high-profile data. Cache state information is exposed to the batch system both for managing caches and scheduling jobs. As a result, users directly work with a regular, adequately sized storage system. However, their automated batch processes are presented with
... mehr


Verlagsausgabe §
DOI: 10.5445/IR/110103470
Veröffentlicht am 05.08.2019
Originalveröffentlichung
DOI: 10.1088/1742-6596/608/1/012018
Scopus
Zitationen: 4
Dimensions
Zitationen: 4
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Kernphysik (IKP)
Scientific Computing Center (SCC)
Universität Karlsruhe (TH) – Zentrale Einrichtungen (Zentrale Einrichtungen)
Publikationstyp Zeitschriftenaufsatz
Publikationsjahr 2015
Sprache Englisch
Identifikator ISSN: 1742-6588, 1742-6596
KITopen-ID: 110103470
HGF-Programm 53.52.02 (POF III, LK 02) GridKa
Erschienen in Journal of physics / Conference Series
Verlag Institute of Physics Publishing Ltd (IOP Publishing Ltd)
Band 608
Heft 1
Seiten Art.Nr. 012018/1-5
Nachgewiesen in Dimensions
Scopus
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page