KIT | KIT-Bibliothek | Impressum | Datenschutz

Building a parallel pipelined external memory algorithm library

Beckmann, Andreas; Dementiev, Roman; Singler, Johannes

Large and fast hard disks for little money have enabled the processing of huge amounts of data on a single machine. For this purpose, the well-established STXXL library provides a framework for external memory algorithms with an easy-to-use interface. However, the clock speed of processors cannot keep up with the increasing bandwidth of parallel disks, making many algorithms actually compute-bound. To overcome this steadily worsening limitation, we exploit today's multi-core processors with two new approaches. First, we parallelize the internal computation of the encapsulated external memory algorithms by utilizing the MCSTL library. Second, we augment the unique pipelining feature of the STXXL, to enable automatic task parallelization. We show using synthetic and practical use cases that the combination of both techniques increases performance greatly.

DOI: 10.1109/IPDPS.2009.5161001
Zitationen: 15
Zugehörige Institution(en) am KIT Institut für Theoretische Informatik (ITI)
Publikationstyp Proceedingsbeitrag
Publikationsmonat/-jahr 05.2009
Sprache Englisch
Identifikator ISBN: 978-1-4244-3751-1
KITopen-ID: 1000097654
Erschienen in 23rd IEEE International Parallel and Distributed Processing Symposium (IPDPS 2009), Rome, I, May 23-29, 2009
Verlag Institute of Electrical and Electronics Engineers (IEEE)
Seiten Article no: 5161001
Nachgewiesen in Scopus
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page