KIT | KIT-Bibliothek | Impressum | Datenschutz

Seeing through events: Real-time moving object sonification for visually impaired people using event-based camera

Ji, Zihao; Hu, Weijian; Wang, Ze; Yang, Kailun 1; Wang, Kaiwei
1 Institut für Anthropomatik und Robotik (IAR), Karlsruher Institut für Technologie (KIT)

Scene sonification is a powerful technique to help Visually Impaired People (VIP) understand their surroundings. Existing methods usually perform sonification on the entire images of the surrounding scene acquired by a standard camera or on the priori static obstacles acquired by image processing algorithms on the RGB image of the surrounding scene. However, if all the information in the scene are delivered to VIP simultaneously, it will cause information redundancy. In fact, biological vision is more sensitive to moving objects in the scene than static objects, which is also the original intention of the event-based camera. In this paper, we propose a real-time sonification framework to help VIP understand the moving objects in the scene. First, we capture the events in the scene using an event-based camera and cluster them into multiple moving objects without relying on any prior knowledge. Then, sonification based on MIDI is enabled on these objects synchronously. Finally, we conduct comprehensive experiments on the scene video with sonification audio attended by 20 VIP and 20 Sighted People (SP). The results show that our method allows both participants to clearly distinguish the number, size, motion speed, and motion trajectories of multiple objects. ... mehr

Verlagsausgabe §
DOI: 10.5445/IR/1000133651
Veröffentlicht am 08.06.2021
DOI: 10.3390/s21103558
Zitationen: 4
Zitationen: 4
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Anthropomatik und Robotik (IAR)
Publikationstyp Zeitschriftenaufsatz
Publikationsjahr 2021
Sprache Englisch
Identifikator ISSN: 1424-8220
KITopen-ID: 1000133651
Erschienen in Sensors
Verlag MDPI
Band 21
Heft 10
Seiten Art.-Nr.: 3558
Schlagwörter event-based camera; computer vision for visually impaired people; sonification; unsupervised object tracking
Nachgewiesen in Scopus
Web of Science
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page