XRMusic4VIP: Enabling Simultaneous Sheet Music Reading and Playing for Visually Impaired Musicians through Extended Reality

The use of assistive technologies for visually impaired musicians has been well documented, but the ability to visually read music and play simultaneously is still largely unexplored. Extended Reality (XR) could offer promising technology to support visually impaired people in this area. We address this research gap by examining the effectiveness of XR as a tool to read and play music at the same time, specifically for keyboard instruments. To achieve this, we chose a dual-phase human-centered approach that involved user participation. First, we conducted need-finding interviews with ten visually impaired participants to gather information about persisting challenges and their ideas on how to improve sheet music reading. Based on the results of these interviews, we then developed the XRMusic4VIP prototype. In the second phase, we conducted a user study with the same participants to evaluate the effectiveness of the prototype. Our findings indicate that the XRMusic4VIP prototype successfully enables the simultaneous reading of music and the playing of a keyboard instrument. Participants reported regaining their joy in making music and some of those who stopped playing because of their visual impairment can even imagine starting again.