Towards Accessible Visualizations with Vision-Language Models

Data Visualizations such as charts/plots, and diagrams are multi-dimensional representations commonly used to explore data and communicate insights. They are available in diverse layouts and styles, each tailored to specific analytical needs. For example, with a smartwatch, one may monitor monthly sleeping cycles with a quick glance at a visual plot.

Unfortunately, it is estimated that in 2020, approximately 70\% of visual content existed in inaccessible modalities for readers with visual impairments. People sharing this content might lack the expertise to make their content accessible or fear the time and labor required to achieve such goals. On the other hand, People with Visual Impairment (PVI) might not be confident in the available assistive tools to enable them to interpret the content independently.

In this thesis, our research focuses on developing visual content analysis systems to assist sighted individuals to make their content accessible and to provide end-to-end access for PVI. More specifically, we investigate how to digitize documents and visuals while ensuring adherence to accessibility guidelines. To this end, we first investigate how to construct and convey layout information from deep learning models to tactile modalities. ... mehrThis approach is based on the idea that diverse categories of documents and variations in visuals can be primarily differentiated by their layout. Similarly, the advantage of tactile materials, compared to mere textual descriptions, lies mainly in their presentation layout. We further explore an application designed to jointly involve inexperienced users and deep learning models to make the conversion process more flexible.

Tactile materials and alternative text are not merely visual drawings or plain texts; they must comply with established standards. Assisting users and training models to author high-quality accessible content involves ensuring both comprehensiveness and adherence to accessibility standards. In a setup where the model is running end-to-end with a blind participant, or a sighted individual is authoring a chart description, we investigate how deep learning can support adherence to these standards and enhance the overall quality.

Given the diverse nature of documents and visuals, we finally ask how to cope with this diversity and adapt models' performance accordingly. Capturing this diversity within the training process is crucial to ensure the robustness of the models in real-life scenarios, such as captured and scanned samples.

Our investigations have unfolded significant insights into accessible digitization, including new benchmarks for connecting vision-language models with assistive technologies. We have developed valuable and intelligent systems from which both inexperienced sighted and blind individuals can benefit. These advancements promise to enhance the usability and accessibility of digital content, making it more inclusive for all users.