KIT | KIT-Bibliothek | Impressum | Datenschutz

Is This System Biased? – How Users React to Gender Bias in an Explainable AI System

Jussupow, Ekaterina; Meza Martínez, Miguel Angel; Mädche, Alexander; Heinzl, Armin


Biases in Artificial Intelligence (AI) can reinforce social inequality. Increasing transparency of AI systems through explanations can help to avoid the negative consequences of those biases. However, little is known about how users evaluate explanations of biased AI systems. Thus, we apply the Psychological Contract Violation Theory to investigate the implications of a gender-biased AI system on user trust. We allocated 339 participants into three experimental groups, each with a different loan forecasting AI system version: explainable gender-biased, explainable neutral, and non-explainable AI system. We demonstrate that only users with moderate to high general awareness of gender stereotypes in society, i.e., stigma consciousness, perceive the gender-biased AI system as not trustworthy. However, users with low stigma consciousness perceive the gender-biased AI system as trustworthy as it is more transparent than a system without explanations. Our findings show that AI biases can reinforce social inequality if they match with human stereotypes.

Zugehörige Institution(en) am KIT Fakultät für Wirtschaftswissenschaften – Institut für Informationswirtschaft und Marketing (IISM)
Publikationstyp Proceedingsbeitrag
Publikationsjahr 2021
Sprache Englisch
Identifikator ISBN: 978-1-7336325-9-1
KITopen-ID: 1000139962
Erschienen in 42nd International Conference on Information Systems
Veranstaltung 42nd International Conference on Information Systems (ICIS 2021), Austin, TX, USA, 12.12.2021 – 15.12.2021
Verlag Association for Information Systems (AIS)
Schlagwörter Artificial intelligence, explanations, trust, psychological contract violation, bias
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page