Auditing Biases: From Bias to Unfairness in LLM-Generated Explanations

Large language models (LLMs) have rapidly become central to user-facing/front-end systems across various domains. Generating understandable and trustworthy explanations for these systems is of utmost importance. Explanations are not only critical for transparency and trust, but also for improving user decision-making, emotional impact, and perceived fairness. While growing research interest has explored bias in LLM-generated responses, relatively little is known about how such biases may manifest specifically in explanations---the justifications or clarifications that models offer to users. This paper defines the notions of explanation bias and explanation unfairness. We propose a vision for auditing demographic and social biases in LLM-generated explanations using controlled scenario design, multi-dimensional evaluation, and cross-model comparison. We argue that bias might sometimes be desirable in explanations. However, biased explanations leading to explanation unfairness are undesirable.