What do algorithms explain? The issue of the goals and capabilities of Explainable Artificial Intelligence (XAI)

The increasing ubiquity of machine learning (ML) motivates research on algorithms to
“explain” models and their predictions—so-called Explainable Artificial Intelligence (XAI).
Despite many publications and discussions, the goals and capabilities of such algorithms are
far from being well understood. We argue that this is because of a problematic reasoning
scheme in the literature: Such algorithms are said to complement machine learning models
with desired capabilities, such as interpretability or explainability. These capabilities are in turn
assumed to contribute to a goal, such as trust in a system. But most capabilities lack precise
definitions and their relationship to such goals is far from obvious. The result is a reasoning
scheme that obfuscates research results and leaves an important question unanswered:
What can one expect from XAI algorithms? In this paper, we clarify the modest capabilities of
these algorithms from a concrete perspective: that of their users. We show that current
algorithms can only answer user questions that can be traced back to the question: “How can
one represent an ML model as a simple function that uses interpreted attributes?”.
... mehr
Answering this core question can be trivial, difficult or even impossible, depending on the
application. The result of the paper is the identification of two key challenges for XAI
research: the approximation and the translation of ML models.