Explainability Versus Accuracy of Machine Learning Models: The Role of Task Uncertainty and Need for Interaction with the Machine Learning Model

This paper investigates the importance of explainability versus accuracy of machine learning
(ML) models. We propose that greater task uncertainty makes people want to interact more with the ML
model, which increases the importance of explainability relative to accuracy. We focus on the use of ML
models for product cost estimation during new product development. The paper provides mixed-methods
evidence on the trade-off between explainability and accuracy of ML models. Specifically, we find support for an inverse relationship between explainability and accuracy from the perspective of cost experts. We also find that the accurate but complex and less explainable ML model of gradient boosted regression (GBR) was preferred in only a few situations; mostly, the more basic, better explainable models of multiple linear regression (MLR) and case-based reasoning (CBR) were preferred, although these were less accurate. This suggests that lack of explainability can indeed be a major limitation for the application of ML models. Furthermore, we investigate specific characteristics that could increase task uncertainty and the importance of explainability in our context: project unpredictability, product cost granularity, predecessor product availability, target cost gap, and product development phase.