An Application-Oriented Process Model for Selecting Uncertainty Quantification Methods in Machine Learning

Uncertainty quantification (UQ) is increasingly recognized as essential when machine learning (ML) is employed in domains that are safety-relevant, cost-intensive, or legally binding, such as the product engineering of battery electric vehicle (BEV) energy systems. UQ methods aim to estimate the aleatoric, epistemic or both uncertainties associated with the predictions of a machine learning model. However, the landscape of UQ methods is diverse and rapidly evolving, with no single approach proving optimal across all tasks. Consequently, the selection of methods in practice is often driven by experience, constrained by limited comprehensive knowledge, time, and implementation capacity.
This paper introduces an application-oriented process model supporting data scientists in selecting UQ methods in ML by adapting the SPALTEN [1] problem-solving methodology and the Algorithm Selection Process Model (ASPM) into an Algorithm Selection Process Model for Uncertainty Quantification (UQ-ASPM). This model can be integrated into the modeling phase of a data mining process, such as the Cross Industry Standard Process for Data Mining (CRISP-DM).
... mehr
Ethnographic observations and expert interviews conducted within the research environment of BEV energy system development were analyzed using inductive qualitative content analysis to identify practical barriers, motivations, and requirements. The resulting process translates task requirements and boundary conditions into UQ-specific criteria, primarily including the source of uncertainty, integration depth, and output type. It employs a funnel-like narrowing from method families to candidate algorithms and utilizes a transparent evaluation matrix with weighted criteria, consequence analysis, and learning through a continuous information pool. An illustrative predictive-maintenance example demonstrates the instantiation of the process when an existing deterministic ML model must be retained.
The contribution is made at a meta-level, facilitating structured navigation of the method space rather than providing a direct comparison of individual UQ algorithms.