Leaky Distilled Monte Carlo for Efficient Uncertainty Quantification

Deep Neural Networks (DNNs) often produce overconfident predictions even when their inferences are incorrect. This kind of unexpected behavior is especially likely and possibly harmful when dealing with Out-of-Distribution (OOD) samples. The occurrence of these situations underscores the need to quantify the uncertainty of the model's output to mitigate its overconfidence. In this context, Monte Carlo (MC) sampling is a widely used approach for quantifying uncertainty in estimators like DNNs. However, this methodology requires processing a data point with the DNN multiple times, resulting in a computationally expensive and time-consuming process that is unsuitable for time-sensitive applications. The teacher–student framework is designed to deal with these situation by distilling a teacher model knowledge into a more efficient student model, offering a mitigation to the MC sampling cost. Yet, this framework typically fails to capture the teacher’s sampling behavior in the student and, thus, to generalize beyond the training data. In this work, we propose a revised teacher–student framework, \emph{Leaky Distilled Monte-Carlo} (LDMC), that distills both the MC sampling process and the teacher’s predictive knowledge. ... mehrLDMC improves the accuracy of uncertainty estimation across the evaluated datasets, achieving gains of up to $30\%$ relative to existing single-pass methods when compared to the full MC estimate, while increasing the inference time by only about $7-20\%$ of a single forward pass.