Enabling Uncertainty Estimation in
Iterative Neural Networks

Nikita Durasov1, Doruk Oner1, Jonathan Donier2, Hieu Le1, Pascal Fua1,
1Computer Vision Laboratory, EPFL 2Neural Concept SA
ICML 2024
EPFL logo EPFL logo ICML logo
Paper arXiv Notebook Code
Teaser Image

Uncertainty in recursive models. Such models use their initial predictions as inputs to produce subsequent predictions. We display the output of three consecutive iterations of a network trained to compute distance maps to road pixels. (Top:) All roads are clearly visible. The three maps are similar and the per pixel variance is low. (Bottom:) The road in the red square is tree-covered. It is eventually detected properly but the variance is high.

Video

Abstract

Turning pass-through network architectures into iterative ones, which use their own output as input, is a well-known approach for boosting performance. In this paper, we argue that such architectures offer an additional benefit: The convergence rate of their successive outputs is highly correlated with the accuracy of the value to which they converge. Thus, we can use the convergence rate as a useful proxy for uncertainty. This results in an approach to uncertainty estimation that provides state-of-the-art estimates at a much lower computational cost than techniques like Ensembles, and without requiring any modifications to the original iterative model. We demonstrate its practical value by embedding it in two application domains: road detection in aerial images and the estimation of aerodynamic properties of 2D and 3D shapes.

Teaser Image

1D Regression Example. The task (Left) is to regress \( \mathbf{y} \)-axis values for \( \mathbf{x} \)-axis data points drawn from the range \( \mathbf{x} \in [-1, 1.3]\) using a third-degree polynomial with added Gaussian noise. The method displays higher uncertainty or iteration variance (Right) for out-of-distribution inputs and lower for in-distribution samples.

BibTeX

@article{durasov2024enabling,
  title = {Enabling Uncertainty Estimation in Iterative Neural Networks},
  author = {Durasov, Nikita and Oner, Doruk and Donier, Jonathan and Le, Hieu and Fua, Pascal}, 
  journal = {Proceedings of the International Conference on Machine Learning (ICML)},
  year = {2024}
}