IT³: Idempotent Test-Time Training

This paper introduces Idempotent Test-Time Training (IT³), a novel approach to addressing the challenge of distribution shift. While supervised-learning methods assume matching train and test distributions, this is rarely the case for machine learning systems deployed in the real world. Test-Time Training (TTT) approaches address this by adapting models during inference, but they are limited by a domain-specific auxiliary task. \itt{} is based on the universal property of idempotence. An idempotent operator is one that can be applied sequentially without changing the result beyond the initial application, that is \( f(f(\mathbf{x})) = f(\mathbf{x}) \). At training, the model receives an input \( \mathbf{x} \) along with another signal that can either be the ground truth label \( \mathbf{y} \) or a neutral "don't know" signal \( \mathbf{0} \). At test time, the additional signal can only be \( \mathbf{0} \). When sequentially applying the model, first predicting \( \mathbf{y}_0 = f(\mathbf{x}, \mathbf{0}) \) and then \( \mathbf{y}_1 = f(\mathbf{x}, \mathbf{y}_0) \), the distance between \( \mathbf{y}_0 \) and \( \mathbf{y}_1 \) measures certainty and indicates out-of-distribution input \( \mathbf{x} \) if high. We use this distance, that can be expressed as \( ||f(\mathbf{x}, f(\mathbf{x}, \mathbf{0})) - f(\mathbf{x}, \mathbf{0})|| \) as our TTT loss during inference. By carefully optimizing this objective, we effectively train \( f(\mathbf{x}, \cdot) \) to be idempotent, projecting the internal representation of the input onto the training distribution. We demonstrate the versatility of our approach across various tasks, including corrupted image classification, aerodynamic predictions, tabular data with missing information, age prediction from face, and large-scale aerial photo segmentation. Moreover, these tasks span different architectures such as MLPs, CNNs, and GNNs.