Label Shift Estimation With Incremental Prior Update

2026-04-02

AI Summary

This work addresses the challenge of label shift—mismatch between training and test label distributions—in real-world scenarios by proposing a posterior correction method that does not rely on strong calibration assumptions. Built upon the invariance of conditional distributions, the approach introduces an incremental prior update mechanism under weak calibration conditions to dynamically adjust the posterior probabilities output by a black-box classifier, thereby accurately estimating the test-time label distribution. Unlike conventional methods that depend on confusion matrices or maximum likelihood estimation, the proposed technique is applicable to any probabilistic classifier without requiring retraining. Experimental results on CIFAR-10 and MNIST demonstrate that the method consistently outperforms existing state-of-the-art approaches across varying degrees of calibration quality and label shift intensity.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationReasoning under Uncertainty: Relational Probabilistic ModelsComputer Vision: Adversarial Attacks & Robustness

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systemsEconomics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

An assumption often made in supervised learning is that the training and testing sets have the same label distribution. However, in real-life scenarios, this assumption rarely holds. For example, medical diagnosis result distributions change over time and across locations; fraud detection models must adapt as patterns of fraudulent activity shift; the category distribution of social media posts changes based on trending topics and user demographics. In the task of label shift estimation, the goal is to estimate the changing label distribution $p_t(y)$ in the testing set, assuming the likelihood $p(x|y)$ does not change, implying no concept drift. In this paper, we propose a new approach for post-hoc label shift estimation, unlike previous methods that perform moment matching with confusion matrix estimated from a validation set or maximize the likelihood of the new data with an expectation-maximization algorithm. We aim to incrementally update the prior on each sample, adjusting each posterior for more accurate label shift estimation. The proposed method is based on intuitive assumptions on classifiers that are generally true for modern probabilistic classifiers. The proposed method relies on a weaker notion of calibration compared to other methods. As a post-hoc approach for label shift estimation, the proposed method is versatile and can be applied to any black-box probabilistic classifier. Experiments on CIFAR-10 and MNIST show that the proposed method consistently outperforms the current state-of-the-art maximum likelihood-based methods under different calibrations and varying intensities of label shift.

Problem

• label shift
• distribution estimation
• supervised learning
• post-hoc calibration
• prior update

Innovation

• label shift estimation
• incremental prior update
• post-hoc calibration
• black-box classifier
• probabilistic classification