Know When to Abstain: Optimal Selective Classification with Likelihood Ratios

2025-05-21

AI Summary

This work addresses the reliability challenge of selective classification under covariate shift. We propose a likelihood-ratio-based rejection mechanism grounded in the Neyman–Pearson lemma: when the test distribution deviates from the training distribution, the model abstains from prediction upon insufficient predictive confidence. To our knowledge, this is the first systematic integration of NP-optimal hypothesis testing theory into selective classification—unifying multi-class baselines and designing a novel selection function tailored for covariate shift. Our approach synergistically combines posterior calibration with out-of-distribution detection to enhance the robustness of rejection decisions. Extensive experiments across vision, language, and vision-language modeling (VLM) tasks demonstrate significant improvements over state-of-the-art methods. Results validate that the likelihood-ratio strategy effectively enhances both predictive reliability and generalization under distributional shift, exhibiting broad applicability across modalities and architectures.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationComputer Vision: Adversarial Attacks & RobustnessReasoning under Uncertainty: Relational Probabilistic Models

Application Category

User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systemsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Selective classification enhances the reliability of predictive models by allowing them to abstain from making uncertain predictions. In this work, we revisit the design of optimal selection functions through the lens of the Neyman--Pearson lemma, a classical result in statistics that characterizes the optimal rejection rule as a likelihood ratio test. We show that this perspective not only unifies the behavior of several post-hoc selection baselines, but also motivates new approaches to selective classification which we propose here. A central focus of our work is the setting of covariate shift, where the input distribution at test time differs from that at training. This realistic and challenging scenario remains relatively underexplored in the context of selective classification. We evaluate our proposed methods across a range of vision and language tasks, including both supervised learning and vision-language models. Our experiments demonstrate that our Neyman--Pearson-informed methods consistently outperform existing baselines, indicating that likelihood ratio-based selection offers a robust mechanism for improving selective classification under covariate shifts. Our code is publicly available at https://github.com/clear-nus/sc-likelihood-ratios.

Problem

• Optimal selective classification using likelihood ratios
• Addressing covariate shift in selective classification
• Improving model reliability under distribution shifts

Innovation

• Uses likelihood ratio test for optimal rejection
• Addresses covariate shift in selective classification
• Outperforms baselines in vision and language tasks