🤖 AI Summary
This paper addresses selective classification in high-stakes settings, aiming to minimize the rejection (indecision) rate under a user-specified misclassification constraint—potentially stricter than the Bayes optimal error. We propose a threshold-adaptive framework grounded in statistical learning theory and risk-controlling optimization. By constructing tight confidence sets and dynamically adjusting decision boundaries, we establish, for the first time, theoretical guarantees on achieving the optimal rejection rate under stringent misclassification constraints. Our work challenges the conventional belief that the Bayes error is an insurmountable lower bound, and instead derives the fundamental trade-off between misclassification rate and rejection cost. Experiments demonstrate substantial reductions in misclassification—approaching zero—on hard classification tasks, while incurring only negligible rejection rates; the gain in misclassification reduction far outweighs the cost introduced by rejection.
📝 Abstract
Selective classification frameworks are useful tools for automated decision making in highly risky scenarios, since they allow for a classifier to only make highly confident decisions, while abstaining from making a decision when it is not confident enough to do so, which is otherwise known as an indecision. For a given level of classification accuracy, we aim to make as many decisions as possible. For many problems, this can be achieved without abstaining from making decisions. But when the problem is hard enough, we show that we can still control the misclassification rate of a classifier up to any user specified level, while only abstaining from the minimum necessary amount of decisions, even if this level of misclassification is smaller than the Bayes optimal error rate. In many problem settings, the user could obtain a dramatic decrease in misclassification while only paying a comparatively small price in terms of indecisions.