Hierarchical Selective Classification

2024-05-19

arXiv.org

AI Summary

Existing classifiers struggle to avoid overconfident predictions under high uncertainty, particularly in risk-sensitive classification scenarios. Method: We propose a hierarchical selective classification framework that leverages semantic category hierarchies to automatically retreat to broader, more general superclasses when prediction confidence falls below a threshold. We formally define hierarchical risk and coverage metrics, and design a statistically grounded inference rule that enforces target accuracy guarantees—compatible out-of-the-box with diverse pretraining paradigms (e.g., CLIP, ImageNet21k, knowledge distillation). Contribution/Results: By integrating hierarchical risk modeling, risk–coverage curve analysis, and confidence-bound-driven inference, our method demonstrates consistent efficacy across >1,000 ImageNet classifiers: it substantially improves accuracy guarantees at high coverage levels and enables strictly controllable, uncertainty-aware decision-making.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Deploying deep neural networks for risk-sensitive tasks necessitates an uncertainty estimation mechanism. This paper introduces hierarchical selective classification, extending selective classification to a hierarchical setting. Our approach leverages the inherent structure of class relationships, enabling models to reduce the specificity of their predictions when faced with uncertainty. In this paper, we first formalize hierarchical risk and coverage, and introduce hierarchical risk-coverage curves. Next, we develop algorithms for hierarchical selective classification (which we refer to as"inference rules"), and propose an efficient algorithm that guarantees a target accuracy constraint with high probability. Lastly, we conduct extensive empirical studies on over a thousand ImageNet classifiers, revealing that training regimes such as CLIP, pretraining on ImageNet21k and knowledge distillation boost hierarchical selective performance.

Problem

• Deep Learning Uncertainty
• Risk Management
• Classification Tasks

Innovation

• Hierarchical Selective Classification
• Uncertainty-aware Prediction
• Category Relationship Exploitation