Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach

2025-02-27

AI Summary

In open-set active learning, existing query strategies based on epistemic uncertainty (EU) or aleatoric uncertainty (AU) fail due to interference from unknown-class samples: low-EU instances lack informativeness, while closed-set AU provides no discriminative signal for unknown classes. To address this, we propose an energy-driven uncertainty co-modeling framework—the first to jointly model EU and AU. We design a unified (C+1)-class energy-based detector and classifier with joint optimization, and introduce a target-driven adaptive two-stage sampling strategy: (i) energy scores quantify EU; (ii) energy margin loss enforces out-of-distribution detection; and (iii) energy-based AU estimation enables robust unknown-class filtering, coupled with dynamic candidate set shrinkage. Our method achieves state-of-the-art performance across multiple benchmarks, significantly improving query precision and annotation efficiency. The code is publicly available.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Active learning (AL), which iteratively queries the most informative examples from a large pool of unlabeled candidates for model training, faces significant challenges in the presence of open-set classes. Existing methods either prioritize query examples likely to belong to known classes, indicating low epistemic uncertainty (EU), or focus on querying those with highly uncertain predictions, reflecting high aleatoric uncertainty (AU). However, they both yield suboptimal performance, as low EU corresponds to limited useful information, and closed-set AU metrics for unknown class examples are less meaningful. In this paper, we propose an Energy-based Active Open-set Annotation (EAOA) framework, which effectively integrates EU and AU to achieve superior performance. EAOA features a $(C+1)$-class detector and a target classifier, incorporating an energy-based EU measure and a margin-based energy loss designed for the detector, alongside an energy-based AU measure for the target classifier. Another crucial component is the target-driven adaptive sampling strategy. It first forms a smaller candidate set with low EU scores to ensure closed-set properties, making AU metrics meaningful. Subsequently, examples with high AU scores are queried to form the final query set, with the candidate set size adjusted adaptively. Extensive experiments show that EAOA achieves state-of-the-art performance while maintaining high query precision and low training overhead. The code is available at https://github.com/chenchenzong/EAOA.

Problem

• Active learning with open-set classes
• Integrate epistemic and aleatoric uncertainty
• Energy-based framework for superior performance

Innovation

• Energy-based Active Open-set Annotation
• Target-driven adaptive sampling strategy
• Integration of epistemic and aleatoric uncertainty