BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learning

2026-03-13

AI Summary

Existing active learning strategies often lack robustness across different models, datasets, and annotation budgets, and conventional oracle methods are ill-suited for large-scale deep learning settings. To address this, this work proposes BoSS—a scalable and efficient oracle strategy that constructs candidate batches by integrating multiple selection heuristics and selects the optimal batch based on performance gains evaluated using ground-truth labels. BoSS establishes the first practical oracle benchmark tailored for large-scale deep active learning and enables flexible incorporation of new strategies. Experimental results demonstrate that BoSS significantly outperforms existing oracle approaches, revealing a substantial performance gap between current state-of-the-art active learning algorithms and the achievable upper bound—particularly on large-scale multi-class datasets—and further validating the effectiveness of strategy ensembling.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Active learning (AL) aims to reduce annotation costs while maximizing model performance by iteratively selecting valuable instances. While foundation models have made it easier to identify these instances, existing selection strategies still lack robustness across different models, annotation budgets, and datasets. To highlight the potential weaknesses of existing AL strategies and provide a reference point for research, we explore oracle strategies, i.e., strategies that approximate the optimal selection by accessing ground-truth information unavailable in practical AL scenarios. Current oracle strategies, however, fail to scale effectively to large datasets and complex deep neural networks. To tackle these limitations, we introduce the Best-of-Strategy Selector (BoSS), a scalable oracle strategy designed for large-scale AL scenarios. BoSS constructs a set of candidate batches through an ensemble of selection strategies and then selects the batch yielding the highest performance gain. As an ensemble of selection strategies, BoSS can be easily extended with new state-of-the-art strategies as they emerge, ensuring it remains a reliable oracle strategy in the future. Our evaluation demonstrates that i) BoSS outperforms existing oracle strategies, ii) state-of-the-art AL strategies still fall noticeably short of oracle performance, especially in large-scale datasets with many classes, and iii) one possible solution to counteract the inconsistent performance of AL strategies might be to employ an ensemble-based approach for the selection.

Problem

• active learning
• oracle strategy
• selection robustness
• scalability
• annotation efficiency

Innovation

• oracle strategy
• active learning
• ensemble selection
• scalable AL
• BoSS