Boosting Active Learning with Knowledge Transfer

2025-09-19

AI Summary

In active learning (AL), existing uncertainty estimation methods rely on complex auxiliary models and task-specific training, limiting generalizability—especially in data-scarce domains like computational biology. To address this, we propose a task-agnostic knowledge transfer framework: at each AL iteration, a task-specific model and a universal student model are jointly trained; uncertainty of unlabeled samples is quantified via distributional divergence (e.g., KL divergence) between teacher and student outputs. Theoretically, we establish that this divergence bounds the task loss—not its exact value—enhancing interpretability and cross-task applicability without requiring specialized training strategies or domain-specific design. Experiments across image classification and cryo-electron tomography classification demonstrate significant improvements in query sample accuracy and model convergence efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Uncertainty estimation is at the core of Active Learning (AL). Most existing methods resort to complex auxiliary models and advanced training fashions to estimate uncertainty for unlabeled data. These models need special design and hence are difficult to train especially for domain tasks, such as Cryo-Electron Tomography (cryo-ET) classification in computational biology. To address this challenge, we propose a novel method using knowledge transfer to boost uncertainty estimation in AL. Specifically, we exploit the teacher-student mode where the teacher is the task model in AL and the student is an auxiliary model that learns from the teacher. We train the two models simultaneously in each AL cycle and adopt a certain distance between the model outputs to measure uncertainty for unlabeled data. The student model is task-agnostic and does not rely on special training fashions (e.g. adversarial), making our method suitable for various tasks. More importantly, we demonstrate that data uncertainty is not tied to concrete value of task loss but closely related to the upper-bound of task loss. We conduct extensive experiments to validate the proposed method on classical computer vision tasks and cryo-ET challenges. The results demonstrate its efficacy and efficiency.

Problem

• Improving uncertainty estimation in active learning
• Reducing complex model dependency for domain tasks
• Enhancing cryo-ET classification through knowledge transfer

Innovation

• Teacher-student knowledge transfer for uncertainty
• Task-agnostic auxiliary model without special training
• Distance-based uncertainty measurement using model outputs