Distributionally Robust Coreset Selection under Covariate Shift

2025-01-24

AI Summary

To address the practical challenge of unknown and shifted covariate distributions during deployment, this paper proposes a distributionally robust coreset selection method. It models worst-case distributional shifts within a Wasserstein ball, theoretically derives an upper bound on generalization error, and optimizes the selection of a representative subset by minimizing this bound. This work is the first to integrate distributionally robust optimization into coreset selection, explicitly ensuring robustness of training data selection against covariate shift. By leveraging convex optimization analysis and gradient approximation techniques, the method scales effectively to deep learning models. Empirical evaluation across multiple covariate shift benchmarks demonstrates substantial improvements over conventional approaches: under severe shift conditions, average test accuracy increases by 5.2%, significantly enhancing out-of-distribution generalization capability.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Coreset selection, which involves selecting a small subset from an existing training dataset, is an approach to reducing training data, and various approaches have been proposed for this method. In practical situations where these methods are employed, it is often the case that the data distributions differ between the development phase and the deployment phase, with the latter being unknown. Thus, it is challenging to select an effective subset of training data that performs well across all deployment scenarios. We therefore propose Distributionally Robust Coreset Selection (DRCS). DRCS theoretically derives an estimate of the upper bound for the worst-case test error, assuming that the future covariate distribution may deviate within a defined range from the training distribution. Furthermore, by selecting instances in a way that suppresses the estimate of the upper bound for the worst-case test error, DRCS achieves distributionally robust training instance selection. This study is primarily applicable to convex training computation, but we demonstrate that it can also be applied to deep learning under appropriate approximations. In this paper, we focus on covariate shift, a type of data distribution shift, and demonstrate the effectiveness of DRCS through experiments.

Problem

• Robust Selection
• Representative Dataset
• Feature Change

Innovation

• Distributionally Robust Core Set Selection
• Adaptive Learning
• Worst-case Error Rate