Effective Data Pruning through Score Extrapolation

2025-06-10

AI Summary

Existing data pruning methods require full initial training to evaluate sample importance, undermining the efficiency benefits of single-stage training. Method: We propose the Score Extrapolation Framework (SEF), which accurately predicts global sample importance from only a small subset training run—enabling, for the first time, importance estimation without full training. SEF jointly leverages k-nearest-neighbor similarity modeling and graph neural network propagation, and is compatible with mainstream pruning strategies (e.g., Dynamic Uncertainty, TDDS) across supervised, unsupervised, and adversarial training paradigms. Results: Evaluated on CIFAR-10/100, Places-365, and ImageNet, SEF reduces pre-pruning computational overhead by up to 87% while preserving model accuracy with negligible degradation (<0.3%). This breaks the long-standing dependency of data pruning on full-model training, establishing a new efficiency frontier for scalable dataset pruning.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationSearch and Optimization: Learning to SearchComputer Vision: Diffusion Models for Vision

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search enginesWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

Training advanced machine learning models demands massive datasets, resulting in prohibitive computational costs. To address this challenge, data pruning techniques identify and remove redundant training samples while preserving model performance. Yet, existing pruning techniques predominantly require a full initial training pass to identify removable samples, negating any efficiency benefits for single training runs. To overcome this limitation, we introduce a novel importance score extrapolation framework that requires training on only a small subset of data. We present two initial approaches in this framework - k-nearest neighbors and graph neural networks - to accurately predict sample importance for the entire dataset using patterns learned from this minimal subset. We demonstrate the effectiveness of our approach for 2 state-of-the-art pruning methods (Dynamic Uncertainty and TDDS), 4 different datasets (CIFAR-10, CIFAR-100, Places-365, and ImageNet), and 3 training paradigms (supervised, unsupervised, and adversarial). Our results indicate that score extrapolation is a promising direction to scale expensive score calculation methods, such as pruning, data attribution, or other tasks.

Problem

• Reducing computational costs in machine learning training
• Predicting data importance without full training
• Scaling pruning methods across diverse datasets

Innovation

• Score extrapolation framework for data pruning
• Uses k-nearest neighbors and graph networks
• Predicts importance from small data subset