Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation

2025-10-31

AI Summary

Existing data distillation methods often overlook intrinsic instance quality disparities, treating redundant or harmful samples equivalently to high-quality ones—thereby limiting distillation performance. To address this, we propose Influence-Weighted Distillation (IWD), the first framework to incorporate influence functions into data distillation. IWD quantifies each real-sample’s gradient-level contribution to the target model’s performance and dynamically assigns sample weights accordingly, enabling quality-aware sample retention and end-to-end synthetic optimization. Designed modularly, IWD is plug-and-play compatible with mainstream distillation pipelines. Extensive experiments demonstrate that IWD substantially enhances distilled dataset quality: it achieves up to a 7.8% absolute accuracy gain across multiple benchmarks and effectively distinguishes beneficial from harmful instances.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Distributed Search
• Computer Vision: Diffusion Models for Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Dataset distillation condenses large datasets into synthetic subsets, achieving performance comparable to training on the full dataset while substantially reducing storage and computation costs. Most existing dataset distillation methods assume that all real instances contribute equally to the process. In practice, real-world datasets contain both informative and redundant or even harmful instances, and directly distilling the full dataset without considering data quality can degrade model performance. In this work, we present Influence-Weighted Distillation IWD, a principled framework that leverages influence functions to explicitly account for data quality in the distillation process. IWD assigns adaptive weights to each instance based on its estimated impact on the distillation objective, prioritizing beneficial data while downweighting less useful or harmful ones. Owing to its modular design, IWD can be seamlessly integrated into diverse dataset distillation frameworks. Our empirical results suggest that integrating IWD tends to improve the quality of distilled datasets and enhance model performance, with accuracy gains of up to 7.8%.

Problem

• Dataset distillation methods ignore varying data instance quality
• Influence functions quantify instance impact for weighted distillation
• Modular framework improves distilled dataset quality and model accuracy

Innovation

• Assigns adaptive weights using influence functions
• Prioritizes beneficial data while downweighting harmful instances
• Seamlessly integrates into diverse dataset distillation frameworks