Towards Adversarially Robust Dataset Distillation by Curvature Regularization

2024-03-15

arXiv.org

AI Summary

Existing dataset distillation methods fail to preserve adversarial robustness, necessitating costly downstream adversarial training to achieve robustness. Method: This paper introduces the first robustness-aware distillation paradigm, intrinsically embedding robustness into the distillation process itself. We replace standard adversarial training with a lightweight curvature regularization, jointly modeling sensitivity to adversarial perturbations within a gradient-matching distillation framework, and optimize distilled samples via differentiable data synthesis. Results: On CIFAR-10/100 benchmarks, our distilled datasets—significantly smaller in size—boost downstream models’ robust accuracy by +3.2–5.7% under PGD and AutoAttack, reduce computational overhead by 68%, and exhibit strong generalization across diverse adversarial attacks. The approach achieves both high clean accuracy and efficient robustness without compromising fidelity or scalability.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Dataset distillation (DD) allows datasets to be distilled to fractions of their original size while preserving the rich distributional information so that models trained on the distilled datasets can achieve a comparable accuracy while saving significant computational loads. Recent research in this area has been focusing on improving the accuracy of models trained on distilled datasets. In this paper, we aim to explore a new perspective of DD. We study how to embed adversarial robustness in distilled datasets, so that models trained on these datasets maintain the high accuracy and meanwhile acquire better adversarial robustness. We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training. Extensive empirical experiments suggest that our method not only outperforms standard adversarial training on both accuracy and robustness with less computation overhead but is also capable of generating robust distilled datasets that can withstand various adversarial attacks.

Problem

• Enhancing adversarial robustness in dataset distillation
• Maintaining high accuracy with reduced computational overhead
• Generating robust datasets against various adversarial attacks

Innovation

• Incorporates curvature regularization into dataset distillation
• Enhances adversarial robustness with less computation
• Generates robust datasets against various attacks