Data Unlearning via Inverse Distillation

2026-09-28

AI Summary

This study addresses the high inference costs of multi-step generative models and their difficulty in forgetting specific training data by proposing the IDU framework, which unifies machine unlearning and knowledge distillation in unconditional flow matching and diffusion models for the first time. Based on mixture distribution modeling and a minimax optimization objective, this method compresses a multi-step teacher model into a single-step student model via inverse distillation while simultaneously suppressing outputs from the forget set, without requiring access to retained data or introducing auxiliary classifiers. Experimental results demonstrate that the proposed framework significantly reduces the generation frequency of samples from forgotten classes while effectively preserving high-quality generative performance on retained classes.

Abstract

Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.

Problem

• Machine Unlearning
• Knowledge Distillation
• Diffusion Models
• Flow Matching

Innovation

• Inverse Distillation Unlearning
• Data Unlearning
• Knowledge Distillation
• Flow Matching
• Diffusion Models