ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory

2025-09-16

AI Summary

Diffusion models risk memorizing training data, compromising user privacy and model security. To address this, we propose an efficient data unlearning method specifically designed for diffusion models—without retraining. Our approach employs importance sampling to construct an interpretable fine-tuning loss that selectively emphasizes dominant denoising terms. Crucially, we redirect the target denoising trajectory toward the *k*-nearest neighbors of the to-be-forgotten sample, enabling semantically consistent unlearning guidance. By jointly optimizing an approximate loss and steering the denoising trajectory, our method achieves superior unlearning performance on MNIST T-Shirt, CelebA-HQ, CIFAR-10, and Stable Diffusion—outperforming existing approaches across diverse benchmarks. Notably, it maintains high generation fidelity even under strong unlearning intensity, marking the first work to achieve an optimal trade-off between unlearning effectiveness and generative fidelity in diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Privacy
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models excel at generating high-quality, diverse images but suffer from training data memorization, raising critical privacy and safety concerns. Data unlearning has emerged to mitigate this issue by removing the influence of specific data without retraining from scratch. We propose ReTrack, a fast and effective data unlearning method for diffusion models. ReTrack employs importance sampling to construct a more efficient fine-tuning loss, which we approximate by retaining only dominant terms. This yields an interpretable objective that redirects denoising trajectories toward the $k$-nearest neighbors, enabling efficient unlearning while preserving generative quality. Experiments on MNIST T-Shirt, CelebA-HQ, CIFAR-10, and Stable Diffusion show that ReTrack achieves state-of-the-art performance, striking the best trade-off between unlearning strength and generation quality preservation.

Problem

• Mitigate training data memorization in diffusion models
• Remove specific data influence without full retraining
• Redirect denoising trajectories for efficient unlearning

Innovation

• Redirects denoising trajectories
• Uses importance sampling loss
• Approximates dominant terms efficiently