Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions

2024-09-15

arXiv.org

AI Summary

Machine unlearning for non-convex models remains challenging due to the lack of certified, efficient algorithms applicable to general non-convex loss functions. Method: This paper proposes the first first-order, black-box certified unlearning algorithm for generic non-convex losses, built upon a gradient-based backtracking retraining framework: it restores an early training checkpoint and resumes optimization solely on retained data, integrating differential privacy analysis with the Polyak–Łojasiewicz (PL) inequality to rigorously characterize certified unlearning complexity. Contribution/Results: It establishes the first $(varepsilon,delta)$-certified unlearning guarantee and generalization error bound for non-convex models without strong convexity assumptions, theoretically formalizing the trade-off among privacy, utility, and computational cost. Experiments demonstrate that the algorithm achieves significantly higher unlearning accuracy and model utility than state-of-the-art baselines in realistic privacy-sensitive settings.

Technology Category

Machine Learning: PrivacySearch and Optimization: Non-convex OptimizationReasoning under Uncertainty: Stochastic Optimization

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: User privacy protection in personalized systemsSecurity and Privacy: Data transparency and provenance

Abstract

Machine unlearning algorithms aim to efficiently remove data from a model without retraining it from scratch, in order to remove corrupted or outdated data or respect a user's ``right to be forgotten."Certified machine unlearning is a strong theoretical guarantee based on differential privacy that quantifies the extent to which an algorithm erases data from the model weights. In contrast to existing works in certified unlearning for convex or strongly convex loss functions, or nonconvex objectives with limiting assumptions, we propose the first, first-order, black-box (i.e., can be applied to models pretrained with vanilla gradient descent) algorithm for unlearning on general nonconvex loss functions, which unlearns by ``rewinding"to an earlier step during the learning process before performing gradient descent on the loss function of the retained data points. We prove $(epsilon, delta)$ certified unlearning and performance guarantees that establish the privacy-utility-complexity tradeoff of our algorithm, and we prove generalization guarantees for nonconvex functions that satisfy the Polyak-Lojasiewicz inequality. Finally, we implement our algorithm under a new experimental framework that more accurately reflects real-world use cases for preserving user privacy.

Problem

• Machine Unlearning
• Irregular Shaped Function
• Privacy Protection

Innovation

• Rewinding Deletion
• Differential Privacy
• Irregular Function Learning