On Optimization Complexity of Second-Order Certified Unlearning

2026-07-22

AI Summary

This work addresses the problem of efficiently and reliably removing the memory of specific data from trained models while preserving model performance and providing provable unlearning guarantees. To this end, the authors propose a second-order certified unlearning algorithm based on uniformly convex regularization and an anisotropic Gaussian mechanism. This approach uniquely integrates uniform convexity with surrogate generalization error to analyze the optimization complexity of unlearning. Theoretically, the study shows that when the samples to be forgotten are well-predicted by the model, the corresponding optimization problem becomes easier to solve. The proposed algorithm achieves global convergence for generalized linear models—such as logistic and exponential regression—and demonstrates significant improvements over existing first-order methods.

Abstract

We study machine unlearning: the removal of memorized training data from a trained model. Specifically, we investigate the algorithmic complexity of certified unlearning from an optimization perspective. We formalize the goal of an unlearning algorithm as simultaneously achieving certified unlearning and optimization accuracy. Utilizing the notion of uniformly convex regularizers, we prove new bounds on the distance between initial and unlearned models using a novel substitute for generalization error. Thus we theoretically demonstrate that if the removed data is well-predicted by the unlearned model, the corresponding optimization problem is simple. Furthermore, we develop a new second-order unlearning algorithm with an anisotropic Gaussian mechanism and state-of-the-art global convergence. We prove fast rates for our method in achieving certified unlearning for linear models with quasi-self-concordant losses. As a direct application, our theory covers unlearning for logistic and exponential regressions and shows a provable benefit of utilizing second-order information compared to first-order unlearning methods.

Problem

• machine unlearning
• certified unlearning
• optimization complexity
• second-order methods
• uniformly convex regularizers

Innovation

• certified unlearning
• second-order optimization
• uniformly convex regularizers
• anisotropic Gaussian mechanism
• quasi-self-concordant losses