Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers

2026-04-24

AI Summary

This work addresses the challenge of achieving complete machine unlearning under second-order optimizers, where residual information from deleted data often persists. By modeling model memory through eigendecomposition and integrating state perturbation, geometric analysis, and counterfactual evaluation, the study systematically compares the unlearning behavior of first- and second-order learners. It reveals that although second-order optimizers can align gradients and performance with an ideal counterfactual model, their internal states retain latent, imperceptible memory traces. The authors demonstrate that only through carefully controlled perturbations that precisely erase geometric information can stable and effective forgetting be achieved. This work uncovers the geometric nature of memory in second-order optimization and establishes geometric information erasure as a critical mechanism for reliable machine unlearning.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn't detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased.

Problem

• machine unlearning
• second-order optimizers
• optimizer state
• residual information
• geometric analysis

Innovation

• machine unlearning
• second-order optimizers
• geometric analysis
• optimizer state
• eigendecay