Machine Unlearning via Information Theoretic Regularization

2025-02-08

AI Summary

This paper addresses the core challenge of *feature-level* and *data-point-level machine unlearning*—achieving strict forgetting guarantees while minimizing utility loss. Methodologically, it introduces the first information-theoretic regularization framework unifying both settings: (i) it incorporates diversity learning into feature unlearning to enable verifiable feature disentanglement; (ii) it formally defines a data-point forgetting criterion grounded in inference privacy, with rigorous theoretical guarantees. The framework jointly optimizes conditional entropy, enforces KL-divergence constraints, and applies probabilistic energy regularization, augmented by differential-privacy-inspired verification. Evaluated across diverse learning tasks, it achieves low utility degradation alongside strong, quantifiable forgetting assurance. Designed for flexibility, scalability, and formal verifiability, the framework is broadly applicable to general AI systems.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

How can we effectively remove or"unlearn"undesirable information, such as specific features or individual data points, from a learning outcome while minimizing utility loss and ensuring rigorous guarantees? We introduce a mathematical framework based on information-theoretic regularization to address both feature and data point unlearning. For feature unlearning, we derive a unified solution that simultaneously optimizes diverse learning objectives, including entropy, conditional entropy, KL-divergence, and the energy of conditional probability. For data point unlearning, we first propose a novel definition that serves as a practical condition for unlearning via retraining, is easy to verify, and aligns with the principles of differential privacy from an inference perspective. Then, we provide provable guarantees for our framework on data point unlearning. By combining flexibility in learning objectives with simplicity in regularization design, our approach is highly adaptable and practical for a wide range of machine learning and AI applications.

Problem

• Effective removal of undesirable information
• Minimizing utility loss in unlearning
• Ensuring rigorous unlearning guarantees

Innovation

• Information-theoretic regularization framework
• Unified solution for feature unlearning
• Provable guarantees for data point unlearning