Machine Unlearning for Gibbs Supervised Learning Algorithms

2026-09-24

AI Summary

This study addresses the challenge of achieving exact machine unlearning in Gibbs supervised learning algorithms. Building upon the ERM-RER variational formulation, we propose a novel Gibbs algorithm that constructs an exact unlearning mechanism guaranteeing distributional consistency by maximizing the expected empirical risk over the data to be forgotten while introducing relative entropy regularization, thereby effectively circumventing retraining from scratch. Furthermore, the theoretical framework is extended into a general data-point reweighting scheme. By integrating variational inference with probability measure optimization, this work achieves exact unlearning whose output distribution is fully consistent with that of the proposed algorithm. While controlling generalization error, it establishes a novel data-reweighting perspective for machine unlearning.

Abstract

In this paper, a method for achieving exact unlearning for Gibbs supervised learning algorithms is proposed using a variational formulation inspired by empirical risk minimization subject to relative entropy regularization (ERM-RER). Such a method consists of maximizing the expected empirical risk over the dataset to be unlearned subject to a regularization by relative entropy with respect to the original algorithm. The optimization variable is a probability measure on the models; and the solution is another Gibbs probability measure that represents a new Gibbs supervised learning algorithm. The method guarantees exact unlearning in the sense that the new Gibbs algorithm coincides in distribution with the algorithm that would have been obtained by retraining from scratch on the dataset to be retained. As a byproduct, a framework for reweighting data points in ERM-RER by strategically choosing both the reference measure and the regularization factor is obtained. In this framework, exact unlearning is the special case in which zero-weight is assigned to the contribution of the data points to be unlearned. More generally, depending on the choice of certain parameters, data points can be up-weighted or down-weighted in ERM-RER problems for particular purposes, e.g., controlling the generalization error of Gibbs algorithms. This paves the way for new constructive or adversarial views on classical reweighting data points in ERM-RER.

Problem

• Machine Unlearning
• Gibbs Supervised Learning
• Exact Unlearning
• Empirical Risk Minimization

Innovation

• Machine Unlearning
• Gibbs Supervised Learning
• Relative Entropy Regularization
• Data Reweighting
• Variational Formulation