Exact Unlearning in Reinforcement Learning

2026-06-02

AI Summary

This work formally introduces the problem of exact unlearning in reinforcement learning, aiming to efficiently remove specified user data such that the model’s output becomes statistically indistinguishable from that of a model never trained on the data. To this end, the authors propose a ρ-TV stable learning framework that integrates differential privacy with stability analysis, yielding the first exact unlearning algorithm with theoretical guarantees. The method avoids full retraining, incurring only a computational overhead of ρ√(ln T) times that of retraining from scratch. It achieves a regret bound of Õ(H²√(SAT) + H³S²A + H^{2.5}S²A/ρ) and establishes a fundamental lower bound of Ω(H√(SAT) + SAH/ρ), nearly matching the minimax optimal rate.

Abstract

We formulate the problem of \emph{exact unlearning} in reinforcement learning, where the goal is to design an efficient framework that enables the removal of any user's data upon deletion request, i.e., the online learner's output after unlearning is \emph{indistinguishable} from what would have been produced had the deleted user never interacted with the learner. For any $ρ>0$, we show that there exists a reinforcement learning (RL) algorithm that is $ρ$-TV-stable and supports an exact unlearning procedure whose expected computational cost is only a $ρ\sqrt{\ln T}$ fraction of the computational cost of retraining from scratch. We construct such a $ρ$-TV-stable RL algorithm for tabular Markov decision processes (MDPs), which achieves a regret bound of $\mathcal{O}(H^2 \sqrt{SAT} + H^3 S^2 A + {H^{2.5} S^2 A}/ρ)$, where $S, A, H$, and $T$ denote the number of states, the number of actions, the episode horizon, and the number of episodes, respectively. We also establish a lower bound of $Ω(H\sqrt{\!SAT}\! +\! {SAH}/ρ)$ for $ρ$-TV-stable RL algorithms, showing that our algorithm is nearly minimax optimal.

Problem

• exact unlearning
• reinforcement learning
• data deletion
• privacy
• TV-stability

Innovation

• exact unlearning
• TV-stability
• reinforcement learning
• efficient deletion
• minimax optimality