Erase at the Core: Representation Unlearning for Machine Unlearning

2026-02-05

AI Summary

Existing machine unlearning methods often achieve only superficial "output-level forgetting," leaving intermediate-layer representations vulnerable to leaking information about the data intended to be forgotten, thereby posing privacy risks. To address this limitation, this work proposes Erase at the Core (EC), a framework that systematically realizes representation-level unlearning for the first time. EC inserts pluggable, model-agnostic auxiliary modules across multiple intermediate layers of the network and jointly optimizes representations throughout the entire architecture via a hierarchically weighted contrastive unlearning loss combined with deep supervision from the retention set. Experiments demonstrate that EC significantly reduces the similarity between intermediate representations and those of the original model while preserving performance on retained data, thereby achieving effective unlearning at both the logit and representation levels.

Technology Category

Machine Learning: PrivacyComputer Vision: Representation Learning for VisionReasoning under Uncertainty: Uncertainty Representations

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphsUser Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Many approximate machine unlearning methods demonstrate strong logit-level forgetting -- such as near-zero accuracy on the forget set -- yet continue to preserve substantial information within their internal feature representations. We refer to this discrepancy as superficial forgetting. Recent studies indicate that most existing unlearning approaches primarily alter the final classifier, leaving intermediate representations largely unchanged and highly similar to those of the original model. To address this limitation, we introduce the Erase at the Core (EC), a framework designed to enforce forgetting throughout the entire network hierarchy. EC integrates multi-layer contrastive unlearning on the forget set with retain set preservation through deeply supervised learning. Concretely, EC attaches auxiliary modules to intermediate layers and applies both contrastive unlearning and cross-entropy losses at each supervision point, with layer-wise weighted losses. Experimental results show that EC not only achieves effective logit-level forgetting, but also substantially reduces representational similarity to the original model across intermediate layers. Furthermore, EC is model-agnostic and can be incorporated as a plug-in module into existing unlearning methods, improving representation-level forgetting while maintaining performance on the retain set.

Problem

• machine unlearning
• superficial forgetting
• representation unlearning
• feature representations
• forgetting

Innovation

• machine unlearning
• representation unlearning
• contrastive unlearning
• superficial forgetting
• model-agnostic