Understanding Empirical Unlearning with Combinatorial Interpretability

2026-02-22

AI Summary

This work addresses a critical limitation in current machine unlearning methods, which often suppress rather than truly erase target knowledge, leaving it potentially recoverable. Due to the opacity of large models, verifying the effectiveness of such methods remains challenging. To overcome this, the study implements mainstream unlearning approaches within a fully interpretable two-layer neural network framework, enabling direct analysis of concept representations encoded in model weights. The authors systematically evaluate whether these methods genuinely remove targeted knowledge and assess its recoverability after fine-tuning. Their findings reveal, for the first time in an interpretable setting, that most unlearning techniques merely obscure knowledge rather than eliminate it—“forgotten” information can be efficiently restored through simple fine-tuning, thereby exposing the fragility and fundamental limitations of existing unlearning mechanisms.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Graphical Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large foundation models are far from interpretable, understanding whether and how such knowledge persists remains a significant challenge. To address this, we turn to the recently developed framework of combinatorial interpretability. This framework, designed for two-layer neural networks, enables direct inspection of the knowledge encoded in the model weights. We reproduce baseline unlearning methods within the combinatorial interpretability setting and examine their behavior along two dimensions: (i) whether they truly remove knowledge of a target concept (the concept we wish to remove) or merely inhibit its expression while retaining the underlying information, and (ii) how easily the supposedly erased knowledge can be recovered through various fine-tuning operations. Our results shed light within a fully interpretable setting on how knowledge can persist despite unlearning and when it might resurface.

Problem

• unlearning
• knowledge persistence
• model interpretability
• foundation models
• combinatorial interpretability

Innovation

• combinatorial interpretability
• model unlearning
• knowledge persistence
• neural network interpretability
• empirical unlearning