Understanding Machine Unlearning Through the Lens of Mode Connectivity

2026-07-26

AI Summary

This work addresses the limited understanding of loss landscapes and optimization geometry in existing machine unlearning methods, which hinders their effective evaluation and improvement. Introducing mode connectivity to this domain for the first time, the paper proposes the MCU framework, which analyzes connectivity paths in parameter space to reveal that unlearned models reside in connected basins exhibiting smooth retention and forgetting behaviors. The study further demonstrates a strong correlation between MCU smoothness and unlearning difficulty. By integrating curriculum learning, second-order optimization, and linear interpolation paths, the proposed MCU-based approach not only enhances model robustness and generalization but also effectively resists retraining attacks.

Abstract

Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch. Despite recent progress, the loss landscape and optimization geometry of unlearning are poorly understood. In this paper, we study machine unlearning through the lens of mode connectivity--the phenomenon that independently trained models can often be connected by smooth low-loss paths in parameter space. We introduce {\em mode connectivity in unlearning} (MCU) and evaluate it across a range of settings, including curriculum learning, second-order optimization, and connectivity across different unlearning methods. We find that many unlearned models lie in connected basins with smooth retain/forget behavior, while changes in training dynamics can move solutions into different basins. MCU also reveals that models within the same basin can differ substantially on privacy metrics, and that unlearning progresses nonlinearly from the original model to the unlearned model. In addition, linear connectivity suggests that most approximate unlearning methods are mechanistically distinct from retraining. Finally, MCU-based ensembling can improve generalization and robustness to relearning attacks, and MCU smoothness correlates with unlearning difficulty. To our knowledge, this is the first study of machine unlearning through the lens of mode connectivity.

Problem

• Machine Unlearning
• Mode Connectivity
• Loss Landscape
• Optimization Geometry
• Unlearning Difficulty

Innovation

• mode connectivity
• machine unlearning
• loss landscape
• unlearning basins
• retraining attacks