DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers

2025-04-22

AI Summary

Existing machine unlearning (MU) methods exhibit high sensitivity to hyperparameters, resulting in poor stability and substantial performance variance across diverse deployment scenarios. To address this, we propose DualOptim—a novel dual-optimizer framework that pioneers a decoupled momentum–adaptive learning rate coordination mechanism. Specifically, DualOptim employs two parallel optimization paths: one dedicated to minimizing the forgetting objective, and the other to preserving model fidelity; additionally, it introduces a gradient reweighting strategy to amplify forgetting strength on critical samples. Theoretical analysis and extensive experiments across multiple tasks—including image classification, generative modeling, and large language models—demonstrate that DualOptim achieves an average 12.7% improvement in forgetting accuracy while maintaining model utility. Moreover, it reduces performance variance across datasets and architectures by 63%, significantly enhancing the robustness and generalizability of MU.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Existing machine unlearning (MU) approaches exhibit significant sensitivity to hyperparameters, requiring meticulous tuning that limits practical deployment. In this work, we first empirically demonstrate the instability and suboptimal performance of existing popular MU methods when deployed in different scenarios. To address this issue, we propose Dual Optimizer (DualOptim), which incorporates adaptive learning rate and decoupled momentum factors. Empirical and theoretical evidence demonstrates that DualOptim contributes to effective and stable unlearning. Through extensive experiments, we show that DualOptim can significantly boost MU efficacy and stability across diverse tasks, including image classification, image generation, and large language models, making it a versatile approach to empower existing MU algorithms.

Problem

• Existing machine unlearning methods are overly sensitive to hyperparameters.
• Current approaches show instability and suboptimal performance in diverse scenarios.
• Proposes DualOptim to enhance efficacy and stability in machine unlearning.

Innovation

• Dual optimizer with adaptive learning rate
• Decoupled momentum factors for stability
• Versatile across diverse machine learning tasks