Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction

2026-09-23

AI Summary

本文提出了一种量化鲁棒的遗忘框架，通过敏感性引导的噪声正则化和遗忘关键优化方法，在保持模型实用性的同时，提高模型在量化后的遗忘效果。

Abstract

Unlearning ensures LLM compliance by removing the influence of private or copyrighted training data. However, since LLM models typically undergo post-training compression, like quantization, in practical deployment, it has been observed that the unlearning effect can be substantially weakened, with the forgetting behavior degrading more severely than that of model utility. This paper proposes a quantization-robust unlearning framework that makes forgetting robust to quantization while maintaining overall model utility. We analyze this gap through the lens of loss landscape. Specifically, our analysis reveals a curvature-based criteria that pinpoints sensitive weights in the unlearned model that leads to both non-robust forgetting and reduced utility. We therefore propose sensitivity-guided noisy regularization, which is applied on the sensitive parameters to steer the model convergence towards a smoother minima of uniformly low forget and retain losses. Balancing unlearning and utility, we further propose forget-critical optimization, which updates only forget-critical layers, preserving most of the network to retain useful knowledge. Extensive experiments on the MUSE and TOFU benchmarks across multiple LLM unlearning algorithms show that our approach achieves substantially more quantization-resilient forgetting while maintaining utility.

Problem

• unlearning
• quantization
• forgetting
• model utility
• loss landscape

Innovation

• quantization-robust unlearning
• sensitivity-guided noisy regularization
• forget-critical optimization