GROM: Gradient-Free Rapid One-Shot Machine Unlearning

2026-08-06

AI Summary

This work addresses the limitations of existing gradient-based machine unlearning methods, which incur high computational costs, merely mask rather than truly erase sensitive knowledge, and remain vulnerable to recovery via quantization attacks. The authors propose a novel, gradient-free, single-step analytical unlearning approach that formulates a ridge-regularized least squares problem to compute a closed-form additive weight update. This update precisely removes target knowledge while preserving the model’s remaining functionality. Notably, the method achieves efficient unlearning without backpropagation or iterative optimization, guarantees complete removal of sensitive information from model weights, and exhibits robustness against quantization attacks. Evaluated on benchmarks such as TOFU, MUSE, and WMDP, it outperforms prior methods in the trade-off between forgetting efficacy and utility retention, operates orders of magnitude faster than fine-tuning, and maintains stable unlearning performance even under low-bit quantization.

Abstract

Machine unlearning has become a critical capability for safely removing specific, sensitive knowledge from large language models (LLMs). Current state-of-the-art approaches primarily rely on iterative, training-time unlearning via fine-tuning. However, even when utilizing parameter-efficient dimensionality reduction techniques like LoRA, gradient-based optimization remains computationally expensive and lacks explicit analytical formulations. It can also leave the targeted knowledge merely hidden rather than removed, to the point that simply quantizing the unlearned model restores much of what it was supposed to have erased. To resolve this, we propose a novel one-shot unlearning approach, abandoning iterative optimization in favor of a direct, exact analytical solution. We frame the unlearning process as a ridge-regularized least-squares optimization problem, deriving a closed-form additive update for targeted weight matrices. This update forces the selected layer to suppress unwanted content while strictly preserving its behavior on retained data. Computed from gradient-free forward passes alone, with no backpropagation and no iteration to convergence, GROM applies the weight edit in mere seconds, which makes it orders of magnitude faster than traditional fine-tuning. Extensive evaluations demonstrate that GROM achieves state-of-the-art forgetting-utility trade-offs on TOFU-5%, TOFU-10%, MUSE-Books, MUSE-News and WMDP, significantly reducing computational overhead without sacrificing overall model performance. Because the update removes the targeted content from the weights instead of masking it, GROM also withstands the low-bit quantization attack that recovers much of the content a gradient-based baseline had appeared to forget. Our code is publicly available at https://github.com/Batorskq/GROM.

Problem

• machine unlearning
• large language models
• gradient-free
• one-shot unlearning
• knowledge removal

Innovation

• machine unlearning
• gradient-free
• one-shot editing
• closed-form solution
• model quantization robustness