QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localization

2026-08-02

AI Summary

This work addresses the challenge of efficient and precise machine unlearning by proposing the QR-Erase framework. It introduces Pivoted QR decomposition—used for the first time in this context—to identify and remove task-relevant subspaces, replacing conventional SVD-based approaches. This substitution achieves near-optimal unlearning performance while substantially improving computational efficiency. Additionally, a layer localization strategy is incorporated to update only those network layers that concentrate the most relevant information, thereby minimizing unnecessary degradation of unrelated capabilities. By effectively leveraging low-rank structures and spectral gap analysis, QR-Erase achieves a superior trade-off between forgetting and retention across diverse tasks. For instance, on speech-related benchmarks, it reduces accuracy on the forget set from 53.1% to 15.7%, matching the unlearning efficacy of SVD-based methods at a significantly lower computational cost.

Abstract

Machine unlearning seeks to remove targeted information from trained models without requiring costly retraining. Existing optimization-based methods often degrade unrelated capabilities, while subspace-based approaches rely on computationally expensive singular value decompositions (SVD). We introduce QR-Erase, a subspace-based framework that uses Pivoted QR decomposition to identify and remove task-specific representations directly from model parameters. We further propose Layer-Localized QR-Erase, which restricts updates to layers containing the highest concentration of task-specific information. We show that Pivoted QR provides accurate subspace recovery with bounded error, and that under a mild spectral gap condition, the recovered subspace approaches the optimal SVD solution. Across task-level, cross-lingual, and speech unlearning, QR-Erase achieves a stronger forgetting-retention tradeoff than optimization-based methods while remaining within 5% of SVD across all metrics. Exploiting low-rank and layer-localized structure further improves forgetting (for example, reducing speech forget-set accuracy from 53.1% to 15.7%). These results demonstrate that accurate subspace recovery, rather than optimal reconstruction, is sufficient for effective unlearning and provides an efficient and general alternative to SVD-based methods for modern foundation models.

Problem

• machine unlearning
• subspace recovery
• forgetting-retention tradeoff
• singular value decomposition
• model parameters

Innovation

• machine unlearning
• subspace recovery
• Pivoted QR decomposition
• layer localization
• foundation models