Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning

2025-11-25

AI Summary

Existing privacy forgetting methods for multimodal large language models (MLLMs) often degrade general visual understanding capabilities. Method: We propose a selective forgetting framework featuring the Sculpted Memory Forgetting Adapter, which integrates an anchor-guided masking mechanism and memory isolation to precisely erase sensitive knowledge; we further enhance forgetting robustness via rejection-response training. Contribution/Results: We introduce S-MLLMUn Bench—the first benchmark jointly evaluating forgetting efficacy and visual understanding preservation. Experiments across multiple MLLMs demonstrate 100% sensitive information removal while retaining over 98% of image understanding performance, significantly outperforming state-of-the-art forgetting approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Large Language Models (MLLMs) achieve remarkable capabilities but can inadvertently memorize privacy-sensitive information. Although existing unlearning methods can remove such knowledge, they fail to achieve benign forgetting because they often degrade the model's general image understanding performance. To address this, we propose the Sculpted Memory Forgetting Adapter (SMFA), which confines forgetting to targeted memory regions while preserving overall capabilities. SMFA first fine-tunes the model to replace sensitive responses with refusals, yielding a memory forgetting adapter, and then applies a retaining anchor-guided masking mechanism to prevent interference with unrelated knowledge and understanding ability. To systematically evaluate selective MLLM unlearning, we introduce S-MLLMUn Bench, the first benchmark designed to jointly assess the removal of sensitive knowledge and retention of general visual understanding. Extensive experiments show that, unlike prior methods, SMFA achieves precise and controllable unlearning while maintaining the model's foundational image understanding.

Problem

• Preventing privacy leakage from memorized sensitive data
• Maintaining general image understanding during unlearning
• Achieving precise targeted forgetting without performance degradation

Innovation

• SMFA confines forgetting to targeted memory regions
• SMFA uses retaining anchor-guided masking mechanism
• SMFA achieves precise unlearning while maintaining understanding