Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

2026-07-20

AI Summary

This work addresses the limitations of existing unlearning methods for vision-language models (VLMs), which operate solely at the textual level on the language backbone and fail to guarantee complete removal of target information in multimodal settings. To overcome this, we propose Stochastic Meta-Unlearning (SMU), a novel framework that, for the first time, incorporates holistic VLM feedback into the unlearning process of the language backbone. SMU employs a bilevel optimization strategy: the inner loop performs minimal unlearning updates on the language backbone, while the outer loop evaluates unlearning efficacy and knowledge retention using the full VLM. Experiments demonstrate that SMU significantly outperforms the strongest baselines across two VLMs and two multimodal datasets, reducing average unlearning accuracy by 10.52 points while simultaneously improving retained and generalization accuracies by 20.10 and 17.01 points, respectively, achieving an optimal trade-off between forgetting and preservation.

Abstract

Machine unlearning for vision-language models (VLMs) remains underexplored. Unlike language models, VLMs combine a language backbone with visual components, which makes unlearning more complex. There is a surprising phenomenon when moving from single-modality unlearning to VLM unlearning: a target forgotten by the standalone language backbone can still be recovered when image information is given to the full VLM. This shows that text-only feedback is not enough for reliable VLM unlearning. Motivated by this observation, we propose Stochastic Meta-Unlearning (SMU), a bilevel framework that uses VLM-level feedback to learn an unlearning-ready initialization. In the inner loop, SMU applies a few unlearning steps to the language backbone using text data. In the outer loop, SMU recomposes the updated backbone with the frozen VLM and evaluates forgetting and utility at the VLM level. This design makes the unlearning update aware of the final multimodal behavior, while still keeping the update local to the language backbone. Experiments on two VLMs, two multimodal meme datasets, and three baselines show that SMU achieves the best overall forget-retain trade-off. Compared with the strongest baseline for each metric, SMU reduces average Forget accuracy by 10.52 points and improves average Retain and Test accuracy by 20.10 and 17.01 points, respectively. More importantly, SMU also transfers to new forgetting targets and to different meta-test unlearning methods. These results suggest that VLM-level feedback can make language-backbone unlearning more reliable and more transferable for VLMs.

Problem

• machine unlearning
• vision-language models
• multimodal forgetting
• language backbone
• unlearning reliability

Innovation

• Stochastic Meta-Unlearning
• Vision-Language Models
• Machine Unlearning
• Multimodal Feedback
• Bilevel Optimization