One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models

2026-07-17

AI Summary

This study investigates the efficacy of unimodal forgetting transfer to the other modality in vision-language models and its robustness under typographic attacks. It reveals, for the first time, that cross-modal forgetting exhibits asymmetry and fragility. To address this, the authors propose CrossInf, an influence function–based method that selectively targets critical Transformer modules to enhance forgetting transfer. Through comprehensive validation—including influence function analysis, CKA representation alignment, human evaluation (κ=0.77), and simulated typographic attacks—CrossInf reduces the forgetting transfer gap by over 50% in strongly fused architectures while driving typographic attack success rates nearly to zero, thereby substantially improving both model security and utility preservation.

Abstract

Machine unlearning is widely used to remove hazardous knowledge from large language models. Modern Vision-Language Models (VLMs), however, process both text and visual inputs, raising a fundamental security question: does unlearning in one modality transfer to the other? We present the first systematic, bidirectional study of cross-modal unlearning transfer across three VLM architectures: LLaVA-1.5 (MLP projection), InstructBLIP (Q-Former), and IDEFICS (gated cross-attention). We find that unlearning transfers across modalities, but the transfer is asymmetric and incomplete. In some cases, text unlearning strongly transfers to vision. However, this robustness is not preserved under typographic attacks that manipulate the visual presentation of text. Under such attacks, previously unlearned knowledge can be readily recovered, indicating shallow unlearning. To address the transfer gap and shallow robustness, we propose \textsc{CrossInf}, an influence-guided mitigation strategy. Motivated by the observation that different model components contribute unequally to cross-modal transfer, \textsc{CrossInf} focuses unlearning on transformer blocks that most influence cross-modal generalization. It reduces the transfer gap by more than half in architectures with strong fusion, while preserving model utility. It also improves robustness under typographic attacks, reducing the attack success rate to near zero. We further conduct human evaluation with three annotators ($κ{=}0.77$) to validate our findings. Finally, we analyze shallow unlearning using Centered Kernel Alignment (CKA), providing insights into the observed transfer behavior and robustness limitations.

Problem

• cross-modal unlearning
• vision-language models
• typographic attacks
• unlearning transfer
• machine unlearning

Innovation

• cross-modal unlearning
• vision-language models
• typographic attacks
• influence-guided unlearning
• Centered Kernel Alignment