CLEAR: Character Unlearning in Textual and Visual Modalities

2024-10-23

arXiv.org

AI Summary

Existing machine unlearning research lacks standardized benchmarks and effective methods for multimodal (text+image) settings, particularly for precise cross-modal forgetting of entity-specific information. Method: This paper introduces CLEAR, the first open-source multimodal unlearning benchmark, centered on accurate forgetting of fictional-character knowledge across modalities. CLEAR comprises 200 fictional characters and 3,700 text–image QA pairs; it formally defines the multimodal unlearning task and identifies novel challenges—including cross-modal semantic coupling and forgetting leakage. The authors adapt and systematically evaluate ten unimodal unlearning methods, and propose three novel techniques—feature disentanglement, gradient masking, and contrastive distillation—to enable coordinated multimodal unlearning. Contribution/Results: Experiments reveal pervasive cross-modal residual knowledge under existing methods, validating CLEAR’s diagnostic utility. The benchmark dataset and implementation code are publicly released, establishing a foundation for privacy-preserving multimodal learning research.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Data transparency and provenance

Abstract

Machine Unlearning (MU) is critical for enhancing privacy and security in deep learning models, particularly in large multimodal language models (MLLMs), by removing specific private or hazardous information. While MU has made significant progress in textual and visual modalities, multimodal unlearning (MMU) remains significantly underexplored, partially due to the absence of a suitable open-source benchmark. To address this, we introduce CLEAR, a new benchmark designed to evaluate MMU methods. CLEAR contains 200 fictitious individuals and 3,700 images linked with corresponding question-answer pairs, enabling a thorough evaluation across modalities. We assess 10 MU methods, adapting them for MMU, and highlight new challenges specific to multimodal forgetting. The dataset is available at https://huggingface.co/datasets/therem/CLEAR

Problem

• Multimodal unlearning lacks benchmarks
• CLEAR introduces open-source MMU benchmark
• Joint unlearning outperforms single-modality approaches

Innovation

• Introduces CLEAR for multimodal unlearning
• Evaluates 11 unlearning methods cross-modally
• Demonstrates joint unlearning superiority over single