Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

2026-07-23

AI Summary

This work addresses the overlooked issue of imbalanced distribution of unlearning requests across demographic groups in multimodal large language models, which can exacerbate model bias. To tackle this challenge, the authors introduce FAIRGET, the first fairness-aware evaluation benchmark tailored for imbalanced unlearning scenarios, and propose FAUN, a novel unlearning algorithm that jointly optimizes forgetting efficacy and group fairness. FAUN incorporates a bias-aware activation steering mechanism that dynamically adjusts internal model representations during the unlearning of identity-specific data to preserve equitable performance across groups. Experimental results demonstrate that FAUN consistently outperforms existing methods on both FAIRGET and FIUBench, achieving effective unlearning while significantly better maintaining model fairness.

Abstract

Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations. To evaluate unlearning effectiveness in multimodal large language models (MLLMs), prior works fine-tune models on fictitious identities, simulating unlearning requests on subsets of these IDs, which are typically uniformly distributed. However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies, potentially altering the model's internal beliefs for these groups and leading to biased behaviors. To fill this gap, we propose FAIRGET, the first Visual Question Answering benchmark that evaluates unlearning under unbalanced, realistic, forget requests. These requests are designed to simulate multiple realistic scenarios, ranging from simple to challenging settings, that lead to biased unlearned models if fairness is not accounted for. Additionally, we propose FAUN, the first unlearning algorithm for MLLMs that forgets unlearning data while preserving model fairness. FAUN exploits a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data. Experiments on FAIRGET and the established FIUBench demonstrate our method's superiority both in unlearning quality and fairness.

Problem

• machine unlearning
• fairness
• multimodal LLMs
• unbalanced data
• bias

Innovation

• machine unlearning
• multimodal LLMs
• fairness
• unbalanced forgetting
• bias-aware activation steering