Group Preference Collapse in Personalized Multimodal Large Language Models

2026-06-12

arXiv.org

AI Summary

This study addresses the issue of population preference collapse in existing multimodal large language model personalization tasks, where neglecting individual differences leads to homogenized group-level preferences. To mitigate this, it proposes PrefMoE, a framework that introduces a novel preference-centric architecture decoupling stable user profiles from dynamic preference representations. The method achieves preference-sensitive generation through shared prototype decomposition, counterfactual pseudo-user augmentation, and dual-path LoRA adaptation. Furthermore, imbalance-aware learning and residual decorrelation mechanisms are incorporated to suppress dominant-choice drift. Extensive experiments demonstrate that PrefMoE significantly enhances preference sensitivity across various backbone models while effectively alleviating population preference collapse.

Abstract

Personalized multimodal large language models (MLLMs) aim to generate user-specific responses, but existing methods mainly rely on profile-level information and overlook diverse user preferences. We identify group preference collapse, where multi-user personalized MLLMs become insensitive to individual preferences and drift toward dominant population-level choices due to suppressed preference signals and unreliable preference use during generation. We propose PrefMoE, a preference-centric framework that separates stable profile information from preference-related representations. PrefMoE decomposes preferences into shared prototypes and personalized residuals, preserves individualized residuals with imbalance-aware learning, counterfactual pseudo-user augmentation, and residual decorrelation, and routes profile and preference factors through separate LoRA adaptation paths. Experiments across multiple MLLM backbones show that PrefMoE improves preference-sensitive personalization while substantially reducing preference collapse. Project page: https://prefmoe.github.io/.

Problem

• Group Preference Collapse
• Personalized Multimodal Large Language Models
• Individual Preferences
• Preference-sensitive Personalization

Innovation

• Group Preference Collapse
• Personalized MLLMs
• PrefMoE
• Preference Decoupling
• LoRA Adaptation