MC-LLaVA: Multi-Concept Personalized Vision-Language Model

2026-04-11

AI Summary

Existing vision-language models (VLMs) for personalization are limited to single-concept adaptation, failing to capture users’ multi-concept collaborative understanding—hindering real-world applicability. This work introduces MC-LLaVA, the first multi-concept personalization paradigm for VLMs. Methodologically: (1) we construct a high-diversity instruction dataset featuring multi-role and multi-object scenarios; (2) we design a multi-concept instruction-tuning strategy; (3) we propose a visual-token-initialized concept word embedding method; and (4) we introduce an aggregatable, localization-aware personalized visual prompting mechanism. Evaluated on multi-concept visual question answering and grounding tasks, MC-LLaVA significantly outperforms single-concept baselines, generating more accurate and interpretable responses. Our code and dataset are publicly released.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Computer Vision: Large Vision ModelsNatural Language Processing: Language Grounding & Multi-modal NLP

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies investigate VLM personalization to understand user-provided concepts. However, they mainly focus on single-concept personalization, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes the first multi-concept personalization paradigm, MC-LLaVA. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the costs related to joint training, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location confidence maps for enhanced recognition and grounding capabilities. To advance multi-concept personalization research, we further contribute a high-quality instruction tuning dataset. We carefully collect images with multiple characters and objects from movies and manually generate question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses, paving the way for VLMs to become better user-specific assistants. The code and dataset will be publicly available at https://github.com/arctanxarc/MC-LLaVA.

Problem

• Enables multi-concept personalization in vision-language models
• Reduces costs with personalized textual and visual prompts
• Advances research with high-quality multi-concept dataset

Innovation

• Multi-concept instruction tuning strategy
• Personalized textual prompt initialization
• Personalized visual prompt aggregation