Personalize Your Large Vision-language Models With In-context Prompt Tuning

2026-05-29

AI Summary

This work addresses the inefficiency of existing large vision-language models in personalization, which rely on training during inference and suffer from semantic confusion and identity drift in multi-image, multi-concept scenarios. To overcome these limitations, the authors propose Identity-aware Contextual Prompt Tuning (ICPT), a method that employs a lightweight projection module to extract fine-grained visual semantics from multiple reference images. These features are combined with identity labels to generate continuous prompts, augmented by an adaptive prompt length mechanism and two geometric regularization strategies. This design effectively disentangles identity-specific features from environmental distractions and separates multi-concept semantics. Evaluated across diverse backbone architectures and tasks, ICPT significantly improves personalization accuracy while maintaining high efficiency and robustness in complex scenarios.

Abstract

Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing interest in LVLM personalization, which aims to enable models to quickly and effectively learn out-of-distribution multimodal concepts to meet user-specific needs. However, many existing methods rely on inference-time training, which reduces efficiency. They also struggle to maintain accuracy in complex multi-image, multi-concept settings. These limitations restrict the broader deployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweight projection module capable of operating in complex scenarios to extract fine-grained visual semantics from multiple reference images, seamlessly transforming these features alongside identity-label mappings into continuous prompts. To maximize computational efficiency, this module adaptively determines the prompt length based on the intrinsic visual complexity of each concept. Crucially, to overcome the environmental biases and cross-concept interference prevalent in real-world applications, we introduce two novel geometric regularizations. These constraints refine prompt representations by decoupling key identities from transient environmental states and separating concepts to avoid semantic confusion. Extensive experiments show that ICPT achieves state-of-the-art personalization accuracy across diverse tasks and LVLM backbones.

Problem

• personalization
• large vision-language models
• in-context learning
• multi-image
• multi-concept

Innovation

• in-context prompt tuning
• vision-language personalization
• geometric regularization
• adaptive prompt length
• multimodal concept disentanglement