Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models

2026-06-09

AI Summary

This work addresses the lack of systematic evaluation of multimodal large language models under complex personality conditions. It proposes the first personality modeling evaluation framework tailored for multimodal settings, explicitly injecting personality conditions to systematically investigate the effects of single-personality induction, multi-personality composition, and dynamic personality switching on vision-language tasks. Experiments reveal that personality induction enhances image captioning performance but impairs accuracy in precise reasoning tasks such as visual question answering. Multi-personality combinations exhibit a balancing effect, while dynamic switching demonstrates residual influence, indicating that model behavior is jointly modulated by both current and historical personality states. These findings highlight the limited transferability of existing prompting strategies in multimodal scenarios.

Abstract

With the widespread deployment of Multimodal Large Language Models (MLLMs) in social interaction, understanding and controlling their behavior under complex personality conditions is essential. This paper introduces explicit personality conditioning and establishes a systematic evaluation framework encompassing single-personality induction, multi-personality induction, and personality switching. Experiments show that personality induction improves image captioning performance but can impair performance on tasks requiring precise reasoning, such as visual question answering (VQA). Balancing and residual effects are observed during multi-trait composition and dynamic switching, indicating that model behavior is co-modulated by both previous and current personality constraints. Existing prompt-based personality induction methods show limited transferability to multimodal settings. Our work reveals the dynamic and complex nature of personality modeling in MLLMs and underscores the need for robust, tailored methods for personality induction and evaluation. The code will be released when the paper is accepted.

Problem

• personality modeling
• multimodal large language models
• personality switching
• behavior control
• complex personality composition

Innovation

• personality conditioning
• multimodal large language models
• dynamic personality switching
• multi-personality composition
• behavioral evaluation framework