MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement

2026-07-23

AI Summary

This work addresses the lack of systematic evaluation of visual emotional intelligence in existing multimodal large language models (MLLMs), stemming from their open-ended nature that mismatches traditional sentiment analysis paradigms. To bridge this gap, we propose a novel Emotion Statement Judgment (ESJ) framework, introduce a large-scale dataset named INSETS-462k, and establish a multidimensional visual emotion benchmark, MVEI. We further develop EmObserver, an emotion-aware MLLM that leverages multi-stage fine-tuning and integrates affective polarity, scene context, and subjective perception into its modeling strategy. EmObserver demonstrates superior performance on both MVEI and multiple AICA benchmarks, achieving the first fine-grained assessment of visual emotion understanding in MLLMs. This study thus provides a new evaluation paradigm, a comprehensive benchmark, and a strong baseline for future research in affective multimodal intelligence.

Abstract

Affective Image Content Analysis (AICA) aims to recognize and understand emotions elicited by visual content, representing an indispensable step toward Artificial General Intelligence (AGI). However, despite the rapid progress of Multimodal Large Language Models (MLLMs), systematic evaluation of their visual emotional intelligence remains largely absent from recent model releases. We attribute this gap to a structural mismatch between conventional AICA paradigms and the open-ended, instruction-driven nature of MLLMs, where further analysis reveals four major limitations: omission of plausible responses, limited emotion taxonomies, neglect of contextual factors, and labor-intensive annotation. To overcome these barriers, we introduce Emotion Statement Judgement (ESJ), a statement-verification formulation that preserves the expressiveness of the input space while constraining outputs to discriminative judgements. We further develop INSETS, a labor-efficient pipeline that instantiates ESJ at scale by constructing INSETS-462k and supporting MVEI, a rigorously refined benchmark spanning sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Beyond evaluation, we build EmObserver, an emotion-oriented MLLM optimized on ESJ through an elaborate multi-stage recipe. Extensive evaluation of broad-spectrum MLLMs on MVEI reveals fine-grained insights into current artificial visual emotional intelligence, while experiments on multiple AICA benchmarks demonstrate the accuracy, generalization, and reasoning faithfulness of EmObserver. Collectively, these results establish ESJ as a practical formulation, MVEI as a comprehensive benchmark, and EmObserver as an advanced baseline for advancing MLLM-oriented visual emotional intelligence. Code will be released at: https://github.com/wdqqdw/EmObserver.

Problem

• Visual Emotional Intelligence
• Multimodal Large Language Models
• Affective Image Content Analysis
• Emotion Recognition
• Model Evaluation

Innovation

• Emotion Statement Judgement
• Multimodal Large Language Models
• Visual Emotional Intelligence
• MVEI Benchmark
• EmObserver