EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

2026-07-23

AI Summary

Existing multimodal large language models rely on fixed prompts for emotion recognition, struggling to capture the dynamic and complex nature of emotional expression. To address this limitation, this work proposes a reinforcement learning–based dynamic agent specialization framework featuring a two-stage agent selection and specialization mechanism. We introduce a progressive group relative policy optimization algorithm (P-GRPO) to enable fine-grained reward allocation and stable training. The approach leverages synthetically generated chain-of-thought data conditioned on target answers for cold-start initialization and jointly optimizes agent routing with P-GRPO. Evaluated on multimodal emotion recognition benchmarks, the method significantly enhances emotion reasoning performance, generalization capability, and training stability.

Abstract

Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks and lifted MER to a new level that is complex emotion understanding with advanced video understanding abilities and natural language description. However, existing MLLM-based methods often use a fixed prompt to perceive the emotions, ignoring the dynamicity and complexity of the emotion source in the multimodal inputs. To address these issues, we propose a novel Reinforcement Learning-based Dynamic Agent Specialization framework (\textbf{EmoAgent-R1}) to optimize the emotion recognition, reasoning, and generalization abilities of an MLLM with dynamic agent specialization based on reinforcement learning. Specifically, we first adopt a cold start strategy to endow an MLLM with preliminary emotion recognition, reasoning, and agent routing ability by training with synthetic answer-conditioned chain-of-thought data and agent routing data. Then, we further train the MLLM with reinforcement learning to perceive emotions in a two-step agentic workflow with agent selection and agent specialization. To effectively train EmoAgent-R1, we propose a novel Progressive Group-Relative Policy Optimization (P-GRPO) to combine group-based relative advantages with a PMI-inspired progressive token-level modulation to transform sparse rewards into fine-grained learning signals, mitigating the coarse-grained uniform credit assignment issue in GRPO. Extensive experiments on MER benchmarks demonstrate the superiority of our EmoAgent-R1 in stronger emotion reasoning performance and improved optimization stability.

Problem

• multimodal emotion recognition
• dynamicity
• complexity
• emotion understanding
• fixed prompt

Innovation

• Reinforcement Learning
• Dynamic Agent Specialization
• Multimodal Emotion Recognition
• Progressive Group-Relative Policy Optimization
• Multimodal Large Language Models