Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents

2026-03-10

AI Summary

This work proposes an emotion-aware virtual reality (VR) interaction system that addresses the limitation of existing VR conversational agents, which predominantly rely on textual semantics while neglecting the rich emotional cues embedded in vocal prosody, often resulting in emotionally inconsistent responses. The proposed system explicitly integrates real-time emotion labels—derived from speech prosody analysis and affect recognition—into the dialogue context of a large language model (LLM), thereby enabling emotionally aligned response generation. By unifying vocal prosody analysis, real-time emotion recognition, and LLM-driven dialogue mechanisms, the system significantly enhances perceived dialogue quality, naturalness, engagement, warmth, and human-likeness in user studies involving 30 participants, with 93.3% of users expressing a clear preference for the emotion-aware agent.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Natural Language Processing: Conversational AI/Dialog Systems
• Humans and AI: Emotional Intelligence

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.

Problem

• emotion recognition
• prosody
• virtual reality
• conversational agents
• emotional context

Innovation

• emotion-aware dialogue
• prosody-based emotion recognition
• virtual reality agents
• LLM-based conversational agent
• emotional context integration