ChatMuse: Supporting In-Person Small-Group Conversation Experience with a Proactive Assistive AI Agent in Mixed Reality

2026-07-20

AI Summary

Face-to-face group conversations are often hindered by high cognitive load and the absence of real-time support mechanisms. This work proposes ChatMuse, a mixed reality (MR)-based proactive AI assistance system that leverages multimodal sensing of users’ verbal and nonverbal behaviors to generate context-aware conversational suggestions in real time, while adaptively refining its outputs through user feedback. As the first study to integrate a proactive AI agent within MR for dynamically supporting co-located group interactions, this research establishes a novel paradigm for human-AI collaborative social engagement. A within-subjects experiment demonstrates that ChatMuse significantly enhances participants’ engagement and contribution, thereby validating its feasibility and effectiveness.

Abstract

In-person small-group conversations occur across nearly every aspect of daily life and play a crucial role in social interaction. However, achieving effective in-person group conversations can be challenging and cognitively demanding. While recent Mixed Reality (MR) headsets show promise as a conversational support system by presenting relevant information through overlays, it remains unclear how such supporting information should be designed and generated for in-person group conversations. We propose ChatMuse, a novel MR-based proactive assistive system for in-person small-group conversation experience. ChatMuse analyzes verbal and non-verbal cues from all conversation participants and proactively provides real-time guidance on the user's verbal and non-verbal behaviors. The behavioral responses of the supported users are then used to improve ChatMuse's support capabilities in subsequent interactions. We conducted a within-subject study to evaluate and demonstrate the feasibility and effectiveness of ChatMuse in assisting users to engage in and contribute to in-person small-group conversations. Our research around ChatMuse represents a design exploration of a new interaction space that investigates the feasibility of supporting in-person small-group conversations through a proactive assistive AI agent in MR.

Problem

• small-group conversation
• conversational support
• mixed reality
• proactive assistive AI
• in-person interaction

Innovation

• proactive assistive AI
• mixed reality
• small-group conversation
• real-time behavioral guidance
• multimodal cue analysis