A study on the effects of mixed explicit and implicit communications in human-virtual-agent interactions

2024-09-27

arXiv.org

AI Summary

This study investigates whether multimodal communication—integrating explicit modalities (speech, gesture, on-screen prompts) with implicit ones (gaze, facial expressions, eyebrow movements)—outperforms purely explicit communication in human–virtual agent interaction. A controlled behavioral experiment was conducted, employing Bayesian parameter estimation, Likert-scale subjective assessments, and multimodal interaction modeling to evaluate both objective task performance (error rate, execution time, perceived efficiency) and subjective experience (acceptance, social presence, transparency). Results demonstrate that hybrid communication significantly improves acceptance (88.3% posterior probability exceeding the equivalence region), social presence (92%), and transparency (92.9%), without compromising task efficiency. This work provides the first empirical validation of a hybrid communication paradigm that jointly optimizes subjective experience and objective performance—challenging the conventional human–computer interaction design paradigm centered exclusively on task-oriented metrics.

Technology Category

• Multiagent Systems: Agent Communication
• Computer Vision: Multi-modal Vision
• Humans and AI: Interaction Techniques and Devices

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Communication between humans and robots (or virtual agents) is essential for interaction and often inspired by human communication, which uses gestures, facial expressions, gaze direction, and other explicit and implicit means. This work presents an interaction experiment where humans and virtual agents interact through explicit (gestures, manual entries using mouse and keyboard, voice, sound, and information on screen) and implicit (gaze direction, location, facial expressions, and raise of eyebrows) communication to evaluate the effect of mixed explicit-implicit communication against purely explicit communication. Results obtained using Bayesian parameter estimation show that the number of errors and task execution time did not significantly change when mixed explicit and implicit communications were used, and neither the perceived efficiency of the interaction. In contrast, acceptance, sociability, and transparency of the virtual agent increased when using mixed communication modalities (88.3%, 92%, and 92.9% of the effect size posterior distribution of each variable, respectively, were above the upper limit of the region of practical equivalence). This suggests that task-related measures, such as time, number of errors, and perceived efficiency of the interaction, have not been influenced by the communication type in our particular experiment. However, the improvement of subjective measures related to the virtual agent, such as acceptance, sociability, and transparency, suggests that humans are more receptive to mixed explicit and implicit communications.

Problem

• Evaluates mixed explicit-implicit communication in human-agent interactions
• Compares task performance and subjective perceptions across communication types
• Assesses impact on agent acceptance, sociability, and transparency

Innovation

• Mixed explicit-implicit communication in human-agent interaction
• Bayesian parameter estimation for evaluating effects
• Improved agent acceptance and sociability via mixed modalities