Predicting and Understanding Turn-Taking Behavior in Open-Ended Group Activities in Virtual Reality

2024-07-03

arXiv.org

AI Summary

This study addresses the prediction and interpretation of turn-taking behavior—specifically “who speaks, when they speak, and what they say”—in open-group virtual reality (VR) activities. We propose a social-dynamics-theory-driven multimodal feature modeling approach, integrating group gaze patterns, head pose (e.g., pitch), hand spatial coordinates (e.g., left-hand Y-axis position), personality traits, and temporal interaction dynamics. For the first time, we systematically identify and validate spatiotemporally robust turn-taking predictors on real-world VR classroom data (77 sessions, 1,660 minutes). Using XGBoost and LightGBM, we build a multi-task classification model achieving AUCs of 0.71–0.78 across the three tasks. The model demonstrates strong generalizability across unseen groups, activity types, and time periods. Key findings reveal that nonverbal bodily signals—particularly group gaze and head orientation—are the most predictive indicators of speaking-turn transitions.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In networked virtual reality (VR), user behaviors, individual differences, and group dynamics can serve as important signals into future speech behaviors, such as who the next speaker will be and the timing of turn-taking behaviors. The ability to predict and understand these behaviors offers opportunities to provide adaptive and personalized assistance, for example helping users with varying sensory abilities navigate complex social scenes and instantiating virtual moderators with natural behaviors. In this work, we predict turn-taking behaviors using features extracted based on social dynamics literature. We discuss results from a large-scale VR classroom dataset consisting of 77 sessions and 1660 minutes of small-group social interactions collected over four weeks. In our evaluation, gradient boosting classifiers achieved the best performance, with accuracies of 0.71--0.78 AUC (area under the ROC curve) across three tasks concerning the"what","who", and"when"of turn-taking behaviors. In interpreting these models, we found that group size, listener personality, speech-related behavior (e.g., time elapsed since the listener's last speech event), group gaze (e.g., how much the group looks at the speaker), as well as the listener's and previous speaker's head pitch, head y-axis position, and left hand y-axis position more saliently influenced predictions. Results suggested that these features remain reliable indicators in novel social VR settings, as prediction performance is robust over time and with groups and activities not used in the training dataset. We discuss theoretical and practical implications of the work.

Problem

• Predict turn-taking behaviors in VR group activities
• Understand social dynamics influencing speech timing
• Enable adaptive assistance for diverse VR users

Innovation

• Predict turn-taking using social dynamics features
• Analyze VR classroom dataset with 77 sessions
• Use gradient boosting for best prediction accuracy