Effects of Virtual Controller Representation and Virtuality on Selection Performance in Extended Reality

2025-10-21

AI Summary

This study investigates how visual representations of virtual controllers—specifically virtual hands, virtual controllers, their fusion, or no representation—affect target selection performance in mixed reality (MR) and virtual reality (VR). A controlled user experiment quantifies selection accuracy, completion time, and subjective workload across these four configurations. Results show that embodied representations—particularly the fusion of virtual hands and controllers—significantly improve selection efficiency. While overall performance is comparable between MR and VR, users exhibit heightened spatial awareness in MR, necessitating distinct visual design optimizations for this modality. Crucially, switching between MR and VR does not impair baseline task performance. The core contribution lies in empirically revealing a fundamental divergence in controller representation requirements between MR and VR, leading to the formulation of the “perception-driven embodiment” design principle. This work provides empirical foundations and actionable design guidelines for cross-platform extended reality (XR) interaction.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We present an experiment exploring how the controller's virtual representation impacts target acquisition performance across MR and VR contexts. Participants performed selection tasks comparing four visual configurations: a virtual controller, a virtual hand, both the controller and the hand, and neither representation. We found performance comparable between VR and MR, and switching between them did not impact the user's ability to perform basic tasks. Controller representations mimicking reality enhanced performance across both modes. However, users perceived performance differently in MR, indicating the need for unique MR design considerations, particularly regarding spatial awareness.

Problem

• Examining virtual controller representation effects on selection performance
• Comparing target acquisition in mixed reality and virtual reality
• Investigating user perception differences between MR and VR interfaces

Innovation

• Virtual hand and controller representations tested
• Reality-mimicking controllers enhanced performance
• Unique MR design considerations for spatial awareness