A Study of Performance and Interaction Patterns in Hand and Tangible Interaction in Tabletop Mixed Reality

2025-11-14

AI Summary

This study investigates performance differences between hand-based and tangible interaction for manipulating virtual 3D objects with four degrees of freedom (3D translation + 1D rotation) in desktop mixed reality. We designed a cylindrical tangible proxy integrating a physical knob to support translation, rotation, and scaling, and compared both modalities across isolated and composite manipulation tasks, measuring accuracy, efficiency, and correction behavior. Results show that tangible interaction—leveraging desktop support and structural constraints—significantly improves precision, particularly in composite tasks, where error growth remains markedly lower than with gesture-based interaction. Although hand-based interaction yields marginally smaller rotational errors, it requires frequent corrective actions, undermining overall stability. This work is the first to systematically uncover the mechanistic basis for tangible proxies’ accuracy advantages in desktop MR multi-DOF manipulation, providing empirical evidence and design guidelines for principled interaction modality selection in mixed reality interfaces.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Data transparency and provenance

Abstract

This paper presents a comprehensive study of virtual 3D object manipulation along 4DoF on real surfaces in mixed reality (MR), using hand-based and tangible interactions. A custom cylindrical tangible proxy leverages affordances of physical knobs and tabletop support for stable input. We evaluate both modalities across isolated tasks (2DoF translation, 1DoF rotation scaling), semicombined (3DoF translation rotation), and full 4DoF compound manipulation. We offer analyses of hand interactions, tangible interactions, and their comparison in MR tasks. For hand interactions, compound tasks required repetitive corrections, increasing completion times yet surprisingly, rotation errors were smaller in compound tasks than in rotation only tasks. Tangible interactions exhibited significantly larger errors in translation, rotation, and scaling during compound tasks compared to isolated tasks. Crucially, tangible interactions outperformed hand interactions in precision, likely due to tabletop support and constrained 4DoF design. These findings inform designers opting for hand-only interaction (highlighting tradeoffs in compound tasks) and those leveraging tangibles (emphasizing precision gains despite compound-task challenges).

Problem

• Evaluating hand versus tangible interaction for 4DoF object manipulation in mixed reality
• Comparing performance precision between tangible proxies and direct hand interactions
• Analyzing interaction patterns across isolated and compound manipulation tasks

Innovation

• Tangible proxy uses physical knob affordances
• Hand and tangible interaction comparison in MR
• Constrained 4DoF design enhances manipulation precision