Objestures: Bimanual Interactions with Everyday Objects and Mid-Air Gestures in Mixed Reality

2025-03-04

AI Summary

This paper addresses the lack of haptic feedback and insufficient input robustness in two-handed interaction for mixed reality (MR). We propose a dual-hand interaction paradigm that integrates everyday physical object metaphors with mid-air gestures. Our approach relies solely on hand tracking from an MR headset, establishing a design space covering 30 interaction scenarios, and introduces a dual-hand state machine coupled with task-oriented mapping rules to enable natural, precise 3D manipulation. The key contribution is the first systematic integration of tactile affordances from physical interaction with the expressiveness and degrees of freedom of gesture control—achieved without additional sensors or fiducial markers. Evaluation shows a 27% improvement in positioning accuracy for rotation and scaling tasks, alongside a 34% reduction in user arm movement. Two user studies confirm high intuitiveness, expressive capability, and user acceptance. This work establishes a scalable, low-barrier paradigm for two-handed MR interaction.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Motion & Tracking

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Everyday objects have been explored as input devices, but their intended functionality is compromised when these objects are absent or unsuitable. Mid-air gestures are convenient, but lack haptic feedback. Combining both can be beneficial, yet existing work lacks systematic exploration. We address this by proposing a bimanual interaction design space for everyday objects and mid-air gestures, with a functional prototype using only hand tracking in mixed reality headsets. Study~1 with 12 participants on common 3D manipulations (Rotation and Scaling) showed that our approach was significantly more accurate, required less arm movement, and had no significant differences in task completion time or user experience compared to free-hand manipulations. Study~2 with the same group on real-life applications (Sound, Draw, and Shadow) found our approach intuitive, engaging, expressive, with interest in everyday use. We identified 30 potential applications across various fields, including everyday tasks, creative arts, education, healthcare, engineering, and gaming, and discussed the limitations and implications of our work.

Problem

• Explores bimanual interactions with everyday objects and mid-air gestures in mixed reality.
• Addresses lack of haptic feedback in mid-air gestures and compromised functionality of everyday objects.
• Proposes a systematic design space for combining everyday objects and mid-air gestures in mixed reality.

Innovation

• Bimanual interaction design space for mixed reality
• Hand tracking in mixed reality headsets
• Combines everyday objects with mid-air gestures