GhostObjects: Instructing Robots by Manipulating Spatially Aligned Virtual Twins in Augmented Reality

2025-08-14

AI Summary

Traditional robot teaching-by-demonstration and teleoperation suffer from low intuitiveness, poor spatial precision, and difficulty in specifying complex 3D tasks. To address these challenges, this paper introduces GhostObjects—a spatially aligned virtual twin interaction paradigm enabled by augmented reality (AR). Users manipulate virtual objects via natural hand gestures to non-invasively specify target poses and spatial constraints for physical robots. The system integrates high-fidelity spatial registration, geometric snapping, multi-object lasso selection, and pose rebounding, achieving millisecond-scale responsiveness and sub-centimeter positioning accuracy. Crucially, GhostObjects requires no programming expertise or specialized hardware, significantly enhancing the intuitiveness, flexibility, and expressive power of spatial task specification. It supports not only grasp-and-place operations but also multi-stage collaborative spatial tasks. By providing a scalable, intuitive interaction infrastructure, GhostObjects advances natural human-robot coexistence and shared autonomy.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Interaction Techniques and Devices
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Robots are increasingly capable of autonomous operations, yet human interaction remains essential for issuing personalized instructions. Instead of directly controlling robots through Programming by Demonstration (PbD) or teleoperation, we propose giving instructions by interacting with GhostObjects-world-aligned, life-size virtual twins of physical objects-in augmented reality (AR). By direct manipulation of GhostObjects, users can precisely specify physical goals and spatial parameters, with features including real-world lasso selection of multiple objects and snapping back to default positions, enabling tasks beyond simple pick-and-place.

Problem

• Enabling precise human instruction for robots via AR
• Replacing direct control with virtual object manipulation
• Expanding robot tasks beyond basic pick-and-place operations

Innovation

• Interact with GhostObjects in AR
• Manipulate virtual twins for precise goals
• Lasso selection and position snapping