Reference-Based Manipulation: A Framework and Pipeline for Multimodal Spatial Reasoning

2026-08-05

AI Summary

This study addresses the challenge of accurately interpreting users’ multimodal spatial reference expressions—combining speech and gestures—in immersive environments. Drawing on spatial cognition theory, the authors analyze unconstrained multimodal interactions in virtual reality through a Wizard-of-Oz experiment. They formally introduce, for the first time, a triadic structure for spatial referencing composed of Source, Anchor, and Frame, along with associated composition strategies. Building upon this foundation, they develop a multimodal interaction pipeline integrating large language models to enable generalizable and extensible spatial reasoning. The resulting prototype system demonstrates improved accuracy and practicality in understanding user spatial intent, as validated through technical evaluation.

Abstract

When manipulating objects in immersive platforms through speech and gesture, users naturally construct spatial references, referring to scene entities, their bodies, or the environment. Leveraging spatial cognition theories, this work systematically examines how users construct and communicate spatial intent. Using a custom toolkit, we conducted a Wizard-of-Oz study to observe unconstrained multimodal (speech + gesture) input patterns in Virtual Reality for scene construction. Based on these findings, we formalize a framework that decomposes spatial references into three core components: Source, Anchor, and Frame, while characterizing their compositional strategies and explicitness. We demonstrate the utility of this Reference-based Manipulation framework by implementing an LLM-based pipeline featuring a set of example interaction techniques with a preliminary technical evaluation. Finally, we discuss key lessons learned for supporting reference-based spatial interaction.

Problem

• spatial reference
• multimodal interaction
• virtual reality
• spatial reasoning
• reference-based manipulation

Innovation

• Reference-Based Manipulation
• Multimodal Interaction
• Spatial Reasoning
• Virtual Reality
• Large Language Models