"Just Like This": Manner Deixis at the Graphics Interface

2026-10-06

AI Summary

This study addresses the challenge in 3D conversational authoring where voice alone is insufficient to precisely specify complex motions. To overcome this limitation, this work proposes a multimodal interaction framework that integrates speech, pointing gestures, and spatiotemporal demonstrations. By embedding motion demonstrations within continuous authoring dialogues and leveraging gestures to designate editable behaviors, the system establishes an iterative “demonstrate–preview–correct” closed loop, supported by dedicated mechanisms for parsing spatiotemporal demonstrations and previewing animations. The primary contribution lies in enabling precise behavioral control and efficient revision through multimodal synergy. Experimental results demonstrate a task completion rate of 89%, with accuracy significantly outperforming voice-only baselines, while user studies confirm a strong preference for the proposed approach among the majority of participants.

Abstract

People communicate how things should move by combining words with demonstrations: "open it like this." We present an interaction technique that brings this expressive resource to conversational 3D authoring. Building on "Put-That-There," our system combines speech, pointing, and spatiotemporal demonstrations to specify editable behavior. A hand movement supplies evidence for a mechanism's axis, pivot, range, and pace; an animated preview makes the interpretation inspectable. Users refine behavior through further words or demonstrations, and the system can request a demonstration to clarify intent. In a twelve-participant study comparing three input configurations, our system achieved 89% participant-declared completion versus 47% with speech alone, had the highest observed match rates on six categorical accuracy measures and tied on two, and was preferred by nine participants. This work makes demonstration part of an ongoing authoring conversation: behavior can be shown, inspected, and revised.

Problem

• 3D authoring
• manner deixis
• multimodal interaction
• motion specification
• demonstration

Innovation

• Manner Deixis
• Multimodal Interaction
• 3D Authoring
• Spatiotemporal Demonstration
• Conversational Interface