InSituTale: Enhancing Augmented Data Storytelling with Physical Objects

2025-07-28

AI Summary

Existing data storytelling systems predominantly rely on gesture- or voice-based interaction, leaving physical-object-driven visualization control unexplored. Method: This paper introduces an augmented physical data storytelling approach that systematically establishes the first formal mapping between physical manipulations and visualization commands, enabling a real-world-operation-based visualization control system. The system integrates depth-camera-based object tracking with Vision-Language Large Models (Vision-LLMs) for event recognition, supporting real-time physical action detection and natural, bidirectional coupling with digital feedback. Contribution/Results: A user study demonstrates that the method significantly enhances interaction intuitiveness, presentation fluency, and audience engagement—thereby empirically validating the effectiveness and practicality of physical interaction in data storytelling.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Computer Vision: Low Level & Physics-based Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Web data visualization

Abstract

Augmented data storytelling enhances narrative delivery by integrating visualizations with physical environments and presenter actions. Existing systems predominantly rely on body gestures or speech to control visualizations, leaving interactions with physical objects largely underexplored. We introduce augmented physical data storytelling, an approach enabling presenters to manipulate visualizations through physical object interactions. To inform this approach, we first conducted a survey of data-driven presentations to identify common visualization commands. We then conducted workshops with nine HCI/VIS researchers to collect mappings between physical manipulations and these commands. Guided by these insights, we developed InSituTale, a prototype that combines object tracking via a depth camera with Vision-LLM for detecting real-world events. Through physical manipulations, presenters can dynamically execute various visualization commands, delivering cohesive data storytelling experiences that blend physical and digital elements. A user study with 12 participants demonstrated that InSituTale enables intuitive interactions, offers high utility, and facilitates an engaging presentation experience.

Problem

• Exploring physical object interactions for augmented data storytelling
• Identifying common visualization commands for data-driven presentations
• Developing intuitive system for dynamic visualization manipulation

Innovation

• Augmented physical data storytelling approach
• Object tracking with depth camera
• Vision-LLM for real-world event detection