Semantic Reality: Interactive Context-Aware Visualization of Inter-Object Relationships in Augmented Reality

2026-04-06

AI Summary

Existing augmented reality (AR) systems are largely confined to single-object interactions, limiting their ability to support planning, comparison, and assembly tasks that rely on multi-object relationships. This work proposes a context-aware AR system that fuses spatial anchoring, physical action recognition, and multimodal reasoning to construct and dynamically visualize semantic connections among objects in real time. The system introduces, for the first time in AR, a connectivity-centered interaction paradigm grounded in real-time graph-structured modeling, enabling users to intuitively perceive and manipulate relational structures among multiple objects. User studies demonstrate that, compared to single-object baselines, the proposed approach significantly enhances users’ understanding of object relationships, engagement, and satisfaction, without imposing additional cognitive load.

Abstract

Bridging the physical and digital world through interaction remains a core challenge in augmented reality (AR). Existing systems target single objects, limiting support for planning, comparison, and assembly tasks that depend on relationships among multiple items. We present Semantic Reality, an AR system focused on surfacing inter-object connectivity and making it interactive. Leveraging multimodal reasoning, spatial anchoring, and physical action recognition, Semantic Reality maintains a persistent model of objects around the user and their relationships. Connections are visualized in-situ to highlight compatibility, reveal next steps, and reduce ambiguity during tasks. We contribute a connectivity-centered interaction paradigm and a system architecture that couples anchor tracking, action sensing, and model inference to construct a live connectivity graph. In an exploratory study comparing Semantic Reality to a single-object baseline, participants reported clearer inter-object understanding and higher engagement and satisfaction, without increased workload. A scenario study illustrates where connectivity aids planning, sequencing, and disambiguation.

Problem

• augmented reality
• inter-object relationships
• context-aware visualization
• connectivity
• multimodal interaction

Innovation

• Semantic Reality
• inter-object relationships
• connectivity graph
• multimodal reasoning
• context-aware AR