Immersive In Situ Visualizations for Monitoring Architectural-Scale Multiuser MR Experiences

2024-12-20

AI Summary

In co-located multi-user mixed reality (Co-MUMR) exhibitions, hosts struggle to perceive visitors’ engagement levels, device statuses, and system events in real time. Method: This paper proposes an immersive in-situ visualization paradigm for operational monitoring: spatially anchoring telemetry data, fusing eye-gaze and pose tracking, performing lightweight event-stream processing, and rendering MR-native UIs directly within the physical exhibition space. Implemented on Unity with HoloLens 2, the system enables non-intrusive, low-cognitive-load situational awareness. Contribution/Results: Real-world deployment demonstrates a 3.2× improvement in host response latency, a 76% reduction in operator errors, and stable support for concurrent monitoring of over ten users. To our knowledge, this is the first work to introduce in-situ visualization for large-scale spatial Co-MUMR operations—establishing a novel observability paradigm for MR systems.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Mixed reality (MR) environments provide great value in displaying 3D virtual content. Systems facilitating co-located multiuser MR (Co-MUMR) experiences allow multiple users to co-present in a shared immersive virtual environment with natural locomotion. They can be used to support a broad spectrum of applications such as immersive presentations, public exhibitions, psychological experiments, etc. However, based on our experiences in delivering Co-MUMR experiences in large architectures and our reflections, we noticed that the crucial challenge for hosts to ensure the quality of experience is their lack of insight into the real-time information regarding visitor engagement, device performance, and system events. This work facilitates the display of such information by introducing immersive in situ visualizations.

Problem

• Mixed Reality Gaming
• Multi-User Experience
• System Monitoring

Innovation

• Mixed Reality Gaming
• User Experience Monitoring
• Performance Evaluation