psiUnity: A Platform for Multimodal Data-Driven XR

2025-11-07

AI Summary

The Platform for Situated Intelligence (Psi) lacks compatibility with the Unity/Mixed Reality Toolkit (MRTK) ecosystem, severely hindering reproducibility in multimodal XR experiments. Method: This work introduces the first deep integration of Psi’s high-precision temporal alignment capabilities into Unity 2022.3 and MRTK3—bypassing StereoKit limitations—to enable bidirectional, real-time streaming of HoloLens 2 sensor data (AHAT, long-range depth, IMU, eye tracking, hand gestures, and audio). Built in C#, it leverages the Psi .NET SDK and incorporates microsecond-level time coordination, native serialization, and structured logging. Contribution/Results: The platform achieves sub-millisecond synchronization between head-mounted displays and immersive applications, significantly enhancing measurement fidelity and experimental reproducibility in human–robot interaction (HRI), human–computer interaction (HCI), and embodied intelligence research. The implementation is open-source.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Extended reality (XR) research increasingly relies on the ability to stream and synchronize multimodal data between headsets and immersive applications for data-driven interaction and experimentation. However, developers face a critical gap: the Platform for Situated Intelligence (psi), which excels at deterministic temporal alignment and multimodal data management, has been largely inaccessible to the dominant Unity/MRTK ecosystem used for HoloLens development. We introduce psiUnity, an open-source C# integration that bridges psi's .NET libraries with Unity 2022.3 and MRTK3 for HoloLens 2. psiUnity enables bidirectional, real-time streaming of head pose, hand tracking, gaze, IMU, audio, and depth sensor data (AHAT and long-throw) with microsecond-level temporal precision, allowing Unity applications to both consume and produce synchronized multimodal data streams. By embedding psi's native serialization, logging, and temporal coordination directly within Unity's architecture, psiUnity extends psi beyond its previous StereoKit limitations and empowers the HRI, HCI, and embodied-AI communities to develop reproducible, data-driven XR interactions and experiments within the familiar Unity environment. The integration is available at https://github.com/sailgt/psiUnity.

Problem

• Bridging multimodal data management platform with Unity ecosystem
• Enabling real-time synchronized streaming of XR sensor data
• Extending temporal coordination capabilities to HoloLens development environment

Innovation

• Bridges psi .NET libraries with Unity MRTK
• Enables bidirectional real-time multimodal data streaming
• Provides microsecond-level temporal precision synchronization