Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

2026-10-01

AI Summary

This study addresses the lack of large-scale, multi-category, multi-camera 3D perception benchmarks for indoor intelligent spaces by proposing the first such benchmark encompassing environments like warehouses. Methodologically, it leverages Isaac Sim to generate synthetic data, integrates Cosmos Transfer for appearance enhancement, and employs VGGT for automatic calibration. Furthermore, it introduces a 3D instantiation-based Higher Order Tracking Accuracy (HOTA) metric, extending conventional 2D evaluation into 3D space. The project releases a synchronized video dataset exceeding 280 hours with automatic annotations, establishes a standardized Sim2Real evaluation protocol, and constructs performance baselines spanning from single-person to multi-category 3D bounding box tracking.

Abstract

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.

Problem

• Physical AI
• Smart Spaces
• Multi-Camera 3D Perception
• Benchmark
• 3D Tracking

Innovation

• Multi-camera 3D Perception
• Physical AI Benchmark
• 3D HOTA Metric
• Sim2Real
• Automatic Camera Calibration