Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset

2025-04-11

AI Summary

Current 3D reconstruction research is hindered by the absence of large-scale, high-fidelity, real-world digital twin–grade datasets—particularly benchmarks supporting egocentric image inputs—leading to inconsistent evaluation, poor generalization, and challenges in edge-device deployment. To address this, we introduce the first open-source, digital twin–oriented 3D object dataset: comprising 2,000 high-precision scanned objects, with synchronized multi-view, multi-illumination DSLR imagery and first-person video sequences captured via AR glasses. We propose a standardized evaluation protocol integrating Neural Radiance Fields (NeRF) and inverse rendering techniques, and release accompanying baseline code. This work establishes the first real-scene digital twin 3D benchmark, enables egocentric reconstruction evaluation, and validates lightweight adaptation on AR platforms—achieving breakthroughs in reconstruction fidelity, method comparability, and edge-device applicability.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We introduce Digital Twin Catalog (DTC), a new large-scale photorealistic 3D object digital twin dataset. A digital twin of a 3D object is a highly detailed, virtually indistinguishable representation of a physical object, accurately capturing its shape, appearance, physical properties, and other attributes. Recent advances in neural-based 3D reconstruction and inverse rendering have significantly improved the quality of 3D object reconstruction. Despite these advancements, there remains a lack of a large-scale, digital twin quality real-world dataset and benchmark that can quantitatively assess and compare the performance of different reconstruction methods, as well as improve reconstruction quality through training or fine-tuning. Moreover, to democratize 3D digital twin creation, it is essential to integrate creation techniques with next-generation egocentric computing platforms, such as AR glasses. Currently, there is no dataset available to evaluate 3D object reconstruction using egocentric captured images. To address these gaps, the DTC dataset features 2,000 scanned digital twin-quality 3D objects, along with image sequences captured under different lighting conditions using DSLR cameras and egocentric AR glasses. This dataset establishes the first comprehensive real-world evaluation benchmark for 3D digital twin creation tasks, offering a robust foundation for comparing and improving existing reconstruction methods. The DTC dataset is already released at https://www.projectaria.com/datasets/dtc/ and we will also make the baseline evaluations open-source.

Problem

• Lack of large-scale digital twin dataset for 3D reconstruction benchmarking
• No dataset for evaluating egocentric AR-based 3D object reconstruction
• Need for democratizing photorealistic 3D digital twin creation techniques

Innovation

• Large-scale photorealistic 3D digital twin dataset
• Integration with AR glasses for egocentric capture
• Comprehensive benchmark for reconstruction methods