LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction

2026-10-01

AI Summary

This study addresses the challenge of reconstructing interactive, simulation-ready 3D indoor scenes from RGB-D scans. It introduces a pioneering code generation-based paradigm that reformulates 3D reconstruction as a programming task. Specifically, the authors design an agent system equipped with specialized tools to gather evidence and iteratively edit Python scripts for generating digital twins. This architecture incorporates an "observe-edit-verify" closed-loop mechanism, enabling continuous refinement of both scene layout and reconstruction quality. Compared to existing state-of-the-art models, the proposed approach yields substantial improvements in geometric accuracy, photorealism, and simulation compatibility. The source code has been made publicly available.

Abstract

We present LiteReality-Agent, an agentic system for reconstructing real indoor environments as realistic, articulated, and simulation-ready 3D scenes from RGB-D scans. At its core, LiteReality-Agent formulates 3D reconstruction as a coding problem, in which a coding agent gathers evidence using specialised tools and iteratively edits a Python script, Room.py, which can be executed to produce a 3D digital twin of the room. With this formulation, we develop a robust observe-edit-verify harness that supports evidence gathering, measurement, verification, layout optimisation, simulation readiness, and quality control throughout the reconstruction process. LiteReality-Agent produces high-quality reconstructions suitable for simulation and downstream embodied AI tasks. Furthermore, as agent capabilities continue to improve rapidly, the system introduced by LiteReality-Agent remains a strong orchestration framework for future agents: it equips them with specialised tools, structured workflows, and robust verification mechanisms that substantially improve reconstruction quality and reliability. We demonstrate that LiteReality-Agent produces reconstructions that are more geometrically accurate, visually realistic, and simulation-compatible than those generated by recent frontier models, such as Astra and Fable. We therefore view LiteReality-Agent as a practical and important building block for robust real-to-sim systems. Both the source code and the data-capture application are publicly available. Code:https://github.com/LiteReality/LiteReality-Agent/

Problem

• 3D indoor scene reconstruction
• RGB-D scans
• digital twin
• simulation-ready
• articulated scenes

Innovation

• Agentic 3D Reconstruction
• Code-based Formulation
• Observe-Edit-Verify Harness
• Digital Twin
• Simulation-ready